Re: very poor ext3 write performance on big filesystems?
From: Andreas Dilger <hidden>
Date: 2008-02-27 20:03:49
I'm CCing the linux-raid mailing list, since I suspect they will be interested in this result. I would suspect that the "journal guided RAID recovery" mechanism developed by U.Wisconsin may significantly benefit this workload because the filesystem journal is already recording all of these block numbers and the MD bitmap mechanism is pure overhead. On Feb 27, 2008 12:20 +0100, Tomasz Chmielewski wrote:
Theodore Tso schrieb:quoted
On Mon, Feb 18, 2008 at 03:03:44PM +0100, Andi Kleen wrote:quoted
Tomasz Chmielewski [off-list ref] writes:quoted
Is it normal to expect the write speed go down to only few dozens of kilobytes/s? Is it because of that many seeks? Can it be somehow optimized?I have similar problems on my linux source partition which also has a lot of hard linked files (although probably not quite as many as you do). It seems like hard linking prevents some of the heuristics ext* uses to generate non fragmented disk layouts and the resulting seeking makes things slow.A follow-up to this thread. Using small optimizations like playing with /proc/sys/vm/* didn't help much, increasing "commit=" ext3 mount option helped only a tiny bit. What *did* help a lot was... disabling the internal bitmap of the RAID-5 array. "rm -rf" doesn't "pause" for several seconds any more. If md and dm supported barriers, it would be even better I guess (I could enable write cache with some degree of confidence). This is "iostat sda -d 10" output without the internal bitmap. The system mostly tries to read (Blk_read/s), and once in a while it does a big commit (Blk_wrtn/s): Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 164,67 2088,62 0,00 20928 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 180,12 1999,60 0,00 20016 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 172,63 2587,01 0,00 25896 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 156,53 2054,64 0,00 20608 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 170,20 3013,60 0,00 30136 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 119,46 1377,25 5264,67 13800 52752 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 154,05 1897,10 0,00 18952 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 197,70 2177,02 0,00 21792 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 166,47 1805,19 0,00 18088 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 150,95 1552,05 0,00 15536 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 158,44 1792,61 0,00 17944 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 132,47 1399,40 3781,82 14008 37856 With the bitmap enabled, it sometimes behave similarly, but mostly, I can see as reads compete with writes, and both have very low numbers then: Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 112,57 946,11 5837,13 9480 58488 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 157,24 1858,94 0,00 18608 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 116,90 1173,60 44,00 11736 440 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 24,05 85,43 172,46 856 1728 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 25,60 90,40 165,60 904 1656 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 25,05 276,25 180,44 2768 1808 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 22,70 65,60 229,60 656 2296 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 21,66 202,79 786,43 2032 7880 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 20,90 83,20 1800,00 832 18000 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 51,75 237,36 479,52 2376 4800 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 35,43 129,34 245,91 1296 2464 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 34,50 88,00 270,40 880 2704 Now, let's disable the bitmap in the RAID-5 array: Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 110,59 536,26 973,43 5368 9744 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 119,68 533,07 1574,43 5336 15760 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 123,78 368,43 2335,26 3688 23376 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 122,48 315,68 1990,01 3160 19920 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 117,08 580,22 1009,39 5808 10104 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 119,50 324,00 1080,80 3240 10808 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 118,36 353,69 1926,55 3544 19304 And let's enable it again - after a while, it degrades again, and I can see "rm -rf" stops for longer periods: Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 162,70 2213,60 0,00 22136 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 165,73 1639,16 0,00 16408 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 119,76 1192,81 3722,16 11952 37296 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 178,70 1855,20 0,00 18552 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 162,64 1528,07 0,80 15296 8 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 182,87 2082,07 0,00 20904 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 168,93 1692,71 0,00 16944 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 177,45 1572,06 0,00 15752 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 123,10 1436,00 4941,60 14360 49416 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 201,30 1984,03 0,00 19880 0 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 165,50 1555,20 22,40 15552 224 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 25,35 273,05 189,22 2736 1896 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 22,58 63,94 165,43 640 1656 Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn sda 69,40 435,20 262,40 4352 2624 There is a related thread (although not much kernel-related) on a BackupPC mailing list: http://thread.gmane.org/gmane.comp.sysutils.backup.backuppc.general/14009 As it's BackupPC software which makes this amount of hardlinks (but hey, I can keep ~14 TB of data on a 1.2 TB filesystem which is not even 65% full). -- Tomasz Chmielewski http://wpkg.org - To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc.