Quadrant write performance degradation - kernel3.10 vs kernel3.4
From: Tanya Brokhman <hidden>
Date: 2014-06-16 06:02:13
Also in:
linux-fsdevel
Hello, Recently we encountered a performance degradation on 3.10kernel based build, compared to 3.4 based one, when running the fs_write Quadrant benchmark. We profiled the test and came to the conclusion that the root cause of the degradation is in the vfs_write call stack (overhead of 2611.2us is observed in 3.10 kernel compared to 3.4): ret_fast_syscall SyS_write vfs_write (total time spent: 3.10kernel-21295us, 3.4kernel-18683.79us) do_sync_write ext4_file_write generic_file_aio_write (total time spent: 3.10kernel-19124.4us, 3.4kernel-16815us) __generic_file_aio_write generic_file_buffered_write ext4_da_write_begin (total time spent: 3.10kernel-10935.2us, 3.4kernel-8444.6us) __block_write_begin ext4_da_get_block_prep (total time spent: 3.10kernel-5402.6us, 3.4kernel-3576.8us) ext4_es_lookup_extent (total time spent: 3.10kernel-2219.7us, 3.4kernel-0us) We tried to revert just the ext4 code back to 3.4 (on a 3.10 kernel) build and got an improvement of 50% in the test result. When looking deeper into the changes made to the ext4 FS between 3.4 and 3.10 versions we stumbled across two major features making an explicit tradeoff in favor of robustness and good design over performance in some use cases: 1) Metadata Checksums http://kernelnewbies.org/Linux_3.5#head-e8ea0d70436ea63590eac3dc25a7b417333147f8 “As far as performance impact goes, it shouldn't be noticeable for common desktop and server workloads. A mail server ffsb simulation show nearly no change. On a test doing only file creation and deletion and extent tree modifications, a performance drop of about 20 percent was measured. However, it's a workload very heavily oriented towards metadata, in most real-world workloads metadata is usually a small fraction of total IO, so unless your workload is metadata-oriented, the cost of enabling this feature should be negligible.” 2) Extents status tracking: https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ext4/extents_status.c?id=refs/tags/v3.10.42#n20 “There is a cache extent for write access, so if writes are not very random, adding space operations are in O(1) time.” We tried pick up several performance-enhancement patches from the community, released between 3.10 and 3.14 kernel versions. The performance was almost the same. I was wondering what performance tests were performed on these features? Has anyone encountered same issue? Best Regards Tanya Brokhman -- QUALCOMM ISRAEL, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html