Re: [PATCH 3/5] cifsd: add file operations

On Mon, Mar 22, 2021 at 02:57:18PM +0100, Christoph Hellwig wrote:

On Mon, Mar 22, 2021 at 06:03:21PM +0900, Sergey Senozhatsky wrote:

quoted

On (21/03/22 08:15), Matthew Wilcox wrote:

quoted

What's the scenario for which your allocator performs better than slub

IIRC request and reply buffers can be up to 4M in size. So this stuff
just allocates a number of fat buffers and keeps them around so that
it doesn't have to vmalloc(4M) for every request and every response.

Do we have any data suggesting it is faster than vmalloc?

Oh, I have no trouble believing it's faster than vmalloc.  Here's
the fast(!) path that always has memory available, never does retries.
I'm calling out the things I perceive as expensive on the right hand side.
Also, I'm taking the 4MB size as the example.

vmalloc()
  __vmalloc_node()
    __vmalloc_node_range()
      __get_vm_area_node()
				[allocates vm_struct]
	alloc_vmap_area()
				[allocates vmap_area]
				[takes free_vmap_area_lock]
	  __alloc_vmap_area()
	    find_vmap_lowest_match
				[walks free_vmap_area_root]
				[takes vmap_area_lock]
      __vmalloc_area_node()
				... array_size is 8KiB, we call __vmalloc_node
	__vmalloc_node
				[everything we did above, all over again,
				 two more allocations, two more lock acquire]
	alloc_pages_node(), 1024 times
	vmap_pages_range_noflush()
	  vmap_range_noflush()
				[allocate at least two pages for PTEs]

There's definitely some low handling fruit here.  __vmalloc_area_node()
should probably call kvmalloc_node() instead of __vmalloc_node() for
table sizes > 4KiB.  But a lot of this is inherent to how vmalloc works,
and we need to put a cache in front of it.  Just not this one.