[Buildroot] Uclibc shm performance issue

From: Kenneth Adam Miller <hidden>
Date: 2016-11-23 00:57:32

Hey, so as it would turn out, I turned on multi core scheduler preemption
and up the highest it would go, and now the performance in our custom linux
is comparable to what I get running tests locally. That makes sense, since
I ruled everything else out.

On Nov 22, 2016 4:18 AM, "Kenneth Adam Miller" [off-list ref]
wrote:

Well, I did do a strace on my benchmark test that runs so fast on mine.

Any important point of my particular shared memory data structure
using shm_open is to tell the process how many messages total have
been written to it after a coordinated startup, but to still be able
to wake it up if there is need be. For this, I use atomics in my
shared memory region, and strongly preference keeping the userland
from making unnecessary trips to the kernel. But what I'm seeing is
that the strace file grows to be megabytes in size with read/writes to
named pipes (which I use as a semaphore since I have nothing else
smaller, as I don't want to use a domain socket to pass an eventfd).

I wonder if maybe there's something to do with atomics or the memory
operations that are preventing it from using the hardware to the best
of it's abilities?

On Mon, Nov 21, 2016 at 6:25 PM, Kenneth Adam Miller
[off-list ref] wrote:

On Mon, Nov 21, 2016 at 6:20 PM, Arnout Vandecappelle [off-list ref]

wrote:

quoted


On 21-11-16 23:23, Kenneth Adam Miller wrote:

quoted

No no, my uclibc is different between my target and my host. So, I
can't just run what buildroot built.

 Yes you can: chroot into output/target.

And run a uclibc linked target binary on my glibc host?? I didn't know

that.

quoted

But I did try a statically compiled benchmark that leveraged musl, and
performance is phenominally terrible.

 Why do you drag musl in the equation now?

Well, it's statically linked; it helps to eliminate the what the
source of the slowdown is. So, if I link it statically with musl or
glibc it doesn't matter, so long as I can eliminate whether uclibc is
the source of problem or not.

quoted

By the way, this is just shm files backed by /dev/shm using shm_open.
Just to make sure that that is clear, I'm not sure what it is that is
causing such awful performance.

So far, this has isolated the issue from involving either the libc
implementation or the actual test software itself. I think it is
something to do with the way I built my custom linux image, possibly
the architecture like you said. But one more thing is, I test my image
by running it under qemu and by running it on dedicated hardware.

 Aren't you contradicting yourself here? You're saying you have been

able to

quoted

isolate it to either libc or the the test software itself, but then you

say it's

quoted

due to the custom linux image or the architecture.

Well, I only just ran my test as a statically compiled target with a
different libc implementation, so that kind of narrows down about
everything else since it's still slow as hell.

quoted


 Regards,
 Arnout

quoted

On Mon, Nov 21, 2016 at 5:19 PM, Arnout Vandecappelle [off-list ref]

wrote:

quoted


On 21-11-16 23:01, Kenneth Adam Miller wrote:

quoted

On Mon, Nov 21, 2016 at 4:40 PM, Arnout Vandecappelle [off-list ref]

wrote:

quoted

 Hi Kenneth,

On 21-11-16 20:10, Kenneth Adam Miller wrote:

quoted

Hello,

I'm using an absolutely miniscule fully compiled test inside a

custom linux

quoted
quoted
quoted
quoted
quoted
quoted
built by buildroot with uclibc, and I'm opening and memcpy'ing raw

data into a

quoted
quoted
quoted
quoted
quoted
quoted
shm region shared between two processes. On my host, I get

fantastic performance

quoted
quoted
quoted
quoted
quoted
quoted
of course, gigabytes per second as expected. Therefore I know that

the code that

quoted
quoted
quoted
quoted
quoted
quoted
I wrote is fast. But when I move it to run inside our booted image,

I take a

quoted
quoted
quoted
quoted
quoted
quoted
huge performance hit. I'm not sure what the source is, but I wanted

to know if

quoted
quoted
quoted
quoted
quoted
quoted
anybody would think that uclibc would affect performance so

dramatically, or if

quoted
quoted
quoted
quoted
quoted
quoted
there could be some buildroot specific option that I could change

that would

quoted
quoted
quoted
quoted
quoted
quoted
improve things. There are two layers of performance penalty in

fact, one just

quoted
quoted
quoted
quoted
quoted
quoted
moving into the custom linux, and another when we drive our C

library with Ruby.

quoted
quoted
quoted
quoted
quoted
quoted
I understand that that is to be expected but for me, i dont think

that the

quoted
quoted
quoted
quoted
quoted
quoted
penalty should be so enormous for that as well. Perhaps grsec could

be affecting it?

quoted
quoted
quoted
quoted
quoted
 It sounds like you're stacking a huge number of changes on top of

each other,

quoted
quoted
quoted
quoted
quoted
and any of them could be the cause. So I think it's better to test

each of them

quoted

separately.

- Buildroot: change your configuration so it can run on your host,

select glibc

quoted
quoted
quoted
quoted
quoted
as your library and the same gcc version as you use on your host,

chroot into

quoted
quoted
quoted
quoted
quoted
the rootfs and execute your test application. Do you get the same

performance?

quoted

So, there's several different performances that I've already measured
- what you're talking about has already been measured, and it's the
basis against which I want everything else to compare. I understand if
there's some minor performance difference, but I don't expect to fall
out of the gigabytes per second range just going into a custom linux
image.

quoted

- uClibc: like above, but use uClibc as the C library

I don't know what you're saying here.

quoted

- gcc: switch to the gcc version you use for your target

Gcc versions between host and the target produced are each 4.8.4

quoted

- linux: build the same linux version (but the upstream one) as for

your target

quoted

with the configuration for your host, and boot into it natively

Could something like linux 3.13 to 3.14 really make such a huge

difference?

quoted

- grsec: like above but use the grsec-patches linux

I only just started down the path of building a grsec disabled linux
to test with.

quoted

- CPU arch: now run it on your target

Both my host and the produced linux use x86_64, so we're good there.

 In that case, you can just chroot into the output/target directory

and execute

quoted

your test application on the host. That immediately will tell you if

it's

quoted

Buildroot+uClibc that is the culprit, or the CPU/kernel (config or

grsec).

quoted

 Note that a different CPU + memory architecture, even if it's still

x86_64, can

quoted

make a lot of difference.

 Regards,
 Arnout

quoted

and you can do all that in a different order of course.

 Regards,
 Arnout

--
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR

Leuven

quoted

LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

--
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

--
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20161122/fa229cef/attachment.html>

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help