Thank you for the help!

It ended up being a problem with sqlite3's journaling system waiting
for the writer to unlock the OS buffers before creating a rollback
file. Changing the journal configuration to stay in memory rather than
disk made the journaling way faster and fixed the problem.

I will keep the potential priority problems in mind for the future!

Thanks again,
Sabar

On Mon, Oct 3, 2016 at 11:44 AM, Brian Silverman [off-list ref] wrote:

Have you looked for any internal locks sqlite3 is using? Keeping in
mind I have no experience with that library whatsoever, it sounds like
you may be running into sqlite3async_run holding a lock while it does
IO, which causes sqlite3_step to block waiting for it to finish. Even
if the disk IO thread doesn't do much with the lock held, you could be
running into priority inversion
(https://en.wikipedia.org/wiki/Priority_inversion has a description
and some solutions).

You might have less problems if you split anything realtime (like
reading your data from shared memory) and all the sqlite3 things out
into separate processes. Pipes and POSIX shared memory queues should
work.

If you're still having problems, it might be worth asking sqlite3
people. Specifically, how they do locking etc sounds relevant.

Best of luck,
Brian