From: Daniel Black <hidden> Date: 2021-10-22 03:13:17
Sometime after 5.11 and is fixed in 5.15-rcX (rc6 extensively tested
over last few days) is a kernel regression we are tracing in
https://jira.mariadb.org/browse/MDEV-26674 and
https://jira.mariadb.org/browse/MDEV-26555
5.10 and early across many distros and hardware appear not to have a problem.
I'd appreciate some help identifying a 5.14 linux stable patch
suitable as I observe the fault in mainline 5.14.14 (built
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.14.14/). This is of
interest to both Debian (sid)
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=996951 , Ubuntu
(Impish) and Fedora fc33-35 (TODO bug report)..
Marko in https://jira.mariadb.org/browse/MDEV-26555?focusedCommentId=198601&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-198601
traced this down to a io_uring_wait_cqe never returning after a
request was pushed.
The observed behavior uses a mariadb-test package for 10.6
dan@impish:~$ uname -a
Linux impish 5.14.14-051414-generic #202110201037 SMP Wed Oct 20
11:04:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
dan@impish:~$ cd /usr/share/mysql/mysql-test/
dan@impish:/usr/share/mysql/mysql-test$ ./mtr --vardir=/tmp/var
--parallel=4 stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
stress.ddl_innodb
Logging: ./mtr --vardir=/tmp/var --parallel=4 stress.ddl_innodb
stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
vardir: /tmp/var
Removing old var directory...
Creating var directory '/tmp/var'...
Checking supported features...
MariaDB Version 10.6.5-MariaDB-1:10.6.5+maria~impish
- SSL connections supported
- binaries built with wsrep patch
Collecting tests...
Installing system database...
==============================================================================
TEST WORKER RESULT TIME (ms) or COMMENT
--------------------------------------------------------------------------
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
worker[4] Using MTR_BUILD_THREAD 301, with reserved ports 16020..16039
worker[3] Using MTR_BUILD_THREAD 302, with reserved ports 16040..16059
worker[2] Using MTR_BUILD_THREAD 303, with reserved ports 16060..16079
stress.ddl_innodb 'innodb' w3 [ pass ] 185605
stress.ddl_innodb 'innodb' w4 [ pass ] 186292
stress.ddl_innodb 'innodb' w2 [ pass ] 193053
stress.ddl_innodb 'innodb' w1 [ pass ] 202529
stress.ddl_innodb 'innodb' w4 [ pass ] 213972
stress.ddl_innodb 'innodb' w3 [ pass ] 214661
stress.ddl_innodb 'innodb' w1 [ pass ] 213266
stress.ddl_innodb 'innodb' w4 [ pass ] 181716
stress.ddl_innodb 'innodb' w3 [ pass ] 194047
stress.ddl_innodb 'innodb' w1 [ pass ] 208319
stress.ddl_innodb 'innodb' w2 [ fail ]
Test ended at 2021-10-22 01:24:22
----------SERVER LOG START-----------
2021-10-22 1:24:20 0 [ERROR] [FATAL] InnoDB:
innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch.
Please refer to
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
This threshold is 10 minutes so its not like the hardware is that slow.
To my frustration, the hirsuite based container (below) created as a
test framework for you has never produced a fault even though running
on the same 5.14.14-200.fc34.x86_64 kernel that would fail after 2-3
stress.ddl_innodb tests.
$ podman run --rm --privileged=true
quay.io/danielgblack/mariadb-test:uring --vardir=/var/tmp
stress.ddl_innodb{,,,,,,,,,,,,,}
...
--------------------------------------------------------------------------
The servers were restarted 0 times
Spent 2908.065 of 822 seconds executing testcases
Completed: All 18 tests were successful.
Looking at server test logs in /var/tmp/[0-9]/*/*err* the mariadbd
process are using uring.
I hope provides a hint.
In the mean time, the complete reproduce is to pull a 10.6 disto
package from https://mariadb.org/download/?tab=repo-config
It has to be a distro that provides liburing like:
Debian sid
Ubuntu - groovy+
Rhel8
Fedora
(centos/rhel are doing the incorrect baseurl currently, replace the
last fragement of the path, with [rhel|centos][7|8]-$arch )
Install repo.
Install Package mariadb-test (pull in MariaDB server as dependency).
ldd /usr/{s}bin/mariadbd to check liburing is there.
cd /usr/share/mysql/mysql-test
./mtr --vardir=/tmp/var --parallel=4 encryption.innochecksum{,,,,,}
./mtr --vardir=/tmp/var --parallel=4 stress.ddl_innodb{,,,,,}
Should generate a backtrace like above.
As an mtr argument with gdb and xterm installed the following will
breakpoint the application at this state.
--gdb='b ib::fatal::~fatal;r'
I'm happy to build from a tree like
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=io_uring-5.15
if you'd like to to test something locally.
I can also run bpftrace scripts to pull out info if required.
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-10-22 09:10:46
On 10/22/21 04:12, Daniel Black wrote:
Sometime after 5.11 and is fixed in 5.15-rcX (rc6 extensively tested
over last few days) is a kernel regression we are tracing in
https://jira.mariadb.org/browse/MDEV-26674 and
https://jira.mariadb.org/browse/MDEV-26555
5.10 and early across many distros and hardware appear not to have a problem.
I'd appreciate some help identifying a 5.14 linux stable patch
suitable as I observe the fault in mainline 5.14.14 (built
Cc: io-uring@vger.kernel.org
Let me try to remember anything relevant from 5.15,
Thanks for letting know
--
Pavel Begunkov
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-10-25 09:59:21
On 10/22/21 10:10, Pavel Begunkov wrote:
On 10/22/21 04:12, Daniel Black wrote:
quoted
Sometime after 5.11 and is fixed in 5.15-rcX (rc6 extensively tested
over last few days) is a kernel regression we are tracing in
https://jira.mariadb.org/browse/MDEV-26674 and
https://jira.mariadb.org/browse/MDEV-26555
5.10 and early across many distros and hardware appear not to have a problem.
I'd appreciate some help identifying a 5.14 linux stable patch
suitable as I observe the fault in mainline 5.14.14 (built
Cc: io-uring@vger.kernel.org
Let me try to remember anything relevant from 5.15,
Thanks for letting know
Daniel, following the links I found this:
"From: Daniel Black [off-list ref]
...
The good news is I've validated that the linux mainline 5.14.14 build
from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.14.14/ has
actually fixed this problem."
To be clear, is the mainline 5.14 kernel affected with the issue?
Or does the problem exists only in debian/etc. kernel trees?
--
Pavel Begunkov
From: Daniel Black <hidden> Date: 2021-10-25 11:09:42
On Mon, Oct 25, 2021 at 8:59 PM Pavel Begunkov [off-list ref] wrote:
On 10/22/21 10:10, Pavel Begunkov wrote:
quoted
On 10/22/21 04:12, Daniel Black wrote:
quoted
Sometime after 5.11 and is fixed in 5.15-rcX (rc6 extensively tested
over last few days) is a kernel regression we are tracing in
https://jira.mariadb.org/browse/MDEV-26674 and
https://jira.mariadb.org/browse/MDEV-26555
5.10 and early across many distros and hardware appear not to have a problem.
I'd appreciate some help identifying a 5.14 linux stable patch
suitable as I observe the fault in mainline 5.14.14 (built
Cc: io-uring@vger.kernel.org
Let me try to remember anything relevant from 5.15,
Thanks for letting know
Daniel, following the links I found this:
"From: Daniel Black [off-list ref]
...
The good news is I've validated that the linux mainline 5.14.14 build
from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.14.14/ has
actually fixed this problem."
To be clear, is the mainline 5.14 kernel affected with the issue?
Or does the problem exists only in debian/etc. kernel trees?
--
Pavel Begunkov
Thanks Pavel for looking.
I'm retesting https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.14.14/
in earnest. I did get some assertions, but they may have been
unrelated. The testing continues...
The problem with debian trees on 5.14.12 (as
linux-image-5.14.0-3-amd64_5.14.12-1_amd64.deb) was quite real
https://jira.mariadb.org/browse/MDEV-26674?focusedCommentId=203155&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-203155
What is concrete is the fc34 package of 5.14.14 (which obviously does
have a Red Hat delta
https://src.fedoraproject.org/rpms/kernel/blob/f34/f/patch-5.14-redhat.patch),
but unsure of significance. Output below:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1847210
$ uname -a
Linux localhost.localdomain 5.14.14-200.fc34.x86_64 #1 SMP Wed Oct 20
16:15:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
~/repos/mariadb-server-10.6 10.6
$ bd
~/repos/build-mariadb-server-10.6
$ mysql-test/mtr --parallel=4 encryption.innochecksum{,,,,,}
Logging: /home/dan/repos/mariadb-server-10.6/mysql-test/mariadb-test-run.pl
--parallel=4 encryption.innochecksum encryption.innochecksum
encryption.innochecksum encryption.innochecksum
encryption.innochecksum encryption.innochecksum
vardir: /home/dan/repos/build-mariadb-server-10.6/mysql-test/var
Removing old var directory...
- WARNING: Using the 'mysql-test/var' symlink
The destination for symlink
/home/dan/repos/build-mariadb-server-10.6/mysql-test/var does not
exist; Removing it and creating a new var directory
Creating var directory
'/home/dan/repos/build-mariadb-server-10.6/mysql-test/var'...
Checking supported features...
MariaDB Version 10.6.5-MariaDB
- SSL connections supported
- binaries built with wsrep patch
Collecting tests...
Installing system database...
==============================================================================
TEST WORKER RESULT TIME (ms) or COMMENT
--------------------------------------------------------------------------
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
worker[3] Using MTR_BUILD_THREAD 302, with reserved ports 16040..16059
worker[2] Using MTR_BUILD_THREAD 301, with reserved ports 16020..16039
worker[4] Using MTR_BUILD_THREAD 303, with reserved ports 16060..16079
encryption.innochecksum '16k,cbc,innodb,strict_crc32' w3 [ pass ] 5460
encryption.innochecksum '16k,cbc,innodb,strict_crc32' w2 [ pass ] 5418
encryption.innochecksum '16k,cbc,innodb,strict_crc32' w1 [ pass ] 9391
encryption.innochecksum '16k,cbc,innodb,strict_crc32' w3 [ pass ] 8682
encryption.innochecksum '16k,cbc,innodb,strict_crc32' w3 [ pass ] 3873
encryption.innochecksum '8k,cbc,innodb,strict_crc32' w1 [ pass ] 9133
encryption.innochecksum '4k,cbc,innodb,strict_crc32' w2 [ pass ] 11074
encryption.innochecksum '8k,cbc,innodb,strict_crc32' w1 [ pass ] 5253
encryption.innochecksum '16k,cbc,innodb,strict_full_crc32' w3 [ pass ] 4019
encryption.innochecksum '4k,cbc,innodb,strict_crc32' w2 [ pass ] 6318
encryption.innochecksum '16k,cbc,innodb,strict_full_crc32' w3 [ pass ] 6176
encryption.innochecksum '8k,cbc,innodb,strict_crc32' w1 [ pass ] 7305
encryption.innochecksum '16k,cbc,innodb,strict_full_crc32' w3 [ pass ] 4430
encryption.innochecksum '4k,cbc,innodb,strict_crc32' w2 [ pass ] 10005
encryption.innochecksum '8k,cbc,innodb,strict_crc32' w1 [ pass ] 6878
encryption.innochecksum '16k,cbc,innodb,strict_full_crc32' w3 [ pass ] 3613
encryption.innochecksum '16k,cbc,innodb,strict_full_crc32' w3 [ pass ] 3875
encryption.innochecksum '4k,cbc,innodb,strict_crc32' w2 [ pass ] 6612
encryption.innochecksum '8k,cbc,innodb,strict_crc32' w1 [ pass ] 4901
encryption.innochecksum '16k,cbc,innodb,strict_full_crc32' w3 [ pass ] 3853
encryption.innochecksum '8k,cbc,innodb,strict_crc32' w1 [ pass ] 5080
encryption.innochecksum '4k,cbc,innodb,strict_crc32' w2 [ pass ] 7072
encryption.innochecksum '4k,cbc,innodb,strict_crc32' w2 [ pass ] 6774
encryption.innochecksum '4k,cbc,innodb,strict_full_crc32' w3 [ pass ] 7037
encryption.innochecksum '8k,cbc,innodb,strict_full_crc32' w1 [ pass ] 4961
encryption.innochecksum '8k,cbc,innodb,strict_full_crc32' w1 [ pass ] 5692
encryption.innochecksum '4k,cbc,innodb,strict_full_crc32' w3 [ pass ] 8449
encryption.innochecksum '16k,ctr,innodb,strict_crc32' w2 [ pass ] 5515
encryption.innochecksum '8k,cbc,innodb,strict_full_crc32' w1 [ pass ] 5650
encryption.innochecksum '16k,ctr,innodb,strict_crc32' w2 [ pass ] 3722
encryption.innochecksum '4k,cbc,innodb,strict_full_crc32' w3 [ pass ] 6691
encryption.innochecksum '8k,cbc,innodb,strict_full_crc32' w1 [ pass ] 4611
encryption.innochecksum '16k,ctr,innodb,strict_crc32' w2 [ pass ] 4587
encryption.innochecksum '16k,ctr,innodb,strict_crc32' w2 [ pass ] 5465
encryption.innochecksum '8k,cbc,innodb,strict_full_crc32' w1 [ pass ] 6900
encryption.innochecksum '4k,cbc,innodb,strict_full_crc32' w3 [ pass ] 8333
encryption.innochecksum '16k,ctr,innodb,strict_crc32' w2 [ pass ] 4691
encryption.innochecksum '8k,cbc,innodb,strict_full_crc32' w1 [ pass ] 5077
encryption.innochecksum '4k,cbc,innodb,strict_full_crc32' w3 [ pass ] 6319
encryption.innochecksum '16k,ctr,innodb,strict_crc32' w2 [ pass ] 4590
encryption.innochecksum '4k,ctr,innodb,strict_crc32' w1 [ pass ] 9683
encryption.innochecksum '8k,ctr,innodb,strict_crc32' w2 [ pass ] 5404
encryption.innochecksum '4k,ctr,innodb,strict_crc32' w1 [ pass ] 6775
encryption.innochecksum '8k,ctr,innodb,strict_crc32' w2 [ pass ] 6190
encryption.innochecksum '4k,ctr,innodb,strict_crc32' w1 [ pass ] 9354
encryption.innochecksum '8k,ctr,innodb,strict_crc32' w2 [ pass ] 7734
encryption.innochecksum '8k,ctr,innodb,strict_crc32' w2 [ pass ] 4993
encryption.innochecksum '4k,ctr,innodb,strict_crc32' w1 [ pass ] 6280
encryption.innochecksum '8k,ctr,innodb,strict_crc32' w2 [ pass ] 4487
encryption.innochecksum '4k,ctr,innodb,strict_crc32' w1 [ pass ] 6971
encryption.innochecksum '8k,ctr,innodb,strict_crc32' w2 [ pass ] 5172
encryption.innochecksum '4k,ctr,innodb,strict_crc32' w1 [ pass ] 6317
encryption.innochecksum '16k,ctr,innodb,strict_full_crc32' w2 [ pass ] 3371
encryption.innochecksum '16k,ctr,innodb,strict_full_crc32' w2 [ pass ] 3472
encryption.innochecksum '16k,ctr,innodb,strict_full_crc32' w2 [ pass ] 6707
encryption.innochecksum '4k,ctr,innodb,strict_full_crc32' w1 [ pass ] 9337
encryption.innochecksum '16k,ctr,innodb,strict_full_crc32' w2 [ pass ] 9176
encryption.innochecksum '4k,ctr,innodb,strict_full_crc32' w1 [ pass ] 11817
encryption.innochecksum '16k,ctr,innodb,strict_full_crc32' w2 [ pass ] 3419
encryption.innochecksum '16k,ctr,innodb,strict_full_crc32' w2 [ pass ] 5256
encryption.innochecksum '4k,ctr,innodb,strict_full_crc32' w1 [ pass ] 9291
encryption.innochecksum '4k,ctr,innodb,strict_full_crc32' w1 [ pass ] 6508
encryption.innochecksum '4k,ctr,innodb,strict_full_crc32' w2 [ pass ] 6294
encryption.innochecksum '4k,ctr,innodb,strict_full_crc32' w1 [ pass ] 6327
encryption.innochecksum '8k,ctr,innodb,strict_full_crc32' w2 [ pass ] 4579
encryption.innochecksum '8k,ctr,innodb,strict_full_crc32' w1 [ pass ] 4764
encryption.innochecksum '8k,ctr,innodb,strict_full_crc32' w2 [ pass ] 4469
encryption.innochecksum '8k,ctr,innodb,strict_full_crc32' w1 [ pass ] 4677
encryption.innochecksum '8k,ctr,innodb,strict_full_crc32' w2 [ pass ] 4696
encryption.innochecksum '8k,ctr,innodb,strict_full_crc32' w1 [ pass ] 3898
encryption.innochecksum '4k,cbc,innodb,strict_full_crc32' w3 [ pass ] 127358
encryption.innochecksum '16k,cbc,innodb,strict_crc32' w4 [ fail ]
Test ended at 2021-10-25 21:39:13
CURRENT_TEST: encryption.innochecksum
mysqltest: At line 41: query 'INSERT INTO t3 SELECT * FROM t1' failed:
<Unknown> (2013): Lost connection to server during query
The result from queries just before the failure was:
SET GLOBAL innodb_file_per_table = ON;
set global innodb_compression_algorithm = 1;
# Create and populate a tables
CREATE TABLE t1 (a INT AUTO_INCREMENT PRIMARY KEY, b TEXT)
ENGINE=InnoDB ENCRYPTED=YES ENCRYPTION_KEY_ID=4;
CREATE TABLE t2 (a INT AUTO_INCREMENT PRIMARY KEY, b TEXT)
ENGINE=InnoDB ROW_FORMAT=COMPRESSED ENCRYPTED=YES ENCRYPTION_KEY_ID=4;
CREATE TABLE t3 (a INT AUTO_INCREMENT PRIMARY KEY, b TEXT)
ENGINE=InnoDB ROW_FORMAT=COMPRESSED ENCRYPTED=NO;
CREATE TABLE t4 (a INT AUTO_INCREMENT PRIMARY KEY, b TEXT)
ENGINE=InnoDB PAGE_COMPRESSED=1;
CREATE TABLE t5 (a INT AUTO_INCREMENT PRIMARY KEY, b TEXT)
ENGINE=InnoDB PAGE_COMPRESSED=1 ENCRYPTED=YES ENCRYPTION_KEY_ID=4;
CREATE TABLE t6 (a INT AUTO_INCREMENT PRIMARY KEY, b TEXT) ENGINE=InnoDB;
Server [mysqld.1 - pid: 15380, winpid: 15380, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
$ /home/dan/repos/build-mariadb-server-10.6/sql/mariadbd
--defaults-group-suffix=.1
--defaults-file=/home/dan/repos/build-mariadb-server-10.6/mysql-test/var/4/my.cnf
--log-output=file --innodb-page-size=16K
--skip-innodb-read-only-compressed
--innodb-checksum-algorithm=strict_crc32 --innodb-flush-sync=OFF
--innodb --innodb-cmpmem --innodb-cmp-per-index --innodb-trx
--innodb-locks --innodb-lock-waits --innodb-metrics
--innodb-buffer-pool-stats --innodb-buffer-page
--innodb-buffer-page-lru --innodb-sys-columns --innodb-sys-fields
--innodb-sys-foreign --innodb-sys-foreign-cols --innodb-sys-indexes
--innodb-sys-tables --innodb-sys-virtual
--plugin-load-add=file_key_management.so --loose-file-key-management
--loose-file-key-management-filename=/home/dan/repos/mariadb-server-10.6/mysql-test/std_data/keys.txt
--file-key-management-encryption-algorithm=aes_cbc
--skip-innodb-read-only-compressed --core-file
--loose-debug-sync-timeout=300
2021-10-25 21:28:56 0 [Note]
/home/dan/repos/build-mariadb-server-10.6/sql/mariadbd (server
10.6.5-MariaDB-log) starting as process 15381 ...
2021-10-25 21:28:56 0 [Warning] Could not increase number of
max_open_files to more than 1024 (request: 32190)
2021-10-25 21:28:56 0 [Warning] Changed limits: max_open_files: 1024
max_connections: 151 (was 151) table_cache: 421 (was 2000)
2021-10-25 21:28:56 0 [Note] Plugin 'partition' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'SEQUENCE' is disabled.
2021-10-25 21:28:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-10-25 21:28:56 0 [Note] InnoDB: Number of pools: 1
2021-10-25 21:28:56 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-10-25 21:28:56 0 [Note] InnoDB: Using liburing
2021-10-25 21:28:56 0 [Note] InnoDB: Initializing buffer pool, total
size = 8388608, chunk size = 8388608
2021-10-25 21:28:56 0 [Note] InnoDB: Completed initialization of buffer pool
2021-10-25 21:28:56 0 [Note] InnoDB: 128 rollback segments are active.
2021-10-25 21:28:56 0 [Note] InnoDB: Creating shared tablespace for
temporary tables
2021-10-25 21:28:56 0 [Note] InnoDB: Setting file './ibtmp1' size to
12 MB. Physically writing the file full; Please wait ...
2021-10-25 21:28:56 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-10-25 21:28:56 0 [Note] InnoDB: 10.6.5 started; log sequence
number 43637; transaction id 17
2021-10-25 21:28:56 0 [Note] InnoDB: Loading buffer pool(s) from
/home/dan/repos/build-mariadb-server-10.6/mysql-test/var/4/mysqld.1/data/ib_buffer_pool
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_CMP' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'THREAD_POOL_WAITS' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'THREAD_POOL_QUEUES' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'THREAD_POOL_GROUPS' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'user_variables' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'THREAD_POOL_STATS' is disabled.
2021-10-25 21:28:56 0 [Note] Plugin 'unix_socket' is disabled.
2021-10-25 21:28:56 0 [Warning]
/home/dan/repos/build-mariadb-server-10.6/sql/mariadbd: unknown
variable 'loose-feedback-debug-startup-interval=20'
2021-10-25 21:28:56 0 [Warning]
/home/dan/repos/build-mariadb-server-10.6/sql/mariadbd: unknown
variable 'loose-feedback-debug-first-interval=60'
2021-10-25 21:28:56 0 [Warning]
/home/dan/repos/build-mariadb-server-10.6/sql/mariadbd: unknown
variable 'loose-feedback-debug-interval=60'
2021-10-25 21:28:56 0 [Warning]
/home/dan/repos/build-mariadb-server-10.6/sql/mariadbd: unknown option
'--loose-pam-debug'
2021-10-25 21:28:56 0 [Warning]
/home/dan/repos/build-mariadb-server-10.6/sql/mariadbd: unknown option
'--loose-aria'
2021-10-25 21:28:56 0 [Warning]
/home/dan/repos/build-mariadb-server-10.6/sql/mariadbd: unknown
variable 'loose-debug-sync-timeout=300'
2021-10-25 21:28:56 0 [Note] Server socket created on IP: '127.0.0.1'.
2021-10-25 21:28:56 0 [Note]
/home/dan/repos/build-mariadb-server-10.6/sql/mariadbd: ready for
connections.
Version: '10.6.5-MariaDB-log' socket:
'/home/dan/repos/build-mariadb-server-10.6/mysql-test/var/tmp/4/mysqld.1.sock'
port: 16060 Source distribution
2021-10-25 21:28:56 0 [Note] InnoDB: Buffer pool(s) load completed at
211025 21:28:56
2021-10-25 21:39:11 0 [ERROR] [FATAL] InnoDB:
innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch
From: Pavel Begunkov <asml.silence@gmail.com> Date: 2021-10-25 11:26:47
On 10/25/21 12:09, Daniel Black wrote:
On Mon, Oct 25, 2021 at 8:59 PM Pavel Begunkov [off-list ref] wrote:
quoted
On 10/22/21 10:10, Pavel Begunkov wrote:
quoted
On 10/22/21 04:12, Daniel Black wrote:
quoted
Sometime after 5.11 and is fixed in 5.15-rcX (rc6 extensively tested
over last few days) is a kernel regression we are tracing in
https://jira.mariadb.org/browse/MDEV-26674 and
https://jira.mariadb.org/browse/MDEV-26555
5.10 and early across many distros and hardware appear not to have a problem.
I'd appreciate some help identifying a 5.14 linux stable patch
suitable as I observe the fault in mainline 5.14.14 (built
Cc: io-uring@vger.kernel.org
Let me try to remember anything relevant from 5.15,
Thanks for letting know
Daniel, following the links I found this:
"From: Daniel Black [off-list ref]
...
The good news is I've validated that the linux mainline 5.14.14 build
from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.14.14/ has
actually fixed this problem."
To be clear, is the mainline 5.14 kernel affected with the issue?
Or does the problem exists only in debian/etc. kernel trees?
Hi Daniel,
On Mon, Oct 25, 2021 at 12:25:01PM +0100, Pavel Begunkov wrote:
On 10/25/21 12:09, Daniel Black wrote:
quoted
On Mon, Oct 25, 2021 at 8:59 PM Pavel Begunkov [off-list ref] wrote:
quoted
On 10/22/21 10:10, Pavel Begunkov wrote:
quoted
On 10/22/21 04:12, Daniel Black wrote:
quoted
Sometime after 5.11 and is fixed in 5.15-rcX (rc6 extensively tested
over last few days) is a kernel regression we are tracing in
https://jira.mariadb.org/browse/MDEV-26674 and
https://jira.mariadb.org/browse/MDEV-26555
5.10 and early across many distros and hardware appear not to have a problem.
I'd appreciate some help identifying a 5.14 linux stable patch
suitable as I observe the fault in mainline 5.14.14 (built
Cc: io-uring@vger.kernel.org
Let me try to remember anything relevant from 5.15,
Thanks for letting know
Daniel, following the links I found this:
"From: Daniel Black [off-list ref]
...
The good news is I've validated that the linux mainline 5.14.14 build
from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.14.14/ has
actually fixed this problem."
To be clear, is the mainline 5.14 kernel affected with the issue?
Or does the problem exists only in debian/etc. kernel trees?
Thanks for the work on pinpointing it. I'll wait for your conclusion
then, it'll give us an idea what we should look for.
Were you able to pinpoint the issue?
Retesting on the ubuntu mainline 5.14.14 and 5.14.15 was unable to
reproduce the issue in a VM.
Using Fedora (34) 5.14.14 and 5.14.15 kernel I am reasonably able to
reproduce this, and it is now reported as
https://bugzilla.redhat.com/show_bug.cgi?id=2018882.
I've so far been unable to reproduce this issue on 5.15.0-rc7 inside a
(Ubuntu-21.10) VM.
Marko did using a other heavy flushing sysbench script (modified
version attached - slightly lower specs, and can be used on distro
install) was able to see the fault (qps goes to 0) using Debian sid
userspace and 5.15-rc6/5.15-rc7 Ubuntu mainline kernels.
https://jira.mariadb.org/browse/MDEV-26674?focusedCommentId=203645&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-203645
Note if using a mariadb-10.6.5 (not quite released), there's a change
of defaults to avoid this bug, mtr options
--mysqld=--innodb_use_native_aio=1 --nowarnings will test this
however.
Thanks, this is most useful! I'll take a look at this.
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
The script also seems to assume that various things are setup
appropriately, like SRCTREE, MDIR, etc.
--
Jens Axboe
From: Daniel Black <hidden> Date: 2021-11-11 06:52:49
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
Sorry Jens.
Hope containers are ok.
mkdir ~/mdbtest/
$ podman run -d -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 -e
MARIADB_USER=sbtest -e MARIADB_PASSWORD=sbtest -e
MARIADB_DATABASE=sbtest --name mdb10.6-uring_test -v
$HOME/mdbtest:/var/lib/mysql:Z --security-opt seccomp=unconfined
quay.io/danielgblack/mariadb-test:10.6-impish-sysbench
--innodb_log_file_size=1G --innodb_buffer_pool_size=50G
--innodb_io_capacity=5000 --innodb_io_capacity_max=9000
--innodb_flush_log_at_trx_commit=0 --innodb_adaptive_flushing_lwm=0
--innodb-adaptive-flushing=1 --innodb_flush_neighbors=1
--innodb-use-native-aio=1 --innodb_file-per-table=1
--innodb-fast-shutdown=0 --innodb-flush-method=O_DIRECT
--innodb_lru_scan_depth=1024 --innodb_lru_flush_size=256
# drop 50G pool size down if you don't have it. Not critical to
reproduction. IO capacity here should be about what the hardware is.
Otherwise gaps of 0 tps will appear without it being the cause of the
bug.
$ podman logs mdb10.6-uring_test
...
2021-11-11 6:06:49 0 [Warning] innodb_use_native_aio may cause hangs
with this kernel 5.15.0-0.rc7.20211028git1fc596a56b33.56.fc36.x86_64;
see https://jira.mariadb.org/browse/MDEV-26674
2021-11-11 6:06:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-11-11 6:06:49 0 [Note] InnoDB: Number of pools: 1
2021-11-11 6:06:49 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-11-11 6:06:49 0 [Note] mysqld: O_TMPFILE is not supported on
/tmp (disabling future attempts)
2021-11-11 6:06:49 0 [Note] InnoDB: Using liburing
Should contain first and last line here:
$ podman exec mdb10.6-uring_test sysbench
/usr/share/sysbench/oltp_update_index.lua --mysql-password=sbtest
--percentile=99 --tables=8 --table_size=2000000 prepare
Creating table 'sbtest1'...
Inserting 2000000 records into 'sbtest1'
Creating a secondary index on 'sbtest1'...
Creating table 'sbtest2'...
Inserting 2000000 records into 'sbtest2'
Creating a secondary index on 'sbtest2'...
Creating table 'sbtest3'...
Inserting 2000000 records into 'sbtest3'
Creating a secondary index on 'sbtest3'...
Creating table 'sbtest4'...
Inserting 2000000 records into 'sbtest4'
Creating a secondary index on 'sbtest4'...
Creating table 'sbtest5'...
Inserting 2000000 records into 'sbtest5'
Creating a secondary index on 'sbtest5'...
Creating table 'sbtest6'...
Inserting 2000000 records into 'sbtest6'
Creating a secondary index on 'sbtest6'...
Creating table 'sbtest7'...
Inserting 2000000 records into 'sbtest7'
Creating a secondary index on 'sbtest7'...
Creating table 'sbtest8'...
Inserting 2000000 records into 'sbtest8'
Creating a secondary index on 'sbtest8'...
# Adjust threads there to the amount of hardware threads available.
time is the length of the test.
$ podman exec mdb10.6-uring_test sysbench
/usr/share/sysbench/oltp_update_index.lua --mysql-password=sbtest
--percentile=99 --tables=8 --table_size=2000000 --rand-seed=42
--rand-type=uniform --max-requests=0 --time=600 --report-interval=5
--threads=64 run
Eventually after
https://mariadb.com/kb/en/innodb-system-variables/#innodb_fatal_semaphore_wait_threshold
of 600 seconds the podman logs mdb10.6-uring_test will contains an
error like:
2021-10-07 17:06:43 0 [ERROR] [FATAL] InnoDB:
innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch.
Please refer to
https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
211007 17:06:43 [ERROR] mysqld got signal 6 ;
Restarting the container on the same populated ~/mdbtest volume could
be slow due to recovery time. Remove contents and repeat prepare step.
cleanup:
podman kill mdb10.6-uring_test
podman rm mdb10.6-uring_test
sudo rm -rf ~/mdbtest
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
Sorry Jens.
Hope containers are ok.
Don't think I have a way to run that, don't even know what podman is
and nor does my distro. I'll google a bit and see if I can get this
running.
I'm fine building from source and running from there, as long as I
know what to do. Would that make it any easier? It definitely would
for me :-)
--
Jens Axboe
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
Sorry Jens.
Hope containers are ok.
Don't think I have a way to run that, don't even know what podman is
and nor does my distro. I'll google a bit and see if I can get this
running.
I'm fine building from source and running from there, as long as I
know what to do. Would that make it any easier? It definitely would
for me :-)
The podman approach seemed to work, and I was able to run all three
steps. Didn't see any hangs. I'm going to try again dropping down
the innodb pool size (box only has 32G of RAM).
The storage can do a lot more than 5k IOPS, I'm going to try ramping
that up.
Does your reproducer box have multiple NUMA nodes, or is it a single
socket/nod box?
--
Jens Axboe
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
Sorry Jens.
Hope containers are ok.
Don't think I have a way to run that, don't even know what podman is
and nor does my distro. I'll google a bit and see if I can get this
running.
I'm fine building from source and running from there, as long as I
know what to do. Would that make it any easier? It definitely would
for me :-)
The podman approach seemed to work, and I was able to run all three
steps. Didn't see any hangs. I'm going to try again dropping down
the innodb pool size (box only has 32G of RAM).
The storage can do a lot more than 5k IOPS, I'm going to try ramping
that up.
Does your reproducer box have multiple NUMA nodes, or is it a single
socket/nod box?
Doesn't seem to reproduce for me on current -git. What file system are
you using?
--
Jens Axboe
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
Sorry Jens.
Hope containers are ok.
Don't think I have a way to run that, don't even know what podman is
and nor does my distro. I'll google a bit and see if I can get this
running.
I'm fine building from source and running from there, as long as I
know what to do. Would that make it any easier? It definitely would
for me :-)
The podman approach seemed to work, and I was able to run all three
steps. Didn't see any hangs. I'm going to try again dropping down
the innodb pool size (box only has 32G of RAM).
The storage can do a lot more than 5k IOPS, I'm going to try ramping
that up.
Does your reproducer box have multiple NUMA nodes, or is it a single
socket/nod box?
Doesn't seem to reproduce for me on current -git. What file system are
you using?
I seem to be able to hit it with ext4, guessing it has more cases that
punt to buffered IO. As I initially suspected, I think this is a race
with buffered file write hashing. I have a debug patch that just turns
a regular non-numa box into multi nodes, may or may not be needed be
needed to hit this, but I definitely can now. Looks like this:
Node7 DUMP
index=0, nr_w=1, max=128, r=0, f=1, h=0
w=ffff8f5e8b8470c0, hashed=1/0, flags=2
w=ffff8f5e95a9b8c0, hashed=1/0, flags=2
index=1, nr_w=0, max=127877, r=0, f=0, h=0
free_list
worker=ffff8f5eaf2e0540
all_list
worker=ffff8f5eaf2e0540
where we seed node7 in this case having two work items pending, but the
worker state is stalled on hash.
The hash logic was rewritten as part of the io-wq worker threads being
changed for 5.11 iirc, which is why that was my initial suspicion here.
I'll take a look at this and make a test patch. Looks like you are able
to test self-built kernels, is that correct?
--
Jens Axboe
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
Sorry Jens.
Hope containers are ok.
Don't think I have a way to run that, don't even know what podman is
and nor does my distro. I'll google a bit and see if I can get this
running.
I'm fine building from source and running from there, as long as I
know what to do. Would that make it any easier? It definitely would
for me :-)
The podman approach seemed to work, and I was able to run all three
steps. Didn't see any hangs. I'm going to try again dropping down
the innodb pool size (box only has 32G of RAM).
The storage can do a lot more than 5k IOPS, I'm going to try ramping
that up.
Does your reproducer box have multiple NUMA nodes, or is it a single
socket/nod box?
Doesn't seem to reproduce for me on current -git. What file system are
you using?
I seem to be able to hit it with ext4, guessing it has more cases that
punt to buffered IO. As I initially suspected, I think this is a race
with buffered file write hashing. I have a debug patch that just turns
a regular non-numa box into multi nodes, may or may not be needed be
needed to hit this, but I definitely can now. Looks like this:
Node7 DUMP
index=0, nr_w=1, max=128, r=0, f=1, h=0
w=ffff8f5e8b8470c0, hashed=1/0, flags=2
w=ffff8f5e95a9b8c0, hashed=1/0, flags=2
index=1, nr_w=0, max=127877, r=0, f=0, h=0
free_list
worker=ffff8f5eaf2e0540
all_list
worker=ffff8f5eaf2e0540
where we seed node7 in this case having two work items pending, but the
worker state is stalled on hash.
The hash logic was rewritten as part of the io-wq worker threads being
changed for 5.11 iirc, which is why that was my initial suspicion here.
I'll take a look at this and make a test patch. Looks like you are able
to test self-built kernels, is that correct?
Can you try with this patch? It's against -git, but it will apply to
5.15 as well.
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
Sorry Jens.
Hope containers are ok.
Don't think I have a way to run that, don't even know what podman is
and nor does my distro. I'll google a bit and see if I can get this
running.
I'm fine building from source and running from there, as long as I
know what to do. Would that make it any easier? It definitely would
for me :-)
The podman approach seemed to work, and I was able to run all three
steps. Didn't see any hangs. I'm going to try again dropping down
the innodb pool size (box only has 32G of RAM).
The storage can do a lot more than 5k IOPS, I'm going to try ramping
that up.
Does your reproducer box have multiple NUMA nodes, or is it a single
socket/nod box?
Doesn't seem to reproduce for me on current -git. What file system are
you using?
I seem to be able to hit it with ext4, guessing it has more cases that
punt to buffered IO. As I initially suspected, I think this is a race
with buffered file write hashing. I have a debug patch that just turns
a regular non-numa box into multi nodes, may or may not be needed be
needed to hit this, but I definitely can now. Looks like this:
Node7 DUMP
index=0, nr_w=1, max=128, r=0, f=1, h=0
w=ffff8f5e8b8470c0, hashed=1/0, flags=2
w=ffff8f5e95a9b8c0, hashed=1/0, flags=2
index=1, nr_w=0, max=127877, r=0, f=0, h=0
free_list
worker=ffff8f5eaf2e0540
all_list
worker=ffff8f5eaf2e0540
where we seed node7 in this case having two work items pending, but the
worker state is stalled on hash.
The hash logic was rewritten as part of the io-wq worker threads being
changed for 5.11 iirc, which is why that was my initial suspicion here.
I'll take a look at this and make a test patch. Looks like you are able
to test self-built kernels, is that correct?
Can you try with this patch? It's against -git, but it will apply to
5.15 as well.
I think that one covered one potential gap, but I just managed to
reproduce a stall even with it. So hang on testing that one, I'll send
you something more complete when I have confidence in it.
--
Jens Axboe
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
Sorry Jens.
Hope containers are ok.
Don't think I have a way to run that, don't even know what podman is
and nor does my distro. I'll google a bit and see if I can get this
running.
I'm fine building from source and running from there, as long as I
know what to do. Would that make it any easier? It definitely would
for me :-)
The podman approach seemed to work, and I was able to run all three
steps. Didn't see any hangs. I'm going to try again dropping down
the innodb pool size (box only has 32G of RAM).
The storage can do a lot more than 5k IOPS, I'm going to try ramping
that up.
Does your reproducer box have multiple NUMA nodes, or is it a single
socket/nod box?
Doesn't seem to reproduce for me on current -git. What file system are
you using?
I seem to be able to hit it with ext4, guessing it has more cases that
punt to buffered IO. As I initially suspected, I think this is a race
with buffered file write hashing. I have a debug patch that just turns
a regular non-numa box into multi nodes, may or may not be needed be
needed to hit this, but I definitely can now. Looks like this:
Node7 DUMP
index=0, nr_w=1, max=128, r=0, f=1, h=0
w=ffff8f5e8b8470c0, hashed=1/0, flags=2
w=ffff8f5e95a9b8c0, hashed=1/0, flags=2
index=1, nr_w=0, max=127877, r=0, f=0, h=0
free_list
worker=ffff8f5eaf2e0540
all_list
worker=ffff8f5eaf2e0540
where we seed node7 in this case having two work items pending, but the
worker state is stalled on hash.
The hash logic was rewritten as part of the io-wq worker threads being
changed for 5.11 iirc, which is why that was my initial suspicion here.
I'll take a look at this and make a test patch. Looks like you are able
to test self-built kernels, is that correct?
Can you try with this patch? It's against -git, but it will apply to
5.15 as well.
I think that one covered one potential gap, but I just managed to
reproduce a stall even with it. So hang on testing that one, I'll send
you something more complete when I have confidence in it.
Alright, give this one a go if you can. Against -git, but will apply to
5.15 as well.
From: Daniel Black <hidden> Date: 2021-11-12 06:25:47
On Fri, Nov 12, 2021 at 10:44 AM Jens Axboe [off-list ref] wrote:
On 11/11/21 10:28 AM, Jens Axboe wrote:
quoted
On 11/11/21 9:55 AM, Jens Axboe wrote:
quoted
On 11/11/21 9:19 AM, Jens Axboe wrote:
quoted
On 11/11/21 8:29 AM, Jens Axboe wrote:
quoted
On 11/11/21 7:58 AM, Jens Axboe wrote:
quoted
On 11/11/21 7:30 AM, Jens Axboe wrote:
quoted
On 11/10/21 11:52 PM, Daniel Black wrote:
quoted
quoted
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
Sorry Jens.
Hope containers are ok.
Don't think I have a way to run that, don't even know what podman is
and nor does my distro. I'll google a bit and see if I can get this
running.
I'm fine building from source and running from there, as long as I
know what to do. Would that make it any easier? It definitely would
for me :-)
The podman approach seemed to work,
Thanks for bearing with it.
quoted
quoted
quoted
quoted
quoted
and I was able to run all three
steps. Didn't see any hangs. I'm going to try again dropping down
the innodb pool size (box only has 32G of RAM).
The storage can do a lot more than 5k IOPS, I'm going to try ramping
that up.
Good.
quoted
quoted
quoted
quoted
quoted
Does your reproducer box have multiple NUMA nodes, or is it a single
socket/nod box?
It was NUMA. Pre 5.14.14 I could produce it on a simpler test on a single node.
quoted
quoted
quoted
quoted
Doesn't seem to reproduce for me on current -git. What file system are
you using?
Yes ext4.
quoted
quoted
quoted
I seem to be able to hit it with ext4, guessing it has more cases that
punt to buffered IO. As I initially suspected, I think this is a race
with buffered file write hashing. I have a debug patch that just turns
a regular non-numa box into multi nodes, may or may not be needed be
needed to hit this, but I definitely can now. Looks like this:
Node7 DUMP
index=0, nr_w=1, max=128, r=0, f=1, h=0
w=ffff8f5e8b8470c0, hashed=1/0, flags=2
w=ffff8f5e95a9b8c0, hashed=1/0, flags=2
index=1, nr_w=0, max=127877, r=0, f=0, h=0
free_list
worker=ffff8f5eaf2e0540
all_list
worker=ffff8f5eaf2e0540
where we seed node7 in this case having two work items pending, but the
worker state is stalled on hash.
The hash logic was rewritten as part of the io-wq worker threads being
changed for 5.11 iirc, which is why that was my initial suspicion here.
I'll take a look at this and make a test patch. Looks like you are able
to test self-built kernels, is that correct?
Can you try with this patch? It's against -git, but it will apply to
5.15 as well.
I think that one covered one potential gap, but I just managed to
reproduce a stall even with it. So hang on testing that one, I'll send
you something more complete when I have confidence in it.
Alright, give this one a go if you can. Against -git, but will apply to
5.15 as well.
Daniel,
On Fri, Nov 12, 2021 at 05:25:31PM +1100, Daniel Black wrote:
On Fri, Nov 12, 2021 at 10:44 AM Jens Axboe [off-list ref] wrote:
quoted
On 11/11/21 10:28 AM, Jens Axboe wrote:
quoted
On 11/11/21 9:55 AM, Jens Axboe wrote:
quoted
On 11/11/21 9:19 AM, Jens Axboe wrote:
quoted
On 11/11/21 8:29 AM, Jens Axboe wrote:
quoted
On 11/11/21 7:58 AM, Jens Axboe wrote:
quoted
On 11/11/21 7:30 AM, Jens Axboe wrote:
quoted
On 11/10/21 11:52 PM, Daniel Black wrote:
quoted
quoted
Would it be possible to turn this into a full reproducer script?
Something that someone that knows nothing about mysqld/mariadb can just
run and have it reproduce. If I install the 10.6 packages from above,
then it doesn't seem to use io_uring or be linked against liburing.
Sorry Jens.
Hope containers are ok.
Don't think I have a way to run that, don't even know what podman is
and nor does my distro. I'll google a bit and see if I can get this
running.
I'm fine building from source and running from there, as long as I
know what to do. Would that make it any easier? It definitely would
for me :-)
The podman approach seemed to work,
Thanks for bearing with it.
quoted
quoted
quoted
quoted
quoted
quoted
and I was able to run all three
steps. Didn't see any hangs. I'm going to try again dropping down
the innodb pool size (box only has 32G of RAM).
The storage can do a lot more than 5k IOPS, I'm going to try ramping
that up.
Good.
quoted
quoted
quoted
quoted
quoted
quoted
Does your reproducer box have multiple NUMA nodes, or is it a single
socket/nod box?
It was NUMA. Pre 5.14.14 I could produce it on a simpler test on a single node.
quoted
quoted
quoted
quoted
quoted
Doesn't seem to reproduce for me on current -git. What file system are
you using?
Yes ext4.
quoted
quoted
quoted
quoted
I seem to be able to hit it with ext4, guessing it has more cases that
punt to buffered IO. As I initially suspected, I think this is a race
with buffered file write hashing. I have a debug patch that just turns
a regular non-numa box into multi nodes, may or may not be needed be
needed to hit this, but I definitely can now. Looks like this:
Node7 DUMP
index=0, nr_w=1, max=128, r=0, f=1, h=0
w=ffff8f5e8b8470c0, hashed=1/0, flags=2
w=ffff8f5e95a9b8c0, hashed=1/0, flags=2
index=1, nr_w=0, max=127877, r=0, f=0, h=0
free_list
worker=ffff8f5eaf2e0540
all_list
worker=ffff8f5eaf2e0540
where we seed node7 in this case having two work items pending, but the
worker state is stalled on hash.
The hash logic was rewritten as part of the io-wq worker threads being
changed for 5.11 iirc, which is why that was my initial suspicion here.
I'll take a look at this and make a test patch. Looks like you are able
to test self-built kernels, is that correct?
Can you try with this patch? It's against -git, but it will apply to
5.15 as well.
I think that one covered one potential gap, but I just managed to
reproduce a stall even with it. So hang on testing that one, I'll send
you something more complete when I have confidence in it.
Alright, give this one a go if you can. Against -git, but will apply to
5.15 as well.
Applied, built, attempting to boot....
If you want to do the same for Debian based system, the following
might help to get the package built:
https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s4.2.2
I might be able to provide you otherwise a prebuild package with the
patch (unsigned though, but best if you built and test it directly)
Regards,
Salvatore
Awesome, thanks so much for reporting and testing. All bugs are shallow
when given a reproducer, that certainly helped a ton in figuring out
what this was and nailing a fix.
The patch is already upstream (and in the 5.15 stable queue), and I
provided 5.14 patches too.
--
Jens Axboe
Awesome, thanks so much for reporting and testing. All bugs are shallow
when given a reproducer, that certainly helped a ton in figuring out
what this was and nailing a fix.
The patch is already upstream (and in the 5.15 stable queue), and I
provided 5.14 patches too.
Awesome, thanks so much for reporting and testing. All bugs are shallow
when given a reproducer, that certainly helped a ton in figuring out
what this was and nailing a fix.
The patch is already upstream (and in the 5.15 stable queue), and I
provided 5.14 patches too.
The patch is already upstream (and in the 5.15 stable queue), and I
provided 5.14 patches too.
Jens,
I'm getting the same reproducer on 5.14.20
(https://bugzilla.redhat.com/show_bug.cgi?id=2018882#c3) though the
backport change logs indicate 5.14.19 has the patch.
Anything missing?
ext4 again (my mount is /dev/mapper/fedora_localhost--live-home on
/home type ext4 (rw,relatime,seclabel)).
previous container should work, thought a source option is there:
build deps: liburing-dev, bison, libevent-dev, ncurses-dev, c++
libraries/compiler
git clone --branch 10.6 --single-branch
https://github.com/MariaDB/server mariadb-server
(cd mariadb-server; git submodule update --init --recursive)
mkdir build-mariadb-server
cd build-mariadb-server
cmake -DPLUGIN_{MROONGA,ROCKSDB,CONNECT,SPIDER,SPHINX,S3,COLUMNSTORE}=NO
../mariadb-server
(ensure liburing userspace is picked up)
cmake --build . --parallel
mysql-test/mtr --mysqld=--innodb_use_native_aio=1 --nowarnings
--parallel=4 --force encryption.innochecksum{,,,,,}
Adding to mtr: --mysqld=--innodb_io_capacity=50000
--mysqld=--innodb_io_capacity_max=90000 will probably trip this
quicker.
5.15.3 is good (https://jira.mariadb.org/browse/MDEV-26674?focusedCommentId=206787&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-206787).
We might also need another patch that isn't in stable, I'm attaching
it here. Any chance you can run 5.14.20/21 with this applied? If not,
I'll do some sanity checking here and push it to -stable.
--
Jens Axboe
We might also need another patch that isn't in stable, I'm attaching
it here. Any chance you can run 5.14.20/21 with this applied? If not,
I'll do some sanity checking here and push it to -stable.
Looks good to me - Greg, would you mind queueing this up for
5.14-stable?
--
Jens Axboe
We might also need another patch that isn't in stable, I'm attaching
it here. Any chance you can run 5.14.20/21 with this applied? If not,
I'll do some sanity checking here and push it to -stable.
Looks good to me - Greg, would you mind queueing this up for
5.14-stable?
5.14 is end-of-life and not getting any more releases (the front page of
kernel.org should show that.)
If this needs to go anywhere else, please let me know.
thanks,
greg k-h
We might also need another patch that isn't in stable, I'm attaching
it here. Any chance you can run 5.14.20/21 with this applied? If not,
I'll do some sanity checking here and push it to -stable.
Looks good to me - Greg, would you mind queueing this up for
5.14-stable?
5.14 is end-of-life and not getting any more releases (the front page of
kernel.org should show that.)
Oh, well I guess that settles that...
If this needs to go anywhere else, please let me know.
Should be fine, previous 5.10 isn't affected and 5.15 is fine too as it
already has the patch.
--
Jens Axboe
Looks good to me - Greg, would you mind queueing this up for
5.14-stable?
5.14 is end-of-life and not getting any more releases (the front page of
kernel.org should show that.)
Oh, well I guess that settles that...
quoted
If this needs to go anywhere else, please let me know.
Should be fine, previous 5.10 isn't affected and 5.15 is fine too as it
already has the patch.
Are 5.11 and 5.13 are affected, these are hwe kernels for ubuntu,
I may need to open a bug for them...
Please do, then we can help get the appropriate patches lined up for
5.11/13. They should need the same set, basically what ended up in 5.14
plus the one I posted today.
--
Jens Axboe
From: Stefan Metzmacher <metze@samba.org> Date: 2021-11-25 16:48:59
Am 25.11.21 um 01:58 schrieb Jens Axboe:
On 11/24/21 3:52 PM, Stefan Metzmacher wrote:
quoted
Hi Jens,
quoted
quoted
quoted
Looks good to me - Greg, would you mind queueing this up for
5.14-stable?
5.14 is end-of-life and not getting any more releases (the front page of
kernel.org should show that.)
Oh, well I guess that settles that...
quoted
If this needs to go anywhere else, please let me know.
Should be fine, previous 5.10 isn't affected and 5.15 is fine too as it
already has the patch.
Are 5.11 and 5.13 are affected, these are hwe kernels for ubuntu,
I may need to open a bug for them...
Please do, then we can help get the appropriate patches lined up for
5.11/13. They should need the same set, basically what ended up in 5.14
plus the one I posted today.
Looks good to me - Greg, would you mind queueing this up for
5.14-stable?
5.14 is end-of-life and not getting any more releases (the front page of
kernel.org should show that.)
Oh, well I guess that settles that...
quoted
If this needs to go anywhere else, please let me know.
Should be fine, previous 5.10 isn't affected and 5.15 is fine too as it
already has the patch.
Are 5.11 and 5.13 are affected, these are hwe kernels for ubuntu,
I may need to open a bug for them...
Please do, then we can help get the appropriate patches lined up for
5.11/13. They should need the same set, basically what ended up in 5.14
plus the one I posted today.
Let me know if I can help, should probably prepare a set for 5.11-stable
and 5.13-stable, but I don't know if the above kernels already have some
patches applied past last stable release of each...
--
Jens Axboe
From: Stefan Metzmacher <metze@samba.org> Date: 2022-02-10 01:25:19
Hi Jens,
quoted
quoted
quoted
quoted
quoted
Looks good to me - Greg, would you mind queueing this up for
5.14-stable?
5.14 is end-of-life and not getting any more releases (the front page of
kernel.org should show that.)
Oh, well I guess that settles that...
quoted
If this needs to go anywhere else, please let me know.
Should be fine, previous 5.10 isn't affected and 5.15 is fine too as it
already has the patch.
Are 5.11 and 5.13 are affected, these are hwe kernels for ubuntu,
I may need to open a bug for them...
Please do, then we can help get the appropriate patches lined up for
5.11/13. They should need the same set, basically what ended up in 5.14
plus the one I posted today.
We might also need another patch that isn't in stable, I'm attaching
it here. Any chance you can run 5.14.20/21 with this applied? If not,
I'll do some sanity checking here and push it to -stable.
Looks good to me - Greg, would you mind queueing this up for
5.14-stable?
5.14 is end-of-life and not getting any more releases (the front page of
kernel.org should show that.)
Oh, well I guess that settles that...
Certainly does. Thanks for looking and finding the patch.
quoted
If this needs to go anywhere else, please let me know.
Should be fine, previous 5.10 isn't affected and 5.15 is fine too as it
already has the patch.