Thread (2 messages) 2 messages, 1 author, 2017-10-23

[linux-mips] [openwrt] [sc16is7xx] [bitbang I2C] Any suggestions on board reboot caused by "sched: RT throttling activated"?

From: Xuebing Wang <hidden>
Date: 2017-10-19 05:40:40

Hi community,

Any suggestions on how should I proceed with my issue?

Description of my issue:
-  My board kernel prints: "sched: RT throttling activated", then reboot 
(there is no back-trace in UART console).
-  I have sc16is752 (I2C based UART) on my board.
-  As Atheros AR9331 does not have dedicated I2C controller, I am using 
bitbang I2C.
-  This is likely related to below line of sc16is752 driver 
(drivers//tty/serial/sc16is7xx.c)
         ret = devm_request_threaded_irq(dev, irq, NULL, sc16is7xx_ist,
-  './ps -eo pid,nice,policy,cmd' confirms that above is a realtime 
kernel thread (with SCHED_FIFO scheduling policy)
-  Occurrence rate is about once or several times a day, after I stress 
I2C/UART traffic.
-  Occurrence rate is much lower if I do *NOT* stress I2C/UART traffic.

My setup:
-  Atheros AR9331 chip, which is MIPS32 24kc
-  OpenWRT tag 15.05
-  Linux kernel v3.18.29
-  Not preempt kernel (CONFIG_PREEMPT_NONE=y)
-  As AR9331 does not have dedicated I2C controller, I am using bitbang 
I2C as below:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/tree/drivers/i2c/algos/i2c-algo-bit.c?h=linux-3.18.y
-  source code of sc16is752 driver sc16is7xx.c is here:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/tree/drivers/tty/serial/sc16is7xx.c?h=linux-3.18.y

Question 1): There seems a fundamental question about OpenWRT mips linux 
kernel?
-  OpenWRT tag 15.05
-  In OpenWRT file target/linux/ar71xx/Makefile, there is CPU_TYPE=34kc
-  Atheros AR9331 datasheet says it is 24kc (rather than 34kc)
-  With CPU_TYPE=34kc, it uses gcc options "-mips32r2 -mtune=34kc"

Question 2): I have no idea what causes the reboot? It is a chip 
hardware reboot, or function similar to emergency_restart() is called 
somewhere in the kernel? Kernel console only shows "sched: RT throttling 
activated" and reboot, there is no back-trace.

If I enable preempt kernel (CONFIG_PREEMPT=y), "RT throttling activated" 
still happens, and there is no reboot.
-  After "RT throttling activated" happens, I use "top -d 1" to check 
CPU usage, I observed at least once sc16is752 threaded-irq handler 
kernel thread occupies 95% of CPU for some seconds (not sure how many 
seconds), then sc16is752 threaded-irq handler CPU usage drops to normal, 
and the whole OpenWRT/Linux system is normal.
-  I did observe once that sc16is752 threaded-irq handler CPU usage 
stays at 95% forever, it seems like a dead-lock inside sc16is752 
threaded-irq handler.
-  It seems a dead-lock similar (in some way?) to below patch:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-4.4.y&id=d78006d2345f87889918a8a7aa3764628ff84263

Question 3) Is there any explanation why board does not reboot when I 
enable preempt kernel (CONFIG_PREEMPT=y)?

Question 4) I am not sure if this is related to MIPS32r2 34kc (with 
9-stage pipeline multi-threaded processor core), while Atheros AR9331 is 
24kc (with 8-stage pipeline processor core)?
https://www.imgtec.com/mips/classic/


I have a temporary work-around:
-  In bitbang I2C driver, use hrtimer-backed usleep_range() (rather than 
busywait udelay()), then this issue is gone (verified by stress test).
-  As I do not know how long it takes to arm hrtimer on AR9331 platform, 
I use usleep_range(10us, 20us), this greatly reduces bitbang I2C 
throughput, and reduces sc16is752 (I2C based UART) baudrate.

Thanks.
Xuebing Wang
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help