Re: 82571EB: Detected Hardware Unit Hang

On 07/11/12 03:02, Dave, Tushar N wrote:

quoted

-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
On Behalf Of Joe Jin
Sent: Tuesday, July 10, 2012 12:40 AM
To: Joe Jin
Cc: e1000-devel@lists.sf.net; netdev@vger.kernel.org; linux-
kernel@vger.kernel.org
Subject: Re: 82571EB: Detected Hardware Unit Hang

When I debug the driver I found before Detected HW hang, driver unable to
clean and reclaim the resources:

1457         while ((eop_desc->upper.data &
cpu_to_le32(E1000_TXD_STAT_DD)) &&  <== at here upper.data always is 0x300
1458                (count < tx_ring->count)) {
    <--- snip --->
1487         }


I checked all driver codes I did not found anywhere will set the
upper.data with E1000_TXD_STAT_DD, I guess upper.data be set by hardware?

Yes upper.data (part of it is STATUS byte) is set by HW. Basically driver checks E1000_TXD_STAT_DD (Descriptor Done) bit. If this bit is set that means HW has processed that descriptor and driver can now clean that descriptor.
With value 0x300 , DD bit is not set. That means HW has not processed that descriptor.

Thanks for the clarify, might be firmware issue?

How fast does tx hang reproduce? I suggest you to enable debug code in driver so when tx hang occurs it will dump the HW desc ring info into kernel log.

Once I copy a file from other server, issue to be reproduced at once.
I'll enable the debug to get more debug info.

You can run "ethtool -s ethx msglvl 0x2c00" to enable debug.
Once tx hang occurs please send me the full dmesg log.

Does tx hang occur with in-kernel e1000e driver too?

I tried several drivers included rhel5 the latest, Intel the latest,
rhel6 the latest, issue see on all those drivers.

Thanks,
Joe 

Thanks.

-Tushar

quoted

If OS is 32bit system, what which happen?

quoted

Thanks in advance,
Joe

On 07/09/12 16:51, Joe Jin wrote:

quoted

Hi list,

I'm seeing a Unit Hang even with the latest e1000e driver 2.0.0 when
doing scp test. this issue is easy do reproduced on SUN FIRE X2270 M2,
just copy a big file (>500M) from another server will hit it at once.

Would you please help on this?

device info:
# lspci -s 05:00.0
05:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit
Ethernet Controller (Copper) (rev 06)

# lspci -s 05:00.0 -n
05:00.0 0200: 8086:10bc (rev 06)

# ethtool -i eth0
driver: e1000e
version: 2.0.0-NAPI
firmware-version: 5.10-2
bus-info: 0000:05:00.0

# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
udp fragmentation offload: off
generic segmentation offload: on
generic-receive-offload: on

kernel log:
-----------
e1000e 0000:05:00.0: eth0: Detected Hardware Unit Hang:
  TDH                  <6c>
  TDT                  <81>
  next_to_use          <81>
  next_to_clean        <6b>
buffer_info[next_to_clean]:
  time_stamp           <fffc7a23>
  next_to_watch        <71>
  jiffies              <fffc8c0c>
  next_to_watch.status <0>
MAC Status             <80387>
PHY Status             <792d>
PHY 1000BASE-T Status  <3c00>
PHY Extended Status    <3000>
PCI Status             <10>
e1000e 0000:05:00.0: eth0: Detected Hardware Unit Hang:
  TDH                  <6c>
  TDT                  <81>
  next_to_use          <81>
  next_to_clean        <6b>
buffer_info[next_to_clean]:
  time_stamp           <fffc7a23>
  next_to_watch        <71>
  jiffies              <fffc9bac>
  next_to_watch.status <0>
MAC Status             <80387>
PHY Status             <792d>
PHY 1000BASE-T Status  <3c00>
PHY Extended Status    <3000>
PCI Status             <10>
------------[ cut here ]------------
WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x225/0x230()
Hardware name: SUN FIRE X2270 M2 NETDEV WATCHDOG: eth0 (e1000e):
transmit queue 0 timed out Modules linked in: autofs4 hidp rfcomm
bluetooth rfkill lockd sunrpc cpufreq_ondemand acpi_cpufreq mperf
be2iscsi iscsi_boot_sysfs ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad
ib_core ib_addr iscsi_tcp bnx2i cnic uio ipv6 cxgb3i libcxgbi cxgb3
mdio libiscsi_tcp libiscsi scsi_transport_iscsi video sbs sbshc
acpi_pad acpi_ipmi ipmi_msghandler parport_pc lp parport e1000e(U)
snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device
igb snd_pcm_oss serio_raw snd_mixer_oss snd_pcm tpm_infineon snd_timer
snd soundcore snd_page_alloc i2c_i801 iTCO_wdt i2c_core pcspkr
i7core_edac iTCO_vendor_support ioatdma ghes dca edac_core hed
dm_snapshot dm_zero dm_mirror dm_region_hash dm_log dm_mod usb_storage
sd_mod crc_t10dif sg ahci libahci ext3 jbd mbcache [last unloaded:
microcode]
Pid: 0, comm: swapper Not tainted 2.6.39-200.24.1.el5uek #1 Call
Trace:
 [<c07d9ac5>] ? dev_watchdog+0x225/0x230  [<c045ba61>]
warn_slowpath_common+0x81/0xa0  [<c07d9ac5>] ?
dev_watchdog+0x225/0x230  [<c045bb23>] warn_slowpath_fmt+0x33/0x40
[<c07d9ac5>] dev_watchdog+0x225/0x230  [<c07d98a0>] ?
dev_activate+0xb0/0xb0  [<c0468e82>] call_timer_fn+0x32/0xf0
[<c04bceb0>] ? rcu_check_callbacks+0x80/0x80  [<c046a76d>]
run_timer_softirq+0xed/0x1b0  [<c07d98a0>] ? dev_activate+0xb0/0xb0
[<c0461a81>] __do_softirq+0x91/0x1a0  [<c04619f0>] ?
local_bh_enable+0x80/0x80  <IRQ>  [<c0462295>] ? irq_exit+0x95/0xa0
[<c087f8b8>] ? smp_apic_timer_interrupt+0x38/0x42
 [<c08784f5>] ? apic_timer_interrupt+0x31/0x38  [<c046007b>] ?
do_exit+0x11b/0x370  [<c065eae4>] ? intel_idle+0xa4/0x100
[<c078d9b9>] ? cpuidle_idle_call+0xb9/0x1e0  [<c0411d77>] ?
cpu_idle+0x97/0xd0  [<c085cbbd>] ? rest_init+0x5d/0x70  [<c0b07a7a>] ?
start_kernel+0x28a/0x340  [<c0b074b0>] ? obsolete_checksetup+0xb0/0xb0
[<c0b070a4>] ? i386_start_kernel+0x64/0xb0 ---[ end trace
5502b55cd4d4e5cb ]--- e1000e 0000:05:00.0: eth0: Reset adapter
e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

Thanks,
Joe

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired