@@ -1257,6 +1257,10 @@ struct vio_dev *vio_register_device_node(struct device_node *of_node)viodev->dev.parent=&vio_bus_device.dev;viodev->dev.bus=&vio_bus_type;viodev->dev.release=vio_dev_release;+/* needed to ensure proper operation of coherent allocations+*later,incasedriverdoesn'tsetitexplicitly*/+dma_set_mask(&viodev->dev,DMA_BIT_MASK(64));+dma_set_coherent_mask(&viodev->dev,DMA_BIT_MASK(64));/* register with generic device framework */if(device_register(&viodev->dev)){
Milton, Sonny, any thoughts?
A bit more detail after trying a few more kernels on the box that
originally showed the error:
1) This doesn't actually prevent booting, afaict. I think it "just"
disables DMA, which is bad, but not a boot fail, technically.
2) Reverting the above commit definitely prevents those messages.
3) I'm seeing a separate issue with 2.6.37-git17 (that's not present in
2.6.37):
sd 0:4:2:0: [sda] Aborting command: 2A
sd 0:4:2:0: Abort timed out. Resetting bus.
At which point the box locks up :)
So testing fixes is a bit of a challenge right now.
Ben, if you're ok with waiting to see if Milton or Sonny have any ideas,
I'd like to hold off on asking for a revert. In the case they do, I'll
be able to test and send out any proposed fix rapidly.
Thanks,
Nish
--
Nishanth Aravamudan [off-list ref]
IBM Linux Technology Center
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-01-19 04:06:33
On Tue, 2011-01-18 at 14:47 -0800, Nishanth Aravamudan wrote:
On 18.01.2011 [12:31:52 +1100], Anton Blanchard wrote:
quoted
Hi,
I was testing 2.6.37-git17 on a POWER7 with virtual IO and hit this:
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 7446k freed
vio 30000000: Warning: IOMMU dma not supported: mask
0xffffffffffffffff, table unavailable
vio 4000: Warning: IOMMU dma not supported: mask 0xffffffffffffffff,
table unavailable
vio 4001: Warning: IOMMU dma not supported: mask 0xffffffffffffffff,
table unavailable
vio 4002: Warning: IOMMU dma not supported: mask 0xffffffffffffffff,
table unavailable
vio 4004: Warning: IOMMU dma not supported: mask 0xffffffffffffffff,
table unavailable
audit: initializing netlink socket (disabled)
Haven't had a chance to look closer yet.
Well, this causes messages for vdevices that don't do DMA at all (such
as vterm etc...) and don't have the necessary properties. However, it
didn't -break- anything for me in my tests so far, just spurrious
messages. Not sure what's up with Anton's setup. Anton, can you hack the
printk to display the OF path to the device so we see what devices are
complaining ? It could be a different issue that prevents booting.
Cheers,
Ben.
quoted hunk
After debugging a bit, this would appear to be due to the second hunk of
b3c73856ae47d43d0d181f9de1c1c6c0820c4515.
On 19.01.2011 [15:06:20 +1100], Benjamin Herrenschmidt wrote:
On Tue, 2011-01-18 at 14:47 -0800, Nishanth Aravamudan wrote:
quoted
On 18.01.2011 [12:31:52 +1100], Anton Blanchard wrote:
quoted
Hi,
I was testing 2.6.37-git17 on a POWER7 with virtual IO and hit this:
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 7446k freed
vio 30000000: Warning: IOMMU dma not supported: mask
0xffffffffffffffff, table unavailable
vio 4000: Warning: IOMMU dma not supported: mask 0xffffffffffffffff,
table unavailable
vio 4001: Warning: IOMMU dma not supported: mask 0xffffffffffffffff,
table unavailable
vio 4002: Warning: IOMMU dma not supported: mask 0xffffffffffffffff,
table unavailable
vio 4004: Warning: IOMMU dma not supported: mask 0xffffffffffffffff,
table unavailable
audit: initializing netlink socket (disabled)
Haven't had a chance to look closer yet.
Well, this causes messages for vdevices that don't do DMA at all (such
as vterm etc...) and don't have the necessary properties. However, it
didn't -break- anything for me in my tests so far, just spurrious
messages. Not sure what's up with Anton's setup. Anton, can you hack the
printk to display the OF path to the device so we see what devices are
complaining ? It could be a different issue that prevents booting.
Is this what you were looking for?
vio 30000000: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
vio 30000000: Path: /vdevice/vty@30000000
vio 4000: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
vio 4000: Path: /vdevice/IBM,sp@4000
vio 4001: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
vio 4001: Path: /vdevice/rtc@4001
vio 4002: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
vio 4002: Path: /vdevice/nvram@4002
vio 4004: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
vio 4004: Path: /vdevice/gscsi@4004
FWIW, I looked at Anton's logs, and I don't think the boot failed, per
se. I think it may have timed out (but not positive on that). I was able
to boot 2.6.27-git17 on the exact same box, albeit it locks up at a
later point (the sd abort I e-mailed about in a follow-up).
Cheers,
Ben.
quoted
After debugging a bit, this would appear to be due to the second hunk of
b3c73856ae47d43d0d181f9de1c1c6c0820c4515.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-01-19 04:54:46
On Tue, 2011-01-18 at 20:37 -0800, Nishanth Aravamudan wrote:
Is this what you were looking for?
vio 30000000: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
vio 30000000: Path: /vdevice/vty@30000000
vio 4000: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
vio 4000: Path: /vdevice/IBM,sp@4000
vio 4001: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
vio 4001: Path: /vdevice/rtc@4001
vio 4002: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
vio 4002: Path: /vdevice/nvram@4002
vio 4004: Warning: IOMMU dma not supported: mask 0xffffffffffffffff, table unavailable
vio 4004: Path: /vdevice/gscsi@4004
Ok, so they are all harmess (none of those device do DMA, appart maybe
gscsi, I have no idea what it is :-)
FWIW, I looked at Anton's logs, and I don't think the boot failed, per
se. I think it may have timed out (but not positive on that). I was able
to boot 2.6.27-git17 on the exact same box, albeit it locks up at a
later point (the sd abort I e-mailed about in a follow-up).
I haven't seen your email. I'll dig. Have to run now.
Cheers,
Ben.
quoted
Cheers,
Ben.
quoted
After debugging a bit, this would appear to be due to the second hunk of
b3c73856ae47d43d0d181f9de1c1c6c0820c4515.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-01-19 06:06:28
On Tue, 2011-01-18 at 16:48 -0800, Nishanth Aravamudan wrote:
Ben, if you're ok with waiting to see if Milton or Sonny have any
ideas,
I'd like to hold off on asking for a revert. In the case they do, I'll
be able to test and send out any proposed fix rapidly.
I don't believe this specific error is causing the lockup, I think we
only hit a spurrious message on devices that don't have DMA capabilities
in the first place. (But I may be wrong, I'll wait for you guys to dig
more or I'll have a look myself tomorrow if I manage to get out of
meetings).
So there's another problem with SCSI tho it -could- also be a DMA issue,
hard to tell at this point.
BTW. I'm not too happy with those defaults set to 64-bit. Probably not
an issue until your other patches go in, but some devices like veth
cannot do 64-bit DMA. I think we should default to 32-bit in the VIO
base code and explicitely enable 64-bit DMA from drivers that support it
(in theory vscsi but I haven't verified the implementation).
Cheers,
Ben.
On 19.01.2011 [17:06:18 +1100], Benjamin Herrenschmidt wrote:
On Tue, 2011-01-18 at 16:48 -0800, Nishanth Aravamudan wrote:
quoted
Ben, if you're ok with waiting to see if Milton or Sonny have any
ideas,
I'd like to hold off on asking for a revert. In the case they do, I'll
be able to test and send out any proposed fix rapidly.
I don't believe this specific error is causing the lockup, I think we
only hit a spurrious message on devices that don't have DMA
capabilities in the first place. (But I may be wrong, I'll wait for
you guys to dig more or I'll have a look myself tomorrow if I manage
to get out of meetings).
Yes, this seems accurate. Like I mentioned elsewhere, this box came up
ok even with these messages and seemed ok (up until the disk locked up).
So there's another problem with SCSI tho it -could- also be a DMA issue,
hard to tell at this point.
Right, I'm not sure how to determine that. I did see the lockup, though,
with both my patches reverted (the patches for vio, I mean, after
2.6.37)
BTW. I'm not too happy with those defaults set to 64-bit. Probably not
an issue until your other patches go in, but some devices like veth
cannot do 64-bit DMA. I think we should default to 32-bit in the VIO
base code and explicitely enable 64-bit DMA from drivers that support it
(in theory vscsi but I haven't verified the implementation).
Ok, so change the bit-mask to 32-bit? Or would it be appropriate to
attempt 64-bit, if it fails fallback to 32-bit? Seems to be a common
pattern throughout the DMA bit-setting callers.
Thanks,
Nish
--
Nishanth Aravamudan [off-list ref]
IBM Linux Technology Center
From: Anton Blanchard <hidden> Date: 2011-01-29 22:22:17
Hi,
FWIW, I looked at Anton's logs, and I don't think the boot failed, per
se. I think it may have timed out (but not positive on that). I was
able to boot 2.6.27-git17 on the exact same box, albeit it locks up
at a later point (the sd abort I e-mailed about in a follow-up).
This fail bisects down to the VPHN (shared processor affinity) patch.
I've got some fixes on the way.
Anton
From: Anton Blanchard <hidden> Date: 2011-01-29 22:24:34
VPHN supports up to 8 distance fields but the number of entries in
ibm,associativity-reference-points signifies how many are in use.
Don't look at all the VPHN counts, only distance_ref_points_depth
worth.
Since we already cap our distance metrics at MAX_DISTANCE_REF_POINTS,
use that to size the VPHN arrays and add a BUILD_BUG_ON to avoid it growing
larger than the VPHN maximum of 8.
Signed-off-by: Anton Blanchard <redacted>
---
Index: linux-2.6/arch/powerpc/mm/numa.c
===================================================================
@@ -1291,8 +1291,7 @@ u64 memory_hotplug_max(void)/* Vrtual Processor Home Node (VPHN) support */#ifdef CONFIG_PPC_SPLPAR-#define VPHN_NR_CHANGE_CTRS (8)-staticu8vphn_cpu_change_counts[NR_CPUS][VPHN_NR_CHANGE_CTRS];+staticu8vphn_cpu_change_counts[NR_CPUS][MAX_DISTANCE_REF_POINTS];staticcpumask_tcpu_associativity_changes_mask;staticintvphn_enabled;staticvoidset_topology_timer(void);
@@ -1305,12 +1304,15 @@ static void setup_cpu_associativity_chan{intcpu=0;+/* The VPHN feature supports a maximum of 8 reference points */+BUILD_BUG_ON(MAX_DISTANCE_REF_POINTS>8);+for_each_possible_cpu(cpu){inti=0;u8*counts=vphn_cpu_change_counts[cpu];volatileu8*hypervisor_counts=lppaca[cpu].vphn_assoc_counts;-for(i=0;i<VPHN_NR_CHANGE_CTRS;i++){+for(i=0;i<distance_ref_points_depth;i++){counts[i]=hypervisor_counts[i];}}
@@ -1339,7 +1341,7 @@ static int update_cpu_associativity_chanu8*counts=vphn_cpu_change_counts[cpu];volatileu8*hypervisor_counts=lppaca[cpu].vphn_assoc_counts;-for(i=0;i<VPHN_NR_CHANGE_CTRS;i++){+for(i=0;i<distance_ref_points_depth;i++){if(hypervisor_counts[i]>counts[i]){counts[i]=hypervisor_counts[i];changed=1;
From: Anton Blanchard <hidden> Date: 2011-01-29 22:26:19
The hypervisor uses unsigned 1 byte counters to signal topology changes to
the OS. Since they can wrap we need to check for any difference, not just if
the hypervisor count is greater than the previous count.
Signed-off-by: Anton Blanchard <redacted>
---
Index: linux-2.6/arch/powerpc/mm/numa.c
===================================================================
From: Anton Blanchard <hidden> Date: 2011-01-29 22:28:04
The rest of the NUMA code expects an OF associativity property with
the first cell containing the length. Without this fix all topology changes
cause us to misparse the property and put the cpu into node 0.
Signed-off-by: Anton Blanchard <redacted>
---
Index: linux-2.6/arch/powerpc/mm/numa.c
===================================================================
@@ -1373,7 +1376,7 @@ static int vphn_unpack_associativity(con#define VPHN_FIELD_MSB (0x8000)#define VPHN_FIELD_MASK (~VPHN_FIELD_MSB)-for(i=0;i<VPHN_ASSOC_BUFSIZE;i++){+for(i=1;i<VPHN_ASSOC_BUFSIZE;i++){if(*field==VPHN_FIELD_UNUSED){/* All significant fields processed, and remaining*fieldscontainthereservedvalueofall1's.
@@ -1398,6 +1401,9 @@ static int vphn_unpack_associativity(con}}+/* The first cell contains the length of the property */+unpacked[0]=nr_assoc_doms;+returnnr_assoc_doms;}
From: Anton Blanchard <hidden> Date: 2011-01-29 22:35:22
There is no need to start up the timer and monitor topology changes on a
dedicated processor partition, so disable it.
Signed-off-by: Anton Blanchard <redacted>
---
Index: linux-2.6/arch/powerpc/mm/numa.c
===================================================================
From: Anton Blanchard <hidden> Date: 2011-01-29 22:37:16
When converting to the new cpumask code I screwed up:
- if (cpu_isset(cpu, numa_cpumask_lookup_table[node])) {
- cpu_clear(cpu, numa_cpumask_lookup_table[node]);
+ if (cpumask_test_cpu(cpu, node_to_cpumask_map[node])) {
+ cpumask_set_cpu(cpu, node_to_cpumask_map[node]);
This was introduced in commit 25863de07af9 (powerpc/cpumask: Convert NUMA code
to new cpumask API)
Fix it.
Signed-off-by: Anton Blanchard <redacted>
Cc: <stable@kernel.org>
---
Index: linux-2.6/arch/powerpc/mm/numa.c
===================================================================
@@ -186,7 +186,7 @@ static void unmap_cpu_from_node(unsigneddbg("removing cpu %lu from node %d\n",cpu,node);if(cpumask_test_cpu(cpu,node_to_cpumask_map[node])){-cpumask_set_cpu(cpu,node_to_cpumask_map[node]);+cpumask_clear_cpu(cpu,node_to_cpumask_map[node]);}else{printk(KERN_ERR"WARNING: cpu %lu not found in node %d\n",cpu,node);