Re: mpt2sas: /sysfs sas_address entries do not show individual port sas addresses.

On 19/08/11 15:58, Douglas Gilbert wrote:

On 11-08-19 08:30 AM, Benjamin ESTRABAUD wrote:

quoted

On 18/08/11 20:52, Douglas Gilbert wrote:

quoted

On 11-08-18 01:57 PM, Ravi Shankar wrote:

quoted

quoted

Do you know a reason why it is not preferably for every
phy on a SAS HBA to respond with the same SAS address?


As a practical matter a SAS HBA needs a single SAS address,
preferably printed on the board or its box. Then if you
manage to wipe its SAS address (e.g. by erasing its flash
to move from IR to IT firmware) then you know which SAS
address to re-instate :-)

HBA SAS phy could have same SAS address when they are directly 
connected.
however when connected to expanders, each logical port/phy need 
unique SAS
address.

No.

SAS HBAs and expanders should always be trying to maximize
the width of a link. By definition all physical ** phys on an
expander should (must) have the same SAS address. So if
you connect 5 phys from a SAS HBA to the same expander (and
the HBA supports links wider than 4 phys) then those
5 HBA phys should have the same SAS address. Those 5 HBA
phys then form a SCSI port.

Just tested a triangular arrangement: a LSI SAS-2 HBA
(9212-4i4e) connected to one SAS-2 expander (4 phy wide link)
and one SAS-1 expander (narrow link). And the expanders
where connected to each other. Two disks were connected
to the SAS-2 expander.
Both expanders reported the same SAS address for the
5 HBA attached phys. You might argue that is two
separate SCSI initiator ports with the same port
identifier (SAS address) in the same SAS domain.

Anyway there was an interesting difference between the HBA's
BIOS and Linux (lk 3.0.3): the BIOS reported those two
disks twice while Linux only reported them once. That seems
to suggest that the BIOS set up the routing table in the
SAS-1 expander while Linux did not. smp_discover in Linux
confirms that the SAS-1 expander's route table was not set up.

The trouble with testing is that is raises more questions
than it supplies answers.


SAS disks have two phys which are typically given two
different SAS addresses. This stops them forming a wide link
if, for example, they were both connected to the same
expander. Typically the two SAS disk phys would be connected
to different expanders for redundancy. However if the
interest was speed (e.g. with a SAS SSD) then both the
disk phys might be given the same SAS address.


** SAS-2 expanders often have an integrated SES target
on a virtual phy in the expander chip, and that
virtual phy has a different SAS address.

Doug Gilbert


P.S. Why is my post cc-ing to "unlisted-recipients:;"

Hi Douglas, Ravi,

According to the SAS specs, (ISO/IEC 14776-152:200x, sas2r15.pdf), on 
page 45,
they state that that a port is formed by a unique tuple of the SAS 
Phy address
and the attached SAS Phy address.

For instance, if you take 2 * 2 phy wide ports, where all 4 phys from 
these two
ports have the same sas address, let's call it "A" and connect them 
each to
another port that each has a different address, "B" and "C", they 
state that two
ports will be formed, one connecting "A" to "B" and one connecting 
"A" to "C".

This is what Douglas is saying with the SAS disks for instance, that are
typically given two separate SAS addresses to avoid forming a wide 
port with the
expander (since the expander will have the same sas address on all 
phys), and to
allow for dual expander multiplexing for redundancy.

But what I don't understand is that, in the context of two HBAs 
connected
together, things seem to be different:

I configured a 9200-8e HBA (8Phys) and changed all its SAS phys 
addresses from
being the same to being incremental, therefore the last byte of each 
SAS phy
address changed from:

0 1 2 3 4 5 6 7
b0 b0 b0 b0 b0 b0 b0 b0
to:
b0 b1 b2 b3 b4 b5 b6 b7

I also changed the "ports" setup from "Auto" to "Wide", making two 
4*phys ports:

Port 0 | Port 1
b0 b1 b2 b3 | b4 b5 b6 b7

I also set all these ports to Target.

I then connected this HBA to another 9200-8e HBA, which was left 
setup as default:

Auto
Initiator
0 1 2 3 4 5 6 7
10 10 10 10 10 10 10 10

However, when I looked up the SAS topology on either side in LSIUtil, 
I saw that
there was two ports connected on each HBAs, one connected on phy 0 
and one on
phy 4.

On the second (Initiator) HBA, the two ports appeared as b0 and b4, 
with two
separate handles.

On the first (Target) HBA, both ports appeared as 10, with two 
separate handles.

What I don't understand above, is since all phys on the Target HBAs 
have a
different SAS address, and all the ones on the Initiator one have the 
same, 8
narrow ports should have been created there.

However, there is a separate notion of "port" in LSIUtil, does that 
mean that
agglomerating 4 phys with different SAS addresses in a logical 
LSIUtil "port"
forces the HBA FW to transmit the same sas address on these 4 Phys, 
to make them
look like a single port? Or is there an extra separate notion of 
"port", that
does not rely on the phy SAS address and its attached SAS address?

I guess my question is: Is there an extra information ontop of phy 
sas address
and phy id that is transmitted in SAS, like a "port" id or a handle?

Ben,
In my previous post I assumed that the SAS HBA was a SCSI
initiator.

And yes, I also felt that LSI has its own meaning for "port"
and that is not the same as a "SCSI port" or a "SAS port"
defined in the t10 documents.

There is another piece of the identification puzzle but
as far as I can see its has not been properly implemented.
Namely: device name. spl2r02.pdf section 4.2.6:

  "Each expander device and SAS device shall include a
   SAS address (see 4.2.4) as its device name.
   A SAS address used as a device name shall not be used
   as any other name or identifier (e.g., a device name,
   port name, port identifier, or logical unit name (see
   SAM-5)), except:
     a) the SAS address of an expander device is the same
        as the SAS address of the SMP port in that expander
        device."

SAS disks comply with this but not SAS initiators (at least
not the ones that I have checked). So if a SAS HBA in
initiator mode is to be regarded as a SCSI device then
it should have a device name which is a SAS address that
is distinct from the SAS addresses of any of its ports.

Yes, I observed the same thing with all the initiators I have.
It looks like the phys on LSI HBAs are set by reading the SAS ioc or 
that the SAS ioc sas address is read by reading the first phy's sas address.

BTW Why not put an expander between the HBAs set up as
an initiator and a target? That way you can check (via
SMP) that LSIUtil is reporting the same thing as the
phy does "down the wire". The SMP DISCOVER response
reports the attached device name.

That's a good idea,

I'll try if possible.


Thanks for your reply.

Regards,
Ben.

Doug Gilbert

quoted

Also, in the above case, if we assume that the HBA FW was 
transmitting the same
phys for phy 0-3 and phy 4-8 on the Target HBA, it would make sense 
that we have
two ports, since there is two pairs of SAS addresses / attached SAS 
addresses here.

Am I holding on to the right belief?

Thanks in advance for your reply.

Regards,
Ben.

PS: The "Undisclosed recipient" was because Ravi seemed to have 
replied with a
hidden "To:" and CC'ed to the linux-scsi ML.

quoted

To unsubscribe from this list: send the line "unsubscribe 
linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html