On Sun, Nov 30, 2008 at 4:55 AM, Trent Piepho [off-list ref] wrote:

On Sat, 29 Nov 2008, Roland Dreier wrote:

quoted

quoted

The 8b10 encoding is already accounted for in the 2.5 Gbps figure;
the raw bit rate is 3.125 Gbps.

In addition, PCI Express transfers are broken up into packets, usually
with very small payloads (128 or 256 bytes are common).  So the packet
header overhead reduces throughput further, and then the transaction
layer adds further overhead.  Then transferring NIC control structures
over the link adds even more overhead.  So achieving 13.something Gb/sec
of real throughput on a PCIe link theoretically capable of 16 Gb/sec
seems pretty good to me.

I tested it both on PCIE2 and PCIE1 . same results.

For transfers with only 4 byte payloads, I get about 6.5 MB/sec over a PCIe
4x link.  For small transfers, the overhead is huge.

Well, you are correct. I used packet size of 1.3k. when I increased to
9K a packet each port
reached ~7Gbps ( I am using pktgen). I used two cards, so I reached
28Gbps. BIOS settings for pci payload were AUTO,so i set it to 256
bytes but it made no difference, still 28Gbps. well, at least I know
what the problem is.
guys, I truly thank you all.