Hi Florian,

Looking at your patches, it was not entirely clear to me how the buffer
manager on these Marvell SoCs work, but other networking products have
something similar, like Broadcom's Cable Modem SoCs (BCM33xx) FPM, and
maybe Freescale's FMAN/DPAA seems to do something similar.

Does the buffer manager allocation work by giving you a reference/token
to a buffer as opposed to its address? If that is the case, it would be
good to design support for such hardware in a way that it can be used by
more drivers.

It does not operate on a reference/token but buffer pointers (physical
adresses). It's a ring and you cannot control which buffer will be
taken at given moment.

Eric Dumazet suggested a while ago to me that you could get abstract
such allocation using hardware-assisted buffer allocation by either
introducing a new mm zone (instead of ZONE_NORMAL/DMA/HIGHMEM etc.), or
using a different NUMA node id, such that SKB allocation and freeing
helpers could deal with the specifics, and your networking stack and
driver would be mostly unaware of the buffer manager underlying
implementation. The purpose would be to get a 'struct page' reference to
your buffer pool allocation object, so it becomes mostly transparent to
other areas of the kernel, and you could further specialize everything
that needs to be based on this node id or zone.

As this buffer manager is pretty tightly coupled with NIC (please see
below) and the solution is very platform specific, I'm not sure if it
wouldn't be an overdesign, to provide such generic, paralel to DMA
mechanism.

Finally, these hardware-assisted allocation schemes typically work very
well when there is a forwarding/routing workload involved, because you
can easily steal packets and SKBs from the network stack, but that does
not necessarily play nicely with host-terminated/initiated traffic which
wants to have good feedback on what's happening at the NIC level
(queueing, buffering, etc.).

Sure, I can imagine developing some applications that are developed on
top of the proposed patches, but I'm not sure if such things like
cutting network stack in half should be a part of original support.

quoted

Known issues:
- problems with obtaining all mapped buffers from internal SRAM, when
destroying the buffer pointer pool
- problems with unmapping chunk of SRAM during driver removal
Above do not have an impact on the operation, as they are called during
driver removal or in error path.

Humm, what is the reason for using the on-chip SRAM here, is it because
that's the only storage location the Buffer Manager can allocate from,
or is it because it is presumably faster or with constant access times
than DRAM? Would be nice to explain a bit more in details how the buffer
manager works and its interfacing with the network controllers.

Each pool of pointers is a ring maintained in DRAM (called buffer
pointers' pool external). SRAM (called buffer pointers' pool internal
memory, BPPI) ensures smaller latency, but is also the only way to
allocate/fetch buffer pointers from DRAM ring. Transfers between those
two memories are controlled by buffer manager itself.

In the beginning the external pool has to be filled with desired
amount of pointers. NIC (controlled by mvneta driver) has to be
informed, which pools it can use for longer and shorter packets, their
size and also SRAM physical address has to be written to one of NETA
registers. Moreover, in order to be able to provide direct access
between NETA and buffer manager SRAM, special, Marvell-specific
settings have to be configured (so called opening of MBUS window).

After enabling ingress, incoming packet is automatically placed in
next-to-be-used buffer from buffer manager resources and the
controller updates NIC's descriptor contents with pool's number and
buffer addresses. Once the packet is processed, a new buffer has to be
allocated and it's address written to SRAM - this way the pool of
pointers gets refilled.

Can I use the buffer manager with other peripherals as well? Like if I
wanted to do zero-copy or hardware-assisted memcpy DMA, would that be a
suitable scheme?

Other peripherals cannot access SRAM directly - only DMA-based access
to DRAM. If one would like to access buffers using SRAM from other
drivers, it has to be done by read/write operations performed by CPU.
Moreover I see a limitation, which is a lack of control over the
current buffer index.

Best regards,
Marcin