IPng Networks

Prefixing this here with - this is just my experience, I’m sure everyone who’s gone down this rabbit hole has had a different experience. I usually trawl Ebay for a little bit every day when I’m looking for something specific, and start making offers when I figure out exactly what I want. Negotiating with the seller, asking questions about things that are for “parts only”. I have built up a small but very useful set of contacts that liquidate a lot of large/medium tech co’s surplus through this, so shoot me a message (see profile) if you have some specific hardware in mind. If you’re doing 100G ethernet, my advice, biased by personal experience, is to buy “old” Mellanox gear from before the Nvidia acquisition, but went through the phase of still being supported by Nvidia. With NICs I’d avoid Connect-X 4, mostly due to their age, but 5 and 6 are great. There are so many models though so make sure you pick one that meets your needs, pay attention to the form factor especially - FHHL, HHHL, OCP 3, etc. Vendor OEM parts can often be cheaper - there’s a glut of HPE cables/transceivers, cards and switches out there, that are just Mellanox gear out there, or Supermicro AIOM cards that are fully OCP compliant. I’ve not had any issue mixing and matching this stuff. For instance, I have a Gigabyte server with a couple Bergamo CPUs plugged into an HPE SN2700 switch with FS cables, Juniper transceivers, a Supermicro CX-6 OCP 100G card, and a non-OEM CX-6 dual port Infiniband CX-6…

DSA and OpenWRT are the wrong targets for this device class. switchdev & plain Linux is "it". This is how it needs to turn out: https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html

The author mentioned another similar write up which provided more useful explanations and context to me: https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html

All the switches in this series except the one in this review actually have two separate boards inside with a PCIe-over-cable connection between them. For example the SN2700 here (the sleeved cable in the third photo; you can't see the connector from the angle it's taken from): https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html The cable has a SAS (SFF-8087) connector on each end. I bet you can replace it with an SFF-8087-to-Oculink cable: https://www.amazon.com/chenyang-SFF-8611-SFF-8087-PCI-Express-Slimline… and a PCIe-to-oculink card like one of these: https://www.amazon.com/Ableconn-PEX-OL153-OCuLink-SFF-8612-Adapter/dp/… (none of the links are affiliate links)

> https://ipng.ch/s/articles/2021/07/19/pcengines-apu6.html Nice little write-up. I found it interesting that OpenBSD outperformed Linux, but DPDK-based systems blew the roof off.

There was this interesting talk [1] about how this ISP uses OpenBSD as BGP control plane. I used to heard that OpenBSD was often used as Internet facing middlebox (like firewalls/proxy/network load balancer) because of how good pf and relayd are. But it would surprise me if it would still be the case nowadays, OpenBSD seems to perform poorly [2] in terms of network throughput compared to the competition (VyOS/VPP). And I don't think this 7.5 upgrade helped a lot regarding this topic. [1]: https://gregsowell.com/?p=7069 [2]: https://ipng.ch/s/articles/2021/07/19/pcengines-apu6.html

If you want to go down the rabbit hole have a look at VPP. [0] It's a very fast, like 100GBit/s+ and 100 Mpps+, user space routing engine for commodity x86 hardware. Pim van Pelt has many interesting Videos [1] and articles [2] about it. I only tested it briefly before abandoning it because I'm still stuck on measly ~300/120 g.fast for the foreseeable future so I don't need it yet but especially the Linux CP [3] makes it reasonably workable if you have some Linux networking knowledge. [0]: https://fd.io/ [1]: https://video.ipng.ch/search?search=vpp&searchTarget=local [2]: https://ipng.ch/s/articles/ [3]: https://github.com/pimvanpelt/lcpng

A friend of mine runs dns01 thusly: https://ipng.ch/s/articles/2023/03/24/lego-dns01.html

Contact info on SFP review: https://ipng.ch/s/articles/2021/07/19/pcengines-apu6.html

The latest version (APU6) has an SFP port, https://ipng.ch/s/articles/2021/07/19/pcengines-apu6.html M.2 NVME mod, https://github.com/florolf/apu-m2

The funding - I believe organizationally it has now grown into https://lfnetworking.org/about/members/ and fd.io is just one of the projects. Contributions-wise to VPP: varies from time to time. On my very recent memory there were sizable IPSec acceleration patches from folks at Intel. There is a fair few one off bugfixes coming from smaller users who scratch their own itch. Pim van Pelt [0] has contributed/improved a few very cool and sizable features (Linux control plane integration, VPP yaml configurator [1]) As for difference with DPDK - does it do L3 routing? NAT ? MAP ? MPLS ? DHCP ? VPP can do all of this, and of course it’s just off top of my memory… https://docs.fd.io/vpp/22.10/aboutvpp/featurelist.html is the autogenerated feature list based on the “claimed official features” that are tracked via YAML files in the source code. [0] https://ipng.ch/s/articles/2021/08/12/vpp-1.html [1] https://lists.fd.io/g/vpp-dev/topic/feedback_on_a_tool_vppcfg/90202690…