ipSpace.net Blog

> We want to dual bound the network connections to increase performance and reliability. This will allow us to take routers out of service during low traffic times, for example to restart them after a software upgrade. does not really agree with > Each of the two physical network connections will connect to a different top of rack router. Sure, you can do it with something like MLAG, but that's really just moving your SPOF to somewhere else (the router software running MLAG). Router software being super buggy, I wouldn't rely on MLAG being up at all times. > N1 Which router should we purchase? Pick your favorite. For what you're looking for here, everything is largely using the same silicon (broadcom chipsets). > N2 How do we interconnect the routers while keeping the network simple and fast? Don't fall into the trap of extending vlans everywhere. You should definitely be routing (not switching) between different routers. You can read through http://blog.ipspace.net/ for some info on layer 3 only datacenter networks. You'd want to use something like OSPF or BGP between routers. > N3 Should we have a separate network for Ceph traffic? Yes, if you want your Ceph cluster to remain usable during rebuilds. Ceph will peg the internal network during any sort of rebuild event. > N4 Do we need an SDN compatible router or can we purchase something more affordable? You probably don't need SDN unless you actually have a SDN use case in mind. I'd bet you can get away…

Buy some routers that have BGP implementations (there are some cheap ones, Mikrotik for example), or use open source implementations. The article lists bird, another very popular one is FRR (free range routing). You can trivially stand up two docker containers, stand up a BGP session between them, and - for example - propagate static routes you set up within them. If you like guided tutorials, https://blog.ipspace.net/2023/08/bgp-labs-basic-setup/ is rather good and has been extended to somewhat advanced topics. Everything needed to follow along is free software.

For a latest reference on AI and machine learning for network engineer please check this book by Javier Antich [1]. Please also check the review here [2]. For what it's worth, the book is listed in the "10 Books Every Network Engineer Should Read" [3]. [1] Machine Learning for Network and Cloud Engineers: Get ready for the next Era of Network Automation: https://www.goodreads.com/book/show/101180344-machine-learning-for-net… [2] MUST READ: Machine Learning for Network and Cloud Engineers: https://blog.ipspace.net/2023/02/machine-learning-network-cloud/ [3] 10 Books Every Network Engineer Should Read: https://networkphil.com/2024/05/21/10-books-every-network-engineer-sho…

It looks like it never took off[1]; I was working on this exact topic this week and just went with multiple vlan interfaces on the host and bridged in container interfaces via multus. Would love to know if there's a better practice floating around these days. [1]: https://blog.ipspace.net/2012/02/edge-virtual-bridging-8021qbg.html

Ivan Peplnjack (CCIE Emeritus, networking badass) interview the author several times on his podcast. It's really impressive. http://blog.ipspace.net/

I'm still wondering if for deployments of new services deploying IPv6-only with stateless IPv6 to IPv4 translation might be best: http://blog.ipspace.net/2012/10/skip-transitions-build-ipv6-only-data.… The same is possible for the home or office networks, but so far I don't have native IPv6 at home, which I think would makes it kind of a kludge to implement. So I haven't done it. I think MAP for IPv6/IPv4 is also a great idea, because it too is stateless: http://en.wikipedia.org/wiki/Mapping_of_Address_and_Port There are people doing really interesting things with IPv6 as a first citizen or IPv6-only solutions: https://ripe67.ripe.net/archives/video/3/ https://ripe67.ripe.net/archives/video/61/

> Exactly. IPv6 was developed in the ivory towers where it was still assumed that everyone wanted to be a full participant of the internet. IPv6 was developed in the open on mailing lists that anyone could subscribe to: The criteria presented here were culled from several sources, including "IP Version 7" [1], "IESG Deliberations on Routing and Addressing" [2], "Towards the Future Internet Architecture" [3], the IPng Requirements BOF held at the Washington D.C. IETF Meeting in December of 1992, the IPng Working Group meeting at the Seattle IETF meeting in March 1994, the discussions held on the Big-Internet mailing list (big-internet-at-munnari.oz.au, send requests to join to big-internet-request-at-munnari.oz.au), discussions with the IPng Area Directors and Directorate, and the mailing lists devoted to the individual IPng efforts. * https://datatracker.ietf.org/doc/html/rfc1726 Just like all current IETF discussions are in the open and free for all to participate. If you don't like the direction things are going in participate: as Gandhi did (not) say, “Be the change you want to see in the world.” One of the co-authors on that RFC worked at BBN: you know, the folks that actually built the first the routers (IMPs) that created the ARPA/Internet in the first place. I would hazard to guess they have know something about network operations. * https://www.goodreads.com/book/show/281818.Where_Wizards_Stay_Up_Late > But the…

>https://blog.ipspace.net/2011/12/is-nat-security-feature/ >>Basic NAT (as defined in RFC 2663) performs just the IP address translation (one inside host to one IP address in the NAT pool). The moment the inside host starts a session through the NAT, it becomes fully exposed to the outside world. This is a lie. A "session through the NAT" does not really expose the host to the outside world, because in 99% of the cases this is a TCP session, and the NAT machine would drop all "out of order" packets. >Most modern attacks start from a compromised internal host (e.g., from phishing), or through stolen credentials via a remote access method. Your statement is a perfect example of https://en.wikipedia.org/wiki/Survivorship_bias. Most modern attacks start from an internal host exactly because NAT makes external attacks infeasible for the majority of scenarios. >Set your firewall to default deny, then add a rule for allow outgoing connections, followed by only allow incoming connections if they are replies. What about I don't do it, and the system is still _automatically_ secure, because NAT does exactly that while being _required_ for the system to work. >See Figure A-5 LOL. What about I don't see any figures, and the system still works and is secure for the 99% of the cases.

> Also, NAT is desirable for security/network isolation reasons […] This is security theatre. People have been saying that NAT is not a security feature for over a decade: * https://blog.ipspace.net/2011/12/is-nat-security-feature/ but the message still has not sunk in. The "Zero Trust" paper was published by John Kindervag in 2010: * https://media.paloaltonetworks.com/documents/Forrester-No-More-Chewy-C… Most modern attacks start from a compromised internal host (e.g., from phishing), or through stolen credentials via a remote access method. The above is "castle-and-moat" thinking that tends to have weaker internal controls because it is thought the internal network is "hidden" from the dangerous outside network. Set your firewall to default deny, then add a rule for allow outgoing connections, followed by only allow incoming connections if they are replies. For most machines (and networks), most of the time, this is what's needed: the above is applicable for both IPv6 and IPv4 (with or without NAT). The protection comes from filtering (generally) and stateful packet inspection, not from hiding addresses. > […] and having no distinction between a local IP and a public IP has a lot of disadvantages. Just because something has a global addresses does not mean global reachability (see default deny above). Further you can layout your IPv6 address plan so that you can tell at a glance if hosts are externally accessible. Using a /48 a basis, you break out…

Also, IPV6 has far more scalable systems for dynamic address allocation compared to IPv4. SLAAC scales really, really well, and is fairly stateless compared to DHCPv4 or V6. Also, slaac allows one to easily change the global prefix of a vm/host if it lives on another layer 3 network. A good example of this: you have two seperate datacenter networks based on an EVPN-VXLAN solution, and you do no want to stretch layer 2 across both datacenters because stretching layer 2 is a terrible, terrible idea[0]. Before IPv6, moving VM;s across datacenters which have different public ip space was a major pain for two reasons: - IPV4 has no concept of using multiple addresses per interface without it having unexpected bahaviour on a host. - address management at scale is a major PITA because DHCP is not scalable. How to solve this with IPv6? It's fairly easy: - Use slaac + prefix delegation of your globally unique address space (which is different per Datacenter). - Use a different (site local) address to reach the VM for management purposes. Because this address is not globally routable, you are sure it will never leak into the greater internet and be reachable from the outside world. - If you are using anycast, you can easily announce this prefix to upstream BGP neighbours because you can use BGP Neighbour Autodiscovery[1]. Trying to do this with Ipv4 will result in a mess of administration, not to mention have to do some clever technical hacks to make it work on some…

I'm the Snabb Switch originator. The project is new: I and other open source contributors are currently under contract to build a Network Functions Virtualization platform for Deutsche Telekom's TeraStream project [1] [2]. This is called Snabb NFV [3] and it's going to be totally open source and integrated with OpenStack. Currently we are doing a lot of virtualization work from the "outside" of the VM: implementing Intel VMDq hardware acceleration and providing zero-copy Virtio-net to the VMs. So the virtual machine will see normal Virtio-net but we will make that operate really fast. Inside the VMs we can either access a hardware NIC directly (via IOMMU "PCI passthrough") or we can write a device driver for the Virtio-net device. So, early days, first major product being built, and lots of potential both inside and outside VMs, lots of fantastic products to build with nobody yet building them :-) [1] http://blog.ipspace.net/2013/11/deutsche-telekom-terastream-designed.h… [2] https://ripe67.ripe.net/archives/video/3/ [3] https://github.com/SnabbCo/snabbswitch/blob/snabbnfv-readme/src/design…

https://blog.ipspace.net/2009/08/what-went-wrong-socket-api/ A DNS name can have a LOT of A records associated but a socket has to pick one. This is a severe limitation.

There's some important nuances stated by Ivan Pepelnjak who is considered an authority in networking: https://blog.ipspace.net/2026/06/goodbye-leaf-spine-networks/ Although very interesting from a scientific point of view I always consider these use cases to be very unique to AWS / hyperscaler DC's and in no way a new 'blueprint' for how to build DC-networks/fabrics.

there's Prefix Delegation support since Android 11 (so there's some DHCPv6 support, but of course in a fucking worse way with the mandatory "we know better" attitude) https://blog.ipspace.net/2025/09/android-dhcpv6-prefix-delegation/

> Anyhow. I'm not confused about NAT vs. firewalling. No one who dislikes IPv6 is confused by this. "No one"; LOL. I've participated in entire sub-threads on HN with people insisting that NAT = security. I've cited well-regarded network educators/commentators and vendors: * https://blog.ipspace.net/2011/12/is-nat-security-feature/ * https://www.f5.com/resources/white-papers/the-myth-of-network-address-…

There are some address source selection problems if you're still using any ipv4 for the local services https://blog.ipspace.net/2022/05/ipv6-ula-made-useless/

Multihoming is broken for most people in IPv6 space. It's stupid easy in IPv4. https://blog.ipspace.net/2010/12/small-site-multihoming-in-ipv6-missio…

This has already been done at scale with HTTP/3 (QUIC), it's just not widely distributed beyond the largest sites & most popular web browsers. gRPC for example is still on multiplexed TCP via HTTP/2, which is "good enough" for many. Though it doesn't really replace TCP, it's just that the predominant requirements have changed (as Ousterhout points out). Bruce Davie has a series of articles on this: https://systemsapproach.substack.com/p/quic-is-not-a-tcp-replacement Also see Ivan Pepelnjak's commentary (he disagrees with Ousterhout): https://blog.ipspace.net/2023/01/data-center-tcp-replacement/

Depends on what you want to do. Maybe use one network or assign static addresses. Here's a post on /120 networks: https://blog.ipspace.net/2011/05/ipv6-neighbor-discovery-exhaustion.ht…

Early morning here so I don't recall them all, but the latest one I ran across was ULA addresses not being used[1] in dual-stack configurations due to IPv4 being preferred[2]. [1]: https://blog.ipspace.net/2022/05/ipv6-ula-made-useless.html [2]: https://datatracker.ietf.org/doc/html/draft-buraglio-v6ops-ula

`git diff --word-diff` solves this by showing changes inline. Example: https://blog.ipspace.net/2020/04/git-tip-word-diff.html

I got an error message that said it rejects hotlinking, but you can see the picture in this post: https://blog.ipspace.net/2019/11/why-are-you-always-so-negative.html

I don't know if you've seen this, but working with Eastern Europeans for a year, this really hit home: https://blog.ipspace.net/2019/11/s1600-Its-All-Garbage-Short.jpeg

> see TCP/IP as not being the "obvious design It is far from obvious, TCP has it's problems over fast and relatively reliable networks. For a random example see [1] [1] https://blog.ipspace.net/2019/06/do-packet-drops-matter-for-tcp.html

I've never heard this weird qualification for the definition of "line rate" that it somehow requires minimum packet size, so I looked it up. The first three sources for a quoted big-g search all imply or directly state that it's the same as bandwidth: https://blog.ipspace.net/2009/03/line-rate-and-bit-rate.html https://www.reddit.com/r/networking/comments/4tk2to/bandwidth_vs_line_… https://www.fmad.io/blog-what-is-10g-line-rate.html Also, for gigabit networks, ethernet packets are padded to at least 512 bytes because of a bigger slot size: https://www.cse.wustl.edu/~jain/cis788-97/ftp/gigabit_ethernet/index.h…

"If the software was changed to reconcile 2 sensors, it would then lead to a new "AOA Disagree" indicator which would raise doubts to the FAA that Boeing could just give pilots a simple iPad training orientation instead of expensive flight-sim training." I always liked this quote from the "Mythical Man-Month": “Never go to sea with two chronometers, take one or three”. https://blog.ipspace.net/2017/01/never-take-two-chronometers-to-sea.ht…

You should always have at least three and preferably an odd number of similar sensors so that you can get at least a vague idea of consensus. The British Royal Navy rule was: take one chronometer or three, never two because with two you will always be unsure which one is correct while with three you can, usually, just average the two that are closest together and regard the third as an outlier. See also https://blog.ipspace.net/2017/01/never-take-two-chronometers-to-sea.ht…

I'm not a network engineer, rather a security-focused guy, and I'm also very interested in this topic because the basis for a secure network is a well managed network without any unknown/undocumented assets (network equipment incl. layer 2 as well as network participants). Few months ago I attended a talk of Ivan Pepelnjak, the author of a network engineering blog called ipspace. He talked a lot on network automation and enumerated some success story. I remember he talked about people setting up several data centers in a relatively short amount of time by using a lot of network automation / configuration management. http://blog.ipspace.net/2018/03/presentation-and-video-real-life.html

This is one rack of equipment; no spines are needed. http://blog.ipspace.net/2014/10/all-you-need-are-two-top-of-rack.html Cumulus isn't really SDN so I'm not sure what you're saying there. Traditional networking is fine but it's totally different than Linux so you need dedicated netops people to manage it. And Cisco is the most expensive traditional vendor.

Also note Spotify published their own stuff also on Arista hardware https://labs.spotify.com/2016/01/26/sdn-internet-router-part-1/ Podcast where David Barroso talks about it: http://blog.ipspace.net/2015/01/sdn-router-spotify-on-software-gone-wi… Arista blog post: https://eos.arista.com/spotifys-sdn-internet-router/ Disclaimer I work at Arista