@benjojo that firmware download link… I needed this article like last two weeks, it took me hours to find that link :X (and I’m still looking for FS3 failsafe images for a 2700 with empty flash that I bought and doesn’t boot ostensibly because it was never flashed)

@benjojo flash for the ASIC. The x86 part doesn’t boot either so I connected the dataplane directly to another switch. Lspci indicates the asic is in ‘livefish’ mode, which is a firmware flashing mode, and it appears the whole flash is just empty. Mellanox has neat instructions on what to do in this case but unfortunately none of the files they reference are publicly available (and seem to be gated to OEMs… :() - and I assume x86 part doesn’t boot because the bios flash chip is also empty…

@benjojo eBay, for quite cheap (but not cheap enough to deal with this tbh). I assume if I had another 2700 I could dump it’s flash and write it to this one, but I’ve got only one 2700, every other spectrum switch I have is 2410/3700 :\

@benjojo (also I’ve been looking for 2010s but can’t find reasonable priced ones, so I ended up buying a pallet worth of used 3700s for quite cheap :). great switches)

@benjojo

I hope this changes in the future, as Mikrotik’s hardware price point is very competitive, it’s just the software reliability that always turns me off their products, so having an option to not use RouterOS while keeping their very competitive hardware would be a huge deal.

That hit’s hard. Not only is their price point great, their power consumption is also often one of the lowest for a given port count

But even as little as I use RouterOS it drives me nuts

@benjojo "Mikrotik is known to use this hardware, but right now has no official (or known) way to “jailbreak” the hardware to run your own software stack" – I thought the devices allowed you to TFTP-boot whatever, did that change?

@benjojo @grawity from what i can gather there’s no lockdown as such, but the bootloader is a bit weird

@erincandescent @benjojo @grawity are the switch ICs documented well enough? As far as I've seen Marvell is not exactly forthcoming with documentation about their stuff.

It wouldn't help much if you just get basic switching running and can't implement acls, igmp multicast and similar things due to those parts not being documented or part of a published driver.

@electronic_eel @erincandescent @benjojo @grawity I mean, it depends on what you want to do with it.

For my own FPGA-based switch project all I want is port vlans, 802.1q, forcing ports to specific speed/duplex states, and TDR testing on baseT interfaces.

Maybe some basic ACLs eventually but that won't be an initial focus.,

@azonenberg @erincandescent @benjojo @grawity yeah, vlan port assignments are a must.

but i also want to limit dhcp to trusted ports and do ipv6 ra guard incl. RFC7113. that requires deeper acls to implement properly.

@electronic_eel @azonenberg @erincandescent @benjojo @grawity could always punt the packets to broadcast/multicast for certain ranges to the management cpu and do the hard decision there...

@jeroen @azonenberg @erincandescent @benjojo @grawity yes, sure. But in the case of RFC7113 RA guard you quickly arrive at the point where you would have to parse all IPv6 packets from link-local addrs on the management cpu. that would kill performance.

@electronic_eel @azonenberg @erincandescent @benjojo @grawity yes, it gets complex quick indeed. https://qdiv.dev/posts/eth2/ did some nice tricks already and that is in discrete logic ;)

One day will attempt the same, hence following all your progress in amazement ;)

@electronic_eel @erincandescent @benjojo @grawity That said, I also decided to go with an FPGA for my switch project *because* the big switch vendors made it ~impossible to get datasheets.

@electronic_eel @benjojo @grawity theres some pretty decent support in mainline:
https://github.com/Marvell-switching/switchdev-prestera

@erincandescent @benjojo @electronic_eel @grawity Unfortunately, despite the name, that is not actually a Prestera driver.
So called "DENT" switches have a 2nd CPU, and the firmware that driver uploads to the prestera is actually encrypted code for that CPU, which the driver then RPCs high level commands to.
Microtik switches won't have that 2nd CPU, which I usually refer to as the "GPL circumvention co-processor".

@nolanl @benjojo @electronic_eel @grawity ugh, that's profoundly annoying. At least when NXP play this game the coprocessor always accompanies the hardware

@erincandescent @benjojo @electronic_eel @grawity Yeah, that is how Sparx-5 and Spectrum work.

Marvell is as NDA locked down as always. What happened here is that Amazon (or Marvell, on behalf of Amazon) used the proprietary SDK to write a driver that is driven via RPC, and then hired an outside contractor to write a switchdev driver to that RPC interface.

If the "firmware" blob wasn't encrypted, we could run it in CPU emulation and use the in-kernel driver, but alas...

@nolanl @erincandescent @benjojo @grawity ughh, so that is how DENT switches are implemented.

I have seen DENT and considered it interesting, but haven't had the time yet to investigate how they are implemented in more detail.

@erincandescent @benjojo @electronic_eel @grawity Microchip's Sparx-5 driver actually does drive the Sparx-5 chip, but I've not been able to source a switch with one in the US. They're available in China though: https://www.servethehome.com/insane-48-port-2-5gbe-2x-25gbe-2x-10gbe-managed-chinese-e-sports-cafe-and-hotel-switch-microchip-micron-sandisk/

@nolanl @benjojo @electronic_eel @grawity if you ever find any reasonably priced (as in I won't be too sad if I brick it) SPARX-5 devices let me know, I'd really like to see some more interesting stuff in this area

@erincandescent @benjojo @nolanl @electronic_eel @grawity

We (NetDEF, the FRR non-profit) are currently looking into this too, including trying to find an answer to how to source SparX-5 hardware. For the time being, we've ordered one of the $5k devkits… (ETA 2 months or so)

Whether we can push anything in an useful direction remains to be seen, but I'm personally vested in pushing this as far as I (and we) can.

@equinox @erincandescent @benjojo @electronic_eel @grawity Sadly it is beyond my PCB layout skills, but I would love to build a switch around Sparx-5. Ideally with a socket for a raspberry pi compute module as the control plane.

@nolanl @equinox @erincandescent @benjojo @grawity even if you do all the development work in your spare time for free, the final prices for such a switch won't go into reasonable regions unless you have substantial sales volume behind it. There won't be many people buying such a switch if it would cost something like $3k in the end.

So I think it would be much better to invest time into building a contact to those chinese manufacturers linked above or something along those lines instead of designing your own. Testing the devboard to get a feeling for the capabilites and limitations of the chipset is a commendable thing though.

On the other hand I had a peek at the Sparx-5 datasheet. I was missing things like VXLAN and support for 25G ports (they have 25G SERDES, but they are limited to 10G speeds). On something released in 2021 I would have expected those features.

@electronic_eel @equinox @erincandescent @benjojo @grawity No VXLAN (and no ERSPAN) are bummers, but where did you get the idea that the 25G ports only run at 10G? The article I linked earlier has benchmarks that show the 25G ports doing better than 10G. It didn't get to 25G, but that is because the benchmark was all-ports, which ran into the total 160G total throughput limit of that part #.

@nolanl @equinox @erincandescent @benjojo @grawity hmm, seems I looked at the ds for a smaller part, the SparX-5-90. The bigger ones support 25G.

@benjojo @grawity honestly the more difficult bit is probably how many of them have like. An entire 16MiB of flash.

@benjojo
It looks like OpenWRT supports the SN2100. For low end use I've been buying Zyxel switches specifically for their Linux support since I'm tired of limited vendor updates. Thanks for writing this!

@benjojo I'm sorry to be that guy, but you have a typo in “Firmware" in the picture.

@benjojo nice article, but "If you are not able to compile a kernel yourself, and you can try with my pre-compiled kernels (that come with zero support/security updates/guarantee) here:" has no link following it

@benjojo Nice, that looks fun. Reminds me I should get back to my side project of making a gokrazy-based router OS for Ubiquiti EdgeRouter.

@benjojo I run a dozen Dell S5212F-ONs and it’s like a Soviet clone of your switch 😅 fewer ports and twice the power draw, but looks almost identical!