New post! After spending a while with the new switch/router that I use, I wrote down all of the notes I wish existed, and a general review of the Mellanox spectrum/switchdev support! Sysadmin friendly high speed ethernet switching https://blog.benjojo.co.uk/post/sn2010-linux-hacking-switchdev
qwertyoruiop@nso.gro..
replied 18 Apr 2024 10:38 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/BDc9VkBrB8c92615MS
@benjojo that firmware download link… I needed this article like last two weeks, it took me hours to find that link :X (and I’m still looking for FS3 failsafe images for a 2700 with empty flash that I bought and doesn’t boot ostensibly because it was never flashed)
benjojo
replied 18 Apr 2024 10:39 +0000
in reply to: https://nso.group/users/qwertyoruiop/statuses/112291787985986463
@qwertyoruiop Flash for what? the SATA disk? Have you managed to blow away the actual control plane's EFI/Firmware?
qwertyoruiop@nso.gro..
replied 18 Apr 2024 10:42 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/HK4XxtVln4cxq42n11
@benjojo flash for the ASIC. The x86 part doesn’t boot either so I connected the dataplane directly to another switch. Lspci indicates the asic is in ‘livefish’ mode, which is a firmware flashing mode, and it appears the whole flash is just empty. Mellanox has neat instructions on what to do in this case but unfortunately none of the files they reference are publicly available (and seem to be gated to OEMs… :() - and I assume x86 part doesn’t boot because the bios flash chip is also empty…
benjojo
replied 18 Apr 2024 11:01 +0000
in reply to: https://nso.group/users/qwertyoruiop/statuses/112291805306860598
qwertyoruiop@nso.gro..
replied 18 Apr 2024 11:06 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/x9Pl8H59C8Kw1vs5H8
@benjojo eBay, for quite cheap (but not cheap enough to deal with this tbh). I assume if I had another 2700 I could dump it’s flash and write it to this one, but I’ve got only one 2700, every other spectrum switch I have is 2410/3700 :\
qwertyoruiop@nso.gro..
replied 18 Apr 2024 10:39 +0000
in reply to: https://nso.group/users/qwertyoruiop/statuses/112291787985986463
@benjojo (also I’ve been looking for 2010s but can’t find reasonable priced ones, so I ended up buying a pallet worth of used 3700s for quite cheap :). great switches)
benjojo
replied 18 Apr 2024 10:41 +0000
in reply to: https://nso.group/users/qwertyoruiop/statuses/112291792089027445
@qwertyoruiop Yeah, Mellanox knocked it out of the park with the Spectrum switches. The only downside is that people think that they should run SONIC on them, a incredibly bad idea!
erincandescent@akko...
replied 18 Apr 2024 11:06 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/BDc9VkBrB8c92615MS
I hope this changes in the future, as Mikrotik’s hardware price point is very competitive, it’s just the software reliability that always turns me off their products, so having an option to not use RouterOS while keeping their very competitive hardware would be a huge deal. That hit’s hard. Not only is their price point great, their power consumption is also often one of the lowest for a given port count But even as little as I use RouterOS it drives me nuts
benjojo
replied 18 Apr 2024 11:14 +0000
in reply to: https://akko.erincandescent.net/objects/bf4e3dd4-bfb7-4ebb-bfca-5fddaa61fb50
@erincandescent RouterOS does eventually grow on you, but the software stability/inter-op is the larger problem
grawity@social.treeh..
replied 18 Apr 2024 11:31 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/BDc9VkBrB8c92615MS
@benjojo "Mikrotik is known to use this hardware, but right now has no official (or known) way to “jailbreak” the hardware to run your own software stack" – I thought the devices allowed you to TFTP-boot whatever, did that change?
benjojo
replied 18 Apr 2024 11:37 +0000
in reply to: https://social.treehouse.systems/users/grawity/statuses/112291994862903950
@grawity They will only accept (as far as I am aware) tik firmware, for the devices that mattrer in this case at least (The nice marvell chipset ones)
erincandescent@akko...
replied 18 Apr 2024 21:35 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/Lf5Y8k451lN6YQh2Nq
electronic_eel@socia..
replied 18 Apr 2024 22:15 +0000
in reply to: https://akko.erincandescent.net/objects/98511c08-9678-4bcb-bf4d-74cf9f584efb
@erincandescent @benjojo @grawity are the switch ICs documented well enough? As far as I've seen Marvell is not exactly forthcoming with documentation about their stuff. It wouldn't help much if you just get basic switching running and can't implement acls, igmp multicast and similar things due to those parts not being documented or part of a published driver.
azonenberg@ioc.excha..
replied 18 Apr 2024 22:16 +0000
in reply to: https://social.treehouse.systems/users/electronic_eel/statuses/112294528095750185
@electronic_eel @erincandescent @benjojo @grawity I mean, it depends on what you want to do with it. For my own FPGA-based switch project all I want is port vlans, 802.1q, forcing ports to specific speed/duplex states, and TDR testing on baseT interfaces. Maybe some basic ACLs eventually but that won't be an initial focus.,
azonenberg@ioc.excha..
replied 18 Apr 2024 22:17 +0000
in reply to: https://ioc.exchange/users/azonenberg/statuses/112294533633928215
@electronic_eel @erincandescent @benjojo @grawity That said, I also decided to go with an FPGA for my switch project *because* the big switch vendors made it ~impossible to get datasheets.
electronic_eel@socia..
replied 18 Apr 2024 22:20 +0000
in reply to: https://ioc.exchange/users/azonenberg/statuses/112294533633928215
@azonenberg @erincandescent @benjojo @grawity yeah, vlan port assignments are a must. but i also want to limit dhcp to trusted ports and do ipv6 ra guard incl. RFC7113. that requires deeper acls to implement properly.
jeroen@secluded.ch
replied 19 Apr 2024 06:44 +0000
in reply to: https://social.treehouse.systems/users/electronic_eel/statuses/112294550140422004
@electronic_eel @azonenberg @erincandescent @benjojo @grawity could always punt the packets to broadcast/multicast for certain ranges to the management cpu and do the hard decision there...
electronic_eel@socia..
replied 19 Apr 2024 07:39 +0000
in reply to: https://secluded.ch/users/jeroen/statuses/112296528635694360
@jeroen @azonenberg @erincandescent @benjojo @grawity yes, sure. But in the case of RFC7113 RA guard you quickly arrive at the point where you would have to parse all IPv6 packets from link-local addrs on the management cpu. that would kill performance.
jeroen@secluded.ch
replied 19 Apr 2024 08:39 +0000
in reply to: https://social.treehouse.systems/users/electronic_eel/statuses/112296746913570915
@electronic_eel @azonenberg @erincandescent @benjojo @grawity yes, it gets complex quick indeed. https://qdiv.dev/posts/eth2/ did some nice tricks already and that is in discrete logic ;) One day will attempt the same, hence following all your progress in amazement ;)
erincandescent@akko...
replied 18 Apr 2024 22:38 +0000
in reply to: https://social.treehouse.systems/users/electronic_eel/statuses/112294528095750185
@electronic_eel @benjojo @grawity theres some pretty decent support in mainline:
https://github.com/Marvell-switching/switchdev-prestera
nolanl@mastodon.soci..
replied 19 Apr 2024 00:12 +0000
in reply to: https://akko.erincandescent.net/objects/3026d2b7-4a78-4ca5-b799-dcaf5619876a
@erincandescent @benjojo @electronic_eel @grawity Unfortunately, despite the name, that is not actually a Prestera driver.
So called "DENT" switches have a 2nd CPU, and the firmware that driver uploads to the prestera is actually encrypted code for that CPU, which the driver then RPCs high level commands to.
Microtik switches won't have that 2nd CPU, which I usually refer to as the "GPL circumvention co-processor".
erincandescent@akko...
replied 19 Apr 2024 00:33 +0000
in reply to: https://mastodon.social/users/nolanl/statuses/112294990859050226
@nolanl @benjojo @electronic_eel @grawity ugh, that's profoundly annoying. At least when NXP play this game the coprocessor always accompanies the hardware
nolanl@mastodon.soci..
replied 19 Apr 2024 01:39 +0000
in reply to: https://akko.erincandescent.net/objects/d0a4193f-dd9d-4cfd-85d7-f3cd7cd539c3
@erincandescent @benjojo @electronic_eel @grawity Yeah, that is how Sparx-5 and Spectrum work. Marvell is as NDA locked down as always. What happened here is that Amazon (or Marvell, on behalf of Amazon) used the proprietary SDK to write a driver that is driven via RPC, and then hired an outside contractor to write a switchdev driver to that RPC interface. If the "firmware" blob wasn't encrypted, we could run it in CPU emulation and use the in-kernel driver, but alas...
nolanl@mastodon.soci..
replied 19 Apr 2024 00:14 +0000
in reply to: https://mastodon.social/users/nolanl/statuses/112294990859050226
@erincandescent @benjojo @electronic_eel @grawity Microchip's Sparx-5 driver actually does drive the Sparx-5 chip, but I've not been able to source a switch with one in the US. They're available in China though: https://www.servethehome.com/insane-48-port-2-5gbe-2x-25gbe-2x-10gbe-managed-chinese-e-sports-cafe-and-hotel-switch-microchip-micron-sandisk/
erincandescent@akko...
replied 19 Apr 2024 17:10 +0000
in reply to: https://mastodon.social/users/nolanl/statuses/112294995222066130
@nolanl @benjojo @electronic_eel @grawity if you ever find any reasonably priced (as in I won't be too sad if I brick it) SPARX-5 devices let me know, I'd really like to see some more interesting stuff in this area
equinox@chaos.social
replied 20 Apr 2024 13:59 +0000
in reply to: https://akko.erincandescent.net/objects/cbfac555-8f5d-44b1-a77d-b2c780da2463
@erincandescent @benjojo @nolanl @electronic_eel @grawity We (NetDEF, the FRR non-profit) are currently looking into this too, including trying to find an answer to how to source SparX-5 hardware. For the time being, we've ordered one of the $5k devkits… (ETA 2 months or so) Whether we can push anything in an useful direction remains to be seen, but I'm personally vested in pushing this as far as I (and we) can.
nolanl@mastodon.soci..
replied 20 Apr 2024 17:05 +0000
in reply to: https://chaos.social/users/equinox/statuses/112303902155347010
@equinox @erincandescent @benjojo @electronic_eel @grawity Sadly it is beyond my PCB layout skills, but I would love to build a switch around Sparx-5. Ideally with a socket for a raspberry pi compute module as the control plane.
electronic_eel@socia..
replied 20 Apr 2024 18:07 +0000
in reply to: https://mastodon.social/users/nolanl/statuses/112304636269448748
@nolanl @equinox @erincandescent @benjojo @grawity even if you do all the development work in your spare time for free, the final prices for such a switch won't go into reasonable regions unless you have substantial sales volume behind it. There won't be many people buying such a switch if it would cost something like $3k in the end. So I think it would be much better to invest time into building a contact to those chinese manufacturers linked above or something along those lines instead of designing your own. Testing the devboard to get a feeling for the capabilites and limitations of the chipset is a commendable thing though. On the other hand I had a peek at the Sparx-5 datasheet. I was missing things like VXLAN and support for 25G ports (they have 25G SERDES, but they are limited to 10G speeds). On something released in 2021 I would have expected those features.
nolanl@mastodon.soci..
replied 20 Apr 2024 18:25 +0000
in reply to: https://social.treehouse.systems/users/electronic_eel/statuses/112304877457200651
@electronic_eel @equinox @erincandescent @benjojo @grawity No VXLAN (and no ERSPAN) are bummers, but where did you get the idea that the 25G ports only run at 10G? The article I linked earlier has benchmarks that show the 25G ports doing better than 10G. It didn't get to 25G, but that is because the benchmark was all-ports, which ran into the total 160G total throughput limit of that part #.
electronic_eel@socia..
replied 20 Apr 2024 18:39 +0000
in reply to: https://mastodon.social/users/nolanl/statuses/112304949962319554
@nolanl @equinox @erincandescent @benjojo @grawity hmm, seems I looked at the ds for a smaller part, the SparX-5-90. The bigger ones support 25G.
electronic_eel@socia..
replied 19 Apr 2024 05:26 +0000
in reply to: https://mastodon.social/users/nolanl/statuses/112294990859050226
@nolanl @erincandescent @benjojo @grawity ughh, so that is how DENT switches are implemented. I have seen DENT and considered it interesting, but haven't had the time yet to investigate how they are implemented in more detail.
erincandescent@akko...
replied 18 Apr 2024 21:37 +0000
in reply to: https://akko.erincandescent.net/objects/98511c08-9678-4bcb-bf4d-74cf9f584efb
FritzAdalis@infosec...
replied 18 Apr 2024 11:44 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/BDc9VkBrB8c92615MS
@benjojo
It looks like OpenWRT supports the SN2100. For low end use I've been buying Zyxel switches specifically for their Linux support since I'm tired of limited vendor updates. Thanks for writing this!
benjojo
replied 18 Apr 2024 11:48 +0000
in reply to: https://infosec.exchange/users/FritzAdalis/statuses/112292046665625204
Tenzer@s.waq.dk
replied 18 Apr 2024 11:55 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/BDc9VkBrB8c92615MS
benjojo
replied 18 Apr 2024 11:57 +0000
in reply to: https://s.waq.dk/users/Tenzer/statuses/112292091549568062
nabijaczleweli@10101..
replied 18 Apr 2024 13:06 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/BDc9VkBrB8c92615MS
@benjojo nice article, but "If you are not able to compile a kernel yourself, and you can try with my pre-compiled kernels (that come with zero support/security updates/guarantee) here:" has no link following it
benjojo
replied 18 Apr 2024 16:14 +0000
in reply to: https://101010.pl/users/nabijaczleweli/statuses/112292368300305324
karppinen@mastodon.o..
replied 24 Apr 2024 18:37 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/BDc9VkBrB8c92615MS
@benjojo I run a dozen Dell S5212F-ONs and it’s like a Soviet clone of your switch 😅 fewer ports and twice the power draw, but looks almost identical!
benjojo
replied 25 Apr 2024 07:55 +0000
in reply to: https://mastodon.online/users/karppinen/statuses/112327645193725329
@karppinen yeah I had the option to buy those as well but I discovered that they were basically just boring broadcom switches versus the much more exciting option
jamesog@mastodon.soc..
replied 18 Apr 2024 16:35 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/BDc9VkBrB8c92615MS
@benjojo Nice, that looks fun. Reminds me I should get back to my side project of making a gokrazy-based router OS for Ubiquiti EdgeRouter.
benjojo
replied 18 Apr 2024 10:36 +0000
in reply to: https://helvede.net/users/holsta/statuses/112291745278170256
@holsta I'd much rather reviews of kit directly from the people who enjoy/suffer it, than some generic case study that has almost certainly been sterilized of any kind of criticism