home tags events about login
one honk maybe more

benjojo posted 31 Jul 2024 09:59 +0000

SpamHaus's recent "Too big to care? - Our disappointment with Cloudflare’s anti-abuse posture" post does have a bit of a nit in it.

In that SpamHaus uses cloudflare itself:

$ dig www.spamhaus.org +short
www.spamhaus.org.cdn.cloudflare.net.
104.16.199.238
104.16.198.238

Now, I know that is a We Should Improve Society Somewhat type of criticism, but it is worth reflecting on if you are calling a supplier out for being bad... why are you giving them market share and enabling them that little bit more? (Even though I'm reasonably sure these days cloudflare has moved beyond the credibility need for those domains.)

It's complex, I have mixed views on cloudflare (even having worked there for 5~ years a while ago), they do cool stuff, and I still even use them for my blog. But cloudflare is a more of a wider discussion on do you want "individual" accountability on the internet for things like outages/policy/legal, or are we willing to accept the mass market cost decreases by merging that into the hands of a small bundle of players.

The services that the "internet centralisation" player provide are for the most part, genuinely useful and good (otherwise people would not use them). There is often this view from some folk that orgs are being forced at proverbial gunpoint to use AWS/Azure/Google/Cloudflare, but some of the alternatives are either a lot more work or cost significantly more overall, and it's probably the job of a higher power entity (IE, regulation) to control the "outage contagion" risk. Or not, depends on what stake holders want.

faisal@social.lol replied 31 Jul 2024 10:02 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/dq8xS9R2BMQXT62T6v

@benjojo they acknowledge that in the footer (also, the H is not capitalized)

"Spamhaus uses the services of Cloudflare. In fact Cloudflare helped Spamhaus during one of the worst DDoS attacks in the history of the internet, back in 2013. We were, and continue to be grateful, for the services Cloudflare provides us. Our concern is with how Cloudflare handles and prevents abuse."

benjojo replied 31 Jul 2024 10:07 +0000
in reply to: https://social.lol/users/faisal/statuses/112880527349438843

@faisal Sure, but its a bit of a complicated (In my eyes at least) argument there. Cloudflare has readily had this abuse problem for easily 10 years now, and while I'm sure things have changed since I've worked there, the vibe I get is that just short of extreme regulatory intervention, almost certainly nothing will not change it. So why are they consuming services from the "enemy"?

(To be fair, Spamhaus's own authority is waning, as a lot of their own efforts have meant that people just use SaaS email providers because it's too hard to host your own biz's email on-prem now, so Spamhaus's views matter less, the google/microsoft spam team's policy matters the most)

benjojo replied 31 Jul 2024 09:50 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/dq8xS9R2BMQXT62T6v

One of the low stakes conspiracy theories that I have is that AWS has either intentionally or unintentionally become one of the greatest reasons why people use AWS and find themselves trapped in Cloud provider ecosystems even when it makes financial sense to not operate in that environment.

I have spoken to a decent number of enterprise's (partly through contracting and partly through just events) and when people discuss about the inabilities to move from cloud IaaS back to "on-prem" ( even though they understand the it is very financially advantages for them to do so ), almost all of them these days cite the fact that they are unable to hire anybody to manage it for them, or if they do it is too expensive.

I think this is one of the most fascinating issues. The people who used to run such on-prem systems have either left the industry entirely or have moved on to managing AWS setups, and the new people entering the industry have not typically had any pressure to learn how to do such deployments, the result is the people who do need to use "on-prem" now have to pay a significant premium for experienced people who can do such work, thus lending more benefit to AWS even though it is more expensive on the face of it to operate.

It's hard to know if someone like AWS has actually planned this far ahead ( something that honestly I would doubt ) but it's a very interesting "chilling effect"

Don't get me wrong, some of what the IaaS providers offer is awesome, It's hard for me to offer anything close to the "Automatic Scaling Group" function that can just summon compute out of no where. But also, quite a lot of shops just don't need that, or if they do, it's likely cheaper to just have the capacity sitting idle as the overall costs are lower

erikk@chaos.social replied 31 Jul 2024 09:58 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/93PG4469XzXz1Z3Xdg

@benjojo Yea but also like there is less and less offering in the onprem market, so much stuff is now cloud only. And like autoscaling is a bit useless tbh with what you can get for so little money on onprem stuff. A fully specked out 128 core box is not that expensive.

Altho i also think the whole convoluted stuff of having to build relationships with people to get good quotes and keeping pricing a secret does not help the onprem case vs cloud.

benjojo replied 31 Jul 2024 10:02 +0000
in reply to: https://chaos.social/users/erikk/statuses/112880509971857225

@erikk Oh totally, "We" take the piss out of AWS's byzantine pricing structure, but at least it's _in theory_ possible to figure out the pricing of something without getting a sales person involved.

Colo is a weird set of space/bandwidth/hardware contracts/POs that vary from person to person. Even down to the "how much does a Juniper MX204 cost?" and it answer is "dunno, ask your channel partner and they will give you a number"

erikk@chaos.social replied 31 Jul 2024 09:59 +0000
in reply to: https://chaos.social/users/erikk/statuses/112880509971857225

@benjojo And like all onprem hardware stuff is getting retargeted to been or As a Service to offload stuff or beeing so hyperscaler focused that for a small / medium deployment the amount of people you need to make it worth it is not really there.
Now i do like working at a fully onprem gig and would not want to switch to a cloud only bla, but i do totally see why people don't want to go that route.

voltagex@aus.social replied 31 Jul 2024 10:26 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/93PG4469XzXz1Z3Xdg

@benjojo $employer is putting me and others under a bit of pressure to get AWS certifications, yet there's little training available for matching on-prem stuff.

I think it's now a self-perpetuating cycle - because there's less on-prem stuff there's less commercial training for it, less impetus for people to learn the basics, less people available to hire.

wolf480pl@mstdn.io replied 31 Jul 2024 10:48 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/N3g3frgv1dpP68717J

@benjojo @voltagex IME a bigger difference is storage.

Networking is also a thing in the cloud. Maybe you have less control of it and more ready-made solutions, but a misconfigured MTU can still ruin your day.

But storage? In AWS, there are these magical disks that are infinitely resizable, snapshottable, and you can attach them to any computer you like.
You never have to deal with smartctl, raid, lvm, etc.

benolot@comfy.social replied 31 Jul 2024 10:32 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/93PG4469XzXz1Z3Xdg

@benjojo@benjojo.co.uk I notice this a lot in discussions between me and @nasha@catgirls.technology, she knows on-prem and has experience there, I've only ever known and worked with cloud services. Which I will admit, defo leads to me missing some fundamentals about just how to run stuff on prem. There's a lot of "hidden" stuff that just never crossed my mind because you never need to think about it in the cloud.

dashdsrdash@tilde.zo.. replied 31 Jul 2024 11:01 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/93PG4469XzXz1Z3Xdg

@benjojo

I can support that.

I work for a financial software service. All the clients are banks or similar entities. Banks take months, at a minimum, to make any decisions. When they grow in usage, they do so primarily by moving accounts away from other services towards us, and secondarily by attracting new accounts (mostly from other banks).

The result is that growth is very predictable. There is no advantage to us in auto-scaling anything; certainly not worth the hassle of dealing with AWS in any regard. Instead, we buy hardware and install it in colocation data centers. Our peak workloads are not multiple orders of magnitude more than our minimum, and we get to completely sidestep all the problems that come from running your confidential data through other people's computers.

We do have problems hiring sysadminly staff, but we have at least two offsetting factors: staff turnover is extremely low, and related to that, we are willing to invest in training junior people to become senior people.