Now this is a cool new trick that is definitely borderline on the halon razor (is it a fuck up? or is it someone doing something malicious?) This LLM crawler is supplying two different user-agent headers presumably with the idea to confuse systems that restrict or log user-agent values
one honk maybe more
roguelazer@hachyderm..
replied 09 Jun 2025 18:06 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/fTKQv1834WL3CGF7y5
@benjojo Shouldn't repeated headers be normalized into a single header with the values separated by comma? RFC 2616 § 4.2?
benjojo
replied 09 Jun 2025 18:09 +0000
in reply to: https://hachyderm.io/users/roguelazer/statuses/114654734367486083
@roguelazer 🌈It depends🌈, What 2616 suggests and what often happens in reality are quite different. A lot of the HTTP libs treat HTTP headers very similar to SMTP headers (just multi line doesnt work*), so they just keep adding stuff on and on, Go's HTTP parser will supply headers to you as a
map[string][]string to handle this edge case
jamesog@mastodon.soc..
replied 09 Jun 2025 19:51 +0000
in reply to: https://benjojo.co.uk/u/benjojo/h/fTKQv1834WL3CGF7y5