I was trying to scrape a website briefly for research purposes.
Fetching the HTML with curl using the correct combination of headers worked fine, but when I tried to do it from inside my program it returned an error 403, exact same headers, same IP, same URL.
Eventually I found out that the website had Cloudflare in front and Cloudflare did client fingerprinting based on the TLS algorithms advertised or something like that, such that curl and browsers were allowed but not anything else.
It worked when I switched my code to using libcurl.
This is both stupid and dystopic. How did we get to this situation?
fiatjaf
npub180…jh6w6
2026-05-14 23:05:22 UTC
Author Public Key
npub180cvv07tjdrrgpa0j7j7tmnyl2yr6yr7l8j4s3evf6u64th6gkwsyjh6w6Published at
2026-05-14 23:05:22 UTCEvent JSON
{
"id": "3e9c06ac86f1b63c0953438f312c095c4d4290976fb5b87b80f5600684f5959d",
"pubkey": "3bf0c63fcb93463407af97a5e5ee64fa883d107ef9e558472c4eb9aaaefa459d",
"created_at": 1778799922,
"kind": 1,
"tags": [],
"content": "I was trying to scrape a website briefly for research purposes.\n\nFetching the HTML with curl using the correct combination of headers worked fine, but when I tried to do it from inside my program it returned an error 403, exact same headers, same IP, same URL.\n\nEventually I found out that the website had Cloudflare in front and Cloudflare did client fingerprinting based on the TLS algorithms advertised or something like that, such that curl and browsers were allowed but not anything else.\n\nIt worked when I switched my code to using libcurl.\n\nThis is both stupid and dystopic. How did we get to this situation?",
"sig": "085e15fae1b0968d2728c7d876537bc793cc74225fb7da2f95a9553b06d4ea6098c5a996803e897f39b46a2f1b03c4c0c3184dd8afb2cc18b74e8ab3226e3629"
}