Hacker Newsnew | past | comments | ask | show | jobs | submit | QuiDortDine's commentslogin

So custom implementation, then? How very Microsoft.


Even funnier when you remember that they own github, the place where arguably markdown was popularized.


> markdown was popularized.

That was before MS started vibecoding all their software.


I remember back when Anubis came out, some naysayers on here were saying it wouldn't work for long because the scrapers would adapt. Turns out careless, unethical vibecoders aren't very competent.


I still think it is just a matter of time until scrapers catch up. There are more and more scrapers that spin up an full blown chromium.


It seems inevitable, but in the mean time, that's vastly more expensive than running curl in a loop. In fact, it may be expensive enough that it cuts bot traffic down to a level I no longer care about defending against. Like GoogleBot had been crawling my stuff for years without breaking the site. If every bot were like that, I wouldn't care.


Serious question, in 2026 you can actually have a successful crawler with just curl? I just had to create one for a customer - for their own site - and nothing would have worked without using Chromium.


Probably not for most sites. Example of a site where it'd likely work: a blog made with a static site generator. Example of one where it wouldn't: darn near anything made with React.


It works for the majority of things a text mining scraper would care to scrape. It's not just static sites but also any CMS like wordpress, as well as many JS apps that have server-side rendering. SPA-only sites aren't that common anymore, especially for things like blogs, news and text-based social media.


Cool, if they're running full blown chromium maybe the next step can be mining bitcoin on any pages served to bots.


Even that functions as a sort of proof of work, requiring a commitment of compute resources that is table stakes for individual users but multiplies the cost of making millions of requests.


AFAIK you can bypass it with curl because there's an explicit whitelist for it, no need for a headful browser.


Well it's a race, just like security. And as long as anubis is in the front, all looks bright


> Turns out careless, unethical vibecoders aren't very competent.

Well they are scraping web pages from a git forge, where they could just, you know, clone the repo(s) instead.


"Turns out careless, unethical vibecoders aren't very competent." well, they rely on AI, don't they? and AI is trained with already existing bad code, so why should the outcome be different?


Not sure why you're talking like OP pissed in your cheerios. They are a victim of a broken system, it shouldn't be on them to spend more effort protecting their stuff from careless-to-malicious actors.


I'm always fascinated by these growth charts. Isn't everyone who needs GitHub already on GitHub? Are people migrating from GitLab? I don't get it!


This is only true if you weren't paid for your work all those years (which, then, it was just a hobby).

But more importantly, this is only relevant for vomiting boilerplate code. I don't know about you but I always did a lot more than that.


Sales taxes are regressive and nonsensical, why punish people because they contribute to the economy by buying stuff? Just do income tax.


I don't disagree with the first part, but income tax falls into the exact same logic of "why punish people because they contribute to the economy by [working]?"


Well, we do need taxes. Land taxes are fine too I guess, but I doubt they'll catch on in North America (too many politicians own real estate).


You don't contribute by buying stuff but by producing it. And the regressive effect can be reduced by having different rates depending on the type of product.


hahah what are you talking about, there's no such thing as long term!


Surely you mean managers, right? Most developers I interact with would love to do things the right way, but there's just no time, we have to chase this week's priority!


Jesus I just had flashbacks from my last jobs. Non-technical founder always telling me I was being pessimistic (there were no technical founders). It's just not that simple Karen!


Is there a way to ban specific users in your GitHub project?

(I prefer GitLab, I'm sure if it had projects that are as popular it would be similarly inundated.)


IIRC if one of the maintainers of a project blocks a user that prevents them from participating in issues and PRs.

For bigger projects with many maintainers that can also lead to problems if people use the block function as liberally as on Twitter.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: