Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I remember that. A few weeks later ran a script to count all the websites on the Internet.. 324 at that time.
 help



Was your script the very first web crawler or did you just have a list?

I'm also curious because I remember that the first time I used the Internet (not internet, as it is nowadays), I had to buy a paper book with categorized links to websites.

Connecting... Waiting... It was slow, both because of dial-up kbit/s and ping to websites, and every page felt like you were literally sending a request to another part of the planet. It felt like that was actually happening, and it was very different from what we experience now.

But most importantly, there were zero funds/VC in that Internet. Only very niche websites, zero online services, even email was difficult to obtain and felt like a real privilege. Only the fact of being connected made everyone feel not a stranger.

I kind of miss that Internet, but I'm grateful that once I was part of it.


There’s a page “Robert’s comments on Tim’s MIT trip” that says:

“I hope this does not offend Brewster, but I hope, probably in vain, that the commercialists will stay out of the Web world. Selling information is like selling air and water to me, though of course you need to pay the people who provide the information. Your comment already points out some of the bad side-effects of selling per access, or worse, tariffs per type of information or per item! Like: today's newspaper is 10CHF because there is this item in it which everyone wants to know about.”

Interesting too that an article on the front page the other day was about microtransactions for news.


I wonder which Robert said that.

The problem of viable news business models persists, and micro-payments have been proposed, but I have yet to see a viable implementation. Also, I think paying per news story isn't the right level of granularity. Articles that are less popular also need to be written, and the people that wrote them need food, too.


"I'm also curious because I remember that the first time I used the Internet (not internet, as it is nowadays), I had to buy a paper book with categorized links to websites."

I was looking at one of these books the other day called "The Internet Yellow Pages"

In the early 90s McGraw Hill published a book with this title

I found a version published by Que (Pearson Education) from 2007 (I suspect there may even be later editions)

That's 14 years into the public www

This was right before the iPhone

I never used these books. The best resources I remember were lists of sites published via FTP

Some of the nostalgia is still easily accessible via textfiles.com

I still use the internet in much the same way I did when I was first connected through a university. UNIX-like OS, no graphics, 100% command line

The main difference between then and now for me is the hardware and bandwidth

Everything is so much faster

IME, generally any slowness today is due (directly or indirectly) to the commercialisation ("monetisation") of _traffic_, e.g., ads, tracking, or having to use Tor to avoid all the nonsense

Originally the idea of commercial use of the internet was to sell products and services (excluding "advertising services"), not to sell and "monetise" _traffic_

Internet subscriber bandwidth is now used by companies for free to perform data collection, surveillance, telemetry, mostly undetected by the subscriber

For example, the majority of "Big Tech" revenues do not come from selling products and (non-advertising) services but from performing data collection, surveillance and "ad services". Even popular subscription software that predated the web, e.g., MS Windows, is engaged in data collection, surveillance and ads/tracking as a "business". Apple, once a traditional hardware company, is engaged in this activity as well

1. I have been doing some information retrieval experiments and the speed can be mind-blowing


Crawler. Heh.. never thought of it that way.

Wow. Which year was it?

In 1993, you could refresh the home page of Netscape (Mosaic) every day and it would mention new sites that had been added. That became unmanageable quickly, which is when two dudes from Stanford started a directory.


I've been trying to track down "What's New" for a long long time. If memory serves, there was a daily email titled "What's New on the World Wide Web" - very possibly the source for this monthly summary.

It was a fascinating way to experience the early WWW's exponential growth. It started out small, but once it began to grow, you could see it expanding faster and faster practically in real time.

At first it only took seconds to give the daily list a good once over. Over time it started taking minutes, then 20 minutes or half an hour (if things weren't too busy at work), and eventually it morphed into almost another full time job. There was just no way to keep up. Around that time they stopped sending it out.

From a historical point of view, these daily emails and monthly summaries would be a terrific resource for those interested in the early Web. It's hard to believe now that there was once a time when you could literally check out every new Web site as they came online.


If you remove commercial and edu and gov sites it's still doable today to track NEW unique websites today. There are less and less personal webpages due the instagram and fb etc

I keep track of these on my website, Well Made Web. You may like a visit: https://wmw.thran.uk/

It would be nice if a "What's new?" could be implemented by the Web protocol, a challenging task in a decentralized network.

New domain registrations would have to be queryable and the result set merged across all domain registrars globally.

That would make (completed) Web crawling easier, esp. of pages not interlinked (yet) with others.


Why the Web doesn't have a Scottish Accent

I once asked for funding from a Scottish business angel.

He confided to me that his biggest mistake in life was saying no on a phone call by a certain Tim Berners-Lee, who was looking for someone to help implement a browser for the "World Wide Web".

"Why did you reject him?" I asked. "'World Wide Web' sounded pretentious." said the man who got independently wealthy by selling a company that produced hypertext software (incl. browsers) for technical documentation running on Sun workstations...

...TBL turned to the NCSA team in the U.S. instead, and the rest is history.


I belive that an early beowser, possibly Mosaic, had an edit button. Think of that and the fundamental change of internet philosophy it implies!

The original idea behind a protocol which did updates as much as reads was later realised as Wikipedia and similar sites.

One early tool in this space was Navipress (which AOL bought out and made into AOLpress) which is notable for having been used by a certain Tim Berners-Lee to write a book:

https://www.w3.org/People/Berners-Lee/Weaving/

https://www.goodreads.com/book/show/821987.Weaving_the_Web


Just yesterday, this blew my mind: https://news.ycombinator.com/item?id=47096599

Thanks for reminding me of this. Used it once or twice to prank co workers.

Hence GET / POST / PUT. The WWW was designed to edit documents directly.

Yeah, and the original HTTP had PUT and DELETE methods.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: