How to search the internet 2: how a modern web search works

In the first instalment of this guide on how to search the internet, I gave a little history of the search engine: I covered Archie, Gopher, and site directories like the Open Directory Project. Those are the old technologies, all pretty much obsolete now. That brings us to the present day and the modern search engine.
When I write “modern search engine”, I mean web search sites like Google and Bing. Because they all work in pretty much the same way – the only difference seems to be in the algorithms each service uses.
Now I could tell you all about spiders crawling the web and stuff, but I think most of you would just tune out after a couple of lines. So I will give you 2 lovely Youtube videos to watch instead:
The 3 Minute Guide to How Search Works:


A slightly longer video that looks at the subject from the perspective of a webmaster who wants to increase traffic to his site:

Watched them? Good. So now you have the basic idea: little programs called “bots”, “crawlers” or “spiders” are sent out to crawl over the world wide web, following links, and compiling lists of URLs that they consider to contain good information. And how do these mindless software automatons decide that the info is “good”? It all comes down to the algorithms.
It’s Google’s algorithms – the “secret ingredient” – that has made Google the world’s favourite search engine and kept them at the top for so many years. Any coder of sufficient proficiency can create bots to crawl the web; but it’s the secret algorithms that turn a regular bot into a googlebot. And there just hasn’t been another bot that can compete.
At least that’s how it has seemed for some time. Yahoo has a hard core of admirers; Altavista.com has had success mostly due to its “Babel Fish” translation service blowing its rivals out of the water; but it’s only recently that a true contender for the title of Number One Search Engine to step up and challenge Google. That challenger’s name: Bing.
Microsoft has been trying for years to break into the search engine market, with a plethora of products: Live Search, Windows Live Search, MSN Search – they even tried to buy, then made a deal with Yahoo to get that Microsoft name up there with the giants – but nothing was able to make much impact on Google. Then in 2008 Microsoft (following the tried and tested strategy of “embrace, extend, extinguish”) bought a tech company called Powerset and, importantly, its “semantic technology”. Microsoft claim that their improved technology cuts down on the risk of “search overload”, when a user is inundated with millions of barely relevant results – something that can happen when using Google. And Microsoft has used the near-ubiquity of its web browser, by incorporating Bing into Internet Explorer 8. Google is still number one search engine, but Microsoft has certainly made its mark on the territory.
So who’s going to win this battle of the search engines? I think it could still go either way. Google has years of good form and a hell of an online presence; but Microsoft still owns the desktop and the browser. And anyway, someone else might come from the left field and clinch it in the final seconds – Ixquick is a potential outside bet with their whole “ethical privacy” trip; Google’s got the “Don’t be evil” motto but it’s Ixquick who are out there actually being “not evil” (and if privacy is a major concern, don’t forget Scroogle). One thing we should have learnt from IT history is that nothing is set in stone.
I’ll bet you’re thinking “Oh well done Google and Microsoft, give yourselves a pat on the back… but what in hell has any of this got to do with how to use a goddamn search engine?!! I figured it would be useful to cover all this history and present situation stuff. Well, maybe interesting rather than useful… I certainly find this kinda crap fascinating. But you’re right, it doesn’t tell us a great deal about how to use a search engine. So I promise: the next instalment of this howto will actually cover some proper howto material. So keep ’em peeled… you definitely don’t want to miss this!!

_gos=’c4.gostats.com’;_goa=354450;
_got=2;_goi=2;_goz=0;_gol=’Free hit counter’;_GoStatsRun();
Free hit counter
Free hit counter

I <3 Dropbox!

Well, maybe it’s a bit overboard saying that I “heart” Dropbox. I mean it’s just an online storage solution, it hasn’t got breasts or a dazzling personality! But I think it’s pretty cool nevertheless, and today I’m gonna tell you why.
For quite some time now, barely a day has gone by without me seeing or hearing something about “cloud computing”. And although I hate these buzz words that don’t actually mean very much, I finally figured that “the cloud” was something I could use.
I need to access some files an awful lot, wherever I may be. And sometimes that means accessing the files from a library computer, or a computer at a client’s office – in other words, computers that do not belong to me. And even if I do have my netbook on me, I want any alterations made to my files to be synchronized to all my machines automatically.
For reasons too boring to go into here, I can’t access my home machine from the internet. And I am remarkably ill-equipped when it comes to online resources – I use a wordpress.com-hosted blog for crying out loud, I ain’t got a web server of my own kicking around somewhere. And carrying a fistful of USB sticks is not an ideal solution – sticks can easily be misplaced or even stolen. So I decided I needed to sign up for one of those “cloud computing” services, where I put a bunch of files on a third party’s server somewhere out there on the interwebs which I can then access no matter where I am (within reason – if I’m on a camel in the middle of the Sahara and forgot to pack my satellite phone I’d be screwed. But as I own neither a camel or a satellite phone, I think we can rule out that possibility).
Because of my innate stingeyness, I needed a solution that was free. So I fired up my good friend Google, plugged in the search terms “free cloud computing storage” and let ‘er rip. And it turned up a few free solutions, such as G.ho.st, Google’s various products, box.net, oosah.com… There’s a lot out there – if you want a quick list of freebies check out this guide at readwriteweb.com.
But of course, I’m utterly clueless when it comes to all this cloudy Web 2.0 stuff. So I went to my favourite forum and had a look at what folk there were saying on the subject.
Unsurprisingly for an Ubuntu site, a lot of people seemed to rate Ubuntu One. But there were also a bunch who liked DropBox. And I kinda liked what they were saying. So I chose to go with DropBox.
Like a lot of these cloud storage services, DropBox gives you 2GB of space for free. You install this program on the computers you want to be synced (and yes it comes in a linux flavour), create a DropBox folder on each computer, then link those computers to your account. Once that’s done, all you have to do is put files into the DropBox folder on one of the computers, and before you know it those files are accessible from all your synced computers. And you can even access them if you’re on a different computer, as there’s a web interface you can sign into from anywhere!
Another cool feature is the “Public” sub-folder. If you put a file into the Public sub-folder, then right-click on it, you get a link to that file that you can post in a blog, forum, whatever. So you can make chosen files accessible for absolutely anyone you want, without having to tell them your username or password. For instance, here’s a link that will enable you to download a pdf of the novel Neuromancer by William Gibson. If you’ve never read it, I strongly urge you to give it a go. Extremely cool cyberpunk science fiction. And I’ll let you have have it for the very reasonable price of fuck-all.
Cloud computing isn’t for everyone, despite what some characters will try and tell you. A lot of people will have no need for it whatsoever. But if you think it might be useful, go grab yourself a free account and give it a whirl. I’ve certainly been seduced by the sultry maiden called DropBox, as you may have guessed from this gushing love letter. Did I say love letter? That should have said “porn”. Cos DropBox makes me horny as only a sad geek can be!!
Note: Unfortunately, some of the info here is out of date. For instance, g.ho.st no longer provides a free service (though they’ll happily take your money) and for some reason the oosah.com site seems to be unavailable. But there definitely are free services available out there. Go check it out!
———-
I just thought I’d add a footnote to point out there’s another free (as in beer) online storage solution out there: Gspace. This Firefox add-on enables you to use the inbox of a Gmail account as an online disk. Google gives its Gmail users an awful lot of storage – more than 2GB at the moment, and rising all the time – plus you can use any number of Gmail accounts with Gspace. This solution is especially useful if, like me, you own a netbook with limited onboard storage. It works with Windows, OSX and Linux. I use Gspace, and can thoroughly recommend it.

_gos=’c4.gostats.com’;_goa=354450;
_got=2;_goi=2;_goz=0;_gol=’Free hit counter’;_GoStatsRun();
Free hit counter
Free hit counter