A Comparison of Open Source Search Engines « zooie’s blog
http://zooie.wordpress.com/2009/07/06/a-comparison-of-open-source-search-engines-and-indexing-twitter/

a first step to investigate search engine.

Later this month we will be presenting a half day tutorial on Open Search at SIGIR. It’ll basically focus on how to use open source software and cloud services for building and quickly prototyping advanced search applications. Open Search isn’t just about building a Google-like search box on a free technology stack, but encouraging the community to extend and embrace search technology to improve the relevance of any application.

and it's on track to become the open source world's canonical answer to the question of text search. MySQL and Solr, the two popular solutions, are showing their age. MySQL introduced full-text search in late 2000 as a way to more intelligently search blobs of text stored in databases. You can work a full-text clause into a query, and MySQL will rank the result rows by how relevant it thinks they are to the query. MySQL uses textbook search algorithms and doesn't allow for a lot of relevance tuning. It's like a drawing from a five year old: The heart is in the right place, but everybody knows that kids suck at drawing. Implementation details aside, MySQL still suffers from scalability problems. Having ignored the trend of chip manufacturers to build multiple cores into CPUs, hoping that this unpleasant trend that required them to actually think about multi-threading would just blow over sooner or later, MySQL's ability to handle parallelism is, well, see the five year old's drawing.

Sphinx can index 10 megabytes of data per second and can search up to 100 gigabytes of text on a single processor. It also supports multi-machine distributed searching, as in the case of Craigslist.

fast full-text search

Writing Technical Documentation with Sphinx, Paver, and Cog.

I knew in the back of my mind that Doug Hellman had a sphinx workflow for his blog posts. Now, how did I not notice how close it was to my needs? Includes lots of tasty tips for tweaking output etc

"Writing Technical Documentation with Sphinx, Paver, and Cog I've been working on the Python Module of the Week series since March of 2007. During the course of the project, my article style and tool chain have both evolved. I now have a fairly smooth production process in place, so the mechanics of producing a new post don't get in the way of the actual research and writing. Most of the tools are open source, so I thought I would describe the process I go through and how the tools work together."