I was trying to figure out how to write a search engine that could support GamerzCrib forums.

The biggest challenge is that potentially I could be looking at over 10 million posts on the server, and something like 5 million users a month. I've seen vBulletin forums where the search became painfully slow and they had less than a million posts.

After poking around the net, I found Solr, run by the same guys who do Apache web servers. Solr is a Lucene search engine written in Java. It runs as it's own service and accepts updates to the search index, and typically provides XML output as search results, all using http as it's interface.

The documentation is pretty cryptic for someone not familiar with Java, but that's about the only bad thing I can say about it. The version 1.3 'nightly builds' even has a PHP specific interface you can use.

If running Solr on an Internet connected machine, make sure you lock it down with a firewall, because there's no authentication needed to update or query the index. You don't want some script kiddie screwing with your search backend.

So far I've got my PHP scripts talking to it using CURL, so the next step is to write the code to actually add forum posts to the index, delete those entries if the post is deleted, and provide a search interface.