When we initially built Kelpp, we really didn't worry about performance, scalability and all those other "luxuries". See I'm going to scale my foot up your ass by Ted Dziuba to get a sense for where we started from.
...Engineers love to talk about scalability. It makes us feel like the bad a**, dick-swingin' mother*** that we wish we could be....
However scalability wasn't really the problem, it was just that some queries ran slowly and pegged the CPU at 100%. This was a clear case of end user impacting perf slowdown.
It was time to do some improved indexing and searching.
Numerous solutions for fast indexing and optimized searching exist for Ruby on Rails. We picked arguably the most fully featured and best documented one: Ferret.
"Ferret is a high-performance, full-featured text search engine library written for Ruby. It is inspired by Apache Lucene Java project. "
A few days of optimization followed and voila: our query speeds on most queries were atleast an order of magnitude better.
At this point we are a lot happier with query performance but wanted to know if any of you had other suggestions for other indexers that are either faster or have functionality that is missing in Ferret? Do you have a Ruby on Rails blog that addresses this? Let us know and we'll link to you.
Once the performance optimization bug bites, it is hard to resist. Our site, like most could benefit from some speeding up anyway.
Please try out the new and improved query engine in Kelpp and we'd love to hear your suggestions!