Sphinx and Gearman: A Distributed Computing AH-HA! Moment

A week ago I decided to finally get serious about putting gearman to use for search indexing. I had been batting the idea around in my head for a long time (too long, really) but figured I should just write the code and see what happens. It took less than a day to get a prototype working in our development environment, but the end result made me very happy. Today, in our production deployment, when a sphinx cluster pulls new content to index, the master does all the work. It fetches the new and changed postings, massages them into the XML format that sphinx expects (and makes a lot of small changes along the way), invokes the indexer, and makes the new indexes available for the slaves. The second step is usually the most CPU intensive. Processing the raw data into XML involves a lot of other tweaks and changes that are very specific to Criagslist. What I did was turn that into a gearman client/worker pair. The client (or master) simply submits processing tasks and then waits for each of them to complete. The workers fetch the data from the master, transform it, and make the transformed data available. When each task completes, the master grabs the transformed data an informs the worker that it can delete the file. So instead of being stuck at using only the 4 CPU cores on a single box, I can run 4 workers on each of 3 machines and get 12 CPU cores involved. The end result is that I have a solid foundation for a system that can easily scale to many machines. AH-HA! Linear scaling rocks! So does relatively seamless distributed computing. As time allows I'll have to work on deploying this in production. (comments)
Publication date: 2009-12-24

More news from 2009-12-24

On the MyBlogLog Shutdown

Marshall Kirkpatrick is reporting that Yahoo! will shut down MyBlogLog next year. Well, color me unsurprised. The service has languished for years. I removed it from my site a long time ago. It made me a little sad to do so, but it was just slowing things down and not really "adding value" as they like to say. It's sad because I was involved in the MyBlogLog acquisition at Yahoo! and believed in what they were doing. I worked to help get the team on board, nag the right people to make sure they got reasonable hardware on which they could grow, interviewed their first post-Yahoo engineer, and made the trek up to the Berkeley office a few times a week to help them transition into Yahoo and work on plans. I genuinely had high hopes for what MyBlogLog could do both inside and outside of Yahoo. But as I wrote in Watching Yahoo's Transformation: MyBlogLog has all but died on the vine, right? Is there anyone left of the original team of 5 or 6 engineers still working on it? No, I think it fell victim to Yahoo's larger social strategy. FAIL. On the one hand, it's sad that our collective time was wasted, but the members of the MyBlogLog team have all gone on to bigger and better things outside of Yahoo. And I suspect everyone involved learned some important lessons along the way. (comments)
Publication date: 2009-12-24 more

geeker2009 on "How to convert video for iPhone Free?"

This holiday I have been asked often how to watch videos on iPhone 3G and iPhone 3GS. Most people that own an iPhone 3G and iPhone 3GS know they can enjoy videos on their iPhones but they don't know how to exactly. So I wrote this tutorial to help you to watch videos on your iPhone 3G and iPhone 3GS. To watch the holiday videos on your iPhone 3G you need to convert the videos to the iPhone MPEG-4(MP4 for short) format. So you should convert the video files to .mp4 (H.264 encoding) format files before transferring them onto your iPhone 3G using some video converter tool. There are lots of free tools on the Internet, the one I'm using is Free iPhone Video Converter from Aneesoft. Don't know how to convert videos to iPhone mp4 format? It's not that difficult, let me show you step by step. Step 1. Download and install Aneesoft Free iPhone Video Converter Download and install Aneesoft Free iPhone Video Converter, which is easy to use and completely free! There is no spyware or adware. It's absolutely safe to install and run! You can convert videos for your iPhone, iPhone 3G and iPhone 3G S. Step 2. Add videos that you want to convert to iPhone MP4 format Click the button "Add
Publication date: 2009-12-24 more

Mandriva offers 10 second boot with InstantOn

The ability to boot a PC in under ten seconds is still a desirable rather than common feature of current operating systems. But there are solutions which allow for quick boot times by relying on a cut down feature set being present rather than the full OS experience. The latest company to join the fold with [...]
Publication date: 2009-12-24 more