dekstop: weblog

I just finished a first version of a custom database cache for FeedTools 0.2.23 -- basically a drop-in replacement for the default DatabaseFeedCache. I did this for two reasons: first, because I could (always a good reason), and second, because DatabaseFeedCache does not work at all in multi-threaded applications. DatabaseFeedCache is based on ActiveRecord, and while ActiveRecord is supposed to be thread-safe since version 0.9 I've found this not no be true, at least when using SQLite3.

The implementation of SQLite3FeedCache is as basic as it gets, to the point where it's rather inefficient -- it does not even keep a persistent database connection, but reconnects and disconnects with every call to save, find_by_id and find_by_href.

But hey, it's thread-safe. ;)

You can browse the short SQLite3FeedCache documentation for more details.

Setup

SQLite3FeedCache, like the original DatabaseFeedCache, requires you to set up a database cache file yourself before you can start using it. But that's easily done:

$ sqlite3 db/feeds.db < db/schema.sqlite.sql 

If you look at the schema definition you'll see that it matches that of FeedTools -- which means you can use the same database cache with SQLite3FeedCache and FeedTools' default caching mechanism, as long as you're using SQLite 3.

Example

After setting up your database cache you can start using SQLite3FeedCache in your FeedTools script. Start your scripts like this:

# point SQLite3FeedCache to the database file
SQLite3FeedCache.set_db('db/feeds.db')
# tell FeedTools to use the SQLite3FeedCache
FeedTools::configurations[:feed_cache] = SQLite3FeedCache

... and that's all it takes. You now can start requesting and parsing feeds using the convenient FeedTools interface, and all requests will be cached by SQLite3FeedCache. See my earlier article, Using the FeedTools Cache in Plain Ruby Scripts, for some examples of how to request feeds with FeedTools.

Of course the main reason for using SQLite3FeedCache is its thread-safe implementation -- just be patient, I'll probably write a little about multi-threaded feed aggregation later.

Disclaimer

This is very new, and there might be bugs. My main goal for this initial version was to make it work at all.

And it's probably safe to say that this will break with the upcoming FeedTools 0.3 release.

Let me know how it works for you -- or even better, make suggestions for improvement.

Update: I just uploaded a new version that fixes a small bug where feeds behind HTTP redirects were re-requested when they shouldn't be.

Download

• sqlite3_feed_cache.rb.txt
• schema.sqlite.sql (database schema)

---