Home

CacheHarvester was born out of a need to record and log a large amount of data about users hitting a tinyurl type site.

Because many thousands of users hit the site every hour, it was unfeasible to log items directly to the database and then redirect the user. The goal was to quickly log data and then forward the user on to the site they wanted.

CacheHarvester allows just this. A new object can be stored in memcached and harvested later by a generated daemon. The daemon generation is based on daemon-generator by dougal with an extra extension to the daemons gem to ensure that the process is properly killed if it does not respond in the specified amount of time.

All you need to do to get started with CacheHarvester is

  $ sudo gem install daemons
  $ cd /your/rails/app
  $ script/plugin install git://github.com/marksim/cacheharvester.git

and edit your config/environments/*.rb files to add

  config.cache_store = :mem_cache_store

Add to your config/deploy.rb file if you’re using capistrano:

  task :restart_daemons, :roles => :app do
    run "cd #{current_path} && ./script/daemons stop"
    run "cd #{current_path} && RAILS_ENV=production ./script/daemons start"
  end

  after "deploy:restart", "restart_daemons"

Then generate a harvestable model and its daemon as many times as you need

  $ script/generate harvestable_model Log url:string ip:string referer:string user_agent:string

And then in your application, simply

  Log.new(:url => 'http://github.com').store

Then, your new object will be stored in memcache and harvested to the database the next time the daemon runs.