Eliot's Ramblings

curl http://stream.twitter.com/1/statuses/sample.json -u<user>:<pass> | mongoimport -c twitter_live

One thing that you can do with mongo is have 1 streaming master and 1 read/write master

server A:

./mongod —master —dbpath /tmp/a

server B

./mongod —dbpath /tmp/b —master —slave —source localhost:27017 —port 9999

You can then pipe the stream into server a, and it will only process the live stream.

Server B will replicate all changes.  You can also write to it, query on it, etc…  This way you can do operations that block writing on server B, but server A will never backlog.