Code, docs and discussion related to the Aggregator.
The "puller" for the aggregators is called "Aggrivate" and is a node.js application. It uses Iconv, so you will need to install your distribution's "build-essential" package prior to running npm install.
The "parser" for the aggregators is called "Partytime" and is a node.js application.
The puller and parser run independently of each other. Aggrivate is constantly polling for updated feed content. When it finds updated content, it downloads that content into a file named with the feed id
and updates the feed record in the database with updated=<node id of this server>. The parser is then constantly polling the database for feeds marked with updated=<node id of this server>. When any are
returned it looks for their files and parses them into the DB.
Batch DB inserts are used whenever possible.