In the following image is depicted an overview of the Curiosity architecture:
A Curiosity execution consists of the following steps:
- retrieve some information;
- check the gathered data against the related “history” in order to identify the new items - and possibly the modified and deleted ones;
- possibly apply some xslt to the data;
- notify someone somehow of the data modifications identified, possibly bundled in a document formatted at the previous step.
Curiosity will execute the above steps for every information source defined; the main information source type is the Web source.
A web source consists of a starting url from which Curiosity can perform site crawling searching for pages to which apply screen scraping.
A gui named Curiosity Studio is available in order to do engineering of (xpath based) rules for crawling and screen scraping; it also allows for basic editing of web sources.
Other information sources are also available.
In general, for data retrieval we intend the acquisition of a list of information items having a number of attributes (or “slots”); for example, if our web source is the url of a news site, tipically Curiosity will search for a list of news each one having a title, a pubblication date, an abstract, possibly going through the news pagination.
Notification engine
In order to check the retrieved data against the history, Curiosity uses md5 checksum of the whole information slot. In the case of the web source, you can define one of the slots as a “primary key”: this way Curiosity will be able of uniquely identify an information slot and therefore check also for modification and deletions - besides additions.
Then, the Notification Engine will pass the filtered and marked data to each “provider” defined for the current source.
Providers
Providers are in charge of executing steps 3 and 4 of the above list.
You can write you own xsl transformations: learn about the representation used for the data passed to the providers.
The available providers will give you means for sending the data via email and upload it via ftp.
Morevoer, you can also implement in .NET your own providers.