With persistence, Scoopi offers following benefits.
- reduce network usage by reusing the downloaded pages
- recover from the aborted run by skipping the tasks already completed
- avoid expensive parse operation by reuse of persisted data
- set expiry date for each page
If you running Scoopi from docker image then no database setup is required as docker image contains pre-configured MariaDB container. Please refer Scoopi Installation to run Scoopi with MariaDB using docker-compose.
In case you are using release package from GitHub to run Scoopi, then you have to set up MariaDB or HSQLDB database manually. See HSQLDB Setup to setup and configure Hsql database.
Enable Persistence in Scoopi
By default persistence is disabled in Scoopi through configuration
setting scoopi.useDatastore=false in
conf/scoopi.properties file. To
enable persistence, either set the property scoopi.useDatastore=true
or hash it out.
Now run Example 10 and scoopi database should have locator, datadef, data and document tables with data.
When we first run Example-10 with persistence enabled, document table which holds compressed contents of the pages - acme-snapshot-links.html, acme-bs.html and acme-pl.html will end up with three rows and data table which holds the parsed data will have five rows - finLinks, price, snapshot, bs and pl.
If we run Scoopi again and query the tables, the document table will have 6 rows and data table will end up with 10 rows. For each run, Scoopi fetches fresh pages and parse it to create data even though persistence is enabled. This is because, by default, live setting which controls the expiry of page is set to zero days (P0D). So, in each run, Scoopi sees page as expired and fetches new one. Use live property in tasks group to alter this default behavior.
Suppose, we want to fetch new snapshot page once in a week. The Example 11 sets live property to P1W for snapshotGroup as shown below
taskGroups: snapshotGroup: priceTask: dataDef: price snapshotTask: dataDef: snapshot live: P1W
The P0D, P1W are ISO_8601 based representation of duration.
During initial run, Scoopi fetches fresh pages and persists. In subsequent runs, it reuses the persisted page and data till pages are expired. This speeds up the run by many fold.
We can further control persistence of locator or data using persist property.
conf/scoopi.properties we can set following properties
scoopi.useDatastore=true|false scoopi.persist.locator=true|false scoopi.persist.data=true|false
If scoopi.useDatastore is false then there is no need to start database as nothing is stored. If scoopi.useDatastore is true, then
- if scoopi.persist.locator is false then no locator and its documents are stored
- if scoopi.persist.data is false then no data is stored
If scoopi.persist.data is true, then in addition, we can control whether to persist data of each task using persist/data property in task definition. The Example 12
taskGroups: snapshotGroup: priceTask: dataDef: price persist: data: true snapshotTask: dataDef: snapshot persist: data: false live: P1W
The data parsed by priceTask are stored but not the snapshotTask as persist/data is false for that task.
Use Scoopi with other Databases
Scoopi uses JDO and DataNucleus
AccessPlatform which works with variety of
databases. Scoopi is tested with MySQL, MariaDB and MongoDB and
configuration for each of them is in
The next chapter describes spliting the lengthy defintions into multiple files for easy maintainance.