Split Definitions

The Scoopi definition file tends to grow as you scrape more and more data sets or pages. This chapter explains how to split the definitions into multiple files for easy maintenance.

Scoopi allows you to split the definitions either on job types or on top level elements.

Split on Job Types

The examples so far scraped BS, PL, Quote data - price and snapshot - of Acme. We can split the definitions into three job types quote, bs and pl. The Example 13 splits job.yml into quote.yml, bs.yml and pl.yml. When we split on job types, the broad outline of each definition file is as below

locatorGroups:

  quoteGroup:                   -- locator group
    locators:                   -- locators
    ...

taskGroups:

  quoteGroup:                   -- task group
    priceTask:                  -- tasks
    ...

dataDefs:

links:                          -- datadef
...

The files bs.yml and pl.yml do not specify locatorGroups as locators are created from linkGroup tasks in quote.yml.

Split on Top elements

Another way to split is based top elements - locatorGroups, taskGroups and dataDefs. The Example 14 splits top level elements into three files - locators.yml, tasks.yml and datadefs.yml.

In the next chapter we explain the logging framework of Scoopi.