Converters


October 13, 2018 Maithilish

Converters

This chapter explain step override and converters.

Step Override

We can override any step at task level. The Example 8 overrides converter step of jsoupDefault steps in bs task. The relevant snippet is shown below

defs/examples/jsoup/ex-8/job.yml

bsGroup:

  bsTask:
    dataDef: bs
    steps:
      jsoupDefault:
        converter:
          class: "org.codetab.scoopi.step.convert.DataConverter"
          previous: process
          next: appender

The other tasks in the example uses jsoupDefault steps as it is but the bsTask takes all steps from jsoupDefault and replace converter step with the step defined locally at task level.

The structure of steps override is as follows

bsGroup:                     -- task group

  bsTask:                    -- task name
    dataDef: bs              
    steps:                 
      jsoupDefault:          -- steps name
        converter:           -- name of the step to override 


Multiple steps can be overridden in single steps definition.

It is also possible to define completely new steps definition in job.yml file using top level steps definition. To do that, copy steps definition and jsoupDefault steps from steps-default.yml and add it to job.yml and rename jsoupDefault in job.yml to some other name say userStepsX and then make required changes to any of the step. In task, we can refer it as

bsGroup:

  bsTask:
    dataDef: bs
    steps: userStepsX

Converters

Now we can plugin converters to the locally defined step so that they are applied only to bs task and not to other tasks.

The complete overridden step from the example is as show below.

defs/examples/jsoup/ex-8/job.yml

bsGroup:

  bsTask:
    dataDef: bs
    steps:
      jsoupDefault:
        converter:
          class: "org.codetab.scoopi.step.convert.DataConverter"
          previous: process
          next: appender
          plugins: [
            plugin: {
                   name: converter,
                   class: "org.codetab.scoopi.plugin.converter.DateRoller",
                   axis: col,
                   inPattern: "MMM ''YY",
                   outPattern: "yyyy-MM-dd",
                   roll: "DAY_OF_MONTH=ceil" }
         ]

It uses org.codetab.scoopi.plugin.converter.DateRoller which, based on inPattern and outPattern, changes the date pattern and also roll any date field to ceiling and floor. For example, if col axis date text is Dec ’16, the DateRoller first changes the date to format YYYY-MM-dd and date becomes 2016-12-01, but as financial dates are normally month end we need to roll the date field DAY_OF_MONTH to ceiling to get month end date. Refer SimpleDateFormat for in and out patterns and Calendar for date fields.

To change the format we can use org.codetab.scoopi.plugin.converter.DateFormater but, to change format and adjust date we require DateRoller.

The next chapter explains the creation of locators from links parsed from the pages.