Converters

This chapter explain adding new steps to workflow and converter plugins.

Add new Step

We can add new steps at task level. The Example 8 adds converter step to jsoupDefault steps in bs task. The relevant snippet is shown below

defs/examples/fin/jsoup/ex-8/job.yml

bsTask:
  dataDef: bs
  steps:
    jsoupDefault:
      converter:
        class: "org.codetab.scoopi.step.process.DataConverter"
        previous: filter
        next: appender
        ....  

The bsTask uses jsoupDefault built-in step as base and inserts converter step between filter and appender steps. The converter step is added only to baTask and for any other task that uses jsoupDefault step is not affected by this.

The structure of steps add or override is as follows

bsTask:                 -- task name
    dataDef: bs              
    steps:                 
      jsoupDefault:          -- base steps name
        converter:           -- step to add or override

Steps can override or add multiple steps.

Converters

Let’s see how to add plugins to steps. The datadef snippet from the example is as below.

bsTask:
  dataDef: bs
  steps:
    jsoupDefault:
      converter:
        class: "org.codetab.scoopi.step.process.DataConverter"
        previous: filter
        next: appender           
        plugins: <a href="https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html" target="_blank">
          plugin: { 
                 name: dateConverter, 
                 class: "org.codetab.scoopi.plugin.converter.DateRoller", 
                 item: year, 
                 inPattern: "MMM ''YY", 
                 outPattern: "yyyy-MM-dd", 
                 roll: "DAY_OF_MONTH=ceil" }            
       ]

The plugins is an array defined at step level and as array, it can hold multiple plugin objects. The above example defines a single plugin named dateConverter. It uses org.codetab.scoopi.plugin.converter.DateRoller which is applied to item named year. The inPattern and outPattern changes the date format and also roll any date field to ceiling and floor. For example, if axis item’s date text is Dec ‘16, the DateRoller first changes the date to format YYYY-MM-dd and date becomes 2016-12-01, but as financial dates are normally month end we need to roll the date field DAY_OF_MONTH to ceiling to get month end date. Refer [SimpleDateFormat for in and out patterns and Calendar for date fields.

Just to change the format without rolling the date, we can use org.codetab.scoopi.plugin.converter.DateFormater.

New steps definition

Do we need to define multiple steps when multiple tasks require jsoupDefault steps with date converter plugin. No, Scoopi allows you to define new steps in job.yml file using top level steps element.

taskGroups:
  bsGroup:
    bsTask:
      dataDef: bs
      steps: fooSteps

  plGroup:
    plTask:
      dataDef: pl
      steps: fooSteps

steps:
  fooSteps:
    jsoupDefault:
      converter:
        class: "org.codetab.scoopi.step.process.DataConverter"
        previous: filter
        next: appender           
        plugins: <a href="https://github.com/maithilish/scoopi/blob/master/src/main/resources/steps-default.yml" target="_blank">
          plugin: { 
            name: dateConverter, 
            class: "org.codetab.scoopi.plugin.converter.DateRoller", 
            item: year, 
            inPattern: "MMM ''YY", 
            outPattern: "yyyy-MM-dd", 
            roll: "DAY_OF_MONTH=ceil" }            
        ]

The above snippet defines two tasks - bsTask and plTask and both uses an user defined steps - fooSteps. The steps element defines a new fooSteps which uses jsoupDefault as base and adds converter step with DateRoller plugin. The steps element is top level element in job.yml like other top level elements - taskGroups, locatorGroups and dataDefs.

We can also create completely new steps without using default steps as base. To do that, copy any steps definition from [steps-default.yml to job.yml. Rename and alter the step as required.

The next chapter describes scripting plugin which allows data modifications with JavaScript.