Script Plugin

The example in the previous chapter converts date field using built-in DateRoller plugin. However, it is not possible to handle all types of modification with built-in plugins. To cope with myriad possibilities, Scoopi provides Scripter step to which user can hook arbitrary JavaScript to modify data.

Scripter Step

The Example 8a adds process step to jsoupDefault steps in bs task. The relevant snippet is shown below

defs/examples/fin/jsoup/ex-8a/job.yml

bsTask:
  dataDef: bs
  steps:
    jsoupDefault:
      process:
        class: "org.codetab.scoopi.step.script.Scripter"
        previous: filter
        next: appender
        ....

It inserts a new process step between filter and appender step of jsoupDefault steps. The step class is org.codetab.scoopi.step.script.Scripter which executes Script Plugin.

Script Plugin

The Script Plugin can execute any user provided JavaScript. The plugin definition for process step is as below.

defs/examples/fin/jsoup/ex-8a/job.yml

bsTask:
  dataDef: bs
  steps:
    jsoupDefault:
      process:
        class: "org.codetab.scoopi.step.script.Scripter"
        previous: filter
        next: appender
        plugins: [
          plugin: { 
            name: script, 
            class: "org.codetab.scoopi.plugin.script.BasicScript",
            script: "/defs/examples/fin/jsoup/ex-8a/script.js",
            entryPoint: "execute",
            scripts: [ "/defs/examples/fin/jsoup/ex-8a/moment.min.js"] }            
        ]

The plugins/plugin definition is similar to the one used in previous chapter but with following changes.

class - org.codetab.scoopi.plugin.script.BasicScript
script - path to js file which contains the JavaScript code
entryPoint - the name of JavaScript function to call
scripts - array of additional JavaScript libraries to load

The entryPoint property in plugin calls the function defined in js and scripts array is used to add any other required JavaScripts or libraries. In the above example, we add moment.min.js script which is well known library in JavaScript to manipulate dates.

The JavaScript file /defs/examples/fin/jsoup/ex-8a/script.js is where we have put our JavaScript code which is as shown below.

var execute = function(data) {
var items = data.getItems()
for (i = 0; i &lt; items.size(); i++) {
        var item = items.get(i)
        var dateAxis = item.getAxisByItemName("year")
        var date = dateAxis.getValue()
        var x = moment(date, "MMM 'YY")
        var e = moment(x).endOf('month')
        var d = moment(e).format('YYYY-MM-DD')
        dateAxis.setValue(d)
}
    return data
 }

The Scripter step passes data object which contains list of date items, to the execute() function in script.js and js loops through each data item and format the date using Moment library.

While script plugin is the easy and flexible way to modify any sort of data but its execution is inherently slow when compared to the built-in converter plugin. If you are well versed in Java, then better option is to develop your own converter plugin by extending org.codetab.scoopi.plugin.converter class.

Next, we move on to creation of new locators from links parsed from pages and pagination.