Appender and Encoder

Scoopi uses appender to append data to output and encoder to encode or alter the output format. At present, it comes with two appender - FileAppender and ListAppender.

FileAppender

So far, all examples use FileAppender defined in default steps steps-default.yml packaged in scoopi distribution jar. The appender snippet from is as below

appender:
  class: "org.codetab.scoopi.step.load.DataAppender"
  previous: converter
  next: end
  plugins: [
    plugin: {
      name: file,
      class: "org.codetab.scoopi.plugin.appender.FileAppender",
      file: "output/data.txt",
      plugins: [
         plugin: {
           name: csv,
           delimiter: "|",
           class: "org.codetab.scoopi.plugin.encoder.CsvEncoder"
         }
      ]
    }
  ]

We can override the step in task level or create altogether new steps as detailed in Steps and Plugins.

To use appender, first we must define a step that uses class DataAppender and this class is responsible for creation of appender. Then we can define appender in plugins array. For example, say we want two output files with different delimiters then override the step as follows

appender:
  class: "org.codetab.scoopi.step.load.DataAppender"
  previous: converter
  next: end
  plugins: [
    plugin: {
      name: file1,
      class: "org.codetab.scoopi.plugin.appender.FileAppender",
      file: "output/data.csv",
      plugins: [
         plugin: {
           name: csv,
           delimiter: ",",
           class: "org.codetab.scoopi.plugin.encoder.CsvEncoder"
         }
      ]
    },
    plugin: {
      name: file2,
      class: "org.codetab.scoopi.plugin.appender.FileAppender",
      file: "output/data.dump",
      plugins: [
         plugin: {
           name: pipe,
           delimiter: "|",
           class: "org.codetab.scoopi.plugin.encoder.CsvEncoder"
         }
      ]
    }
  ]

Encoder also handles sort. By default, it sorts on col and then by row axis and we can change this by using sortOrder. For example the following sorts by row and then on col

      plugins: [
         plugin: {
           name: csv,
           delimiter: "|",
           sortOrder: "row,col"
           class: "org.codetab.scoopi.plugin.encoder.CsvEncoder"
         }
      ]

In next chapter, we explain how to split job.xml into multiple files when definitions become large and complex.