Filters

The extracted data of the previous example contains many unwanted items and Scoopi provides filters to remove them from output.

The filter is normally applied after parse is over and members are created. The Example 5 uses filter for row axis. The filter snippet is as below

defs/examples/fin/jsoup/ex-5/job.yml

row:
  query:
    region: "table:contains(Sources Of Funds)"
    field: "tr:nth-child(%{row.index}) > td:nth-child(1)"      
  members: [ 
    member: {name: item, index: 5, breakAfter: ["Book Value (Rs)"] },
  ]
  filters: [ 
    filter: { type: value, pattern: "" },
    filter: { type: value, pattern: "Sources Of Funds" },
    filter: { type: value, pattern: "Application Of Funds" },                   
  ]

The filter definition remove the members whose row axis value

• blank
• null
• Sources Of Funds
• Application Of Funds

As the filter specifies type: value, pattern is applied to member value field. The filter property type can be value or match. When type: match pattern is compared with members’ match field instead of value field. Pattern property can be plain text simple comparison or regex for complex pattern matching.

When filter is true for an axis, then the enclosing member is removed from the output. For example, when filter for col axis is

col:

  ...

  filters: [ 
    filter: { type: value, pattern: "Dec 16" },
  ]

and member and its axis are as below

Member
 Col : Dec 16
 Row : Equity
 Fact : 20.00

Then as col axis matches with the pattern and the whole member is removed from data even though filter is not specified for row and fact axis.

The next chapter explains how define multiple tasks for a locator.