Logs

Scoopi uses Logback library for logging. By default, it logs info message to console and log errors to logs/error.log file.

Debug and Trace logs

The logback configuration file logback.xml is located in conf folder. By default root log level is info which outputs only error, warn and info logs. To output debug logs, change root level to debug and un-comment appender-ref ref=“debuglog” element.

Similarly for trace logs. Trace logs output query and parsed html nodes which is helpful to construct query strings. But trace log is very difficult to analyze as many concurrent task threads output the log to logs/trace.log file.

Task logs

Trace output is of little use when multiple tasks concurrently executes. To overcome this limitation, Scoopi provides task logs which can output debug or trace logs of selected task or tasks end to end.

Suppose, we want to output trace of task acme-price and nothing else. To do this, edit conf/logback.xml and set root level as

conf/logback.xml

<root level="trace">
    <appender-ref ref="console" />
    <appender-ref ref="errorlog" />

    <appender-ref ref="tasklog" />

    <!-- <appender-ref ref="debuglog" /> -->
    <!-- <appender-ref ref="tracelog" /> -->
</root>

next alter filter element of tasklog appender as follows

conf/logback.xml

<appender name="tasklog"

    ...

    <filter
        class="ch.qos.logback.core.filter.EvaluatorFilter">
        <evaluator
            class="ch.qos.logback.classic.boolex.OnMarkerEvaluator">
            <marker>task-acme-quoteGroup-price</marker>
        </evaluator>
        <onMismatch>DENY</onMismatch>
        <onMatch>NEUTRAL</onMatch>
    </filter>
</appender>

Now run Scoopi Example 10 and you should have logs/task.log which trace logs of price task for acme.

Another example is that linkTask scrapes bs link from acme-quote.html which in turn executes bsTask to scrape acme-bs.html. Suppose, we want to log both linkTask and bsTask and for analyzes, we also output related dataDef details. To accomplish that, change the filter element as below

conf/logback.xml

<appender name="tasklog"

    ...

    <filter
        class="ch.qos.logback.core.filter.EvaluatorFilter">
        <evaluator
            class="ch.qos.logback.classic.boolex.OnMarkerEvaluator">
            <marker>task-acme-quoteGroup-linkTask</marker>
            <marker>task-acme-bsGroup-bsTask</marker>
            <marker>datadef-links</marker>
            <marker>datadef-bs</marker>
        </evaluator>
        <onMismatch>DENY</onMismatch>
        <onMatch>NEUTRAL</onMatch>
    </filter>
</appender>

The marker syntax is for task is task-locatorName-taskGroupName-taskName and for dataDef it is dataDef-datadefName.