Logs

Scoopi uses Log4j2 library for logging. By default, it logs info message to console and log errors to logs/error.log file.

Debug and Trace logs

The log configuration file log4j2.xml is located in conf folder. By default root log level is info which outputs error, warn and info logs.

To output debug logs, change root level to debug and un-comment appender-ref ref=“debuglog” element. and similarly, for trace logs. The trace logs output query and parsed html nodes which is helpful to construct query strings.

Task logs

The trace log is of little use when multiple tasks concurrently executes and it is very difficult to analyze. To overcome this limitation, Scoopi provides task based logs which can output debug or trace logs of selected task or tasks end to end.

Suppose, we want to output trace of task acme-price and nothing else. To do this, edit conf/logback.xml and set root level as

conf/logback.xml

<root level="trace">
    <appender-ref ref="console" />
    <appender-ref ref="errorlog" />

    <appender-ref ref="tasklog" />

    <!-- <appender-ref ref="debuglog" /> -->
    <!-- <appender-ref ref="tracelog" /> -->
</root>

next, alter filter element of tasklog appender as follows

conf/logback.xml

<appender name="tasklog"

    ...

    <filter
        class="ch.qos.logback.core.filter.EvaluatorFilter">
        <evaluator
            class="ch.qos.logback.classic.boolex.OnMarkerEvaluator">
            <marker>task-acme-snapshotGroup-price</marker>
        </evaluator>
        <onMismatch>DENY</onMismatch>
        <onMatch>NEUTRAL</onMatch>
    </filter>
</appender>

Now run Scoopi Example 10 and you should have trace logs of price task in logs/task.log.

Now, let’s take example where linkTask scrapes bs link from acme-snapshot.html which in turn executes bsTask to scrape acme-bs.html. Suppose, we want to log both linkTask and bsTask and also, want to log related dataDef details. To accomplish that, change the filter element as below

conf/logback.xml

<appender name="tasklog"

    ...

    <filter
        class="ch.qos.logback.core.filter.EvaluatorFilter">
        <evaluator
            class="ch.qos.logback.classic.boolex.OnMarkerEvaluator">
            <marker>task-acme-snapshotGroup-linkTask</marker>
            <marker>task-acme-bsGroup-bsTask</marker>
            <marker>datadef-links</marker>
            <marker>datadef-bs</marker>
        </evaluator>
        <onMismatch>DENY</onMismatch>
        <onMatch>NEUTRAL</onMatch>
    </filter>
</appender>

The marker syntax is for task is task-<locatorName>-<taskGroupName>-<taskName> and for dataDef it is dataDef-<datadefName>.

The next chapter describes Scoopi Dashboard which helps to fine tune thread pools.