metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

View project on GitHub

Documentation Home

Logging

The DriverSetup class will attempt to set logging options based on three parameters:

  • logLevel - Controls log level for all classes in the com.acxiom package.
  • rootLogLevel - Controls log level for all other classes.
  • customLogLevels - Provides log levels for individual classes/packages.

The values for logLevel and rootLogLevel will be valid Log4J log levels. Custom Log Levels is a comma separated list of class/package and level pairs. Below is an example showing the proper format:

--customLogLevels com.acxiom.pipeline:WARN,org.apache.spark:ERROR,com.acxiom.pipeline.DriverSetup:TRACE

These options may be passed with the spark-submit command’s application parameters by prepending – before the name.