metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

View project on GitHub
Documentation Home Common Home

Streaming Monitor

This pipeline is designed to be chained with other pipelines that may produce a streaming DataFrame. The STREAMING_DATAFRAME global is used to retrieve the DataFrame that will be monitored. This pipeline is useful as-is when the streaming-job command line parameter is set to true.

General Information

Id: streaming-monitor

Name: Streaming Monitor

Required Parameters (all parameters should be part of the globals)

  • STREAMING_DATAFRAME - The DataFrame to be monitored.
  • streamingMonitorClassName - The optional fully qualified class name of the streaming monitor class to use for the query.