metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

View project on GitHub

Documentation Home

Advanced Step Features

Several advanced step features are made available by Metalus.

Step Parameter Validation

Additional step parameter validation may be enabled on the command line. More information may be found here.

Pipeline Context Update

The PipelineContext is immutable within a step, however there are a few features in place that allow steps to update globals and the metrics on the step audit. In order to use these features, a step function must return a PipelineStepResponse.

Globals

Using the secondary named map, a step may add a value that begins with $globals. to update or add an entry to the globals object once the step has completed. Any number of these parameters are allowed.

Using the secondary named map, a step may add a value that begins with $globalLink. to update or add an entry to the global links object once the step has completed. Any number of these parameters are allowed.

Metrics

Using the secondary named map, a step may add a value that begins with $metrics. to update or add an entry to the step audit once the step has completed. Any number of these parameters are allowed.