metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

View project on GitHub

Documentation Home

Step Group

A step-group provides a mechanism for embedding pipelines within another pipeline. This feature is in place to leverage building smaller reusable pipelines that can be shared across other pipelines. The pipeline will be provided with a pipeline context with no pipeline parameters and globals that are only populated with the values from the pipelineMappings parameter.

Step Group Result

The stepGroupResult parameter of a pipeline defines a mapping that provides a single primary response that can be referenced easily from other steps. Normal mappings apply as well as scalascript mappings. This will override the default return by placing the mapped value into the primary response and the default return into the namedReturns response.

Parameters

There are five parameters for a step-group:

pipelineId

This parameter must be a pipeline id that is accessible to the PipelineManager within the PipelineContext. This parameter can be made optional by providing the pipeline parameter.

pipeline

This required parameter can contain either a string with expansion variable or a map with the proper pipeline layout. The “className” attribute must be set when using a map. This parameter can be made optional by providing the pipelineId parameter.

pipelineMappings

This optional parameter provides a mechanism for mapping values from the outer pipeline to the globals object accessible to the embedded pipeline.

useParentGlobals

This boolean parameter indicates whether the step group should have access to the parent globals. The default value of false will seed the globals from the pipelineMappings only. When true, the parent globals will be used as a base with the pipelineMappings applied as an addition and override.

output

This optional result parameter works the same as the Step Group Result.

Default Return

A PipelineStepResponse containing a map as the primary return type and nothing for the secondary return type. The map keys will be the pipeline step ids of each step that was executed for the step-group pipeline.

Global Updates

Steps within a step-group may update the root globals.

Execution Audits

Like other pipelines, audits will be captured during execution. These audits will be attached to the pipeline step.