metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

View project on GitHub
Branch Build Coverage
Develop Develop Build Develop Coverage
Master Release Master Coverage

Metalus Pipeline Library

The Metalus library was created as a way to create Spark applications at runtime without the need to write or compile code. The library is written in Scala and provides binaries for different version of Spark and Scala. Developers build applications by providing a JSON configuration file which gets loaded and executed by the metalus core library.

Documentation

Documentation for this project may be found here.

Contributing

Instructions for contributing to this project and instructions on building may be found here.

Projects

There are several sub-projects:

Metalus Pipeline Core

This project contains the core library and is the minimum requirement for any application.

Metalus Common Step Library

This step library contains steps that are considered generic enough to be used in any project.

Metalus AWS Step Library

This step library contains AWS specific components. The Kinesis driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

Metalus GCP Step Library

This step library contains GCP specific components. The Pub/Sub driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

Metalus Kafka Step Library

This step library contains GCP specific components. The Kafka driver provides a basic implementation that gathers data and then initiates the Metalus Pipeline Core for processing of the incoming data.

Metalus Mongo Step Library

This step library adds support for working with Mongo.

Metalus Pipeline Examples

This project provides several examples to help demonstrate how to use the library.

Metalus Utilities

This project provides utilities that help work with the project.

Metalus Application

This project provides a single jar that can be used to run the application. Additional components provide jars that can be added to the classpath.

Examples

Examples of building pipelines can be found in the metalus-examples project.