metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

View project on GitHub

Documentation Home

Connectors

Connectors provide an abstraction for working with data. Connectors are designed to be modeled as JSON to easily include in applications, pipelines and steps. Each implementation implement functions that will be used by steps to work with data in a generic manner.

Connectors

File Connectors

File Connectors are designed to operate with a single file. These connectors provide access to FileManager implementation for the specified file system. More information can be found here.

Data Connectors

Data Connectors provide an abstraction for loading and writing data. The provided functions work specifically on DataFrames. More information can be found here.