metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

View project on GitHub

Table of Contents

Introduction
Getting Started
JSON Pipelines
Applications
Step Libraries
- Metalus Core
  - Steps
- Metalus Common
  - Steps
  - Pipelines/Step Groups
    - Copy File * Uses new connectors api
    - Load to Bronze * Uses new connectors api
    - SFTP to HDFS
    - Download To Bronze HDFS
    - DownloadSFTPToHDFSWithDataFrame
    - WriteDataFrameToHDFS
  - Execution Templates
    - Load SFTP to Bronze HDFS
- Metalus AWS
  - Steps
    - S3Steps
    - KinesisSteps
  - Pipelines/Step Groups
    - LoadS3Data
    - WriteDataFrameToS3
  - Execution Templates
    - S3 Load to Bronze
  - Extensions
    - S3FileManager
    - AWS Secrets Manager Credential Provider
- Metalus GCP
  - Steps
  - Extensions
    - GCSFileManager
    - GCP Secrets Manager Credential Provider
- Metalus Kafka
  - Steps
    - KafkaSteps
- Metalus Mongo
  - Steps
    - Mongo Steps
- Metalus Delta Lake
  - Steps
    - Delta Lake Steps
- Creating your own step library (Step Templates)
  - Step Annotations
Advanced
Contributing