metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

View project on GitHub
Documentation Home AWS Home

S3 Load to Bronze

This execution template will load data from an S3 location into a DataFrame using the LoadS3Data, call the LoadToParquet pipeline, and use the WriteDataFrameToS3 pipeline to store the data in the bronze zone as parquet.

General Information

Id: s3_load_data_bronze

Name: S3 Load to Bronze

Form

A custom form allows configuring the input parameters as well as controlling the pipeline behavior.