Common Step Library
Metalus provides a common step library with a set steps and utilities required to build data processing applications using Spark.
Step Classes
- ApiSteps
- CSVSteps
- DataConnectorSteps
- DataFrameSteps
- DataSteps
- FileManagerSteps
- FlowUtilsSteps
- HDFSSteps
- HiveSteps
- JDBCSteps
- JSONSteps
- QuerySteps
- SFTPSteps
- SparkConfigurationSteps
- StringSteps
- TransformationSteps
Pipelines/Step Groups
- Copy File * Uses new connectors api
- Load To Bronze * Uses new connectors api
- Streaming Monitor
Deprecated
- SFTP to HDFS
- Download to Bronze HDFS
- LoadToParquet
- DownloadSFTPToHDFSWithDataFrame
- WriteDataFrameToHDFS