metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

View project on GitHub
Documentation Home Common Home

Copy File

This pipeline will copy a file from one location to another. The download will make up to 5 attempts when a failure occurs. A verification check will ensure the downloaded file is the same size and if not, will re-download up to 5 times.

General Information

Id: 43bc9450-2689-11ec-9c0c-cbf3549779e5

Name: CopyFile

Required Parameters

  • sourceConnector - The file connector to use as the source. This is expected to be in the globals.
  • sourceCopyPath - The path to get the source file. This is expected to be in the globals.
  • destinationConnector - The file connector to use as the destination. This is expected to be in the globals.
  • destinationCopyPath - The path to write the destination file. This is expected to be in the globals.