metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

View project on GitHub

Documentation Home

Credential Provider

The CredentialProvider provides a single point of access to the Metalus application for obtaining credentials. The provider requires a map of parameters that may be used to initialize Credentials. A single function named getNamedCredential is provided to perform look-ups.

DefaultCredentialProvider

A default implementation is provided that will scan the parameters map for credential-classes. This is a comma separated list of fully qualified class names of Credential_s that should be loaded. Each _Credential will be supplied the same parameters Map used by the provider.

Credential

A Credential is used to provide custom credentials. Each implementation must have a name that can be used by the CredentialProvider to return the custom Credential.

DefaultCredential

This implementation will look in the provided parameters for credentialName to set the name of the credential and credentialValue for the value.

Metalus AWS

A trait named AWSCredential is provided with the basic requirements for a credential: api key and secret. Implementations should extend this trait and override the awsAccessKey and awsAccessSecret functions.

Implementations provided:

  • com.acxiom.aws.utils.DefaultAWSCredential - Reads the values for the name from credentialName and splits the credentialValue on the : character to populate the key and secret.
  • com.acxiom.aws.utils.AWSBasicCredential - Reads the accessKeyId and secretAccessKey parameters to populate the functions. The name used to access this credential is AWSCredential.
  • com.acxiom.aws.utils.AWSCloudWatchCredential - Reads the cloudWatchAccessKeyId and cloudWatchSecretAccessKey parameters to populate the functions. The name used to access this credential is AWSCloudWatchCredential.
  • com.acxiom.aws.utils.AWSDynamoDBCredential - Reads the dynamoDBAccessKeyId and dynamoDBSecretAccessKey parameters to populate the functions. The name used to access this credential is AWSDynamoDBCredential.
  • com.acxiom.aws.pipeline.AWSecretsManagerCredentialProvider -An extension of DefaultCredentialProvider which will use the AWS secrets manager to attempt to find Credentials.

Metalus GCP

A trait named GCPCredential is provided with the basic requirements for a credential: A map of Strings. Implementations should extend this trait and override the authKey function. The PubSubPipelineDriver will attempt to access this credential using the name GCPCredential.

Implementations provided:

  • com.acxiom.gcp.pipeline.DefaultGCPCredential - GCPCredential implementation that looks for the gcpAuthKeyArray parameter to generate the authKey.
  • com.acxiom.gcp.pipeline.Base64GCPCredential - Reads the credentials from a parameter named gcpAuthKey and expects it to be Base64 encoded. It is only recommended for development testing and not production use.
  • com.acxiom.gcp.pipeline.GCPSecretsManagerCredentialProvider -An extension of DefaultCredentialProvider which will use the GCP secrets manager to attempt to find Credentials.