Page Comparison

Eagle Data Services has a facility to define distribution profiles to distribute data to multiple destinations.

Distribution Profiles

The distribution profiles are JSON files stored in eagle/estar/tpe/dynamic/metadata/distributions/profilename_profile.json

In the distribution profile, you can specify one or several destinations (sinks) to distribute the data. For example, to distribute to two different SFTP destinations, you can use the following profile:

{
"data_sinks": [
       { 
       "sink": "sftp_sink",
       "type": "sftp",
       "credentials": {
           "user_id": "eagle",
           "password": "eagle1"
       },
       "cfg": {
           "host": "inno-mlperfapp01",
           "port": 22,
           "compression_codec": "gzip"
       }
     },
       { 
       "sink": "sftp_extracts",
       "type": "sftp",
       "credentials": {
           "user_id": "eagle",
           "password": "eagle1"
       },
       "cfg": {
           "host": "inno-mlperfapp01",
           "port": 22,
           "compression_codec": "gzip",
           "location": "extracts"
       }
     }
   ]
}

You can specify the distribution profile in the RTR either by name using the DistributionProfileName parameter or as an embedded distribution profile in the DistributionProfileJsonDefinition parameter as a JSON document.

For example, if eagle/estar/tpe/dynamic/metadata/distributions/client_destinations_profile.json is defined, you can reference it in the RTR by using the DistributionProfileName=client_destinations parameter.

Encrypted Credentials

You can embed the credentials in the distribution profile as shown in the example above or the credentials may point to an encrypted configuration file.

For example:

"credentials": "clientcredentials/sftpcrendentials"

In this case, the credentials are encrypted and retrieved from a file from

dynamic/dataservices/configs/clientcredentials.json.

The credentials reference has two parts: a credential group name and the specific credential name. In the above example, the credentials group name is clientcredentials and is translated into the credentials file name. The credential name is sftpcredentials and points to a section within the credentials file name.

Each encrypted credentials file has a region designation, which is the Oracle SID of the environment. If the credentials file is copied to a different environment, it fails to work.

There is a SOAP endpoint provided to save encrypted credentials. An example RTR to save the credentials is provided below:

<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest">
<header>
    <messageId>B70DHO72GCLFRHGI</messageId>
    <sentBy>http://www.eagleinvsys.com</sentBy>
    <sendTo>http://www.eagleinvsys.com/eagle_default_ruleservice</sendTo>
</header>
<taskIdentifier>
    <correlationId>MLPERFSFTPCRED_{GUID}</correlationId>
    <businessTaskId>BB0573A416C18678</businessTaskId>
</taskIdentifier>
<taskTypeEnum>NEW</taskTypeEnum>
<synchronousExecution>yes</synchronousExecution>
<taskParameters>
   <taskParameter>
       <name>ActionType</name>
       <value>EXECUTE</value>
   </taskParameter>
   <taskParameter>
       <name>REST</name>
       <value>POST /eagle/v2/configurations/clientcredentials/sftpcredentials</value>
   </taskParameter>
   <taskParameter>
       <name>rest_body</name>
       <value>{
           "user_id": "eagle",
           "password": "eagle1234",
           "passphrase": "test123"
       }</value>
   </taskParameter>
</taskParameters>
</EagleML>

The above RTR creates a credentials configuration which may be referred with the "credentials":  "clientcredentials/sftpcrendentials" element in the distribution profile file.

Data Encryption

The data to be distributed may be encrypted. Only GPG is currently supported. The encryption is enabled using the encryption_policy element in the distribution profile as shown in the example below.

{
"data_sinks": [
       { 
       "sink": "sftp_sink",
       "type": "sftp",
       "credentials": " "clientcredentials/sftpcrendentials",
       "cfg": {
           "host": "inno-mlperfapp01",
           "port": 22,
           "compression_codec": "gzip"
       }
     },
       { 
       "sink": "sftp_extracts",
       "type": "sftp",
       "encryption_policy": { 
		"codec" : "gpg",
		 "credentials" : {
                        "client_id": "eaglekey"
		 }
	},
       "credentials":  "clientcredentials/sftpcrendentials",
       "cfg": {
           "host": "inno-mlperfapp01",
           "port": 22,
           "compression_codec": "gzip",
           "location": "extracts"
       }
     }
   ]
}

• The encryption_policy has a codec element specifying the encryption type. Currently only gpg is supported.
• The credentials/client_id specifies the GPG destination name.
• The "credentials" element may be an embedded configuration as shown in the example or a string with a reference to the encrypted configuration as shown in Encrypted Credentials section.
• The "encryption_policy" may be a string reference to an encryption policy configuration stored in the eagle/estar/tpe/dynamic/metadata/encryptions/polycyname.json files.

For example, a file eagle/estar/tpe/dynamic/metadata/encryptions/gpgpolicy.json can exist with the following content:

{
  "encryption_policies" :  {
	"gpg_policy" : {
		"codec" : "GPG",
		"armor": "gpgarmor",
		"credentials" : "eagle_certificates/gpgcert"
	}
  }
}

In the distribution profile, you can reference it as:

"encryption_policy": "gpgpolicy/gpg_policy"

The GPG encryption policy currently supports the following parameters:

• "armor" - the armor for GPG - please refer to GPG documentation for description.
• "credentials"/"client_id" - the recipient, for which should have a certificate imported in GPG.

Include Sink Definitions

In the Distribution Profile, you can include other sink definitions and redefine their parameters. For example, if the file is named "test_sinks.json" in the eagle/estar/tpe/dynamic/metadata/distributions/ path with the following content:

{
"sinks": {
    "sftp_sink": {
	"type":"SFTP",
	"credentials" : "test_cred/test_credentials",
	"cfg" : {
		"host":"sftp.eagleinvsys.com",
		"port":22,
		"location" : "client1/extract",
		"encryption_policy" :  "test_encryption_policy/enc_policy1",
		"compression_codec": "GZIP"
	}
    }
 }
}

You also can included it in a policy file "send_to_sftp_profile.json" as shown below:

{
	"include": ["test_sinks.json"],
	
	"data_sinks": [{
				"sink" : "sftp_sink"
	}
}

Data Compression

You can compress the distributed data by adding the "compression_codec": "GZIP" parameter to the "cfg" section of the sink definition. Only GZIP is supported presently.

SFTP Distribution Sink Parameters

The SFTP sink is designated with a "type": "SFTP" sink parameter.

The SFTP Distribution may take the following parameters:

• "credentials"/"user_id" - the login user id on the SFTP server
• "credentials"/"password" - the password to login to the SFTP server
• "credentials"/"private_key" - the name to the private key file, if necessary. The private key file should be in the estar/tpe/dynamic/dataservices/certificates folder.
• "credentials"/"passphrase" - the passphrase for the private key
• "credentials"/"private_key_type"  - the private key type, for example RSA or DSA, defaults to RSA
• "cfg"/"host" - the host name or IP address of the SFTP server
• "cfg"/"port" - the port number of the SFTP server, usually set to 22
• "cfg"/"make_dir" - values of true or false.  Set to true to create missing directories in the distribution path on the SFTP server and false to fail if the directory is missing. The default is true.
• "cfg"/"location" - a path on the SFTP server to distribute the files to.

AZURE Publishing Configuration (Starting December 2019 release)

The Azure publishing configuration profile sample is below:

{
    "data_sinks":
    [{
            "sink": "azure_dev",
            "type": "azureblob",
            "cfg": {
                "credentials": {
                    "account": "devdataaccount",
                    "application_id": "11111111-1111-470c-9d01-9999999999",
                    "application_secret": "dalkjflj*SDjhDSljadaok",
                    "tenant_id": "1111111111-1111111-1111-1111-111111111"
                },
                "container_name": "devcontainer",
                "target_dir": "dropdir/dest",
                "compression_codec": "gzip"
            }
        }
    ]
}

The "compression_codec" parameter has to be set to gzip to compress data or none to send uncompressed data.

EMAIL Distribution Sink Parameters

The EMAIL sink is designated with a "type": "EMAIL" sink parameter. (Available as of January 2021 release)

The EMAIL Distribution may take the following parameters:

• "credentials"/"secure_connection" - values of true or false.  Set to true to tells an email server that an email client wants to turn an existing insecure connection into a secure one. The default is false. Should be used only when SMTP server support the secure connection
• "credentials"/"ca_certs" -  the name to the CA certificate, if necessary to setup secure connection and load CA certificates. The CA certificate file should be in the estar/tpe/dynamic/dataservices/certificates folder and in PEM format. Load a set of “certification authority” (CA) certificates used to validate other peers’ certificates.  The CA certificate needs only to verify the validity of SSL certificates that have been signed by this certificate
• "credentials"/"certfile" - the name to the certificate, if necessary to setup secure connection and load certificate. The certificate file should be in the estar/tpe/dynamic/dataservices/certificates folder and in PEM format. Load a private key and the corresponding certificate. Containing the certificate as well as any number of CA certificates needed to establish the certificate’s authenticity. If keyfile is not provided the private key will be taken from certfile as well
• "credentials"/"keyfile" - the name to the key file, if necessary for certfile. The key file should be in the estar/tpe/dynamic/dataservices/certificates folder and in PEM format. If present, must point to a file containing the private key in certfile
• "credentials"/"password"  - string value in profile. It will only be called if the private key is encrypted and a password is necessary. It will be ignored if the keyfile is not encrypted and no password is needed
• "cfg"/"host" - the host name or IP address of the SMTP server. By default got from database for each environment
• "cfg"/"port" - the port number of the SMTP server, usually set to 25.By default got from database for each environment
• "cfg"/"contentType" - entity header is used to indicate the media type of the attached resource in email. Optional parameter
• "cfg"/"mailFrom" -  address that identifies from where the email came. By default EDS@$hostname.com
• "cfg"/"subject"  - the first text which recipients see when  an email reaches their inbox. Next placeholders are available: {env} - return the environment name; {profile_name} – return the name of the used distribution profile
• "cfg"/"mailTo" - required parameter. Include the list of recipients. Used to send an email to a specific addresses

EMAIL publishing configuration profile sample is below: