A facility to define distribution profiles to distribute data to multiple destinations was added to Eagle Data Services.
The distribution profiles are JSON files stored in eagle/estar/tpe/dynamic/metadata/distributions/profilename_profile.json
The distribution profile can specify one or several destinations (sinks) to distribute the data. For example to distribute to two different SFTP destinations the following profile can be used:
{
"data_sinks": [
{
"sink": "sftp_sink",
"type": "sftp",
"credentials": {
"user_id": "eagle",
"password": "eagle1"
},
"cfg": {
"host": "inno-mlperfapp01",
"port": 22,
"compression_codec": "gzip"
}
},
{
"sink": "sftp_extracts",
"type": "sftp",
"credentials": {
"user_id": "eagle",
"password": "eagle1"
},
"cfg": {
"host": "inno-mlperfapp01",
"port": 22,
"compression_codec": "gzip",
"location": "extracts"
}
}
]
}
The distribution profile can be specified in the RTR either by name using the DistributionProfileName parameter or as an embedded distribution profile in the DistributionProfileJsonDefinition parameter as a JSON document.
For example if we have a eagle/estar/tpe/dynamic/metadata/distributions/client_destinations_profile.json we can refer it in the RTR by using DistributionProfileName=client_destinations parameter.
Encrypted Credentials
The credentials can be embedded in the distribution profile as shown in the example above, or the credentials can point to an encrypted configuration file.
For example:
"credentials": "clientcredentials/sftpcrendentials"
In this case the credentials are encrypted and retrieved from a file in
dynamic/dataservices/configs/clientcredentials.json.
The credentials reference has two parts - a credential group name and the specific credential name. In the above example the credentials group name is clientcredentials and is translated into the credentials file name. The credential name is sftpcredentials and points to a section within the credentials file name.
Each encrypted credentials file has a region designation, which is the Oracle SID of the environment. If the credentials file is copied to a different environment it will fail to work.
There is a SOAP endpoint provided to save encrypted credentials. An example RTR to save the credentials is provided below:
<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest">
<header>
<messageId>B70DHO72GCLFRHGI</messageId>
<sentBy>http://www.eagleinvsys.com</sentBy>
<sendTo>http://www.eagleinvsys.com/eagle_default_ruleservice</sendTo>
</header>
<taskIdentifier>
<correlationId>MLPERFSFTPCRED_{GUID}</correlationId>
<businessTaskId>BB0573A416C18678</businessTaskId>
</taskIdentifier>
<taskTypeEnum>NEW</taskTypeEnum>
<synchronousExecution>yes</synchronousExecution>
<taskParameters>
<taskParameter>
<name>ActionType</name>
<value>EXECUTE</value>
</taskParameter>
<taskParameter>
<name>REST</name>
<value>POST /eagle/v2/configurations/clientcredentials/sftpcredentials</value>
</taskParameter>
<taskParameter>
<name>rest_body</name>
<value>{
"user_id": "eagle",
"password": "eagle1234",
"passphrase": "test123"
}</value>
</taskParameter>
</taskParameters>
</EagleML>
The above RTR will create a credentials configuration which can be referred with the "credentials": "clientcredentials/sftpcrendentials" element in the distribution profile file.
Data Encryption
The data to be distributed can be encrypted. Currently only GPG is supported. The encryption is enabled using the encryption_policy element in the distribution profile as shown in the example below.
{
"data_sinks": [
{
"sink": "sftp_sink",
"type": "sftp",
"credentials": " "clientcredentials/sftpcrendentials",
"cfg": {
"host": "inno-mlperfapp01",
"port": 22,
"compression_codec": "gzip"
}
},
{
"sink": "sftp_extracts",
"type": "sftp",
"encryption_policy": {
"codec" : "gpg",
"credentials" : {
"client_id": "eaglekey"
}
},
"credentials": "clientcredentials/sftpcrendentials",
"cfg": {
"host": "inno-mlperfapp01",
"port": 22,
"compression_codec": "gzip",
"location": "extracts"
}
}
]
}
The encryption_policy has a codec element specifying the encryption type. Currently only gpg is supported.
The credentials/client_id specifies the GPG destination name.
The "credentials" element can be an embedded configuration as shown in the example or a string with a reference to the encrypted configuration as shown in Encrypted Credentials section.
The "encryption_policy" can be a string reference to an encryption policy configuration stored in eagle/estar/tpe/dynamic/metadata/encryptions/polycyname.json files.
For example if we have a file eagle/estar/tpe/dynamic/metadata/encryptions/gpgpolicy.json with the following content:
{
"encryption_policies" : {
"gpg_policy" : {
"codec" : "GPG",
"armor": "gpgarmor",
"credentials" : "eagle_certificates/gpgcert"
}
}
}
we can refer to it from the distribution profile as:
"encryption_policy": "gpgpolicy/gpg_policy"
The GPG encryption policy currently supports the following parameters:
"armor" - the armor for GPG - please refer to GPG documentation for description
"credentials"/"client_id" - the recipient, for which we should have a certificate imported in GPG
Including Sink Definitions
The Distribution Profile can include other sink definitions and redefine their parameters. For example if we have a file named "test_sinks.json" in eagle/estar/tpe/dynamic/metadata/distributions/ with the following content:
{
"sinks": {
"sftp_sink": {
"type":"SFTP",
"credentials" : "test_cred/test_credentials",
"cfg" : {
"host":"sftp.eagleinvsys.com",
"port":22,
"location" : "client1/extract",
"encryption_policy" : "test_encryption_policy/enc_policy1",
"compression_codec": "GZIP"
}
}
}
}
We can include it into a policy file "send_to_sftp_profile.json" as:
{
"include": ["test_sinks.json"],
"data_sinks": [{
"sink" : "sftp_sink"
}
}
Data Compression
The distributed data can be compressed by adding the "compression_codec": "GZIP" parameter to the "cfg" section of the sink definition. Currently only GZIP is supported.
SFTP Distribution Sink Parameters
The SFTP sink is designated with a "type": "SFTP" sink parameter.
The SFTP Distribution can take the following parameters:
"credentials"/"user_id" - the login user id on the SFTP server
"credentials"/"password" - the password to login to the SFTP server
"credentials"/"private_key" - the name to the private key file, if necessary. The private key file should be in estar/tpe/dynamic/dataservices/certificates folder.
"credentials"/"passphrase" - the passphrase for the private key
"credentials"/"private_key_type" - the private key type, for example RSA or DSA, defaults to RSA
"cfg"/"host" - the host name or IP address of the SFTP server
"cfg"/"port" - the port number of the SFTP server, usually should be 22
"cfg"/"make_dir" - true to create missing directories in the distribution path on the SFTP server, false to fail if the directory is missing. The default is true
"cfg"/"location" - a path on the SFTP server to distribute the files to.