A facility to define distribution profiles to distribute data to multiple destinations was added to Eagle Data Services.
The distribution profiles are JSON files stored in eagle/estar/tpe/dynamic/metadata/distributions/profilename_profile.json
The distribution profile can specify one or several destinations (sinks) to distribute the data. For example to distribute to two different SFTP destinations the following profile can be used:
{
"data_sinks": [
{
"sink": "sftp_sink",
"type": "sftp",
"credentials": {
"user_id": "eagle",
"password": "eagle1"
},
"cfg": {
"host": "inno-mlperfapp01",
"port": 22,
"compression_codec": "gzip"
}
},
{
"sink": "sftp_extracts",
"type": "sftp",
"credentials": {
"user_id": "eagle",
"password": "eagle1"
},
"cfg": {
"host": "inno-mlperfapp01",
"port": 22,
"compression_codec": "gzip",
"location": "extracts"
}
}
]
}
The distribution profile can be specified in the RTR either by name using the DistributionProfileName parameter or as an embedded distribution profile in the DistributionProfileJsonDefinition parameter as a JSON document.
For example if we have a eagle/estar/tpe/dynamic/metadata/distributions/client_destinations_profile.json we can refer it in the RTR by using DistributionProfileName=client_destinations parameter.
Encrypted Credentials
The credentials can be embedded in the distribution profile as shown in the example above, or the credentials can point to an encrypted configuration file.
For example:
"credentials": "clientcredentials/sftpcrendentials"
In this case the credentials are encrypted and retrieved from a file in
dynamic/dataservices/configs/clientcredentials.json.
The credentials reference has two parts - a credential group name and the specific credential name. In the above example the credentials group name is clientcredentials and is translated into the credentials file name. The credential name is sftpcredentials and points to a section within the credentials file name.
Each encrypted credentials file has a region designation, which is the Oracle SID of the environment. If the credentials file is copied to a different environment it will fail to work.
There is a SOAP endpoint provided to save encrypted credentials. An example RTR to save the credentials is provided below:
<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest">
<header>
<messageId>B70DHO72GCLFRHGI</messageId>
<sentBy>http://www.eagleinvsys.com</sentBy>
<sendTo>http://www.eagleinvsys.com/eagle_default_ruleservice</sendTo>
</header>
<taskIdentifier>
<correlationId>MLPERFSFTPCRED_{GUID}</correlationId>
<businessTaskId>BB0573A416C18678</businessTaskId>
</taskIdentifier>
<taskTypeEnum>NEW</taskTypeEnum>
<synchronousExecution>yes</synchronousExecution>
<taskParameters>
<taskParameter>
<name>ActionType</name>
<value>EXECUTE</value>
</taskParameter>
<taskParameter>
<name>REST</name>
<value>POST /eagle/v2/configurations/clientcredentials/sftpcredentials</value>
</taskParameter>
<taskParameter>
<name>rest_body</name>
<value>{
"user_id": "eagle",
"password": "eagle1234",
"passphrase": "test123"
}</value>
</taskParameter>
</taskParameters>
</EagleML>
The above RTR will create a credentials configuration which can be referred with the "credentials": "clientcredentials/sftpcrendentials" element in the distribution profile file.