Page Comparison

A Eagle Data Services has a facility to define distribution profiles to distribute data to multiple destinations was added to Eagle Data Services.

Distribution Profiles

The distribution profiles are JSON files stored in eagle/estar/tpe/dynamic/metadata/distributions/profilename_profile.json

The In the distribution profile, you can specify one or several destinations (sinks) to distribute the data. For example, to distribute to two different SFTP destinations, you can use the following profile may be used:

{
"data_sinks": [
       { 
       "sink": "sftp_sink",
       "type": "sftp",
       "credentials": {
           "user_id": "eagle",
           "password": "eagle1"
       },
       "cfg": {
           "host": "inno-mlperfapp01",
           "port": 22,
           "compression_codec": "gzip"
       }
     },
       { 
       "sink": "sftp_extracts",
       "type": "sftp",
       "credentials": {
           "user_id": "eagle",
           "password": "eagle1"
       },
       "cfg": {
           "host": "inno-mlperfapp01",
           "port": 22,
           "compression_codec": "gzip",
           "location": "extracts"
       }
     }
   ]
}

The distribution profile may be specified You can specify the distribution profile in the RTR either by name using the DistributionProfileName parameter or as an embedded distribution profile in the DistributionProfileJsonDefinition parameter as a JSON document.

For example, if eagle/estar/tpe/dynamic/metadata/distributions/client_destinations_profile.json is defined, you can reference it may be referenced in the RTR by using the DistributionProfileName=client_destinations parameter.

Encrypted Credentials

The credentials may be embedded You can embed the credentials in the distribution profile as shown in the example above or the credentials may point to an encrypted configuration file.

For example:

"credentials": "clientcredentials/sftpcrendentials"

In this case, the credentials are encrypted and retrieved from a file from

...

<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0" xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest">
<header>
    <messageId>B70DHO72GCLFRHGI</messageId>
    <sentBy>http://www.eagleinvsys.com</sentBy>
    <sendTo>http://www.eagleinvsys.com/eagle_default_ruleservice</sendTo>
</header>
<taskIdentifier>
    <correlationId>MLPERFSFTPCRED_{GUID}</correlationId>
    <businessTaskId>BB0573A416C18678</businessTaskId>
</taskIdentifier>
<taskTypeEnum>NEW</taskTypeEnum>
<synchronousExecution>yes</synchronousExecution>
<taskParameters>
   <taskParameter>
       <name>ActionType</name>
       <value>EXECUTE</value>
   </taskParameter>
   <taskParameter>
       <name>REST</name>
       <value>POST /eagle/v2/configurations/clientcredentials/sftpcredentials</value>
   </taskParameter>
   <taskParameter>
       <name>rest_body</name>
       <value>{
           "user_id": "eagle",
           "password": "eagle1234",
           "passphrase": "test123"
       }</value>
   </taskParameter>
</taskParameters>
</EagleML>

...

The data to be distributed may be encrypted. Only GPG is currently supported currently. The encryption is enabled using the encryption_policy element in the distribution profile as shown in the example below.

{
"data_sinks": [
       { 
       "sink": "sftp_sink",
       "type": "sftp",
       "credentials": " "clientcredentials/sftpcrendentials",
       "cfg": {
           "host": "inno-mlperfapp01",
           "port": 22,
           "compression_codec": "gzip"
       }
     },
       { 
       "sink": "sftp_extracts",
       "type": "sftp",
       "encryption_policy": { 
		"codec" : "gpg",
		 "credentials" : {
                        "client_id": "eaglekey"
		 }
	},
       "credentials":  "clientcredentials/sftpcrendentials",
       "cfg": {
           "host": "inno-mlperfapp01",
           "port": 22,
           "compression_codec": "gzip",
           "location": "extracts"
       }
     }
   ]
}

• The encryption_policy has a codec element specifying the encryption type. Currently only gpg is supported.
• The credentials/client_id specifies the GPG destination name.
• The "credentials" element may be an embedded configuration as shown in the example or a string with a reference to the encrypted configuration as shown in Encrypted Credentials section.
• The "encryption_policy" may be a string reference to an encryption policy configuration stored in the eagle/estar/tpe/dynamic/metadata/encryptions/polycyname.json files.

For example, a file eagle/estar/tpe/dynamic/metadata/encryptions/gpgpolicy.json may can exist with the following content:

{
  "encryption_policies" :  {
	"gpg_policy" : {
		"codec" : "GPG",
		"armor": "gpgarmor",
		"credentials" : "eagle_certificates/gpgcert"
	}
  }
}

 It may be referenced from In the distribution profile, you can reference it as:

"encryption_policy": "gpgpolicy/gpg_policy"

The GPG encryption policy currently supports the following parameters:

• "armor" - the armor for GPG - please refer to GPG documentation for description
• "credentials"/"client_id" - the recipient, for which we should have a certificate imported in GPG

...

Include Sink Definitions

The Distribution Profile may In the Distribution Profile, you can include other sink definitions and redefine their parameters. For example, if the file is named "test_sinks.json" in the eagle/estar/tpe/dynamic/metadata/distributions/ path with the following content:

{
"sinks": {
    "sftp_sink": {
	"type":"SFTP",
	"credentials" : "test_cred/test_credentials",
	"cfg" : {
		"host":"sftp.eagleinvsys.com",
		"port":22,
		"location" : "client1/extract",
		"encryption_policy" :  "test_encryption_policy/enc_policy1",
		"compression_codec": "GZIP"
	}
    }
 }
}

It may be included into You also can included it in a policy file "send_to_sftp_profile.json" as shown below:

{
	"include": ["test_sinks.json"],
	
	"data_sinks": [{
				"sink" : "sftp_sink"
	}
}

Data Compression

The distributed data may be compressed You can compress the distributed data by adding the "compression_codec": "GZIP" parameter to the "cfg" section of the sink definition. Only GZIP is supported presently.

...

The SFTP Distribution may take the following parameters:

• "credentials"/"user_id" - the login user id on the SFTP server
• "credentials"/"password" - the password to login to the SFTP server
• "credentials"/"private_key" - the name to the private key file, if necessary. The private key file should be in the estar/tpe/dynamic/dataservices/certificates folder.
• "credentials"/"passphrase" - the passphrase for the private key
• "credentials"/"private_key_type"  - the private key type, for example RSA or DSA, defaults to RSA
• "cfg"/"host" - the host name or IP address of the SFTP server
• "cfg"/"port" - the port number of the SFTP server, usually set to 22
• "cfg"/"make_dir" - values of true or false.  Set to true to create missing directories in the distribution path on the SFTP server and false to fail if the directory is missing. The default is true.
• "cfg"/"location" - a path on the SFTP server to distribute the files to.