Page Comparison

EDS fieldatrribute extract is similar to EagleML extract. This extract contains meta information about Eagle fields like field type, name, database table, description and etc. This implementation links two metadata storages - MDC which is the Eagle metadata storage and Ontology, EDS metadata storage.

As a result, the data set is enriched with additional details. Core, custom and derived fields are supported.

Prerequisite steps for the extract to work:

Metadata Lineage from Eagle MDC to Ontology - field attributes and element definitions

Set Up Temporary Table for fieldExpressions

Please use eagle_ml-2-0_default_cm_exec_eds stream to run EDS fieldattribute extract:

Expand

Code Block

<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0"  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0"  xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest">
	<header>
		<messageId>E30X5FTN4N317JFF</messageId>
		<sentBy>MC2EJM</sentBy>
		<sendTo>http://www.eagleinvsys.com/eagle_default_ruleservice</sendTo>
		<creationTimestamp>2018-04-13T09:52:39.73-05:00</creationTimestamp>
		<expiryTimestamp/>
		<expiryLoadFlag/>
	</header>
	<taskIdentifier>
		<processingSequenceNumber/>
		<correlationId>EDS_FIELDATTRIBUTE_EXTRACT</correlationId>
		<businessTaskId>EDS_FIELDATTRIBUTE_EXTRACT</businessTaskId>
		<sequenceNumber/>
	</taskIdentifier>
	<taskTypeEnum>NEW</taskTypeEnum>
	<synchronousExecution>yes</synchronousExecution>
	<taskParameters>
		<taskParameter>
			<name>StreamName</name>
			<dataType>S</dataType>
			<value>eagle_default_ruleservice</value>
		</taskParameter>
		<taskParameter>
			<name>ServiceName</name>
			<value>EBS</value>
		</taskParameter>
		<taskParameter>
			<name>ServiceMethodName</name>
			<dataType>S</dataType>
			<value>extract</value>
		</taskParameter>
		<taskParameter>
			<name>ResourceName</name>
			<value>fieldattribute</value>
		</taskParameter>
		<taskParameter>
			<name>OutputFormat</name>
			<dataType>S</dataType>
			<value>EagleJSON</value>
		</taskParameter>
		<taskParameter>
			<name>SaveSQLQuery</name>
			<dataType>S</dataType>
			<value>Y</value>
		</taskParameter>
	</taskParameters>
</EagleML>

EQL REST API is supported, for example: https://o171-e001-ww01.eagleinvsys.com/eagle/v2/eql

Code Block
{ "ResourceName": "fieldattribute", "ServiceName": "EBS", "ServiceMethodName": "extract", "savesqlquery": "Y" }

The following filters are supported:

fieldattributeid
fieldtype
fieldname

Code Block

		<taskParameter>
			<name>fieldattributeid</name>
			<dataType>S</dataType>
			<value>264</value>
		</taskParameter>
		<taskParameter>
			<name>fieldtype</name>
			<dataType>S</dataType>
			<value>fieldtype_VALUE</value>
		</taskParameter>
		<taskParameter>
			<name>fieldname</name>
			<dataType>S</dataType>
			<value>fieldname_value</value>
		</taskParameter>

The processing rule for EDS fieldattribute extract was improved to support new element fieldAttributeOntologyDescription for all fields with direct or simple mapping.

Code Block

	<referenceTransaction>
		<header>
			<objectType>FieldAttribute</objectType>
		</header>
		<fieldAttribute>
			<calcType>di</calcType>
			<databaseName>cash</databaseName>
			<fieldAttributeDescription>i Local Currency</fieldAttributeDescription>
			<fieldAttributeId>138</fieldAttributeId>
			<fieldIndicator>cf</fieldIndicator>
			<fieldName>LOCAL_CURRENCY</fieldName>
			<tableName>CASH_ACTIVITY</tableName>
			<fieldIdentifier>I LOCAL CURRENCY</fieldIdentifier>
			<scope>2</scope>
			<readOnly>Y</readOnly>
			<fieldAttributeOntologyDescription>Local currency. Three-letter currency code.</fieldAttributeOntologyDescription>
		</fieldAttribute>
	</referenceTransaction>

Info
This field will be part of the EDS extract by default. In order to create a mashup file the manual step is required. Described here

A new logic was implemented for derived or calculated fields. Now all user defined expressions for the calculated fields will be part of the extract. Two additional elements are part of the extract:

fieldExpression element includes user friendly expression available in MDC
userData element contains RAW definition of the value stored in the database

New fields are included by default, but they can be excluded by setting RTR parameter includederivedattributes to N.

Code Block

<referenceTransaction>
	<header>
		<objectType>FieldAttribute</objectType>
	</header>
	<fieldAttribute>
		<statField>Y</statField>
		<fieldAttributeDescription>DC for subtraction</fieldAttributeDescription>
		<fieldAttributeId>1429</fieldAttributeId>
		<fieldAttributeProcessId>1014</fieldAttributeProcessId>
		<fieldIndicator>DE</fieldIndicator>
		<fieldType>NUM</fieldType>
		<fieldFormat>2</fieldFormat>
		<fieldCategory>&lt;none&gt;</fieldCategory>
		<currencyProcess>NOT</currencyProcess>
		<fieldValueType>NOT</fieldValueType>
		<fieldIdentifier>DETAIL CAL FOR SUBTRACTION - SUPERUSERS</fieldIdentifier>
		<scope>2</scope>
		<readOnly>N</readOnly>
		<processDescription>Expression Calculator</processDescription>
		<processType>USERDEFINED</processType>
		<fieldExpression>H-PD-Market_Value_Income - H-PD-Local_Market_Value</fieldExpression>
		<userData>#`FP1014~#`FD1170~-~#`FD1155~</userData>
	</fieldAttribute>
</referenceTransaction>

Info
The initial configuration steps are required. Described here

This functionality is limited for the calculated fields below with field_attribute_process_id:

1010 Arithmetic Addition Calculator
1011 Arithmetic Subtraction Calculator
1012 Arithmetic Division Calculator
1013 Arithmetic Multiplication Calculator
1014 Expression Calculator
1015 Adjustments Calculator
1025 Trade Adjustment Calculator
1030 IRR Calculator
1031 Modified Dietz Calculator
1032 Dollar Value Added Calculator
1033 Average Invested Balance Calculator

Example extract is attached.

View file

name	api_04E6ESOMVRM92GTCX_FIELDATTRIBUTE_20220427.7z

RDC Enrichments

Five additional fields are supported:

fieldAttributeReference
enrichmentRule
rdcValidation
rdcFieldGroup
rdcDataStrategy

Additional logic was implemented to group values and use TEMP table for join. Grouped values are separated by ; symbol.

fieldAttributeReference - field attributes which are used in calculations

Extract file:

Image Modified

enrichmentRule - the enrichment rule for the field

Extract file:

Image Modified

RDC

Image Modified

validation - list of used validations

Extract file:

Image Modified

Value in the cell:

First Compounding Date Cannot Be Greater than Maturity Date; First Compounding Date Cannot Be Greater than Termination Date; First Compounding Date Cannot Be Less than Dated Date; First Compounding Date Cannot Be Less than Effective Date

RDC:

Image Modified

rdcFieldGroup - list of selected groups

Extract file:

Image Modified

RDC:

Single value:

Image Modified

Multiple values:

Image Modified

rdcDataStrategy - list of applied data strtategies

Extract file:

Image Modified

Custom Mashup

...

In case you want to use custom file (CSV, JSON, etc.) to mashup it with fieldattributes extract.

Create custom dataset definition for the file in the folder: estar\tpe\dynamic\metadata\core\datasets

For example:

View file

name	Invalid file id - 0ff492d3-7b03-4bdd-84d3-6b5f7b3116c9

This file describe custom CSV file:

View file

name	Invalid file id - e31f59b9-1855-42c0-9b25-34da24fc1b9d

NOTE: pay attention on file path and adjust it correspondently.

Place file for mashup to path specified in dataset definition:

For example in dataset definition, path is defined as:

"file_name": " /apps/eagle/estar/tpe/dynamic/metadata/core/ontology/templates/EADataMapping.csv",

To make this works create custom processing rule in cache:

KEY: eagle.ebs.resource:O17_FIELDATTRIBUTE_2.0.57
VALUE:
View file
name Invalid file id - 3022b10c-aae4-47f3-aad0-ec9ebbc03976

Pay attention on the following:

Add definition with key in the register_for_mashups block
Add mashup in mashups block
Add mapping in taxonomy block

Image Modified

KEY: eagle.ebs.resource:O17_FIELDATTRIBUTE_VERSIONS
VALUE: 2.0.57

Info
NOTE: O17 this is tenant prefix. it can be different for your case.

Run EQL extract:

Code Block

{
	"ServiceName": "EBS",
	"ServiceMethodName": "extract",
	"resourcename": "fieldattribute",
	"outputformat": "EagleJSON",
	"disabledatasetcache": "Y",
	"resourceversion": "2.0.57",
	"filters": {
		"fieldattributeid": "5300,4876,4877,2014,4263,4265,371,4266,5864,2110,108,109"
	}
}

Info

`"resourceversion": "2.0.57",`

Version can be different depends on the rule version you create

Output example with custom mashup:

Code Block

{
	"eagleJSONVersion": "2-0",
	"eagleJSONType": "ReferenceTransactionMessage",
	"eagleJSONRevision": "294",
	"referenceTransactionMessage": {
		"header": {},
		"referenceTransaction": [
			{
				"header": {
					"objectType": "FieldAttribute"
				},
				"fieldAttribute": [
					{
						"objectType": "FieldAttribute",
						"statField": "Y",
						"calcType": "di",
						"databaseName": "security",
						"fieldAttributeDescription": "i Cusip",
						"fieldAttributeId": 108,
						"fieldIndicator": "S",
						"fieldName": "PRIMARY_ASSET_ID",
						"tableName": "SECURITY_MASTER",
						"refCodeId": 99,
						"comments": "Cusip",
						"fieldIdentifier": "I CUSIP",
						"scope": 2,
						"readOnly": "N",
						"fieldAttributeOntologyDescription": "Stores the most appropriate (primary) market security identifier of the financial instrument.",
						"rdcFieldGroup": "IIP_AF_5",
						"ea_mapping.LOB": "NYL Investors",
						"ea_mapping.SUBJECT_AREA_DOMAIN": "SECURITY",
						"ea_mapping.SUB_DOMAIN": "SECURITY",
						"ea_mapping.BUSINESS_NAME": "PRIMARY_ASSET_ID",
						"ea_mapping.BUSINESS_DESCRIPTION": "Current Primary Identifier of a security",
						"ea_mapping.PHASE_1_NYLIM": "-NA-",
						"ea_mapping.REQUIRED_FOR_FUTURE_STATE": "Y",
						"ea_mapping.SOURCE_PLATFORM": "CAMRA",
						"ea_mapping.SOURCE_TABLE": "SECURITY",
						"ea_mapping.SOURCE_COLUMN_SOURCE_COLUMN_STAT": "CUSIP",
						"ea_mapping.FORMAT": "CHAR(100)",
						"ea_mapping.COMMENTS_DERIVATIONS_TRANSFORMATIONS_FOR_SOURCE": "This is CUSIP",
						"ea_mapping.EXTRACT_FIELD_EXTRACT_FIELD_STAT": "CUSIP",
						"ea_mapping.FORMAT": "CHAR(100)",
						"ea_mapping.ML_ELEMENT": "EagleML/referenceTransaction/genericSMF/primaryAssetId",
						"ea_mapping.SUBJECT_AREA_FOR_INTERNAL": "cam_egl_smf",
						"ea_mapping.TARGET_FIELD_NAME_FOR_INTERNAL": "primaryAssetId",
						"ea_mapping.MANDATORY_FIELD": "Y"
					}
				]
			}
		]
	}
}

Add label

Versions Compared

Old Version 13

New Version 14

Key

RDC Enrichments

Custom Mashup

`"resourceversion": "2.0.57",`