MetaData Lineage: EDS Fieldattribute Extract

As a result, the data set is enriched with additional details. Core, custom and derived fields are supported. Prerequisite steps for the extract to work:

Metadata Lineage from Eagle MDC to Ontology - field attributes and element definitions

Set Up Temporary Table for fieldExpressions

Please use eagle_ml-2-0_default_cm_exec_eds stream to run EDS fieldattribute extract:

Click here to expand...

<EagleML xmlns="http://www.eagleinvsys.com/2011/EagleML-2-0"  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequest" eaglemlVersion="2-0"  xsi:schemaLocation="http://www.eagleinvsys.com/2011/EagleML-2-0 eagleml-main-2-0.xsd" eaglemlType="RunTaskRequest">
	<header>
		<messageId>E30X5FTN4N317JFF</messageId>
		<sentBy>MC2EJM</sentBy>
		<sendTo>http://www.eagleinvsys.com/eagle_default_ruleservice</sendTo>
		<creationTimestamp>2018-04-13T09:52:39.73-05:00</creationTimestamp>
		<expiryTimestamp/>
		<expiryLoadFlag/>
	</header>
	<taskIdentifier>
		<processingSequenceNumber/>
		<correlationId>EDS_FIELDATTRIBUTE_EXTRACT</correlationId>
		<businessTaskId>EDS_FIELDATTRIBUTE_EXTRACT</businessTaskId>
		<sequenceNumber/>
	</taskIdentifier>
	<taskTypeEnum>NEW</taskTypeEnum>
	<synchronousExecution>yes</synchronousExecution>
	<taskParameters>
		<taskParameter>
			<name>StreamName</name>
			<dataType>S</dataType>
			<value>eagle_default_ruleservice</value>
		</taskParameter>
		<taskParameter>
			<name>ServiceName</name>
			<value>EBS</value>
		</taskParameter>
		<taskParameter>
			<name>ServiceMethodName</name>
			<dataType>S</dataType>
			<value>extract</value>
		</taskParameter>
		<taskParameter>
			<name>ResourceName</name>
			<value>fieldattribute</value>
		</taskParameter>
		<taskParameter>
			<name>OutputFormat</name>
			<dataType>S</dataType>
			<value>EagleJSON</value>
		</taskParameter>
		<taskParameter>
			<name>SaveSQLQuery</name>
			<dataType>S</dataType>
			<value>Y</value>
		</taskParameter>
	</taskParameters>
</EagleML>

EQL REST API is supported, for example: https://o171-e001-ww01.eagleinvsys.com/eagle/v2/eql

{
    "ResourceName": "fieldattribute",
    "ServiceName": "EBS",
    "ServiceMethodName": "extract",
    "savesqlquery": "Y"
}

The following filters are supported:

fieldattributeid
fieldtype
fieldname

		<taskParameter>
			<name>fieldattributeid</name>
			<dataType>S</dataType>
			<value>264</value>
		</taskParameter>
		<taskParameter>
			<name>fieldtype</name>
			<dataType>S</dataType>
			<value>fieldtype_VALUE</value>
		</taskParameter>
		<taskParameter>
			<name>fieldname</name>
			<dataType>S</dataType>
			<value>fieldname_value</value>
		</taskParameter>

The processing rule for EDS fieldattribute extract was improved to support new element fieldAttributeOntologyDescription for all fields with direct or simple mapping.

	<referenceTransaction>
		<header>
			<objectType>FieldAttribute</objectType>
		</header>
		<fieldAttribute>
			<calcType>di</calcType>
			<databaseName>cash</databaseName>
			<fieldAttributeDescription>i Local Currency</fieldAttributeDescription>
			<fieldAttributeId>138</fieldAttributeId>
			<fieldIndicator>cf</fieldIndicator>
			<fieldName>LOCAL_CURRENCY</fieldName>
			<tableName>CASH_ACTIVITY</tableName>
			<fieldIdentifier>I LOCAL CURRENCY</fieldIdentifier>
			<scope>2</scope>
			<readOnly>Y</readOnly>
			<fieldAttributeOntologyDescription>Local currency. Three-letter currency code.</fieldAttributeOntologyDescription>
		</fieldAttribute>
	</referenceTransaction>

This field will be part of the EDS extract by default. In order to create a mashup file the manual step is required. Described here

A new logic was implemented for derived or calculated fields. Now all user defined expressions for the calculated fields will be part of the extract. Two additional elements are part of the extract:

fieldExpression element includes user friendly expression available in MDC
userData element contains RAW definition of the value stored in the database

New fields are included by default, but they can be excluded by setting RTR parameter includederivedattributes to N.

<referenceTransaction>
	<header>
		<objectType>FieldAttribute</objectType>
	</header>
	<fieldAttribute>
		<statField>Y</statField>
		<fieldAttributeDescription>DC for subtraction</fieldAttributeDescription>
		<fieldAttributeId>1429</fieldAttributeId>
		<fieldAttributeProcessId>1014</fieldAttributeProcessId>
		<fieldIndicator>DE</fieldIndicator>
		<fieldType>NUM</fieldType>
		<fieldFormat>2</fieldFormat>
		<fieldCategory>&lt;none&gt;</fieldCategory>
		<currencyProcess>NOT</currencyProcess>
		<fieldValueType>NOT</fieldValueType>
		<fieldIdentifier>DETAIL CAL FOR SUBTRACTION - SUPERUSERS</fieldIdentifier>
		<scope>2</scope>
		<readOnly>N</readOnly>
		<processDescription>Expression Calculator</processDescription>
		<processType>USERDEFINED</processType>
		<fieldExpression>H-PD-Market_Value_Income - H-PD-Local_Market_Value</fieldExpression>
		<userData>#`FP1014~#`FD1170~-~#`FD1155~</userData>
	</fieldAttribute>
</referenceTransaction>

The initial configuration steps are required. Described here

This functionality is limited for the calculated fields below with field_attribute_process_id:

1010 Arithmetic Addition Calculator
1011 Arithmetic Subtraction Calculator
1012 Arithmetic Division Calculator
1013 Arithmetic Multiplication Calculator
1014 Expression Calculator
1015 Adjustments Calculator
1025 Trade Adjustment Calculator
1030 IRR Calculator
1031 Modified Dietz Calculator
1032 Dollar Value Added Calculator
1033 Average Invested Balance Calculator

Example extract is attached.

RDC Enrichments

Five additional fields are supported:

fieldAttributeReference
enrichmentRule
rdcValidation
rdcFieldGroup
rdcDataStrategy

Additional logic was implemented to group values and use TEMP table for join. Grouped values are separated by ; symbol.

fieldAttributeReference - field attributes which are used in calculations

Extract file:

enrichmentRule - the enrichment rule for the field

Extract file:

RDC

validation - list of used validations

Extract file:

Value in the cell:

First Compounding Date Cannot Be Greater than Maturity Date; First Compounding Date Cannot Be Greater than Termination Date; First Compounding Date Cannot Be Less than Dated Date; First Compounding Date Cannot Be Less than Effective Date

RDC

rdcFieldGroup - list of selected groups

Extract file:

RDC

Single value:

Multiple values:

rdcDataStrategy - list of applied data strtategies

Extract file:

Custom Mashup

In case you want to use custom file (CSV, JSON, etc.) to mashup it with fieldattributes extract.

Create custom dataset definition for the file in the folder: estar\tpe\dynamic\metadata\core\datasets

For example:

This file describe custom CSV file:

NOTE: pay attention on file path and adjust it correspondently.

Place file for mashup to path specified in dataset definition:

For example in dataset definition, path is defined as:

"file_name": " /apps/eagle/estar/tpe/dynamic/metadata/core/ontology/templates/EADataMapping.csv",

To make this works create custom processing rule in cache:

KEY: eagle.ebs.resource:O17_FIELDATTRIBUTE_2.0.57
VALUE: 2.0.57

Pay attention to the following:

Add definition with key in the register_for_mashups block
Add mashup in mashups block
Add mapping in taxonomy block

KEY: eagle.ebs.resource:O17_FIELDATTRIBUTE_VERSIONS
VALUE: 2.0.57
NOTE: O17 this is tenant prefix. it can be different for your case.

Run EQL extract:

{
	"ServiceName": "EBS",
	"ServiceMethodName": "extract",
	"resourcename": "fieldattribute",
	"outputformat": "EagleJSON",
	"disabledatasetcache": "Y",
	"resourceversion": "2.0.57",
	"filters": {
		"fieldattributeid": "5300,4876,4877,2014,4263,4265,371,4266,5864,2110,108,109"
	}
}

"resourceversion": "2.0.57",Version can be different depends on the rule version you create

Output example with custom mashup:

{
	"eagleJSONVersion": "2-0",
	"eagleJSONType": "ReferenceTransactionMessage",
	"eagleJSONRevision": "294",
	"referenceTransactionMessage": {
		"header": {},
		"referenceTransaction": [
			{
				"header": {
					"objectType": "FieldAttribute"
				},
				"fieldAttribute": [
					{
						"objectType": "FieldAttribute",
						"statField": "Y",
						"calcType": "di",
						"databaseName": "security",
						"fieldAttributeDescription": "i Cusip",
						"fieldAttributeId": 108,
						"fieldIndicator": "S",
						"fieldName": "PRIMARY_ASSET_ID",
						"tableName": "SECURITY_MASTER",
						"refCodeId": 99,
						"comments": "Cusip",
						"fieldIdentifier": "I CUSIP",
						"scope": 2,
						"readOnly": "N",
						"fieldAttributeOntologyDescription": "Stores the most appropriate (primary) market security identifier of the financial instrument.",
						"rdcFieldGroup": "IIP_AF_5",
						"ea_mapping.LOB": "NYL Investors",
						"ea_mapping.SUBJECT_AREA_DOMAIN": "SECURITY",
						"ea_mapping.SUB_DOMAIN": "SECURITY",
						"ea_mapping.BUSINESS_NAME": "PRIMARY_ASSET_ID",
						"ea_mapping.BUSINESS_DESCRIPTION": "Current Primary Identifier of a security",
						"ea_mapping.PHASE_1_NYLIM": "-NA-",
						"ea_mapping.REQUIRED_FOR_FUTURE_STATE": "Y",
						"ea_mapping.SOURCE_PLATFORM": "CAMRA",
						"ea_mapping.SOURCE_TABLE": "SECURITY",
						"ea_mapping.SOURCE_COLUMN_SOURCE_COLUMN_STAT": "CUSIP",
						"ea_mapping.FORMAT": "CHAR(100)",
						"ea_mapping.COMMENTS_DERIVATIONS_TRANSFORMATIONS_FOR_SOURCE": "This is CUSIP",
						"ea_mapping.EXTRACT_FIELD_EXTRACT_FIELD_STAT": "CUSIP",
						"ea_mapping.FORMAT": "CHAR(100)",
						"ea_mapping.ML_ELEMENT": "EagleML/referenceTransaction/genericSMF/primaryAssetId",
						"ea_mapping.SUBJECT_AREA_FOR_INTERNAL": "cam_egl_smf",
						"ea_mapping.TARGET_FIELD_NAME_FOR_INTERNAL": "primaryAssetId",
						"ea_mapping.MANDATORY_FIELD": "Y"
					}
				]
			}
		]
	}
}