Data Rules Extensions: A Step-by-Step Guide

Using core ontology for ETL processes is the best way to work with Data Rules, but sometimes it is necessary to extend the core resource to support additional fields and custom data. In this case, the best practice is to create an extension of the Data Rules core resource which will contain all the necessary information about the new fields.

An extension can be generated only of the CORE resource

This page contains guidance on how to work with extensions at Design and at Run time.

Supported Types of Extension Columns

Details below is required minimum to describe additional fields as an extension element:

path - describes a path of the element;
datatype - describes a type of the data ('string', ‘number', 'date’, etc.);
formatdialect - an attribute of the datatype (as example "length": 16 for 'string').

All extension elements are stored in the “extension” complex element.

If the structure of the extension table is defined in advance, then extension element can have extra DB description:

owner - database SCHEMA
table_name - database TABLE
column_name - database COLUMN

In this section

On this page

1 Supported Types of Extension Columns
2 Fields for an Extension Table
3 Fields for Summary Extension Tables
4 Create an Extension
5 To Generate an Extension in Eagle via RTR
6 Process Rule Generation for an Extension
7 Alter DDL for an Extension
8 Load and Extract Data to/from Extension Table
9 Load
10 Extract
11 How to Remove an Extension

Fields for an Extension Table

When you create Data Rules extension which already has existing DB structure in market database, you have to mimic the same structure in Snowflake. Each element must be described to have the same table in Snowflake, as in below example:

"vocabulary":{
    "newField": {
		"path": "extension/newElement",
		"datatype": "integer",
		"owner": "CUSTOMDBO",
		"table_name": "POSITION_DETAIL_EXT",
		"column_name": "NEW_ELEMENT_INT",
		"formatdialect": {
			"precision": "38",
			"scale": "0"
		}
  },
  ...
}

When DB structure of extension is not predefined, you can create extension without DB parameters definition.

DB parameters for this case will be generated automatically:

owner - {in_market_client_id}DBO
table_name - {RESOURCE_NAME}_EXT
column_name - {ELEMENT}

Information about the owner is taken from consul config dict by “in_market_client_id”.

https://consul.dev.az.eagleinvsys.com/ui/az-dev/kv/services/svc-eds/

Example:

"vocabulary":{
  "newElementChar": {
		"path": "extension/newElementChar",
		"datatype": "string",
		"formatdialect": {
			"length": 16
		}
  },
  ...
}

Fields for Summary Extension Tables

Fields for summary extension tables can be created if core resource has summary extension table (warehouseposition)

All these fields are provided in summary_vocabulary block.

Example when you have predefined DB structure:

"vocabulary":{
    "newField": {
		"path": "extension/positionDetailId",
		"datatype": "integer",
		"owner": "CLIENTID",
		"table_name": "POSITION_DETAIL_EXT",
		"column_name": "POSITION_DETAIL_ID",
		"formatdialect": {
			"precision": "38",
			"scale": "0"
		}
  },
  ...
},
"summary_vocabulary":{
    "newSummaryField": {
		"path": "extension/newSummaryField",
		"datatype": "integer",
		"owner": "JANUS",
		"table_name": "POSITION_EXT",
		"column_name": "POSITION_ID",
		"formatdialect": {
			"precision": "38",
			"scale": "0"
	},
	...
}

Example when you don`t have predefined DB structure:

DB parameters for this case will be generated automatically:

owner - {in_market_client_id}DBO
table_name - {RESOURCE_NAME}_EXT
table_name for summary vocabulary - {RESOURCE_NAME}_EXT_SUMMARY
column_name - {ELEMENT}

Create an Extension

To generate an Extension via Rest API Endpoint in Vault:

Use the Ontology Publish Service API:

https://apps.dev.az.eagleinvsys.com/api/vault/ontology-publish/api/doc#/metadata-mapping-rest-api/submitEDSMetadataMappingPOST

Where:

X-Eagle-Context - is Tenant ID;
Request body: see an example of the request below:

‘generateldm’ parameter should be passed as “N”

To Generate an Extension in Eagle via RTR

To generate an extension via RTR the following RTR should be used via stream eagle_ml-2-0_default_cm_exec_eds , the example is below.

Add extension fields to “load”: “dataframe”: “vocabulary” list in resource JSON definition. (see in expand)

Generated extension can be located in the below folder

dynamic/metadata/custom/ontology/{in_market_cliend_id}

Process Rule Generation for an Extension

A Processing rule is automatically generated when you create an extension.
Alter DDL for an extension will be automatically executed when you generate extension via API.
Cache will be stored in Redis and can be found via the following key:

eagle.ebs.resource:{in_market_cliend_id}_{resource_name}_{core_ontology version}

Example: eagle.ebs.resource_ik:CLI_WAREHOUSEPOSITION_1.0.1

The processing rule for the extension can be also generated via RTR for the corresponding core resource.

Alter DDL for an Extension

This is an optional step, the new tables and fields will be created in the DB during the execution of processing rule generation:

The Extension describes tables/fields that do not exist in the database
Additional Summary fields are required/added

Alter DDL RTR for extension resource is the same as Alter DDL RTR for a core resource.

Load and Extract Data to/from Extension Table

Congratulations! Now the extension is successfully generated and ready to use.

Load

To load data, please use the same RTR as for the Data Rules core resource.

New fields or Extensions elements are listed under the “extension” node:

Sample xml:

Sample csv:

Extract

To extract data use the same RTR as for the core resource

How to Remove an Extension

Delete extension ontology files from azure: https://portal.azure.com/#@eagleinvsys.net/resource/subscriptions/030c4927-46ed-4031-934e-d8cb445e75d0/resourceGroups/dev-eastus2-storage-eds/providers/Microsoft.Storage/storageAccounts/deveastus2eds/containersList

metadata/custom/ontology/{in_market_client_id}/{in_market-client_id}_{resource}*.json

Example for EDS tenant warehouseposition extension:
metadata/custom/ontology/da1t1cedstest/da1t1cedstest_warehouseposition.json
metadata/custom/ontology/da1t1cedstest/da1t1cedstest_warehouseposition-models.json

Delete processing rule for extension from cache

https://apps.dev.az.eagleinvsys.com:8443/api/vault/metadata/api/doc#/processing-rule/deleteResource

resource-name = {in_market-client_id}_{resource}

resourceversion = current CORE ontology version (taken from Consul) + ontology versions from blob

Example for EDS tenant rule for warehouseposition extension:

name = DA1T1CEDSTEST_WAREHOUSEPOSITION
version = 2.0.39.1.0.1