Data Rules Extensions: A Step-by-Step Guide
- Pamela Totas (Deactivated)
Using core ontology for ETL processes is the best way to work with Data Rules, but sometimes it is necessary to extend the core resource to support additional fields and custom data. In this case, the best practice is to create an extension of the Data Rules core resource which will contain all the necessary information about the new fields.
An extension can be generated only of the CORE resource
This page contains guidance on how to work with extensions at Design and at Run time.
Supported Types of Extension Columns
Details below is required minimum to describe additional fields as an extension element:
path - describes a path of the element;
datatype - describes a type of the data ('string', ‘number', 'date’, etc.);
formatdialect - an attribute of the datatype (as example
"length": 16for 'string').
All extension elements are stored in the “extension” complex element.
If the structure of the extension table is defined in advance, then extension element can have extra DB description:
owner - database SCHEMA
table_name - database TABLE
column_name - database COLUMN
In this section
On this page
- 1 Supported Types of Extension Columns
- 2 Fields for an Extension Table
- 3 Fields for Summary Extension Tables
- 4 Create an Extension
- 5 To Generate an Extension in Eagle via RTR
- 6 Process Rule Generation for an Extension
- 7 Alter DDL for an Extension
- 8 Load and Extract Data to/from Extension Table
- 9 Load
- 10 Extract
- 11 How to Remove an Extension
Fields for an Extension Table
When you create Data Rules extension which already has existing DB structure in market database, you have to mimic the same structure in Snowflake. Each element must be described to have the same table in Snowflake, as in below example:
"vocabulary":{
"newField": {
"path": "extension/newElement",
"datatype": "integer",
"owner": "CUSTOMDBO",
"table_name": "POSITION_DETAIL_EXT",
"column_name": "NEW_ELEMENT_INT",
"formatdialect": {
"precision": "38",
"scale": "0"
}
},
...
}When DB structure of extension is not predefined, you can create extension without DB parameters definition.
DB parameters for this case will be generated automatically:
owner -
{in_market_client_id}DBOtable_name -
{RESOURCE_NAME}_EXTcolumn_name -
{ELEMENT}
Information about the owner is taken from consul config dict by “in_market_client_id”.
https://consul.dev.az.eagleinvsys.com/ui/az-dev/kv/services/svc-eds/
Example:
"vocabulary":{
"newElementChar": {
"path": "extension/newElementChar",
"datatype": "string",
"formatdialect": {
"length": 16
}
},
...
}Fields for Summary Extension Tables
Fields for summary extension tables can be created if core resource has summary extension table (warehouseposition)
All these fields are provided in summary_vocabulary block.
Example when you have predefined DB structure:
"vocabulary":{
"newField": {
"path": "extension/positionDetailId",
"datatype": "integer",
"owner": "CLIENTID",
"table_name": "POSITION_DETAIL_EXT",
"column_name": "POSITION_DETAIL_ID",
"formatdialect": {
"precision": "38",
"scale": "0"
}
},
...
},
"summary_vocabulary":{
"newSummaryField": {
"path": "extension/newSummaryField",
"datatype": "integer",
"owner": "JANUS",
"table_name": "POSITION_EXT",
"column_name": "POSITION_ID",
"formatdialect": {
"precision": "38",
"scale": "0"
},
...
}Example when you don`t have predefined DB structure:
"vocabulary":{
"newField": {
"path": "extension/positionDetailId",
"datatype": "integer",
"formatdialect": {
"precision": "38",
"scale": "0"
}
},
...
},
"summary_vocabulary":{
"newSummaryField": {
"path": "extension/newSummaryField",
"datatype": "integer",
"formatdialect": {
"precision": "38",
"scale": "0"
},
...
}DB parameters for this case will be generated automatically:
owner -
{in_market_client_id}DBOtable_name -
{RESOURCE_NAME}_EXTtable_name for summary vocabulary -
{RESOURCE_NAME}_EXT_SUMMARYcolumn_name -
{ELEMENT}
Create an Extension
To generate an Extension via Rest API Endpoint in Vault:
Use the Ontology Publish Service API:
Where:
X-Eagle-Context - is Tenant ID;
Request body: see an example of the request below:
‘generateldm’ parameter should be passed as “N”
To Generate an Extension in Eagle via RTR
To generate an extension via RTR the following RTR should be used via stream eagle_ml-2-0_default_cm_exec_eds , the example is below.
Add extension fields to “load”: “dataframe”: “vocabulary” list in resource JSON definition. (see in expand)
Generated extension can be located in the below folder
dynamic/metadata/custom/ontology/{in_market_cliend_id}
Process Rule Generation for an Extension
A Processing rule is automatically generated when you create an extension.
Alter DDL for an extension will be automatically executed when you generate extension via API.
Cache will be stored in Redis and can be found via the following key:
eagle.ebs.resource:{in_market_cliend_id}_{resource_name}_{core_ontology version}
Example: eagle.ebs.resource_ik:CLI_WAREHOUSEPOSITION_1.0.1
The processing rule for the extension can be also generated via RTR for the corresponding core resource.
Alter DDL for an Extension
This is an optional step, the new tables and fields will be created in the DB during the execution of processing rule generation:
The Extension describes tables/fields that do not exist in the database
Additional Summary fields are required/added
Alter DDL RTR for extension resource is the same as Alter DDL RTR for a core resource.
Load and Extract Data to/from Extension Table
Congratulations! Now the extension is successfully generated and ready to use.
Load
To load data, please use the same RTR as for the Data Rules core resource.
New fields or Extensions elements are listed under the “extension” node:
Sample xml:
...
<extension>
<newElementNum>1</newElementNum>
<newElementString>str</newElementString>
</extension>
...Sample csv:
...,extension/newElementNum,extension/newElementString,...
...,1,str,...
Extract
To extract data use the same RTR as for the core resource
Attention: if new DB fields are described in extension, then Alter DDL is mandatory step for Load and Extract. Otherwise, there will be failures.
How to Remove an Extension
Delete extension ontology files from azure: https://portal.azure.com/#@eagleinvsys.net/resource/subscriptions/030c4927-46ed-4031-934e-d8cb445e75d0/resourceGroups/dev-eastus2-storage-eds/providers/Microsoft.Storage/storageAccounts/deveastus2eds/containersList
metadata/custom/ontology/{in_market_client_id}/{in_market-client_id}_{resource}*.json
Example for EDS tenant warehouseposition extension:
metadata/custom/ontology/da1t1cedstest/da1t1cedstest_warehouseposition.json
metadata/custom/ontology/da1t1cedstest/da1t1cedstest_warehouseposition-models.json
Delete processing rule for extension from cache
https://apps.dev.az.eagleinvsys.com:8443/api/vault/metadata/api/doc#/processing-rule/deleteResource
resource-name = {in_market-client_id}_{resource}
resourceversion = current CORE ontology version (taken from Consul) + ontology versions from blob
Example for EDS tenant rule for warehouseposition extension:
name = DA1T1CEDSTEST_WAREHOUSEPOSITION
version = 2.0.39.1.0.1