Fields for an Extension Table
When you create Data Rules extension which already has existing DB structure in market database, you have to mimic the same structure in Snowflake. Each element must be described to have the same table in Snowflake, as in below example:
| Code Block |
|---|
"vocabulary":{
"newField": {
"path": "extension/newElement",
"datatype": "integer",
"owner": "CUSTOMDBO",
"table_name": "POSITION_DETAIL_EXT",
"column_name": "NEW_ELEMENT_INT",
"formatdialect": {
"precision": "38",
"scale": "0"
}
},
...
} |
When DB structure of extension is not predefined, you can create extension without DB parameters definition.
DB parameters for this case will be generated automatically:
owner -
{in_market_client_id}DBOtable_name -
{RESOURCE_NAME}_EXTcolumn_name -
{ELEMENT}
Information about the owner is taken from consul config dict by “in_market_client_id”.
https://consul.dev.az.eagleinvsys.com/ui/az-dev/kv/services/svc-eds/
Example:
| Code Block |
|---|
"vocabulary":{
"newElementChar": {
"path": "extension/newElementChar",
"datatype": "string",
"formatdialect": {
"length": 16
}
},
...
} |
Fields for Summary Extension Tables
Fields for summary extension tables can be created if core resource has summary extension table (warehouseposition)
All these fields are provided in summary_vocabulary block.
Example when you have predefined DB structure:
| Code Block |
|---|
"vocabulary":{
"newField": {
"path": "extension/positionDetailId",
"datatype": "integer",
"owner": "CLIENTID",
"table_name": "POSITION_DETAIL_EXT",
"column_name": "POSITION_DETAIL_ID",
"formatdialect": {
"precision": "38",
"scale": "0"
}
},
...
},
"summary_vocabulary":{
"newSummaryField": {
"path": "extension/newSummaryField",
"datatype": "integer",
"owner": "JANUS",
"table_name": "POSITION_EXT",
"column_name": "POSITION_ID",
"formatdialect": {
"precision": "38",
"scale": "0"
},
...
} |
Example when you don`t have predefined DB structure:
| Code Block |
|---|
"vocabulary":{
"newField": {
"path": "extension/positionDetailId",
"datatype": "integer",
"formatdialect": {
"precision": "38",
"scale": "0"
}
},
...
},
"summary_vocabulary":{
"newSummaryField": {
"path": "extension/newSummaryField",
"datatype": "integer",
"formatdialect": {
"precision": "38",
"scale": "0"
},
...
} |
DB parameters for this case will be generated automatically:
owner -
{in_market_client_id}DBOtable_name -
{RESOURCE_NAME}_EXTtable_name for summary vocabulary -
{RESOURCE_NAME}_EXT_SUMMARYcolumn_name -
{ELEMENT}
Create an Extension
To generate an Extension via Rest API Endpoint in Vault:
Use the Ontology Publish Service API:
Where:
X-Eagle-Context - is Tenant ID;
Request body: see an example of the request below:
| Expand | |||||
|---|---|---|---|---|---|
| |||||
|
| Info |
|---|
‘generateldm’ parameter should be passed as “N” |
To Generate an Extension in Eagle via RTR
To generate an extension via RTR the following RTR should be used via stream eagle_ml-2-0_default_cm_exec_eds , the example is below.
| View file | ||
|---|---|---|
|
Add extension fields to “load”: “dataframe”: “vocabulary” list in resource JSON definition. (see in expand)
| Expand | ||
|---|---|---|
| ||
|
Generated extension can be located in the below folder
dynamic/metadata/custom/ontology/{in_market_cliend_id}
Process Rule Generation for an Extension
A Processing rule is automatically generated when you create an extension.
Alter DDL for an extension will be automatically executed when you generate extension via API.
Cache will be stored in Redis and can be found via the following key:
eagle.ebs.resource:{in_market_cliend_id}_{resource_name}_{core_ontology version}
Example: eagle.ebs.resource_ik:CLI_WAREHOUSEPOSITION_1.0.1
The processing rule for the extension can be also generated via RTR for the corresponding core resource.
| View file | ||
|---|---|---|
|
Alter DDL for an Extension
This is an optional step, the new tables and fields will be created in the DB during the execution of processing rule generation:
The Extension describes tables/fields that do not exist in the database
Additional Summary fields are required/added
Alter DDL RTR for extension resource is the same as Alter DDL RTR for a core resource.
| View file | ||
|---|---|---|
|
Load and Extract Data to/from Extension Table
Congratulations! Now the extension is successfully generated and ready to use.
Load
To load data, please use the same RTR as for the Data Rules core resource.
New fields or Extensions elements are listed under the “extension” node:
Sample xml:
| Code Block |
|---|
... <extension> <newElementNum>1</newElementNum> <newElementString>str</newElementString> </extension> ... |
Sample csv:
| Code Block |
|---|
...,extension/newElementNum,extension/newElementString,... ...,1,str,... |
Extract
To extract data use the same RTR as for the core resource
| Warning |
|---|
Attention: if new DB fields are described in extension, then Alter DDL is mandatory step for Load and Extract. Otherwise, there will be failures. |
How to Remove an Extension
Delete extension ontology files from azure: https://portal.azure.com/#@eagleinvsys.net/resource/subscriptions/030c4927-46ed-4031-934e-d8cb445e75d0/resourceGroups/dev-eastus2-storage-eds/providers/Microsoft.Storage/storageAccounts/deveastus2eds/containersList
metadata/custom/ontology/{in_market_client_id}/{in_market-client_id}_{resource}*.json
Example for EDS tenant warehouseposition extension:
metadata/custom/ontology/da1t1cedstest/da1t1cedstest_warehouseposition.json
metadata/custom/ontology/da1t1cedstest/da1t1cedstest_warehouseposition-models.json
Delete processing rule for extension from cache
https://apps.dev.az.eagleinvsys.com:8443/api/vault/metadata/api/doc#/processing-rule/deleteResource
resource-name = {in_market-client_id}_{resource}
resourceversion = current CORE ontology version (taken from Consul) + ontology versions from blob
Example for EDS tenant rule for warehouseposition extension:
name = DA1T1CEDSTEST_WAREHOUSEPOSITION
version = 2.0.39.1.0.1