Schema drift
...
Schema drift is detection of any schema changes in incoming data file. There are several schema drift modes that handle drifted data in specific ways.
...
Prod - https://apps.prod.az.eagleinvsys.com:8443/api/vault/eds/api/doc/#/inflow-rest/submitV3InflowPOST
IGNORE
...
Schema Drift Mode
Info |
---|
This schema drift mode is set by default |
Schema drift mode IGNORE will ignore any schema drift if it was found or not and will load data without drifted columns.
Example of request:
Code Block |
---|
{ "resourcename": "legalentityid", "dbprovider": "snowflake", "schemadriftmode": "ignore" } |
VALIDATE
...
Schema Drift Mode
Schema drift mode VALIDATE is detecting any schema changes for resource on load.
Example of request:
Code Block |
---|
{ "resourcename": "legalentityid", "dbprovider": "snowflake", "schemadriftmode": "validate" } |
...
If drift was found - user receives response with drift columns.
Example of validate response when schema drift was found:
Expand | ||
---|---|---|
| ||
|
...
Schema drift mode AUTO_APPROVE handles schema drift without any manual steps.
Loading
...
a New Custom Resource with a Schema Drift
When schema drift detects new custom resource, these steps will be executed:
generate ontology for new resource
generate processing rule
execute ddl
publish data models
load data into new table
IGNORE
...
Schema Drift Mode
Info |
---|
This schema drift mode is set by default |
Schema drift mode IGNORE will ignore any schema drift if it was found or not and will load data without drifted columns.
Example of request:
Code Block |
---|
{ "resourcename": "legalentityid", "dbprovider": "snowflake", "schemadriftmode": "ignore" } |
VALIDATE schema drift mode
Schema drift mode VALIDATE is detecting any schema changes for resource on load.
Example of request:
Code Block |
---|
{ "resourcename": "legalentityid", "dbprovider": "snowflake", "schemadriftmode": "validate" } |
...
If drift was found - user receives response with drift columns.
Example of validate response when schema drift was found:
Expand | ||
---|---|---|
| ||
|
...
Schema drift mode AUTO_APPROVE handles schema drift without any manual steps.
...
Load New Custom Resource with Schema Drift
When schema drift detects new custom resource, these steps will be executed:
generate ontology for new resource
generate processing rule
execute ddl
publish data models
load data into new table
Example of request:
Code Block |
---|
{ "resourcename": "priceMaster", "dbprovider": "snowflake", "schemadriftmode": "auto_approve" } |
Example of request with specified vendor and feedsystem:
By default vendor is BNYM, feedsystem is EAGLE
Code Block |
---|
{ "resourcename": "priceMaster", "dbprovider": "snowflake", "schemadriftmode": "auto_approve", "vendor": "VENDOR_NAME", "feedsystem": "FEEDSYSTEM_NAME" } |
Example of request with schema drift tolerance:
This param is used on update of custom resource.
...
Code Block |
---|
{ "resourcename": "priceMaster", "dbprovider": "snowflake", "schemadriftmode": "auto_approve", "schemadrifttolerance": "2" } |
Data file example:
Example of auto_approve response when schema drift was found for new resource:
Expand | ||
---|---|---|
| ||
|
...
Load Core EDS
...
Resource with
...
Schema Drift
When schema drift is detected in data for core resource, these steps will be executed:
generate extension for resource
generate processing rule
execute alter ddl
publish data models
load data into core and extension tables
Example of request:
Code Block |
---|
{ "resourcename": "legalentityid", "dbprovider": "snowflake", "schemadriftmode": "auto_approve" } |
Data file example:
Example of auto_approve response when schema drift was found for core resource:
Expand | ||
---|---|---|
| ||
|
...