Schema drift
Schema drift is detection of any schema changes in incoming data file. There are several schema drift modes that handle drifted data in specific ways.
API for schema drift is inflow API:
Dev - https://apps.dev.az.eagleinvsys.com:8443/api/vault/eds/api/doc/#/inflow-rest/submitV3InflowPOST
Stage - https://apps.stage.az.eagleinvsys.com:8443/api/vault/eds/api/doc/#/inflow-rest/submitV3InflowPOST
Prod - https://apps.prod.az.eagleinvsys.com:8443/api/vault/eds/api/doc/#/inflow-rest/submitV3InflowPOST
IGNORE schema drift mode
This schema drift mode is set by default
Schema drift mode IGNORE will ignore any schema drift if it was found or not and will load data without drifted columns.
Example of request:
{
"resourcename": "legalentityid",
"dbprovider": "snowflake",
"schemadriftmode": "ignore"
}
VALIDATE schema drift mode
Schema drift mode VALIDATE is detecting any schema changes for resource on load.
Example of request:
{
"resourcename": "legalentityid",
"dbprovider": "snowflake",
"schemadriftmode": "validate"
}
If drift was not found - data loads into database.
If drift was found - user receives response with drift columns.
Example of validate response when schema drift was found:
AUTO_APPROVE schema drift mode
Schema drift mode AUTO_APPROVE handles schema drift without any manual steps.
Loading new custom resource with schema drift
When schema drift detects new custom resource, these steps will be executed:
generate ontology for new resource
generate processing rule
execute ddl
publish data models
load data into new table
IGNORE schema drift mode
This schema drift mode is set by default
Schema drift mode IGNORE will ignore any schema drift if it was found or not and will load data without drifted columns.
Example of request:
{
"resourcename": "legalentityid",
"dbprovider": "snowflake",
"schemadriftmode": "ignore"
}
VALIDATE schema drift mode
Schema drift mode VALIDATE is detecting any schema changes for resource on load.
Example of request:
{
"resourcename": "legalentityid",
"dbprovider": "snowflake",
"schemadriftmode": "validate"
}
If drift was not found - data loads into database.
If drift was found - user receives response with drift columns.
Example of validate response when schema drift was found:
AUTO_APPROVE schema drift mode
Schema drift mode AUTO_APPROVE handles schema drift without any manual steps.
Loading new custom resource with schema drift
When schema drift detects new custom resource, these steps will be executed:
generate ontology for new resource
generate processing rule
execute ddl
publish data models
load data into new table
Example of request:
{
"resourcename": "priceMaster",
"dbprovider": "snowflake",
"schemadriftmode": "auto_approve"
}
Example of request with specified vendor and feedsystem:
By default vendor is BNYM, feedsystem is EAGLE
{
"resourcename": "priceMaster",
"dbprovider": "snowflake",
"schemadriftmode": "auto_approve",
"vendor": "VENDOR_NAME",
"feedsystem": "FEEDSYSTEM_NAME"
}
Example of request with schema drift tolerance:
This param is used on update of custom resource.
If drifted columns count is more than tolerance then Error will be added to processing result.
If drifted columns count is less or equal than tolerance - then schema drift process will be done.
{
"resourcename": "priceMaster",
"dbprovider": "snowflake",
"schemadriftmode": "auto_approve",
"schemadrifttolerance": "2"
}
Data file example:
Example of auto_approve response when schema drift was found for new resource:
Loading core EDS resource with schema drift
When schema drift is detected in data for core resource, these steps will be executed:
generate extension for resource
generate processing rule
execute alter ddl
publish data models
load data into core and extension tables
Example of request:
{
"resourcename": "legalentityid",
"dbprovider": "snowflake",
"schemadriftmode": "auto_approve"
}
Data file example:
Example of auto_approve response when schema drift was found for core resource: