EDS Delta Cache Update

The EDS Delta Cache relies on a set of configuration files to implement the delta cache updates. These files define what resources (tables) to monitor for updates and what datasets to refresh in the cache if the resource change.
The Delta Cache process uses two methods for monitoring the changes in the tables:

Database Poll – it runs a query at a predefined interval to determine if there were any changes in the tables since the last cache update.
Oracle Change Notification Subscription – it creates a subscription to Oracle to receive notifications when a table is being changed.

On this page:

1 Configuration
2 Defining a Database Poll Cache Policy
3 Defining multiple datasets in a single cache policy
4 Defining Additional Parameters for Database Poll
5 Defining a Subscription Based Cache Policy
6 Cache Manager State Variables

The Oracle Change Notification Subscription is suitable for tables which do not have a lot of updates. This is a recommendation from Oracle. We have not validated the subscriptions on tables with large amounts of updates.

There is an issue with the update of the cache in delta mode for File Cache. It should not be used before a fix is provided. File cache applies to datasets having both the properties "cache_provider": "cache-manager-file" and "cache_mode": "full".

Configuration

The cache policies are JSON files stored in the eagle/estar/tpe/dynamic/metadata/custom/cachepolicies/ folder. All files with extension .json from this folder will be loaded by the Cache Manager.
Each file will define the cache policies for a set of resources and will define which datasets have to be updated when the resource changes.

The file name e.g. rating.json must match the _id field within the file e.g. "_id": "rating",

Defining a Database Poll Cache Policy

To define a database poll cache policy, use a resource definition with changeNotificationType set to "poll", for example:

{
"_id": "smf",
"datasets": ["genericsmf"],
"resources": [
{
"resource": "eagledb:SECURITY.SECURITY_MASTER",
"changeNotificationType": "poll",
"parameters": {
"field": "UPDATE_DATE",
"pollIntervalSeconds": 10
}
}
],
"cacheUpdate": {
"checkIntervalSeconds": 10,
"noUpdateActivitySeconds": 5
}
}

The resource to be monitored should be specified in the "resource" parameter. The schema for the table should be the logical schema name. For example REFERENCE should be used instead of PACE_MASTERDBO.
The following parameters specific to the resource should be specified:

"field" – this is the field in the table we have to use to track updated records
"pollIntervalSeconds" – how often we should check the table was changed

The following parameters are specific to the cache update:

"checkIntervalSeconds" - specifies how often the Cache Manager will check for changes to the monitored resources
"noUpdateActivitySeconds" - specifies how long to wait for the resource to not change before initiating a cache update. This is important for massive updates and will prevent the cache refresh from happening in the middle of a big data load. It will also prevent the cache update to start if and RDC process is running and updating records.

Defining multiple datasets in a single cache policy

Multiple datasets driven by the same resource should be specified in a single cache policy as in the following example:

{
"_id": "genericsmf",
"datasets": ["genericsmf", "genericsmf_custom1", "eagle_to_matlab2_genericsmf_equitychar5", "cash_aud_dataset"],
"resources": [
{
"resource": "eagledb:SECURITY.SECURITY_MASTER",
"changeNotificationType": "poll",
"parameters": {
"field": "UPDATE_DATE",
"pollIntervalSeconds": 300
}
}
],
"cacheUpdate": {
"checkIntervalSeconds": 120,
"noUpdateActivitySeconds": 60
}
}

Defining Additional Parameters for Database Poll

When polling very big tables with no indexes on update date we can specify additional parameters to limit the amount of data we poll. Usually these parameters will map to table columns which have an index.

{
"_id": "rating",
"datasets": ["rating"],
"resources": [
{
"resource": "eagledb:SECURITY.RATINGS",
"changeNotificationType": "poll",
"parameters": {
"field": "UPDATE_DATE",
"pollIntervalSeconds": 10
},
"filter_definitions": {
"fromeffectivedate": {
"parameters": [
{
"name": "fromeffectivedate"
}
],
"bindings": [
{
"sourceProvider": "eagledb://oracle",
"filter": "MASTERTABLE.EFFECTIVE_DATE >= :fromeffectivedate"
}
]
}
},
"filter_values": {
"fromeffectivedaterule": "Today"
}
}
],
"cacheUpdate": {
"checkIntervalSeconds": 10,
"noUpdateActivitySeconds": 5,
"filter_values": {
"fromeffectivedaterule": "Today"
}
}
}

Common (predefined) filters can also be used in the resource definition:

{
"_id": "rating",
"datasets": ["rating"],
"resources": [
{
"resource": "eagledb:SECURITY.RATINGS",
"changeNotificationType": "poll",
"parameters": {
"field": "UPDATE_DATE",
"pollIntervalSeconds": 10
},
"filter_definitions": {
"common_filters": ["fromeffectivedate"]
},
"filter_values": {
"fromeffectivedaterule": "Today"
}
}
],
"cacheUpdate": {
"checkIntervalSeconds": 10,
"noUpdateActivitySeconds": 5,
"filter_values": {
"fromeffectivedaterule": "Today"
}
}
}

Defining a Subscription Based Cache Policy

To define a Subscription Based cache policy, use a resource definition with changeNotificationType set to "subscribe", for example:

{
"_id": "interfaces",
"datasets": ["interfaces"],
"resources": [
{
"resource": "eagledb:REFERENCE.INTERFACES",
"changeNotificationType": "subscribe"
}
],
"cacheUpdate": {
"checkIntervalSeconds": 10,
"noUpdateActivitySeconds": 5
}
}

The database user needs to have change notification privilege granted to user estar for the subscription method:
GRANT CHANGE NOTIFICATION TO estar
Currently there are no additional parameters for the resource. The cacheUpdate parameters are similar to the parameters specified in the poll.

Cache Manager State Variables

The Delta Cache Manager maintains a set of cache update state variables to track the data and cache state. This state variables are stored in REDIS.
The data state update status is stored in entries named delta_cache_data_update:policyid, where policyid is the policy id, for example "interfaces".
This structure has the information about when the resources where polled, when the resources being changed were changed and how many records were changed.
The cache state update status is stored in entries named delta_cache_update:policyid. This structure keeps information about the last time the cache was updated and the number of records updated with the last cache update.
Pyruleservice should be cycled to pick up changes to cache policy files
(the delta cache manager runs as part of pyruleservice worker 1).

The cache manager logs to starweb/pyruleservice/mc2py_exec_1.log