EDS Delta Cache Update

The EDS Delta Cache relies on a set of configuration files to implement the delta cache updates. These files define what resources (tables) to monitor for updates and what datasets to refresh in the cache if the resource change.
The Delta Cache process uses two methods for monitoring the changes in the tables:

...

Configuration

The cache policies are JSON files stored in eagle/estar/tpe/dynamic/metadata/custom/cachepolicies/ folder. All files with extension .json from this folder will be loaded by the Cache Manager.
Each file will define the cache policies for a set of resources and will define which datasets have to be updated when the resource changes.

Defining a Database Poll Cache Policy

To define a database poll cache policy, use a resource definition with changeNotificationType set to "poll", for example:

...

• "checkIntervalSeconds" - specifies how often the Cache Manager will check for changes to the monitored resources
• "noUpdateActivitySeconds" - specifies how long to wait for the resource to not change before initiating a cache update. This is important for massive updates and will prevent the cache refresh from happening in the middle of a big data load. It will also prevent the cache update to start if and RDC process is running and updating records.

Defining Additional Parameters for Database Poll

When polling very big tables with no indexes on update date we can specify additional parameters to limit the amount of data we poll. Usually these parameters will map to table columns which have an index.

...

{
"_id": "ratings",
"datasets": ["ratings"],
"resources": [
{
"resource": "eagledb:SECURITY.RATINGS",
"changeNotificationType": "poll",
"parameters": {
"field": "UPDATE_DATE",
"pollIntervalSeconds": 10
},
"filter_definitions": {
"common_filters": ["fromeffectivedate"]
},
"filter_values": {
"fromeffectivedaterule": "Today"
}
}
],
"cacheUpdate": {
"checkIntervalSeconds": 10,
"noUpdateActivitySeconds": 5,
"filter_values": {
"fromeffectivedaterule": "Today"
}
}
}

Defining a Subscription Based Cache Policy

To define a Subscription Based cache policy, use a resource definition with changeNotificationType set to "subscribe", for example:

...

The database user needs to have change notification privilege granted to user estar for the subscription method:
GRANT CHANGE NOTIFICATION TO estar
Currently there are no additional parameters for the resource. The cacheUpdate parameters are similar to the parameters specified in the poll.

Cache Manager State Variables

The Delta Cache Manager maintains a set of cache update state variables to track the data and cache state. This state variables are stored in REDIS.
The data state update status is stored in entries named delta_cache_data_update:policyid, where policyid is the policy id, for example "interfaces".
This structure has the information about when the resources where polled, when the resources being changed were changed and how many records were changed.
The cache state update status is stored in entries named delta_cache_update:policyid. This structure keeps information about the last time the cache was updated and the number of records updated with the last cache update.

Pyruleservice should be cycled to pick up changes to cache policy files
(the delta cache manager runs as part of pyruleservice worker 1).

The cache manager logs to starweb/pyruleservice/mc2py_exec_1.log