The EDS Delta Cache relies on a set of configuration files to implement the delta cache updates. These files define what resources (tables) to monitor for updates and what datasets to refresh in the cache if the resource change.
The Delta Cache process uses two methods for monitoring the changes in the tables:
Database Poll – it runs a query at a predefined interval to determine if there were any changes in the tables since the last cache update.
Oracle Change Notification Subscription – it creates a subscription to Oracle to receive notifications when a table is being changed.
| Info |
|---|
The Oracle Change Notification Subscription is suitable for tables which do not have a lot of updates. This is a recommendation from Oracle. We have not validated the subscriptions on tables with large amounts of updates. |
| Note |
|---|
There is an issue with the update of the cache in delta mode for File Cache. It should not be used before a fix is provided. File cache applies to datasets having both the properties "cache_provider": "cache-manager-file" and "cache_mode": "full". |
Configuration
The cache policies are JSON files stored in the eagle/estar/tpe/dynamic/metadata/custom/cachepolicies/ folder. All files with extension .json from this folder will be loaded by the Cache Manager.
Each file will define the cache policies for a set of resources and will define which datasets have to be updated when the resource changes.
| Info |
|---|
The file name e.g. rating.json must match the _id field within the file e.g. "_id": "rating", |
Defining a Database Poll Cache Policy
...
The resource to be monitored should be specified in the "resource" parameter. The schema for the table should be the logical schema name. For example REFERENCE should be used instead of PACE_MASTERDBO.
The following parameters specific to the resource should be specified:
"field" – this is the field in the table we have to use to track updated records
"pollIntervalSeconds" – how often we should check the table was changed
The following parameters are specific to the cache update:
"checkIntervalSeconds" - specifies how often the Cache Manager will check for changes to the monitored resources
"noUpdateActivitySeconds" - specifies how long to wait for the resource to not change before initiating a cache update. This is important for massive updates and will prevent the cache refresh from happening in the middle of a big data load. It will also prevent the cache update to start if and RDC process is running and updating records.
Defining multiple datasets in a single cache policy
...
The Delta Cache Manager maintains a set of cache update state variables to track the data and cache state. This state variables are stored in REDIS.
The data state update status is stored in entries named delta_cache_data_update:policyid, where policyid is the policy id, for example "interfaces".
This structure has the information about when the resources where polled, when the resources being changed were changed and how many records were changed.
The cache state update status is stored in entries named delta_cache_update:policyid. This structure keeps information about the last time the cache was updated and the number of records updated with the last cache update.
Pyruleservice should be cycled to pick up changes to cache policy files
(the delta cache manager runs as part of pyruleservice worker 1).
The cache manager logs to starweb/pyruleservice/mc2py_exec_1.log