We have introduced the concept of mashup definitions to support dynamic snapshot selection and true "as of" views of security and entity datasets. Mashup definitions are maintained centrally and are referenced from within interfaces. Mashup definitions define the logic of selecting the necessary snapshot based on the request parameters and conditions. They will also implement the loading of snapshots dynamically from database if necessary.
Mashup definition examples with genericsmf
...
Now, let us consider each dataframe. The first dataframe, genericsmf_asof, has two parameters required for creating a cache snapshot: effective_date and source, or, max_effective_date and source. It has cache mode set to none. It represents the 'cold' data as people would rarely use security data in the past. In case the condition is true, the mashup processing engine will fetch the security data snapshot and place it into dynamic short-lived cache. The actual fetch from DB is going to be optimized by 2 factors: dynamic query profile, which will be built based on "columns" property in definition and ontology (which would allow one to map canonical elements into DB columns), and the fact that it would be executed concurrently while the main query of interface is performed. The result cursor will be placed into a "smart" descriptor, which uses in-memory compression and is able to effectively store "sparse" data. Due to all these factors, we expect almost no overhead related to this request. This descriptor would then be made available as a mashup dataframe.
The reason for 2 sets of parameters for creating snapshot is due to the fact that some clients keep security data inconsistently, not really following any of the 'full' or 'delta' modes. For those we will have to use 'max_effective_date' queries, which builds snapshot based on unique list of securities up to specified max effective date. Again, the actual processing of this query is still greatly optimized by the previously mentioned factors.
The second dataframe, genericsmf_asof_delta, assumes pure 'delta' mode storage of security data. It has cache mode set to 'full' and would be fully cached and 'effective for' pattern will be used if this dataframe is selected. In order to maintain up to date cache, this dataframe comes with delta cache update policy.
Finally, the third dataframe is genericsmf and it represents current snapshot of security master data. It is "hot" data which is fully cached and readily available and it also comes with delta cache update policy.
Second definition
...
genericsmf_prior dataframe is based on core genericsmf_prior dataset and stands for cached value of security data for a single previous date. This enables extracts for current day transactions to pull SMF data from the
previous day.