Set Up Data Quality Rules

Data quality rules allow you to associate the validation rules with specific entities, performance models, and frequencies. A single data quality rule can have multiple validation rules associated with it. A Data Quality Rule can only have validation rules of one Process Stage type, either Data Quality or Calculation Status. Data Quality validations and Calculations Status validations should be maintained in separate Data Quality Rules.

Once you define the data quality rule, it is run by one of two engines:

  • Data Quality rules with Calculation Status validation are run by the Performance Calculation engine.
  • Data Quality rules with Data Quality validation are run in Automation Center using the Data Quality Monitor Engine event type. For more information, see Schedule a Rule in Automation Center.
Content on this page:

The Data Quality Monitor heat map displays the most recent Data Quality Rule validation results for an entity, effective date, performance model, and frequency. Each time a Data Quality Rule is run, either from Automation Center or the Performance Calculation engine, any previous validation results for that Data Quality Rule, effective date, performance model, and entity are deleted. You cannot run data quality rules or schedule data quality rules from Data Steward.

View Data Quality Rules

You can view all available data quality rules and their components in the Data Quality Rule workspace.

To view data quality rules:

From the Performance Center window, click Setup and click Data Quality Rules in the left navigation. You see the available data quality rules in the workspace.

Create Data Quality Rules

You can create a data quality rule by associating validation rules with specific entities, performance models, and frequencies. Data quality rules use data quality validation rules or calculation status validation rules to test the validity of performance calculations and reference data. For more information, see About Validation Rules.

To create a data quality rule:

  1. From the Performance Center window, click Setup and click Data Quality Rules in the left navigation.
    You see the available data quality rules in the workspace.
  2. Click Create New on the ribbon.
    You see the Creating New Data Quality Rule workspace.
  3. Enter a Name and a Description.
  4. To create a list of entities used by the rule, click the List lookup icon and select the entities. Or, click the ellipsis to conduct a search of entities.
  5. Click the Model drop down and select the performance models used by the rule.

    This is a required field for data quality and reference data if no source rule is chosen. When no source rule is selected, the source rule associated with the performance model is used for cash activity and positions. The Model is always required for Data Quality rules with Calculation Status validations.

  6. Click the Frequency drop down and select a frequency.
    Options include:
    –  Daily
    –  Monthly
    –  Quarterly

    For data quality validations, the frequency defines the days in a range for running the validations. For calculation status validations, the frequency limits the performance calculation that runs the calculation status validations in the data quality rule.

  7. Click the Source Rule drop down and select a source rule to override the source rule in the selected performance model.
  8. Check the Use Business Calendar check box to limit the validations to business dates when you have selected a date range.
  9. Check the Enable This Rule check box to make the rule available for use.
  10. Click the Process Stage drop down and select the process stage for the rule.
    Options include:
    –  Calculation Status. These validation rules focus on the performance calculation process itself by providing important information at various calculation checkpoints.
    –  Data Quality. These validation rules monitor the quality of positions, entities, securities, cash activity, FX rates, and prices.

    Data Quality rules should contain only validations of the same Process Stage.

  11. Click the Stage Detail drop down and select a stage detail.
    For the calculation status process stage, options include:
    –  Performance Commit. If you select this option, you see the Test drop down with system defined tests.
    –  Return Calcs. If you select this option, you see the Test drop down with Performance Return Check, a user defined test.
    For the data quality process stage, options include:
    –  Cash Activity (default). Checks the cash activity database for data such as flow amounts, transaction types, and quantities.
    –  Entity Reference. Checks entity reference data such as inception dates, currencies, and benchmarks.
    –  FX Rates. Checks the FX rates between composites and their constituents.
    –  Performance Commit. Checks for client side commits done for an entity based on effective date and frequency.
    –  Performance Data. Checks the performance database for data such as benchmark returns and existing portfolio returns.
    –  Position. Checks the holdings database for position market value amounts, long or short position, and missing market values.
    –  Price. Checks the security database price data for issues such as price changes and missing prices.
    –  Security Reference. Checks the security database for security classification and reference data such as investment type and base currency.
  12. Click the System Tests drop down and select a system defined test.

    The test options vary depending on the stage detail you selected.

  13. Click the Existing Validation Rules to Apply drop down and select the validation rules you want to associate with the data quality rule.
    You see the names of the selected rules.
  14. Click Add to make the validation rules available for use by the data quality rule.
    You see the selected rules in the Defined Validation Rules section. You can click the X next to a rule to remove it from the list.

    You can set up a new validation rule by clicking Create/Edit Validation Rule and completing the fields in the Validation Rule section. See Set Up Validation Rules for more information.

  15. Click Save.
    You see a message confirming that the data quality rule was saved.

Data Quality Rules for Individual Portfolios

You can run data quality rules for individual portfolios or composites from both the Performance Calculation report and Automation Center using the Data Quality Monitor Engine. Composites are treated as individual portfolios and not enumerated to their membership. The portfolio or composite must be assigned to an enabled data quality rule or be a member of a list which is assigned to a data quality rule. You can run the rules for both Data Quality and Calculation Status process stage validations.

  • Data Quality Process Stage Validations. Automation Center allows you to run data quality rules for individual portfolios or composites for data quality validations. You can build a Data Quality Monitor event for the portfolio/composite or submit an existing event with an override entity. The Data Quality Monitor engine finds enabled data quality rules for that portfolio/composite or for lists with the portfolio/composite as a member and runs validations for the portfolio lists with the entity as a member. For more information, see Schedule a Rule in Automation Center.
  • Calculation Status Process Stage Validations. You can build a Performance Calculation report to run Data Quality Rules for individual portfolios/composites for calculation status validations. You can define a Performance Calculation report for a portfolio/composite or submit an existing Performance Calculation report with an override entity. The Performance Calculation engine finds enabled data quality rules and runs validations for the portfolio/composite or for the lists with the portfolio/composite as members.

Manage Data Quality Rules

While viewing a heat map, you can create a new data quality rule. When you select an individual result at the lowest drill down level, you can also view and edit the underlying data quality rule.

To create a data quality rule:

  1. From the Performance Center window, click Setup and click Data Quality Rules in the left navigation.
    You see the Data Quality Rule workspace in a new tab.
  2. Complete the required fields for a creating a data quality rule.
  3. Click Save.
    You see a message confirming that the data quality rule was saved.

To edit a data quality rule:

  1. From the Performance Center window, click Setup and click Data Quality Rules in the left navigation.
    You see the Data Quality Rule workspace in a new tab.
  2. Search for a different heat map if desired.
  3. In the heat map, select an individual result at the lowest drill down level and then select Edit from the ribbon.
    You see the Data Quality Rule workspace in a new tab with the data quality rule data and the associated validation rules.
  4. Change the fields as needed.
  5. Click Save.
    You see a message confirming that the data quality rule was changed.