SDP-34929 configuration for MC2 to work with RabbitMQ
Enabled RabbitMQ in MC2
It is now possible for MC2 to read from and write to RabbitMQ.
Example entry for RabbitMQ connection:
{
"ConnectionName": "qa.rabbitmq.connection",
"ComponentName": "rabbitmq",
"Parameters": {
"host": "inno-jenever01.eagleinvsys.com",
"port": 5672
}
}
Example entry for RabbitMQ SSL connection:
{
"ConnectionName": "qa.rabbitmq.ssl.connection",
"ComponentName": "rabbitmqssl",
"Parameters": {
"host": "inno-jenever01.eagleinvsys.com",
"port": 5671
},
"SSLTruststoreFile": "${eagle.path.to.root}/estar/tpe/dynamic/mc2/private/rabbitmq/trust-store.p12",
"SSLKeystoreFile" : "${eagle.path.to.root}/estar/tpe/dynamic/mc2/private/rabbitmq/key-store.p12",
"SSLVersion" : "TLSv1.2"
}
SDP-37753 Changes to MC2 Services, MC2 Redis Service and MC2 Kafka Service installation for integration with SMC
Integrated MC2 services with System Management Center
The MC2 services (MC2, REDIS, kafka) are now viewable in the System Management Center Environment/Services window.
SDP-43449 Fixed Parser
Added ability of parsing fixed format files
Parameters for fixed format are optional. Default values are:
"source_format_dialect": {
"headerlines": 1,
"footerlines": 1,
"lineterminator": "n",
"lines_per_row": 1,
"concat_start_pos": 0
}
Supported parameters:
"headerlines" - number of header lines.
"footerlines" - number of footer lines.
"lineterminator" - delimiter between lines.
"lines_per_row" - number of lines on which there is one logical row.
"concat_start_pos" - all lines is trimmed and start at the position specified in the parameter.
There are parameters that can be defined for each column.
Column parameters are optional. Default values are:
"vocabulary": { "COLUMN_NAME": {"startpos": 0, "colwidth": 0} }
Supported column parameters:
"startpos" - start position of column.
"colwidth" - column width
The startpos parameter can be omitted if the required column is located immediately after the end of the previous column.
SDP-44788 New Metadata - genericissueanalytic
Created genericissueanalytic ontology
The Ontology for genericissueanalytic resource was created.
Files added:
estar/tpe/dynamic/metadata/core/ontology/reference/genericissueanalytic.json estar/tpe/dynamic/metadata/core/ontology/reference/genericissueanalytic-models.json estar/tpe/dynamic/metadata/core/ontology/reference/genericissueanalytic_in.json
Extract method is supported for genericissueanalytic resource.
Available filters: effectivedate/fromeffectivedate/toeffectivedate entityfilter fromdate/todate sourcename securityalias processingsecuritytype securitytype investmenttype releasestatus primaryassetid ticker releasedate/fromreleasedate/toreleasedate heldonlyfilter securitysubtype
Mashups used: interfaces genericsmf
SDP-46509 No header/footer in extract if outputorderbyexpr and allowemptyfiles are enabled and source dataset is empty
Corrected EDS issue with allowemptyfiles = Y and an outputorderbyexpr
Corrected an EDS issue with no header or footer in extract when allowemptyfiles = Y and an outputorderbyexpr is specified and no data returned
SDP-46542 Implement into Ontology Canonical Model Output the DB precision and scale
Updated precision and scale values in resourceinfo and model
Added changes to show DB precision and scale values in resourceinfo. Added changes to show in model canonical precision and scale values defined in inventory file. Updated resource validator to check precision and scale.
Example:
{
"element_id": "bidSpreadFactor",
"path": "bidSpreadFactor",
"type": "number",
"type_dialect": { "precision": "26", "scale": "2" } ,
"source_type_dialect": { "precision": "38", "scale": "12" } ,
"label": "Bid Spread Factor element", "definition": ""
},
SDP-46743 Fix data encoding/escaping of special characters in pipe-delimited files
New options introduced for CSV OutputFormatDialect
New options have been introduced for CSV OutputFormatDialect:
"quoting": "QUOTE_ALL"
In this case all values and header will be wrapped in double quote.
Without this option specified only values with delimiter and double quote will be wrapped in double quote.
<taskParameter>
<name>OutputFormatDialect</name>
<value>
{"delimiter":"|","hasheader":"Y", "quoting": "QUOTE_ALL"}
</value>
</taskParameter>
SDP-47026 Implement TSR generation on EDS side
Updated EDS to generate TSR to match EagleML
Updated EDS to generate TSR to match format, name pattern, and delivery options of EagleML TSR, and to include error information.
SDP-47146 Add ability for Catalog API to return available methods
Added ability for Catalog API to return available methods
Added ability for Catalog API to return available methods including custom ontology.
Example of new result:
URL/eagle/v2/metadata/catalog
{
"resources": [
{ "id": "codecategory",
"url": "/eagle/v2/metadata/models/codecategory",
"methods": [ "extract" ],
"scope": "core" }
]
}
SDP-47398 EDS: additional date-time formats in ExtractNamePatternEx parameter
Added optional format codes to 5 keywords for the extractnamepatternex parameter in EDS.
Added optional format codes to 5 keywords for the extractnamepatternex parameter in EDS:
@EFFECTIVEDATE
@FROMEFFECTIVEDATE
@TOEFFECTIVEDATE
@DATETIMESTAMP
@DATESTAMP
All 5 are supported with Distribution Profile distributions.
The first 3 are available with default distribution.
See Date Formats in OutputFile Name for full details
SDP-47575 Research and document how to enable password on MC2 REDIS instance
Enabled optional configuration to require REDIS password
Enabled optional configuration to require REDIS password. See: Enabling REDIS Password
Note: the REDIS password configuration is preserved in the All Services deployment (not in the extractservice deployment).
SDP-47578 Change the MC2 installer to preserve the password configuration in REDIS config files if it is present
Update MC2 Installer to preserve REDIS password if enabled
Updated MC2 Installer to preserve REDIS password if the REDIS configuration file requires and specifies a password.
Note: the REDIS password configuration is preserved in the All Services deployment (not in the extractservice deployment).
SDP-47591 Old EDS codetranslation dataset has bad join definition
Changed join for the eds codetranslation object:
inner join PACE_MASTERDBO.CODE_VALUES FROM_CVAL on FROM_CODE.INSTANCE = FROM_CVAL.CODE_INST AND FROM_CODE.INSTANCE = MASTERTABLE.FROM_CODE -> inner join PACE_MASTERDBO.CODE_VALUES FROM_CVAL on FROM_CODE.INSTANCE = FROM_CVAL.CODE_INST AND FROM_CVAL.INSTANCE = MASTERTABLE.FROM_CODE_VALUE
SDP-47759 Review-Create IssueFXRate EDS Feed and configure to run from Model Office
Created issuefxrate EDS Resource
The Ontology for issuefxrate resource was created.
Files added:
estar/tpe/dynamic/metadata/core/ontology/reference/issuefxrate.json estar/tpe/dynamic/metadata/core/ontology/reference/issuefxrate-models.json
Extract method is supported for issueprice resource.
Available filters :
todate fromdate effectivedate fromeffectivedate toeffectivedate processingsecuritytype securitytype investmenttype releasestatus primaryassetid xrefid ticker securitysubtype securityalias sourcename heldonlyfilter
Mashups used: interfaces genericsmf
SDP-47760 Review - Create IssuePrice EDS Feed and configure to run from Model Office
The Ontology for issueprice resource was created.
Files added:
estar/tpe/dynamic/metadata/core/ontology/reference/issueprice.json estar/tpe/dynamic/metadata/core/ontology/reference/issueprice-models.json
Extract method is supported for issueprice resource.
Available filters :
sourcename fromdate todate effectivedate fromeffectivedate toeffectivedate securityalias processingsecuritytype securitytype investmenttype releasestatus primaryassetid xrefid ticker heldonlyfilter securitysubtype
Mashups used: interfaces genericsmf