Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The extract process is initiated by sending a Control Message file to the stream eagle_ml-2-0_default_cm_control_message.

...

Parameter

Example of value

Comments

ActionType

EXTRACT

Constant value for all of Extracts

StreamName

eagle_ml-2-0_default_out_q

Constant value for all of Extracts.
The name of Q-stream to which data is sent from the control-message stream.

FeedType

ENTITYEXTRACT (for Object “Generic Entity”, etc.)

Value should be corresponding to the Object of extracting

OutputFormat

EagleML or TagValue or CSV 


Specifies the output data format for extracts.

If OutputFormat=EagleML, the extract file will be in EagleML format and the file extension will be *.xml

If OutputFormat=CSV, the extract file will be in TagValue format and the file extension will be *.csv

If OutputFormat=TagValue, the extract file will be in TagValue format and the file extension will be *.txt

If OutputFormat is set to a value other than EagleML, CSV or TagValue, the extract file will be in TagValue format and the file extension will be *.xml

If OutputFormat is not specified, the extract file will be in EagleML format and the file extension will be *.xml

...

Parameter

Example of value

Comments

Anchor
ExtractNamePattern
ExtractNamePattern
ExtractNamePattern

ENTITYDATA

Custom name pattern for the output file. Result file name will be “correlation ID_ExtractNamePattern value_records count_current date”.

Anchor
ExtractNamePatternEx
ExtractNamePatternEx
ExtractNamePatternEx


Code Block
languagexml
<taskParameter>
  <name>ExtractNamePatternEx</name>
  <dataType>S</dataType> 
  <value>tst_@FeedType_my.xml</value>
</taskParameter>


Custom name pattern for the output file. Please use the corresponding feedtype as prefix for the parameter name. Result file name will be “tst_SMFEXTRACT_my.xml” for an SMF extract. List of valid expressions:

  • @CorrelationID replaced by initial CorrID
  • @DateTimeStamp – date-time stamp  in format YYYYMMDDHHMISS
  • @DateStamp – date stamp in format YYYYMMDD
  • @RecordCount – count of extracted records
  • @EntityList – passed entity list filter
  • @EntityID  – passed entity id filter
  • @FeedType – passed feed type
  • @entityselectiontype – passed value of “entityselectiontype” task parameter
  • @entityselectiontype2 – passed value of “entityselectiontype2” task parameter
  • @entityselectionvalue – passed value of “entityselectionvalue” task parameter
  • @entityselectionvalue2 – passed value of “entityselectionvalue2” task parameter


Info
titleImportant Note
This parameter is different from 'FeedType_ExtractNamePatternEx' for Generic Extract workflow, despite a similar name. Please use ExtractNamePatternEx only with common extracts, FeedType_ExtractNamePatternEx only with Generic Extract.


Anchor
ExactExtractFileName
ExactExtractFileName
ExactExtractFileName

CUSTOMEXACTNAME

Customizes the exact name for the output file.

The name will be exact, it will not contain the ID and other info.

Anchor
T_StreamName
T_StreamName
T_StreamName

eagle_default_in_csv

If this parameter is present in the CM, the data received in step (eagle_ml-2-0_default_out_q) will be sent to the specified stream. Default value is eagle_ml-2-0_default_out_t.

Anchor
StreamingT_StreamName
StreamingT_StreamName
StreamingT_StreamName

eagle_ml-2-0_my_custom_streaming_t

Forces using a custom streaming_t stream after the q-step. Default value is eagle_ml-2-0_default_out_streaming_t


Info
titleNote
This parameter is valid only for objects which already support streaming extracts


Anchor
QueryProfileName
QueryProfileName
QueryProfileName

 

Enables Element Reduction feature. See more details here

Anchor
Compress
Compress
Compress
N, Y, ZIP, GZIP, B64, GZIP+B64

Enables/Disables compression. Has a global equivalent - DEFAULT_EAGLEML_EXTRACT_COMPRESS
N - no compression
Y – Default method – ZIP
ZIP - file extension will be .zip
GZIP  - file extension will be .gz
B64 - B64 compression - file extension will be .b64
GZIP + B64 - GZIP and B64 compression - file extension will be .gz.b64


Info
titleNote
Please note, that for ExtractDeliveryMethod = JMS your extract will be compressed using GZIP+B64 by default (if you specify Compress = Y).


Anchor
COMPRESS_WITH_CORR_ID
COMPRESS_WITH_CORR_ID
COMPRESS_WITH_CORR_ID

Y or N

If the value of flag is ‘Y’ then result extract file will be compressed to zip (box with Windows OS) or gz (box with Unix OS) archive AND the

both the  zip file name  and the files within the zip gets created with the correlation ID within the name.  

If value set to ‘N’ then zip file name and the files within the zip will not contain the correlation ID within the name.

Anchor
XSDValidation
XSDValidation
XSDValidation

Y or N

If the value is set to ‘Y’ and OutputFormat is not TagValue, XML Schema validation will be performed for outgoing file. Results are shown in T-Stream status messages and Task Status Response messages.

If the value is set to ‘N’, feature will be disabled.

Otherwise, g_bXSDValidation flag value (1 or 0) is used from ‘w_config.inc’ configuration file.

Anchor
SaveSQLQuery
SaveSQLQuery
SaveSQLQuery

Y or N (default)


Code Block
languagexml
<taskParameter>
   <name>SaveSQLQuery</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>


Setting to Y enables saving query files in the cmw_staging_folder

Anchor
RemoveTaskFiles
RemoveTaskFiles
RemoveTaskFiles

Y (default) or N


Code Block
languagexml
<taskParameter>
   <name>RemoveTaskFiles</name>
   <dataType>S</dataType>
   <value>N</value>
</taskParameter>


This parameter toggles deletion of TSR and extract files. When set to Y, there will be no files left in the cmw folders after the extract.

Anchor
DoNotPublishOnError
DoNotPublishOnError
DoNotPublishOnError

Y (default) or N

If XML Schema validation was performed with errors (see XSDValidation parameter) and value is set to ‘N’, forces publish of output file via FTP, SFTP, EMAIL, etc. if specified.

Anchor
sorting
sorting
sorting

SECURITY_ALIAS,UPDATE_SOURCE,ASC

If this parameter is present in the CM, the sorting will be performed by the specified fields in the descending order of priority. The last part of the parameter (the part after the last comma) determines the sort direction (if the value will be empty, the sort direction is taken by default – in ascending order).

Allowable separators: commas, pipes, spaces.

Format of input parameters:

FIELD_1,FIELD_2,...,FIELD_N,SORT_DIRECTION

FIELD_1|FIELD_2|...|FIELD_N|SORT_DIRECTION

FIELD_1 FIELD_2 ... FIELD_N SORT_DIRECTION

FIELD_1,FIELD_2,...,FIELD_N

Anchor
CustomWhere
CustomWhere
CustomWhere


Code Block
languagexml
<taskParameter>
    <name>CustomWhere</name>
    <dataType>S</dataType>
    <value>securitydbo.security_master.security_alias ='15' and rownum < 10</value>
</taskParameter>


This parameter is usually regarded as a filter. See more details about using CustomWhere on this page 


Anchor
SkipGUNZIPOnSaveFile
SkipGUNZIPOnSaveFile
SkipGUNZIPOnSaveFile

Y or N (default)

Set to Y to skip the GUNZIP runpreprocessor when saving file. As a result, the file will remain archived.

Anchor
RemoveEntityDuplicates
RemoveEntityDuplicates
RemoveEntityDuplicates

Y, N (default)

Code Block
languagexml
<taskParameter>
    <name>RemoveEntityDuplicates</name>
    <dataType>S</dataType>
    <value>Y</value>
</taskParameter>


Enables/disables extraction of duplicate entities for entityselectiontype = ENTITYID filter. You do not get duplicate entities if set to 'Y'.

Anchor
UseAllAcctBasis
UseAllAcctBasis
UseAllAcctBasis

Y, N (default)

Code Block
languagexml
<taskParameter>
    <name>UseAllAcctBasis</name>
    <dataType>S</dataType>
    <value>Y</value>
</taskParameter>


Can be used for EntityAccountingBasis extract only. Defines, whether to extract only Primary Accounting Basis (this is default behavior, for UseAllAcctBasis = N) or all Accounting Basis.
Anchor
ForceExtractFileCreation
ForceExtractFileCreation
ForceExtractFileCreation

Y, N (default)

Code Block
languagexml
<taskParameter>
    <name>ForceExtractFileCreation</name>
    <dataType>S</dataType>
    <value>Y</value>
</taskParameter>


Generates an empty extraction file in case you get NO_DATA. Has a global alternative - W_FORCE_EMPTY_EXTRACTFILE
Anchor
CustomExtractLocation
CustomExtractLocation
CustomExtractLocation

folder path

Code Block
languagexml
<taskParameter>
     <name>CustomExtractLocation</name>
     <dataType>S</dataType>
     <value>../../../cmw/custom_extracts/</value>
</taskParameter>
<taskParameter>
      <name>CustomExtractLocation</name>
      <dataType>S</dataType>
      <value>/home0/qa9999app/eagle/estar/tpe/data/msgcenter/cmw/custom_extracts/</value>
</taskParameter>


Defines a custom folder into which the extract is loaded. Folder path should be either relative or absolute starting with the Message Center root directory.
Customizing Extraction of Common Models

Anchor
ExcludeSecurityAlias
ExcludeSecurityAlias
ExcludeSecurityAlias

Y, empty (default)

Disables extraction of security alias in the Asset Identifiers model. Available as a global parameter EXCLUDE_SECURITY_ALIAS. Can be overriden for Warehouse objects by W_INCLUDE_SECALIAS_FORWHS global parameter.

Anchor
IncludeAssetProperties
IncludeAssetProperties
IncludeAssetProperties

Y, N (default)

Code Block
languagexml
<taskParameter>
    <name>IncludeAssetProperties</name>
    <dataType>S</dataType>
    <value>Y</value>
</taskParameter>


Setting this parameter to Y enables extracting Asset Properties model. Please refer to the corresponding data map table of the EagleML object to understand if it contains this model.

Available as a global parameter W_INCLUDE_ASSET_PROPERTIES.

Anchor
IncludeEntityInfo
IncludeEntityInfo
IncludeEntityInfo

ALL, Y, N (default)

A setting for extracts with Entity fields (such as Entity Id, Name, Xref etc).
N: The following fileds are extracted: Entity Id, Entity Name, Entity Type, Xref Account Id, Xref Account Id Type, Xref Class Code
Y: These fields are extracted along with ones above: Long Entity Name, Legal Entity Name, Total Count Share Class, Partition Number
ALL: Physical Partition fields are extracted along with all fields above: PPTRADES, PPHOLDING, PPCASH, PPPERFORM, PPLEDGER
See a more detailed description of these fields.

Available as a global parameter W_INCLUDE_ENTITY_INFO.

Anchor
IncludeExtendedDetails
IncludeExtendedDetails
IncludeExtendedDetails

(default), Y, ALL

Code Block
languagexml
<taskParameter>
    <name>IncludeExtendedDetails</name>
    <dataType>S</dataType>
    <value>ALL</value>
</taskParameter>


When set to Y, provides a more detailed extraction: 


Info
titleNote
Currently available only for the following feedtypes:
  • WRHSCLOSELOTEXTRACT
  • WRHSGLDETAILPOSTINGEXTRACT
  • WRHSNAVMONTHENDEXTRACT
  • WRHSNAVMULTIEXTRACT
  • WRHSOPENLOTEXTRACT
  • WRHSPOSITIONEXTRACT
  • WRHSTRADEEXTRACT
  • TRADEDCAEXTRACT
  • SETTLEDCAEXTRACT​
  • DISTRIBUTIONACTEXTRACT
  • SUBSCRIBEREEDEMEXTRACT
  • CASHPROJECTIONSEXTRACT

When set to ALL, Entity Properties model is fully extracted

---

For Warehouse Trade, Traded Cash Activity and Settled Cash Activity setting IncludeExtendedDetails to Y or ALL enables extraction of 3 additional elements: 

calcTransactionId, calcMasterTransactionId, calcCancelTransactionId

Anchor
IncludeCommonTransactionsInfo
IncludeCommonTransactionsInfo
IncludeCommonTransactionsInfo

Y, N (default). 

Code Block
languagexml
<taskParameter>
   <name>IncludeCommonTransactionsInfo</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>


Adds a commonTransactionsInformation model with values of dates you have used in your Date filter (please note, that not all date filters are supported by your specific feedtype, refer to this table). Has a global parameter alternative - W_INCLUDE_COMMON_TRANSACTIONS_INFO
Feedtype-Specific Parameters
Anchor
TakeBaseCurrencyFromWRHSPosDet
TakeBaseCurrencyFromWRHSPosDet
TakeBaseCurrencyFromWRHSPosDet

Y, N (default)


Code Block
languagexml
<taskParameter>
    <name>TakeBaseCurrencyFromWRHSPosDet</name>
    <dataType>S</dataType>
    <value>Y</value>
</taskParameter>


A parameter exclusively for Warehouse Position extracts. Forces taking the value for baseCurrency from HOLDINGDBO.POSITION_DETAIL.BASE_CURRENCY instead of HOLDINGDBO.POSITION.BASE_CURRENCY field. You can also set it as global. Available as a global parameter W_TAKE_BASE_CURRENCY_FROM_WRHS_POS_DET.

Anchor
GroupBy
GroupBy
GroupBy

comma-separated list of DB fields in DB_TABLE.DB_FIELD format

Code Block
languagexml
<taskParameter>
    <name>GroupBy</name>
    <dataType>S</dataType>     <value>POSITION.ENTITY_ID,POSITION.EFFECTIVE_DATE,LOT_LEVEL_POSITION.SECURITY_ALIAS,
LOT_LEVEL_POSITION.BROKER_CODE,LOT_LEVEL_POSITION.CURRENCY</value>
</taskParameter>


Works with WRHSOPENLOTBROKERROLLUPEXTRACT. Represents the GROUP BY SQL statement. See an example of RTR with this parameter.

Anchor
includeZeroPositions
includeZeroPositions
includeZeroPositions

Y, N (default)

Code Block
languagexml
<taskParameter>
   <name>includeZeroPositions</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>
 


Works with Warehouse Open Lot and Position objects only. When this flag is set to Y, an additional record can be returned for zero position.

For example:

We have back a traded sell for 8\16 then S2P happened and some records were deleted from position table and after all we have the following picture:

Since 8\17 to 10\7 we still have two records for each effective_date where records with Security_alias = 999497938 have 0 shares.

Note: Additional records are returned only if they exist in the following table:

Code Block
languagesql
select *  FROM HOLDINGDBO.SPD_DELTA_LOG SPD_DELTA_LOG
WHERE 1=1
and SPD_DELTA_LOG.SHARE_PAR_VALUE=0
and SPD_DELTA_LOG.ENTITY_ID in ('KMIC_5_1','')

And the record with 0 share has the max effective date for the (EntityId+SourceName+Effectivedate+SecurityAlias)

Anchor
includeShareholderActivity
includeShareholderActivity
includeShareholderActivity

Y, N (default)

Code Block
languagexml
<taskParameter>
   <name>includeShareholderActivity</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>


Adds the Shareholder Activity model to the extract. Available for SUBSCRIBEREEDEMEXTRACT and WRHSNAVMONTHENDEXTRACT feedtypes.

Anchor
includeEntityShareClass
includeEntityShareClass
includeEntityShareClass

Y, N (default) 

Code Block
languagexml
<taskParameter>
   <name>includeEntityShareClass</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>


Toggle Entity Share Class details in the extract. Available for:

  • SUBSCRIBEREEDEMEXTRACT
  • WRHSNAVMULTIEXTRACT
  • WRHSNAVMONTHENDEXTRACT
Anchor
UseFundsOfFunds
UseFundsOfFunds
UseFundsOfFunds

Y, N (default) 


Code Block
languagexml
<taskParameter>
   <name>UseFundsOfFunds</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>


Setting to Y enables Security information extraction for WRHSNAVMONTHENDEXTRACT
Anchor
includeReportDetails
includeReportDetails
includeReportDetails
N (default), Y

If set to Y, the following fields are added to the extract and <navReportDetails> complex node is added to XSD:
Prior NAV (NAV for the previous Valuation Date),
Mkt Value Cash (elements from the Position Detail table in case Base Currency = Local Currency or sum of elements in case Base Currency != Local Currency),
Share Class Level (elements of ARCH_CTRLCTR.NAV_DETAILS table linked with main table via Valuation Date, Entity Id, Share Class),
Total Fund Level (elements of  ARCH_CTRLCTR.NAV_DETAILS сtable linked with main table via Valuation Date, Entity Id for Share Class = 'TF')

  • WRHSNAVMULTIEXTRACT
  • WRHSNAVMONTHENDEXTRACT

This parameter overrides W_INCLUDE_REPORT_DETAILS parameter from w_config.inc

Anchor
includeCustomReportDetails
includeCustomReportDetails
includeCustomReportDetails
N (default), Y If set to Y, the following fields are added to the extract inside the <navReportDetails> complex node.
Share Class Level (elements of ARCH_CTRLCTR.NAV_DETAILS table linked with main table via Valuation Date, Entity Id, Share Class),
Total Fund Level (elements of  ARCH_CTRLCTR.NAV_DETAILS сtable linked with main table via Valuation Date, Entity Id for Share Class = 'TF')
  • WRHSNAVMULTIEXTRACT
This parameter overrides W_INCLUDE_CUSTOM_REPORT_DETAILS parameter from w_config.inc
Anchor
EnableSignageLogic
EnableSignageLogic
EnableSignageLogic

Y, N (default) 

Code Block
languagexml
<taskParameter>
   <name>EnableSignageLogic</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>


Flips the signage for fields that may be extracted with an incorrect sign. Available for Warehouse Trade, Traded Cash Activity and Settled Cash Activity extracts.
Anchor
IncludeCurveAnalytics
IncludeCurveAnalytics
IncludeCurveAnalytics

Y, N (default) 

Code Block
languagexml
<taskParameter>
   <name>IncludeCurveAnalytics</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>


Setting to Y
- extends Warehouse Position extract to use HOLDINGDBO.POSITION_PLEDGE_DETAIL table
- extends Warehouse Performance extract to use PERFORMDBO.PSR_CURVE_ANALYTICS table
Anchor
IncludeAverageNetAssets
IncludeAverageNetAssets
IncludeAverageNetAssets

Y, N (default)

Code Block
languagexml
<taskParameter>
   <name>IncludeAverageNetAssets</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>


Toggle extraction of average net assets extensions from the ESTAR.AVERAGE_NET_ASSETS TABLE. Works only with WRHSNAVMULTIEXTRACT.
Anchor
IncludePortfolioValuation
IncludePortfolioValuation
IncludePortfolioValuation

ALL, Y, N (default) 

Code Block
languagexml
<taskParameter>
   <name>IncludePortfolioValuation</name>
   <dataType>S</dataType>
   <value>ALL</value>
</taskParameter>


Adds the Extended Portfolio Valuation model to your Warehouse Open Lot or Position extract.
Y - partial extraction (fields before the delimiter on the data map table)
ALL - extracts all fields
Anchor
IncludeWashSales
IncludeWashSales
IncludeWashSales

Y, N (default) 

Code Block
languagexml
<taskParameter>
   <name>IncludeWashSales</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>


When set to Y adds Wash Sale Trade Model to Warehouse Open Lot extract. Works only with WRHSOPENLOTEXTRACT
Anchor
IncludeSubscribeReedemCalcFields
IncludeSubscribeReedemCalcFields
IncludeSubscribeReedemCalcFields

Y, N (default)

Code Block
languagexml
<taskParameter>
   <name>IncludeSubscribeReedemCalcFields</name>
   <dataType>S</dataType>
   <value>Y</value>
</taskParameter>


When set to Y additional calculated fields are added to your Subscribe Redeem extract
Anchor
linkedassetonlydelta
linkedassetonlydelta
linkedassetonlydelta
Y, N (default)If this parameter is set to Y, then whenever we do a delta based dictionary extract, we should only look at SECURITY_DICTIONARY_LINK.SECURITY_DATE to determine the linked assets (and their corresponding dict detail) records to extract. There is no need to check dict detail update_date for delta based dictionary extracts in the selection criteria if that flag is set to Y. Available as a global parameter for w_config_custom.inc W_LINKED_ASSET_ONLY_DELTA
Anchor
ExcludeTradeCash
ExcludeTradeCash
ExcludeTradeCash
Y, NWhen set to Y, excludes Trade related cash activity (identified by records with CASH_EVNT_SPAWNED = Y) from the Traded Cash Activity Extract.