Manage Extract Services

Owned by Octavian Chiriac
Last updated: Jun 28, 2024 by Pamela Totas (Deactivated)
5 min read

Send an Extract to a JMS Destination

Extract Services can send the generated extract to a JMS destination. Before sending the extracts to a JMS destination

To send an extract to a JMS destination:

  1. Configure the connection to the Message Broker as described in Configure a JMS Connection. 

  2. Configure the destination queue or topic in a stream in Message Center Console as shown in Configure a JMS Destination Stream.

  3. Change the w_config_custom.inc to configure the JMS stream name as shown in Configure Extract Service JMS Stream Name.

  4. Restart the Extract Service (MC2) after completing the configuration.

Some examples of REST and SOAP requests are shown below.

When Extract Service publishes the messages it adds the following JMS Headers to each message:

  • EagleExtractRecordCount - the number of records in the message.

  • EagleExtractSequenceNumber - the sequence number of the message when nuggetizing is enabled.

  • EagleExtractFileName - the file name generated for the extract.

  • EagleCorrelationId - the correlation id of the extract request.

Configure a JMS Connection

The JMS Connection is configured by adding a JMS Connection configuration JSON file in the estar/tpe/dynamic/mc2/eaglejms/connections folder. The samples below show how to configure the JMS connections to ActiveMQ and WebSphere MQ servers. For a more detailed description see #JMS Messaging.

To Configure an ActiveMQ Connection:

  1. If there is no ActiveMQ connection already configured, add a new file estar/tpe/dynamic/mc2/eaglejms/connections/extract_jms_connection.json with the following content:

    Sample ActiveMQ Configuration

    { "Region": "oraclesid", /*Database name (Oracle SID) on which the Extract Service is running */ "Connections": [ { "ConnectionName": "extract.jms.connection", "ComponentName": "amq", "Parameters": { "brokerURL": "tcp://activemqhostname:activemqport" } } ] }

    Change the Region parameter to match the oracle SID the Eagle Extract Service is running with.

  2. Change the brokerURL with the specific ActiveMQ instance parameters. The ActiveMQ hostname and port number should be provided by the ActiveMQ broker administrators.

Configure a WebSphere MQ Connection

To configure a websphere MQ Connection: 

  1. If there is no WebSphere MQ Connection already configured, add a new file estar/tpe/dynamic/mc2/eaglejms/connections/extract_jms_connection_wmq.json with the following content:

    Sample WebSphere MQ Connection Configuration

    { "Region": "oraclesid", /*Database name (Oracle SID) on which the Extract Service is running */ "Connections": [ { "ConnectionName": "extract.jms.connection", /* connection name */ "ComponentName": "wmq", /* name of component */ "Parameters": { "transportType": 1, /* sets the transport type */ "port": 1414, /* sets the port for a client connection */ "channel": "DEV.APP.SVRCONN", /* sets the name of the channel - applies to client transport mode only */ "queueManager": "QM1", /* sets the name of the queue manager */ "HostName": "WebSphere MQ host name" /* Sets the name of the host (or IP) */ , "CCSID": 1 /* optional: sets the character set to be used when connecting to the queue manager */ , "clientReconnectTimeout": 10 /* optional: set the amount of time, in seconds, that a client connection will attempt to reconnect */ }, "UseConnectionPool": true, /* optional */ "SessionCacheSize": 50 /* optional: size of connection pool */ } ] }

Change the following parameters:

  1. Set Region to the Oracle SID running the Eagle Extract Service.

  2. Set port to the WebSphere MQ port number.

  3. Set channel to the channel name on the WebSphere MQ.

  4. Set queueManager  to the name of the Queue Manager.

  5. Set HostName to the host name or IP address of the Queue Manager.

In order to use WebSphere MQ, install the WebSphere MQ Client with Java classes.

SSL Connections

To configure a WebSphere MQ SSL Connection add the "SSLCipherSuite" parameter to the "Parameters" section in the connection definition:

      "SSLCipherSuite": "TLS_RSA_WITH_AES_128_CBC_SHA256"

The private keys and public keys should be places in the eagle/estar/tpe/dynamic/mc2/

Eagle will add out of the box support for easier configuration of truststore and keystores for JMS connections.

Configure a JMS Destination Stream

Open Message Center Console and create a MC2JMS stream with the following parameters:

Set the following parameters:

  • Instance - set to the MC2 instance with AppID 10000.

  • Connection Name - the name of the connection as defined in the JSON connection definition file ConnectionName parameter. Use extract.jms.connection if you used the sample JSON Connection files above. 

  • Destination Type - select QUEUE or TOPIC. 

  • Destination Name - the queue or topic name.

  • JMS Connection User and JMS Connection Password - the credentials used to connect to ActiveMQ.

Configure Extract Service JMS Stream Name

Add to eagle/estar/tpe/dynamic/msgcenter/eagle_ml-2-0_custom_cm//w_config_custom.inc a W_JMS_EXTRACT_STREAM variable with the name of the stream created above (mc2_extract_jms_publish):

w_config_custom.inc configuration
<CODE> :W_JMS_EXTRACT_STREAM: := 'mc2_extract_jms_publish'; :W_JMS_EXTRACT_RETRIES: := 5; :W_JMS_EXTRACT_RETRYINTERVAL: := 30; </CODE>

You can use the W_JMS_EXTRACT_RETRIES and W_JMS_EXTRACT_RETRYINTERVAL to redefine the number of delivery retries in case of an error and the interval in seconds between retries.

Specify the Outbound MC2JMS stream name (W_JMS_EXTRACT_STREAM) in the RTR as the JMSExtractStream task parameter.

REST and Soap Extract Examples

Send the REST Extract Results to a JMS Destination

To send the extract to the JMS destination defined above, add a ExtractDeliveryMethod=JMS to the REST request as in this example:

http://eagle.webserver.com/eagle/v2/entities?maxrows=10&OutputFormat=json&ExtractDeliveryMethod=JMS

Send the SOAP Extract Results to a JMS Destination

To send the extract results to the JMS destination add a ExtractDeliveryMethod task parameter with value JMS to the RunTaskRequest:

Sample RunTaskRequest with JMS Delivery
<soc:Envelope xmlns:soc="http://schemas.xmlsoap.org/soap/envelope/"> <soc:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <runTaskRequest xmlns="http://www.eagleinvsys.com/2011/wsdl/EagleML-2-0"> <EagleML xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequestSync" eaglemlType="RunTaskRequest" eaglemlVersion="2.0"> <header> <messageId>CC59ADD40E1E069</messageId> <sentBy>http://www.client.com</sentBy> <sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_out_extract_service</sendTo> <replyTo>http://www.client.com/Eagle Control Messages</replyTo> </header> <taskIdentifier> <correlationId>ENTITY_JMS_1</correlationId> <businessTaskId>ENTITY_JMS_1</businessTaskId> </taskIdentifier> <taskTypeEnum>LOAD</taskTypeEnum> <taskParameters> <taskParameter> <name>ActionType</name> <dataType>S</dataType> <value>EXTRACT</value> </taskParameter> <taskParameter> <name>StreamName</name> <dataType>S</dataType> <value>eagle_ml-2-0_default_out_extract_service</value> </taskParameter> <taskParameter> <name>FeedType</name> <dataType>S</dataType> <value>ENTITYEXTRACT</value> </taskParameter> <taskParameter> <name>maxrows</name> <dataType>I</dataType> <value>10</value> </taskParameter> <taskParameter> <name>outputformat</name> <dataType>S</dataType> <value>JSON</value> </taskParameter> <taskParameter> <name>ExtractDeliveryMethod</name> <dataType>S</dataType> <value>JMS</value> </taskParameter> </taskParameters> </EagleML> </runTaskRequest> </soc:Body> </soc:Envelope>

To split the extract in smaller messages and send to JMS, add the RecordCountPerNugget and DirectTransform=N parameters.

RunTaskRequest - Nuggets - Send to JMS
<soc:Envelope xmlns:soc="http://schemas.xmlsoap.org/soap/envelope/"> <soc:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"> <runTaskRequest xmlns="http://www.eagleinvsys.com/2011/wsdl/EagleML-2-0"> <EagleML xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="RunTaskRequestSync" eaglemlType="RunTaskRequest" eaglemlVersion="2.0"> <header> <messageId>CC59ADD40E1E069</messageId> <sentBy>http://www.client.com</sentBy> <sendTo>http://www.eagleinvsys.com/eagle_ml-2-0_default_out_extract_service</sendTo> <replyTo>http://www.client.com/Eagle Control Messages</replyTo> </header> <taskIdentifier> <correlationId>ENTITY_NUGGET_JMS1</correlationId> <businessTaskId>ENTITY_NUGGET_JMS1</businessTaskId> </taskIdentifier> <taskTypeEnum>LOAD</taskTypeEnum> <synchronousExecution>no</synchronousExecution> <taskParameters> <taskParameter> <name>ActionType</name> <dataType>S</dataType> <value>EXTRACT</value> </taskParameter> <taskParameter> <name>StreamName</name> <dataType>S</dataType> <value>eagle_ml-2-0_default_out_extract_service</value> </taskParameter> <taskParameter> <name>FeedType</name> <dataType>S</dataType> <value>ENTITYEXTRACT</value> </taskParameter> <taskParameter> <name>maxrows</name> <dataType>I</dataType> <value>50</value> </taskParameter> <taskParameter> <name>outputformat</name> <dataType>S</dataType> <value>JSON</value> </taskParameter> <taskParameter> <name>RecordCountPerNugget</name> <dataType>S</dataType> <value>5</value> </taskParameter> <taskParameter> <name>DirectTransform</name> <dataType>S</dataType> <value>N</value> </taskParameter> <taskParameter> <name>ExtractDeliveryMethod</name> <dataType>S</dataType> <value>JMS</value> </taskParameter> </taskParameters> </EagleML> </runTaskRequest> </soc:Body> </soc:Envelope>

Send the TSR and ACK to a JMS Destination

To send the TSR to a JMS Destination, use the ReplyDeliverMethod=JMS and JMSReplyDeliveryStream=stream name parameters.

To send the ACK to a JMS Desitnation, use the ACKDeliveryMethod=JMS and JMSACKDeliveryStream=stream name parameters.