JMS Configuration Recipes

Inbound Queue MC2JMS Stream With Concurrent Processing

The MC2JMS stream for consumption from a queue requires a connection with a pool. MC2 starts consumers for all workers in the cluster and processes the messages in parallel on all workers.

To configure MC2 to consume messages from a queue and process messages in parallel:

Configure the pooled connection to WebSphere or ActiveMQ in a .json file in estar/tpe/dynamic/mc2/eaglejms/connections.

The following ComponentNames for ActiveMQ and WebSphere MQ are defined in the Eagle MC2 out of the box: wmq-no-pool, wmqssl-no-pool, wmq-tds, eagle-amq-tds, eagle-amqssl-no-pool and eagle-amq-no-pool. For example for ActiveMQ, create a amq.json file with the following content:

On this page

1 Inbound Queue MC2JMS Stream With Concurrent Processing
2 Inbound Queue MC2JMS Stream - Preserving The Sequence (available starting July 2019 MC2 Release)
3 Durable Topic Subscription (available starting July 2019 MC2 Release)
4 Publishing To Queue or Topic

ActiveMQ Pooled Connection Configuration

{
    "Region": "BND13XLN", /*Oracle SID of region name to which this config belongs */
    "Connections": [
           {
            "ConnectionName": "activemq.pool", /* name of ActiveMQ connection to be used in the stream*/
            "ComponentName": "eagle-amq", /* name of component */
            "Parameters": {
                "brokerURL": "tcp://eglwwvm22.eagleinvsys.com:61616" /* connection url */
            }
     ]
}

2. Note that the eagle-amq Eagle defined core component is used. If the connection should be encrypted, set ComponentName to eagle-amqssl.
One connection JSON definition file can contain multiple Connection definitions.
The region is the Oracle database SID to which MC2 is connecting (Eagle Oracle Database server).
For WebSphere MQ, the connection has all parameters required for WebSphere MQ connections - port, channel, queueManager name, host name:

WebSphere MQ Non-pooled Connection Configuration

        {
            "ConnectionName": "wmq.pool",
            "ComponentName": "wmq",
	        "Parameters": {
    			"transportType": 1,
			    "port": 1414,
			    "channel": "S_eglwwvm24c",
			    "queueManager": "QM_eglwwvm24c",
			    "HostName": "10.130.36.42"
            }
        }

The ComponentName for pooled connections is wmq. If an SSL connection has to be used use wmqssl.

3. Create an inbound MC2JMS stream in Message Center Console:

4. Set the Connection Name parameter to the ConnectionName used in the connection configuration file.

5. The Destination Type should be QUEUE or ALIAS. If the parameter is set to QUEUE then Destination Name should be set to the physical queue name as defined in ActiveMQ or WebSphere MQ queue manager.

If the Destination Type is set to ALIAS, the name should be an alias defined in the aliases definition file, as described in JMS Logical Names Configuration (Aliases)

Sequential Reading parameter should be set to N.
Concurrent Consumers can be set to 20, which means that one MC2 Worker processes concurrently 20 messages from the queue. If there are 8 MC2 workers (this corresponds to four servers running MC2), MC2 processes 160 messages from the queue concurrently. If this parameter is not set, MC2 processes one message per worker. For example, for 8 workers only 8 messages are processed concurrently.

Inbound Queue MC2JMS Stream - Preserving The Sequence (available starting July 2019 MC2 Release)

To preserve the sequence, make sure to consume messages for one consumer only. The configuration for this type of consumption is the same as the configuration for Queue Concurrent Processing with the following differences:

Set the Sequential Reading parameter to Y. This indicates that MC2 runs only for one consumer in the MC2 cluster.
Set the Concurrent Consumers to 1 or leave blank. This configuration ensures that the messages are processed strictly sequentially.
To improve the performance and have some of the messages processed in parallel, define a Splitting Rule for the JMS stream. If the Split Type parameter in the Message Center Console Stream is defined as Rule Based and a rule is selected, MC2 uses the Split Rule to calculate the key for each record in the message and persists it in its local topic (currently kafka) for further processing. The records/messages with the same key are processed sequentially. Records with different keys can be processed concurrently.

Durable Topic Subscription (available starting July 2019 MC2 Release)

The configuration for the Durable Topic Subscription is almost the same as the configuration for Inbound Queue - Preserving The Sequence, with the following differences:

Set the Destination Type to TOPIC or ALIAS.
Eagle MC2 provides the following Component Names configured specifically for Durable Topic Subscriptions: wmq-tds & eagle-amq-tds.
Set the Durable Subscription Name to a name used for the durable topic subscription.
If Durable Subscription Name is set, MC2 limits the number of consumers to one for the whole cluster. To improve the performance you are strongly advised to use Rule Based splitting with a Splitting Rule.

The Client ID set on the connection is the SID-stream_name, where SID is the oracle database SID MC2 is connected to, and stream_name is the stream name for which the subscription is created.

Publishing To Queue or Topic

Publishing to a JMS Queue or Topic is implemented in MC2 Rules with a new element named SENDADAPTER. For simple extracts, Generic Extracts, Generic Loader distribution of data, TSRs and ACKs this is implemented in EJM distribution rules.

To configure the destination to be used for publishing, you need to create a MC2JMS Outbound stream. This stream is used as a definition of the destination, that is, it defines the JMS connection to be used, the destination name, the credentials, if required, and other parameters defining the connection to JMS. The RuleSet file and the Stream Format is ignored.

When configuring the stream to be used for publishing to JMS, you must use a pooled JMS component. The standard components delivered as part of release for ActiveMQ and WebSphere MQ are wmq, wmqssl, eagle-amqssl and eagle-amq. For example, the connection definition can be:

WebSphere MQ Non-pooled Connection Configuration

        {
            "ConnectionName": "wmq.pool",
            "ComponentName": "wmq",
	        "Parameters": {
    			"transportType": 1,
			    "port": 1414,
			    "channel": "S_eglwwvm24c",
			    "queueManager": "QM_eglwwvm24c",
			    "HostName": "10.130.36.42"
            }
        }

Configure the Outbound MC2JMS stream definition using the above defined connection as shown below: