Inbound Queue MC2JMS Stream With Concurrent Processing
The MC2JMS stream for consumption from a queue should use a connection with no pool. MC2 will start consumers on all workers in the cluster and will process the messages in parallel on all workers.
To configure MC2 to consume messages from a queue and process messages in parallel:
- Configure the non-pooled connection to WebSphere or ActiveMQ in a .json file in estar/tpe/dynamic/mc2/eaglejms/connections. The following non-pooled ComponentNames for ActiveMQ and WebSphere MQ are defined in the Eagle MC2 out of the box: wmq-no-pool, wmqssl-no-pool, eagle-amqssl-no-pool and eagle-amq-no-pool. For example for ActiveMQ, create a amq.json file with the following content:You can note that we use the eagle-amq-no-pool Eagle defined core component. If the connection should be encrypted, set ComponentName to eagle-amqssl-no-pool.ActiveMQ Non-pooled Connection Configuration
{ "Region": "BND13XLN", /*Oracle SID of region name to which this config belongs */ "Connections": [ { "ConnectionName": "activemq.nopool", /* name of ActiveMQ connection to be used in the stream*/ "ComponentName": "eagle-amq-no-pool", /* name of component */ "Parameters": { "brokerURL": "tcp://eglwwvm22.eagleinvsys.com:61616" /* connection url */ } ] }
One connection JSON definition file can contain multiple Connection definitions.
The region is the Oracle database SID to which MC2 is connecting (Eagle Oracle Database server).
For WebSphere MQ the connection will have all parameters required for WebSphere MQ connections - port, channel, queueManager name, host name:
{
"ConnectionName": "wmq.nopool",
"ComponentName": "wmq-no-pool",
"Parameters": {
"transportType": 1,
"port": 1414,
"channel": "S_eglwwvm24c",
"queueManager": "QM_eglwwvm24c",
"HostName": "10.130.36.42"
}
}
The ComponentName for non-pooled connections is wmq-no-pool. If an SSL connection has to be used use wmqssl-no-pool.
2. Create an inbound MC2JMS stream in Message Center Console:
Set the Connection Name parameter to the ConnectionName used in the connection configuration file.
The Destination Type should be QUEUE or ALIAS. If the parameter is set to QUEUE then Destination Name should be set to the physical queue name as defined in ActiveMQ or WebSphere MQ queue manager. If the Destination Type is set to ALIAS, the name should be an alias defined in the aliases definition file, as described in JMS Logical names configuration (Aliases)
Sequential Reading parameter should be set to N.
Concurrent Consumers can be set to 20, which means that one MC2 Worker will process concurrently 20 messages from the queue. If we have 8 MC2 workers (this corresponds to four servers running MC2), MC2 will process 160 messages from the queue concurrently. If this parameter is not set, MC2 will process one message per worker, i.e. for 8 workers only 8 messages will be processed concurrently.
Inbound Queue MC2JMS Stream - Preserving The Sequence (available starting July 2019 MC2 Release)
To preserve the sequence we should make sure we consume messages on one consumer only. The configuration for this type of consumption will be the same as the configuration for Queue Concurrent Processing with the following differences:
The Sequential Reading parameter should be set to Y. This will indicate MC2 to run only one consumer in the MC2 cluster.
The Concurrent Consumers should be set to 1 or should be left blank.
This configuration will ensure that the messages are processed strictly sequentially.
To improve the performance and have some of the messages processed in parallel a Splitting Rule should be defined for the JMS stream. If the Split Type parameter in the Message Center Console Stream is defined as Rule Based and a rule is selected, MC2 will use the Split Rule to calculate the key for each record in the message and will persist it in its local topic (currently kafka) for further processing. The records/messages with same key will be processed sequentially. The records with different keys can be processed concurrently.
Durable Topic Subscription (available starting July 2019 MC2 Release)
The configuration for the Durable Topic Subscription is almost the same as the configuration for Inbound Queue - Preserving The Sequence, with the following differences:
The Destination Type should be set to TOPIC or ALIAS.
The Durable Subscription Name should be set to a name used for the durable topic subscription.
If Durable Subscription Name is set, MC2 will limit the number of consumers to one for the whole cluster. To improve the performance it is strongly advisable to use Rule Based splitting with a Splitting Rule.
NOTE: The Client ID set on the connection will be the "SID-stream_name", where SID is the oracle database SID MC2 is connected to and stream_name is the stream name for which the subscription is created.
Publishing To Queue or Topic
The publishing to a JMS Queue or Topic is implemented in MC2 Rules with a new element named SENDADAPTER For simple extracts, Generic Extracts, Generic Loader distribution of data, TSRs and ACKs this is implemented in EJM distribution rules.
To configure the destination to be used for publishing a MC2JMS Outbound stream has to be created. This stream is used as a definition of the destination, i.e. it defines the JMS connection to be used, the destination name, the credentials, if required, and other parameters defining the connection to JMS - the RuleSet file and the Stream Format is ignored.
When configuring the stream to be used for publishing to JMS a pooled JMS component must be used. The standard components delivered as part of release for ActiveMQ and WebSphere MQ are wmq, wmqssl, eagle-amqssl and eagle-amq.
For example the connection definition can be:
{
"ConnectionName": "wmq.pool",
"ComponentName": "wmq",
"Parameters": {
"transportType": 1,
"port": 1414,
"channel": "S_eglwwvm24c",
"queueManager": "QM_eglwwvm24c",
"HostName": "10.130.36.42"
}
}
The Outbound MC2JMS stream definition using the above defined connection should be configured like this:
