A Brief Introduction into Apache Kafka
...
Kafka is run as a cluster on one or more servers that can span multiple data-centers.
The Kafka cluster stores streams of records in categories called topics.
Each record consists of a key, a value, and a timestamp.
Kafka has five core APIs but for now we pay attention to 2 ones:
The Producer API allows an application to publish a stream of records to one or more Kafka topics.
The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them.
...
Topics
A topic is a category or feed name to which records are published.
...
Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group.
Consumer instances can be in separate processes or on separate machines.
...
If all the consumer instances have different consumer groups, then each record will be broadcast to all the consumer processes.
...
Guarantees
At a high-level Kafka gives the following guarantees:
...
To setup Kafka-Messaging we have to create back-end configuration part (JSON format) and Eagle Streams (Inbound/Outbound) with MC2KAFKA delivery method.
Make sure that your topic already exists in Kafka instance or ask admin to do it for you.
...
Kafka Connections configuration files should be placed as JSON files in: estar/tpe/dynamic/mc2/mc2kafka/connections/*.json
MC2 will read all files with the extension .json from mc2/mc2kafka/connections/ folder.
...
Message Stream Title | your own stream name |
---|---|
Instance | mc2 instance with its AppID (=10000) |
Delivery Method | MC2KAFKA |
Stream Direction | Outbound |
Delivery Format | choose from list (e.g.: XML) |
Ruleset File | your working ruleset file (or any ruleset to stub this field). The ruleset is not used to transform the files. The stream is just a configuration pointing to the right kafka instance and destination. |
Kafka Instance Name | KafkaName reference name from json connection configuration file (required) |
Topic Name | Your Kafka Topic name (required) |
MC events log | MC log level |
URI parameters | leave it empty (advanced configuration) |
Data Encryption | No (default) / SSL |
SSL TrustStore Password SSL KeyStore Password SSL Key Password | leave it empty (support of SSL (Secure Sockets Layer) and its successor, TLS (Transport Layer Security), which are protocols for establishing authenticated and encrypted connections) |
Example of Kafka-Messaging Producer stream:
...
After creation of Eagle Kafka-Messaging Producer Stream you will be able to send data into Kafka topic by using well known SendAdapter rule element.
For Kafka-Messaging we registered kafkamessagingprotocol.
Example of code snippet:
...
Message Stream Title | the stream name |
---|---|
Instance | mc2 instance with its AppID (=10000) |
Delivery Method | MC2KAFKA |
Stream Direction | Inbound |
Delivery Format | choose from list (e.g.: XML) |
Ruleset File | your working ruleset file to process data being read from topic into :IN_MESSAGE: variable; For EJM workflows use: eagle_ml-2-0_cm/mc2/ejm/workflow/workflow_listener.rsf For simple extracts use: eagle_ml-2-0_cm/out/xml/extract_service.rsf |
Kafka Instance Name | KafkaName reference name from connection.json file (required) |
Topic Name | Your Kafka Topic name (required) |
MC events log | MC log level |
Consumers count | The number of consumers that connect to kafka server (default: 2) |
Auto Offset Reset | What to do when there is no initial offset (uniquely identifies each record) or if an offset is out of range: earliest: automatically reset the offset to the earliest offset latest: automatically reset the offset to the latest offset fail: throw exception to the consumer. (default: earliest) |
Batch records size | Max records to be processed at the same time (default: 1) |
Data Encryption | No (default) / SSL |
URI parameters | leave it empty (advanced configuration) |
SSL TrustStore Password SSL KeyStore Password SSL Key Password | leave it empty (support of SSL (Secure Sockets Layer) and its successor, TLS (Transport Layer Security), which are protocols for establishing authenticated and encrypted connections) |
...
Example of Kafka-Messaging Consumer stream:
...
After creation of Eagle Kafka-Messaging Consumer Stream you will be able to receive data from Kafka topic into predefined :IN_MESSAGE: variable to process it.
...
For each topic, the Kafka cluster maintains a partitioned log that looks like this:
...
Each partition is an ordered, immutable sequence of records that is continually appended to — a structured commit log. The records in the partitions are each assigned a sequential id number called the offset that uniquely identifies each record within the partition.
...
The Kafka instance certificate should be imported to the Java trust store file in:
/estar/tpe/dynamic/mc2/private/mc2kafka/client.truststore.jks
The MC2 client certificate and private key should be imported to the Java keystore file in:
/estar/tpe/dynamic/mc2/private/mc2kafka/client.keystore.jks
The MC2KAFKA stream in Message Center Console should be configured so that:
The Data Encryption parameter is set to SSL
The SSL Trust Store, Key Store and Key password should be set in the SSL TrustStore Password, SSL KeyStore Password & SSL Key Password parameters.
For example:.
...
Authentication Using SASL
...
Example of MC2KAFKA outbound stream (marked SASL-SSL parameters):
...