Cleaning Kafka Service Transaction Logs and Configuration

The steps outlined below should not be done as a routine maintenance or upgrade procedure. These are intended for cases when Kafka Service fails to start or form a Kafka cluster and requests in MC2 and/or EQL/EDS are not being processed.

The following symptoms could be indicating of a problem with Kafka Service configuration, requiring a cleanup of Kafka Service logs:

Topic REGION.TOPICNAME not present in metadata after 60000 ms.

or:

Failed sending the request to Kafka for python rule execution. Topic REGION.eagle.eds.requests.v2 not present in metadata after 60000 ms.

Even in this case, before proceeding with log removal, check that Kafka Services is started on ALL servers on the region on which MC2 is running.

The steps for cleaning the logs are to be followed exactly. Cleaning the Kafka Service logs and configuration will result in loss of all messages and data which are in flight and currently processing in MC2.

To clean the Kafka Service logs and configuration follow these steps:

  1. Stop MC2 and kafka service on all nodes on which MC2 is running:

cd eaglemgr

./stop starweb extractservicelb extractserviceworker kafkaservice

2. Only after the services are stopped on ALL nodes, remove the kafka configuration and logs:

cd estar/tpe/servers/kafkaservice

rm -rf kafka zookeeper

cd estar/tpe/cfg/kafkaservice

rm -rf kafka zookeeper

NOTE: Do not start kafka service on any of the nodes before the above steps are completed on ALL nodes.

3. Start Kafka Service on all nodes:

cd eaglemgr

./start starweb kafkaservice

This has to be done on all nodes without big delays. Preferably all kafka service nodes have to be started on all nodes within 1 minute.

4. After Kafka Service is started on all nodes start MC2 on all nodes:

./start starweb extractservicelb extractserviceworker