To onboard a new tenant for EDS Data Rules Kafka Consumption the following steps should be followed:
Create a COPS ticket to create the new topics for tenant:
dna.kafka.client.tenant.prod (80 partitions, 1-week retention)
dna.kafka.client.tenant.prod.schema (1 partition, 1-year retention)
dna.kafka.client.tenant.prod.processed (1 partition, 1-week retention)
...
Request to add CONFLUENT_API_KEY, CONFLUENT_API_SECRET, CONFLUENT_API_KEY_WEST, CONFLUENT_API_SECRET_WEST to vault secrets secrets-tenant-tenantin both eastus2 and westus2 Azure zones for service sd-svc-dna-kafka-client. The prod part of the topic name designates the tenant environment. Can also be dev or stage.
The producer’s secrets should be shared with the producer.
Request the consumer group (client id) dna-kafka-consumer-tenant (replace tenantwith the tenant id) to have access to write and read from all tenant topics.
You can use 2. Onboard the tenant to Data Rules. See Data Rules Implementation Process
3. If data needs to be loaded to existing tables, please follow https://eagleinvsys.atlassian.net/browse/COPS-11087 as a template.
2. Onboard the tenant to EDS.
3/wiki/spaces/SOLUTIONSDEL/pages/3620963123/Load+to+existing+SnowFlake+DB+tables
4. If the producer will use producer side schema validation, publish the schema to dna.kafka.client.tenant.prod.schema
45. Create a COPS ticket to use the DNA Kafka Client swagger UI to create the Kafka Consumer configuration as:
...
In the above configuration replace TENANT with the tenant id. Replace BOOTSTRAP SERVER FOR CONFLUENT WEST CLUSTER and BOOTSTRAP SERVER FOR CONFLUENT EAST CLUSTER with the corresponding bootstrap server and port provided by DEVOPS.
The schemadrift parameter can be added to envConfig section if needed. Please check Schema Drift