About the mission
We are looking for a Kafka consultant to evaluate and optimize existing infrastructure while drafting governance for platform operationalization. The role includes responsibilities in design, development, integration, supervision, security, and knowledge transfer.
Read the full description
Role: Kafka Consultant
Job Summary:
As part of a three-month assignment, we are looking for a Kafka consultant. As an expert, you will evaluate and optimize the existing infrastructure and draft the governance related to the operationalization of the platform. You will also participate in resolving a series of specific issues related to the maintenance of the platform.
Key Responsibilities:
Design and Architecture: Participate in the architecture of data flows and integrate Kafka as the backbone of microservices and CDC.
Development: Create and optimize data pipelines, configure topics, partitions, and retention mechanisms. Operationalize these procedures.
Integration: Assess the architecture of existing Kafka Connect connectors and improve monitoring.
Supervision: Set up monitoring tools (e.g., Grafana).
Establish troubleshooting procedures to ensure high availability of clusters.
Security: Secure data exchanges (TLS, SASL, ACL).
Knowledge transfer and training of the technical team.
Required Skills:
Kafka Expertise: Deep mastery of the Apache Kafka ecosystem (Broker, KRaft, Schema Registry).
Frameworks: Practical experience with Kafka Streams and Kafka Connect.
Infrastructures: Knowledge of containerization technologies (Strimzi, Confluent,...).
Architecture: Good understanding of distributed architectures, microservices principles, and event processing.
Knowledge of Ansible.
Optional: Understanding of data stack on streaming (Iceberg, Spark, Flink,...).
Profile:
Significant experience in the production and optimization of Kafka platforms.
Analytical and problem-solving mindset.
What you'll do
- Participate in the architecture of data flows and integrate Kafka as the backbone of microservices and CDC
- Create and optimize data pipelines, configure topics, partitions, and retention mechanisms
- Assess the architecture of existing Kafka Connect connectors and improve monitoring
- Set up monitoring tools (e.g., Grafana)
- Establish troubleshooting procedures to ensure high availability of clusters
- Secure data exchanges (TLS, SASL, ACL)
- Transfer knowledge and train the technical team
What we're looking for
- Deep mastery of the Apache Kafka ecosystem (Broker, KRaft, Schema Registry)
- Practical experience with Kafka Streams and Kafka Connect
- Knowledge of containerization technologies (Strimzi, Confluent, ...)
- Good understanding of distributed architectures and microservices principles
- Knowledge of Ansible
- Optional: Understanding of data stack on streaming (Iceberg, Spark, Flink, ...)
Skills
- Kafka
- Kafka Streams
- Kafka Connect
- Grafana
- Ansible