kafka-topic-designer

Solid

Designs and optimizes Apache Kafka topics and configurations

AI & Automation 1,919 stars 122 forks Updated 2 weeks ago MIT

Install

View on GitHub

Quality Score: 91/100

Stars 20%
100
Recency 20%
90
Frontmatter 20%
70
Documentation 15%
83
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Kafka Topic Designer ## Overview Designs and optimizes Apache Kafka topics and configurations. This skill provides expertise in topic architecture, partitioning strategies, and producer/consumer configuration for optimal streaming performance. ## Capabilities - Topic naming convention design - Partition strategy optimization - Replication factor recommendations - Retention policy configuration - Compaction strategy design - Schema registry integration - Consumer group design - Throughput capacity planning - Security configuration (ACLs, encryption) ## Input Schema ```json { "requirements": { "throughputMBps": "number", "messageSize": "number", "retentionDays": "number", "orderingRequirements": "string" }, "existingTopics": ["object"], "clusterConfig": "object" } ``` ## Output Schema ```json { "topicDesign": { "name": "string", "partitions": "number", "replicationFactor": "number", "configs": "object" }, "schemaDefinition": "object", "producerConfig": "object", "consumerConfig": "object" } ``` ## Target Processes - Streaming Pipeline - ETL/ELT Pipeline (CDC) - Feature Store Setup ## Usage Guidelines 1. Define throughput and latency requirements 2. Specify message size and retention needs 3. Document ordering requirements (per-partition, global) 4. Include existing topic configurations for consistency ## Best Practices - Use consistent naming conventions across all topics - Size partitions based on consumer paral...

Details

Author
a5c-ai
Repository
a5c-ai/babysitter
Created
10 months ago
Last Updated
2 weeks ago
Language
JavaScript
License
MIT

Similar Skills

Semantically similar based on skill content — not just same category

AI & Automation Listed

kafka-event-driven-design

Kafka event-driven architecture designer and reviewer, at the application/client layer. ALWAYS use when designing, reviewing, or troubleshooting how a service produces or consumes Kafka events — topic and partition-key design, producer and consumer client configuration, consumer group topology, event schema definition (Avro/Protobuf/JSON Schema), Schema Registry compatibility modes, idempotent consumption, dead letter queues, exactly-once semantics, backpressure, and consumer lag. Use even for "just publish an event" — Kafka's partition-ordered-not-globally- ordered semantics, at-least-once default delivery, and consumer rebalance storms are the source of most production event-driven bugs. NOT for cluster operations: broker sizing and tuning, KRaft or ZooKeeper administration, partition reassignment and rebalancing tooling, quotas, ACL administration, MirrorMaker/replication topology, upgrade runbooks, or Kafka Streams and Kafka Connect internals.

30 Updated 1 months ago
johnqtcg
AI & Automation Listed

public-kafka-expert-base

Kafka 知识基座。覆盖 Topics/Partitions、Consumer Groups、Producer/Consumer Config、Serialization、Exactly-Once、Rebalancing。供 devlab-middleware-expert 通过 extends 继承。

26 Updated 2 months ago
seed-forge
AI & Automation Listed

confluent-kafka-developer

Kafka/Confluent developer skill for design, explanation, diagrams, planning, and review — designs event/topic/schema models, explains Kafka/Kafka Streams/ksqlDB/Flink concepts, draws architecture and data-flow diagrams, plans producer/consumer/streaming implementation work, and reviews Kafka client and Streams topology code. Searches official docs (docs.confluent.io, kafka.apache.org, developer.confluent.io) and community sources (Confluent Community Forum, Stack Overflow, relevant GitHub issues/discussions) before answering. Use when explaining a Kafka concept, designing an event-driven system, drawing a Kafka/Streams diagram, planning Kafka feature work, or reviewing Kafka producer/consumer/Streams/Connect code.

1 Updated today
thititongumpun