markdown-chunkerlisted

# Markdown Chunker Skill Use this skill when you need to understand how the `markdown-dedup` workflow parsed markdown files into sections and what the chunk data means. ## Parsing Strategy The workflow uses a **heading-based AST chunker** that processes each file in two passes: 1. **Primary (mistletoe AST)**: If `mistletoe` is importable, the file is parsed to a full AST and sections are extracted from `Heading` nodes. This gives accurate text content free of raw markdown syntax characters. 2. **Fallback (regex)**: If mistletoe is unavailable, a regex-based parser splits on heading lines (`^#{1,6}\s+`) while tracking fenced code blocks to avoid false heading matches inside code examples. Both approaches produce the same output schema. ## Chunk Schema Each chunk object in `md-chunks.json` has: ```json { "file": "workshop/07-your-first-workflow.md", "level": 2, "title": "Steps", "heading_path": ["Write Your First Workflow", "Steps"], "content": "...", "text": "Steps ...", "word_count": 142 } ``` | Field | Description | |---|---| | `file` | Repository-relative path | | `level` | Heading depth (`1`–`6`) | | `title` | Heading text (stripped) | | `heading_path` | Breadcrumb from document root to this section | | `text` | Heading title + content concatenated — used for similarity scoring | | `word_count` | Whitespace-separated word count of `text`; chunks with fewer than 20 words are excluded | ## Cluster Schema Each cluster object in `md-cluste