qdrant-sliding-time-window

Solid

Guides sliding time window scaling in Qdrant. Use when someone asks 'only recent data matters', 'how to expire old vectors', 'time-based data rotation', 'delete old data efficiently', 'social media feed search', 'news search', 'log search with retention', or 'how to keep only last N months of data'.

AI & Automation 222 stars 28 forks Updated yesterday Apache-2.0

Install

View on GitHub

Quality Score: 87/100

Stars 20%
78
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

# Scaling with a Sliding Time Window Use when only recent data needs fast search -- social media posts, news articles, support tickets, logs, job listings. Old data either becomes irrelevant or can tolerate slower access. Three strategies: **shard rotation** (recommended), **collection rotation** (when per-period config differs), and **filter-and-delete** (simplest, for continuous cleanup). ## Shard Rotation (Recommended) Use when: data has natural time boundaries (daily, weekly, monthly). Preferred because queries span all time periods in one request without application-level fan-out. [User-defined sharding](https://skills.qdrant.tech/md/documentation/scaling/distributed_deployment/?s=user-defined-sharding) 1. Create a collection with user-defined sharding enabled 2. Create one shard key per time period (e.g., `2025-01`, `2025-02`, ..., `2025-06`) 3. Ingest data into the current period's shard key 4. When a new period starts, create a new shard key and redirect writes 5. Delete the oldest shard key outside the retention window - Deleting a shard key reclaims all resources instantly (no fragmentation, no optimizer overhead) - Pre-create the next period's shard key before rotation to avoid write disruption - Use `shard_key_selector` at query time to search only specific periods for efficiency - Shard keys can be placed on specific nodes for hot/cold tiering ## Collection Rotation (Alias Swap) Use when: you need per-period collection configuration (e.g., different qua...

Details

Author
qdrant
Repository
qdrant/skills
Created
6 months ago
Last Updated
yesterday
Language
Python
License
Apache-2.0

Integrates with

Bundled in these plugins

Similar Skills

Semantically similar based on skill content — not just same category