databricks-rate-limits

# Databricks Rate Limits ## Overview Handle Databricks API rate limits with exponential backoff, token-bucket queuing, and idempotent job submissions. The API returns HTTP 429 with a `Retry-After` header when limits are exceeded. The SDK has built-in retries for transient errors, but custom logic is needed for bulk operations. ## Prerequisites - `databricks-sdk` installed - Understanding of async patterns for batch operations ## Instructions ### Step 1: Understand Rate Limit Tiers Databricks enforces per-endpoint, per-workspace rate limits. | API Category | Approx. Limit | Notes | |-------------|---------------|-------| | Jobs API (create/run) | ~10 req/sec | Per workspace | | Jobs API (list/get) | ~30 req/sec | Read endpoints more generous | | Clusters API | ~10 req/sec | Create/start are expensive | | DBFS / Files API | ~10 req/sec | Uploads have 1MB/5MB size limits | | SQL Statement API | ~10 concurrent | Concurrent execution limit | | Unity Catalog | ~100 req/min | Permission checks add up fast | | Model Serving | Varies | ITPM/OTPM/QPH limits per endpoint | ```python from databricks.sdk.errors import TooManyRequests, ResourceConflict w = WorkspaceClient() try: w.jobs.run_now(job_id=123) except TooManyRequests as e: print(f"Rate limited. Retry after: {e.retry_after_secs}s") except ResourceConflict as e: print(f"Conflict (409): {e.message}") # Job already running ``` ### Step 2: Exponential Backoff with Jitter ```python import time import random from f...