databricks-local-dev-loop

# Databricks Local Dev Loop ## Overview Set up a fast local development workflow using Databricks Connect v2, Asset Bundles, and VS Code. Databricks Connect lets you run PySpark code locally while executing on a remote Databricks cluster, giving you IDE debugging, fast iteration, and proper test isolation. ## Prerequisites - Completed `databricks-install-auth` setup - Python 3.10+ (must match cluster's Python version) - A running Databricks cluster (DBR 13.3 LTS+) - VS Code or PyCharm ## Instructions ### Step 1: Project Structure ``` my-databricks-project/ ├── src/ │ ├── __init__.py │ ├── pipelines/ │ │ ├── __init__.py │ │ ├── bronze.py # Raw ingestion │ │ ├── silver.py # Cleansing transforms │ │ └── gold.py # Business aggregations │ └── utils/ │ ├── __init__.py │ └── helpers.py ├── tests/ │ ├── conftest.py # Spark fixtures │ ├── unit/ │ │ └── test_transforms.py # Local Spark tests │ └── integration/ │ └── test_pipeline.py # Databricks Connect tests ├── notebooks/ │ └── exploration.py ├── resources/ │ └── daily_etl.yml # Job resource definitions ├── databricks.yml # Asset Bundle root config ├── pyproject.toml └── requirements.txt ``` ### Step 2: Install Development Tools ```bash set -euo pipefail # Create virtual environment python -m venv .venv && source .venv/bin/activate # Databricks Connect v2 — version MUST match cluster DBR pip install "databricks-...