databricks-ci-integration

# Databricks CI Integration ## Overview Automate Databricks deployments with Declarative Automation Bundles (DABs) and GitHub Actions. Covers bundle validation, unit testing PySpark transforms locally, deploying to staging on PR, production on merge, and integration testing against live workspaces. Uses `databricks/setup-cli` action and OAuth M2M for secure CI auth. ## Prerequisites - Databricks workspace with service principal (OAuth M2M) - Asset Bundle (`databricks.yml`) configured - GitHub repo with Actions enabled - GitHub environment secrets: `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET` ## Instructions ### Step 1: GitHub Actions — Validate and Test on PR ```yaml # .github/workflows/databricks-ci.yml name: Databricks CI on: pull_request: paths: ['src/**', 'resources/**', 'databricks.yml', 'tests/**'] jobs: validate-and-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: '3.10' - name: Install dependencies run: | pip install pytest pyspark delta-spark databricks-sdk pip install -e . # If using pyproject.toml - name: Run unit tests (local Spark, no cluster needed) run: pytest tests/unit/ -v --tb=short - name: Install Databricks CLI uses: databricks/setup-cli@main - name: Validate bundle env: DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }} ...