palantir-incident-runbook

# Palantir Incident Runbook ## Overview Rapid incident response for Foundry-related outages: API failures, transform build failures, authentication issues, and data pipeline stalls. ## Prerequisites - Access to application logs and Foundry build history - Foundry service user credentials for health checks - On-call escalation path defined ## Instructions ### Step 1: Triage (First 5 Minutes) ```bash set -euo pipefail echo "=== Foundry Incident Triage ===" echo "Time: $(date -u)" # 1. Check if Foundry itself is down curl -s -o /dev/null -w "Foundry API: HTTP %{http_code}\n" \ -H "Authorization: Bearer $FOUNDRY_TOKEN" \ "https://$FOUNDRY_HOSTNAME/api/v2/ontologies" || echo "FOUNDRY UNREACHABLE" # 2. Check our app health curl -s http://localhost:8080/health | python -m json.tool # 3. Check recent error logs grep -c "ApiError\|status_code.*[45][0-9][0-9]" /var/log/app/app.log | tail -1 ``` ### Step 2: Classify Severity | Severity | Criteria | Response Time | |----------|----------|---------------| | P1 Critical | Foundry API completely unreachable, all operations failing | Immediate | | P2 High | Intermittent 429/5xx errors, degraded performance | 15 minutes | | P3 Medium | Single transform failing, non-critical pipeline stalled | 1 hour | | P4 Low | Deprecation warnings, performance degradation | Next business day | ### Step 3: Common Incident Playbooks **Playbook A: Authentication Failure (401/403)** ```bash # 1. Verify token is set echo "Token set: ${FOUNDRY_TOKEN...