datanalysis-credit-risk

# Data Cleaning and Variable Screening ## Quick Start ```bash # Run the complete data cleaning pipeline python ".github/skills/datanalysis-credit-risk/scripts/example.py" ``` ## Complete Process Description The data cleaning pipeline consists of the following 11 steps, each executed independently without deleting the original data: 1. **Get Data** - Load and format raw data 2. **Organization Sample Analysis** - Statistics of sample count and bad sample rate for each organization 3. **Separate OOS Data** - Separate out-of-sample (OOS) samples from modeling samples 4. **Filter Abnormal Months** - Remove months with insufficient bad sample count or total sample count 5. **Calculate Missing Rate** - Calculate overall and organization-level missing rates for each feature 6. **Drop High Missing Rate Features** - Remove features with overall missing rate exceeding threshold 7. **Drop Low IV Features** - Remove features with overall IV too low or IV too low in too many organizations 8. **Drop High PSI Features** - Remove features with unstable PSI 9. **Null Importance Denoising** - Remove noise features using label permutation method 10. **Drop High Correlation Features** - Remove high correlation features based on original gain 11. **Export Report** - Generate Excel report containing details and statistics of all steps ## Core Functions | Function | Purpose | Module | |------|------|----------| | `get_dataset()` | Load and format data | references.func | | `org_analysis()` | ...