Example: Customer Churn

This example demonstrates binary classification with the Telco Customer Churn dataset.

Dataset

• Source: Kaggle Telco Customer Churn via blastchar/telco-customer-churn
• Task: Predict customer churn (Yes/No)
• Features: 19 (demographics, services, billing)
• Classes: 2 (Yes, No)
• Training samples: 5,634
• Test samples: 1,409

Quick Start

The config files for this example are in examples/churn/:

# 1. Download data (requires kagglehub)
pip install kagglehub
python3 -c "
import kagglehub
path = kagglehub.dataset_download('blastchar/telco-customer-churn')
print(f'Downloaded to: {path}')
"

# 2. Point the settings to your downloaded data
#    Edit examples/churn/train_settings.yaml and update train_files:
#      train_files:
#        - <kagglehub_path>/train.csv
#    Edit examples/churn/eval_settings.yaml and update test_files:
#      test_files:
#        - <kagglehub_path>/test.csv
#    Note: the original dataset is one file; split it into train/test first.

# 3. Train
pilz train \
  --datacard examples/churn/dc_telco_customer.yaml \
  --trainsettings examples/churn/train_settings.yaml

# 4. Evaluate
pilz eval \
  --datacard examples/churn/dc_telco_customer.yaml \
  --evalsettings examples/churn/eval_settings.yaml

Or use the provided script:

cd examples/churn
# After downloading data and updating train_files/test_files paths in the settings
bash run.sh

DataCard Structure

features:
  - name: gender
    statistical: categorial
    type: string
  - name: SeniorCitizen
    statistical: numerical
    type: int
  - name: Partner
    statistical: categorial
    type: string
  - name: Dependents
    statistical: categorial
    type: string
  - name: tenure
    statistical: numerical
    type: int
  - name: PhoneService
    statistical: categorial
    type: string
  - name: MultipleLines
    statistical: categorial
    type: string
  - name: InternetService
    statistical: categorial
    type: string
  # ... 11 more features
  - name: Churn
    statistical: categorial
    type: string

target:
  feature_name: Churn
  values:
    - "Yes"
    - "No"

infos:
  bla: https://www.kaggle.com/datasets/blastchar/telco-customer-churn

Settings (Quick Start)

n: 5                # 5 trees per class (10 trees total)
out_folder: test
max_depth: 8
frac_eval_cat: 0.8
max_eval_fit: 500
min_eval_fit: 20    # Larger leaves protect the minority class
n_dims: 2           # Pairwise feature combinations
n_cat: 3            # 3 bins per numerical feature
calcs_per_dim: 2000
n_rep: 5            # Repetitions per feature
train_files:
  - /path/to/train.csv

The checked-in examples/churn/train_settings.yaml uses a machine-specific absolute train_files path instead of the /path/to/... placeholder.

in_folders:
  - test
out_folder: eval
test_files:
  - /path/to/test.csv
out_file: eval/scored.csv

Training Time

With the checked-in settings on a modern laptop (Apple Silicon):

• Training: ~1 minute
• Evaluation: a few seconds

Actual Results

Overall Accuracy: 78.3%

Per-Class Accuracy

Class	Accuracy
No	84.3%
Yes	60.1%

The "No" class is easier to predict (majority class with ~73% of samples). The "Yes" class is harder due to class imbalance and more varied churn reasons.

ROC Curve

Output Files

test/
├── Yes/0.json    # Model for predicting churn "Yes"
├── ...           # Trees 1-4 per class
├── Yes/4.json
├── No/0.json     # Model for predicting churn "No"
├── ...           # Trees 1-4 per class
├── No/4.json
└── label_stats.json

eval/
├── Yes_roc.html
├── No_roc.html
├── all_roc.html
├── multi_class_result.html
└── scored.csv

Sample Predictions

Churn,Yes,No,predicted_Churn,correct
No,-2.44,2.94,No,1
No,-1.25,1.25,No,1
Yes,-0.37,0.29,No,0
No,-3.71,3.53,No,1
No,-1.96,1.79,No,1

Scores are log-odds (leaf_score="log_odds"): large magnitudes mean strong evidence, values near 0 are undecided. The missed Yes row above (−0.37 vs. +0.29) is a typical close call.

Key Findings

1. Contract type is the strongest predictor
2. Month-to-month customers churn more

3. Two-year contracts have lowest churn

4. Tenure matters

5. New customers (< 12 months) churn more

6. Longer relationships = loyalty

7. Internet service type interacts with contract

8. Fiber optic + month-to-month = high risk
9. DSL customers are more stable

Tips

Quick Start Settings (current)

The checked-in settings (n: 5, max_depth: 8, min_eval_fit: 20) give ~78% accuracy with 60% Yes-class recall (up from ~77% / 54% with single shallow trees).

For Better Accuracy

Beyond the checked-in settings, these directions are worth trying in train_settings.yaml (each roughly doubles training time, gains are not guaranteed — single runs vary by ~±2 points on this data):

max_depth: 13       # Deeper trees
n_dims: 3           # Triple feature combinations
n_cat: 5            # Finer bins
calcs_per_dim: 4000 # More thorough search
max_eval_fit: 5000  # More training samples (dataset has 5634 rows)

Incremental Approach

1. Start with max_depth=5, n_dims=2 to verify the pipeline
2. Increase max_depth to 8, then 13
3. Try n_dims=3 for feature interactions
4. Add more trees with n=5 (checked-in default)
5. For class imbalance, monitor the Yes-class accuracy — and try the youden combination (different_target_pilz_comb_method: youden), which trades overall accuracy for minority recall