Example: MNIST Digits¶

MNIST is a classic multi-class classification problem - recognizing handwritten digits (0-9).

Dataset¶

• Source: MNIST (Modified National Institute of Standards and Technology) via Kaggle mnist-in-csv
• Task: Classify digits (0-9)
• Features: 784 (28×28 pixel values as integers 0-255)
• Classes: 10 (digits 0-9)
• Training samples: 60,000
• Test samples: 10,000

Quick Start¶

The config files for this example are in examples/mnist/:

# 1. Download MNIST CSV data (requires kagglehub)
pip install kagglehub
python3 -c "
import kagglehub
path = kagglehub.dataset_download('oddrationale/mnist-in-csv')
print(f'Downloaded to: {path}')
"

# 2. Point the settings to your downloaded data
#    Edit examples/mnist/train_settings.yaml and update train_files:
#      train_files:
#        - <kagglehub_path>/mnist_train.csv
#    Edit examples/mnist/eval_settings.yaml and update test_files:
#      test_files:
#        - <kagglehub_path>/mnist_test.csv
#    The same paths are also used by the three ensemble train configs
#    (train_entro.yaml, train_smooth1.yaml, train_hellinger_sm1.yaml) and
#    eval_ensemble.yaml, so set them there too if you want the ~96% recipe.

# 3. Train
pilz train \
  --datacard examples/mnist/dc_mninst.yaml \
  --trainsettings examples/mnist/train_settings.yaml

# 4. Evaluate
pilz eval \
  --datacard examples/mnist/dc_mninst.yaml \
  --evalsettings examples/mnist/eval_settings.yaml

Or use the provided scripts:

cd examples/mnist
# After downloading data and updating train_files/test_files paths in the settings
bash run.sh        # single config — ~95.3%
bash run_96.sh     # three variants + ensemble eval — ~96%

DataCard Structure¶

For MNIST, every pixel is a feature (784 total):

features:
  - name: label
    statistical: categorial
    type: int
  - name: 1x1
    statistical: numerical
    type: int
  - name: 1x2
    statistical: numerical
    type: int
  # ... 784 pixels total (1x1 through 28x28)
  - name: 28x28
    statistical: numerical
    type: int

target:
  feature_name: label
  values:
    - 0
    - 1
    - 2
    - 3
    - 4
    - 5
    - 6
    - 7
    - 8
    - 9

infos:
  src: https://www.kaggle.com/datasets/oddrationale/mnist-in-csv

Settings¶

The checked-in train_settings.yaml uses a full search configuration that reaches ~95.3% on its own and is the basis for the ~96% ensemble below:

n: 10               # 10 trees per digit (ensemble within a digit)
out_folder: mnist
max_depth: 20       # Deeper trees capture more patterns
frac_eval_cat: 0.8
max_eval_fit: 5000  # More training samples
min_eval_fit: 5
n_dims: 4           # Feature combinations up to 4 dimensions
n_cat: 5            # Finer pixel value bins
calcs_per_dim: 20000 # Thorough search
n_rep: 5
score_method: hellinger
smoothing: 1.0      # Laplace smoothing steadies the Hellinger score
train_files:
  - /path/to/mnist_train.csv

The checked-in examples/mnist/train_settings.yaml uses a machine-specific absolute train_files path instead of the /path/to/... placeholder.

The eval_settings.yaml evaluates the single model from train_settings.yaml:

in_folders:
  - mnist
out_folder: eval
test_files:
  - /path/to/mnist_test.csv
out_file: eval/scored.csv

Reaching ~96%: the three-variant ensemble¶

A single configuration tops out at ~95.3%. To cross 96% we train the same full search three times with a different score_method/smoothing and then evaluate all three model folders together in a single eval. Because same_target_pilz_comb_method: mean averages the scores of every tree across all three folders, the three variants act as an ensemble. The three train configs are checked in as train_entro.yaml (directional diff_dir_1.5), train_smooth1.yaml (dimension-neutral diff_all_1.0 with smoothing) and train_hellinger_sm1.yaml; the combined eval is eval_ensemble.yaml:

in_folders:
  - mnist_entro
  - mnist_smooth1
  - mnist_hellinger_sm1
out_folder: eval_ensemble
out_file: eval_ensemble/scored.csv
test_files:
  - /path/to/mnist_test.csv
same_target_pilz_comb_method: mean
different_target_pilz_comb_method: max

For a fast first iteration reduce the settings (e.g. n: 1, max_depth: 5, n_dims: 2, n_cat: 3, max_eval_fit: 500, calcs_per_dim: 200).

Training Time¶

With the checked-in settings on a modern laptop (Apple Silicon):

• Training: several hours (10 digits × 10 trees, n_dims: 4, calcs_per_dim: 20000). The multi-dimensional search dominates the cost; lower calcs_per_dim or n_dims cut it sharply.
• Evaluation: ~1 second per config, or a few seconds for the three-folder ensemble.

Training is resumable — if interrupted, it continues where it left off.

Actual Results¶

Measured on the 10 000-row test set (mnist_test.csv) with the full search configuration below (n_dims: 4, calcs_per_dim: 20000).

Single configuration¶

Each value is one independently trained model folder evaluated on its own (same_target: mean, different_target: max):

The checked-in train_settings.yaml is the "hellinger + smoothing" variant and reaches ~95.3% on its own.

Ensemble of the three variants: 96.3%¶

Evaluating mnist_entro, mnist_smooth1 and mnist_hellinger_sm1 together in one eval (eval_ensemble.yaml, scores averaged with same_target_pilz_comb_method: mean) reaches 96.3% — the same models that scored ~95.5% individually push past 96% once their scores are averaged. This is the reproducible recipe for ~96% recognition. (The variant and ensemble numbers predate the log_odds leaf-score default; the hellinger single config was re-measured at 95.28% with current code.)

ROC Curves¶

Per-digit AUC ranges from ~0.99 to ~1.0 across all 10 classes.

Output Files¶

mnist/
├── 0/0.json ... 0/9.json        # 10 trees for digit "0"
├── 1/0.json ... 1/9.json        # 10 trees for digit "1"
├── 2/0.json ... 2/9.json
├── 3/0.json ... 3/9.json
├── 4/0.json ... 4/9.json
├── 5/0.json ... 5/9.json
├── 6/0.json ... 6/9.json
├── 7/0.json ... 7/9.json
├── 8/0.json ... 8/9.json
├── 9/0.json ... 9/9.json
└── label_stats.json

eval/
├── 0_roc.html ... 9_roc.html   # ROC curves per digit
├── all_roc.html                 # Combined ROC overlay
├── multi_class_result.html      # Per-digit accuracy chart
└── scored.csv                   # Per-row scores and predictions

Multi-Class Strategy¶

Pilz uses one-vs-rest for multi-class classification:

For each digit, Pilz trains a binary classifier ("Is this digit or not?"). At prediction time, all 10 models run and the per-target scores are combined into one prediction. By default (different_target_pilz_comb_method="max") the digit with the highest score wins; with "youden" the per-tree Youden thresholds derived on the fly from the stored leaf counts decide the predicted digit.

Combining the per-target scores — what we measured¶

On a single (non-ensemble) model with n=10 trees per digit and calcs_per_dim: 20000, we compared the two combination knobs on the 10 000-row test set:

Take-aways:

• max for the different-target combination wins clearly (~95%): there is no threshold to tune, so the argmax over the per-digit scores is not affected by threshold estimation noise on imbalanced or noisy scores.
• mean across the same-target trees is the safer choice: averaging the n trees per digit is far more robust than max, which lets a single outlier tree dominate the prediction and drops accuracy by ~2-4 points.
• Per-tree margins help exactly where max is weak: with maximum same-target combination, youden (93.34%) beats plain argmax (91.39%), because each tree is judged against its own threshold instead of letting one outlier dominate. With mean combination the plain argmax stays ahead.

For these reasons the checked-in defaults are same_target_pilz_comb_method: mean and different_target_pilz_comb_method: max. The three-variant ensemble reuses the same defaults — mean across the same-target trees now also averages the three folders, which is what lifts the result from ~95.5% to 96.3%.

Tips for MNIST¶

Current Settings¶

The checked-in train_settings.yaml uses the full search configuration — expect ~95.3% as a single model and ~96% with the three-variant ensemble (see Settings above). Training takes several hours.

The pair-correlation filter on pixel data¶

With 784 highly correlated pixel features, the Pair-Correlation Filter is what keeps the search budget on the meaningful combinations: correlated pixel pairs mask the redundant pixel for the rest of the node, so calcs_per_dim is spent on genuinely new signal instead of re-testing the same stroke. For wide data like MNIST you can cap the pair stage explicitly with pair_corr_max_features (e.g. 100) to bound the quadratic number of pairs.

For Faster Iteration¶

Reduce the settings in train_settings.yaml to verify the pipeline first:

n: 1                # 1 tree per digit (10 trees total)
out_folder: mnist_small
max_depth: 5        # Shallow trees for speed
frac_eval_cat: 0.8
max_eval_fit: 500   # Fewer samples per evaluation
min_eval_fit: 5
n_dims: 2           # Pairwise feature combinations
n_cat: 3            # 3 bins per pixel (low/medium/high)
calcs_per_dim: 200  # Limited calculations per dimension

With these settings expect: - Training time: ~15 minutes - Accuracy: ~87%

Incremental Approach¶

1. Start with n=1, max_depth=5, n_dims=2 to verify the pipeline works
2. Increase max_depth to 8, then 13
3. Add feature combinations: n_dims=3
4. Add more trees: n=5, then n=10
5. Fine-tune with more calculations: calcs_per_dim=5000

Monitor Training¶

Training is resumable — if interrupted, it picks up from the last saved tree. Check the out_folder to see progress.