Skip to content

Introduction to Pilz

What is Pilz?

Pilz (German for "mushroom" / "fungus") is a machine learning library for classification tasks. The name reflects its nature: like mushrooms are neither plants nor animals, Pilz is neither a traditional decision tree nor a neural network — it's something in between that captures the best of both worlds.

flowchart TB subgraph Nature P[Plants] --> M[Mushrooms/Fungi] A[Animals] --> M end subgraph ML NN[Neural Networks] --> Pilz[Pilz] DT[Decision Trees] --> Pilz end style M fill:#ccffcc style Pilz fill:#e0f0ff

Unlike traditional ML tools that create "black box" models, Pilz generates readable SQL rules that you can directly deploy to your database.

The core innovation of Pilz is its ability to handle feature correlations naturally through multi-dimensional cuts.

How It Works

Pilz evaluates feature combinations by their ability to separate target from non-target, with the training data processing running natively in Polars. The algorithm is composed of several key concepts:

  • Feature Categorization — Every feature is binned into n_cat categories. Numerical features use quantile binning; categorical features are grouped by target rate similarity.
  • Multi-Dimensional Splits — Pilz splits on feature combinations directly, not one at a time. This captures correlations between features in a single split.
  • SQL-Native Architecture — Training runs natively in Polars; DuckDB SQL is used for legacy model scoring and the multi-class label combination. Filters are a custom boolean-expression AST (pilz.model.expr) that renders to SQL WHERE clauses, and decision thresholds are derived on the fly from the stored leaf counts.
  • Splitting Nodes into Branches — Each node's winning bin combinations fan out into recursive subtrees (merged by spore_diff), plus a residual .n branch for the unobserved combinations.
  • Downsampling — At every node, Pilz independently samples target and non-target rows from a training frame that was shuffled once at load time, taking the first matching rows (.head(max_eval_fit)).
  • Imbalanced Data — Because target and non-target are sampled independently with equal total weight, Pilz handles skewed distributions without SMOTE or class weights.

Traditional vs Pilz

A traditional decision tree needs multiple sequential splits to capture a pairwise correlation. Pilz captures it in a single multi-dimensional cut:

flowchart TD subgraph "Traditional Tree - Needs Multiple Splits" T1[Data] --> T2{Contract=Monthly?} T2 -->|Yes| T3{No TechSupport?} T3 -->|Yes| T4[High Churn] T3 -->|No| T5[Medium Churn] T2 -->|No| T6[Low Churn] end subgraph "Pilz - Single Multi-Dimensional Cut" P1[Data] --> P2{Contract=Monthly AND TechSupport=No?} P2 -->|Yes| P3[High Churn, Score: 0.85] P2 -->|No| P4[Low Churn, Score: 0.15] end style P2 fill:#ccffcc

Comparison with Other Approaches

Aspect Neural Networks Traditional Trees Pilz
Feature correlations Learned implicitly Multiple splits Single cut
Interpretability Low Medium High (SQL)
Output format Opaque weights Tree structure SQL rules
Data preprocessing Often required Minimal Minimal
Imbalanced data Needs tuning Needs tuning Native support

Core Parameters

Parameter What it controls
n_dims How many features to combine in a single split
n_cat Number of bins per feature
max_depth Maximum tree depth
spore_diff Branch-merge distance for the winning split's bin combinations (0 = fan-out, >= 1 = binary split)
max_eval_fit Rows sampled per node
calcs_per_dim Max combinations evaluated per dimension

When to Use Pilz

✓ Ideal For

  • Tabular data — Customer data, transactions, any structured data
  • Interpretability matters — You need to understand and deploy model logic
  • Feature correlations exist — Pilz captures them naturally
  • SQL-native deployment — Model output is runnable SQL

✗ Not Ideal For

  • Image/audio/video — Use neural networks
  • Very large datasets — May need tuning
  • No correlations — Simpler models may suffice

Next Steps

Pilz: Capturing feature correlations naturally through multi-dimensional cuts.