Training Internals¶

This chapter provides a deep dive into how Pilz training works under the hood. If you want to understand every detail of the algorithm, read on.

Architecture Overview¶

flowchart TB subgraph Input DC[DataCard] TS[TrainSettings] end subgraph Training_Service M[Main Loop] --> T[Train Tree] T --> C[Categorize] T --> W[Find Winner] T --> R[Recurse] end subgraph Data_Service DW[Darkwing] --> P[Polars] DW -.-> DB[DuckDB
eval only] end DC --> TS TS --> M M --> DW DW --> P style M fill:#e0f0ff style C fill:#ccffcc style W fill:#ffff99 style R fill:#ffcccc

Data Flow¶

sequenceDiagram participant CLI participant Train participant Darkwing participant Pilz CLI->>Train: run() Train->>Train: for n trees, for each target Train->>Darkwing: read_akt_train(targer_filter, train_settings, akt_filters) Darkwing->>Darkwing: Polars filter + head(max_eval_fit) on shuffled cache Darkwing-->>Train: TrainDataframes Train->>Train: cater() - categorize features Train->>Train: _find_winner() - find best split Train->>Train: train_pilz() - recurse on branches Train->>Pilz: create Pilz object Train-->>CLI: save JSON

DuckDB is not part of the training path: the per-node sample is drawn in Polars, and decision thresholds are derived on the fly from the stored leaf counts. DuckDB is only used to score legacy cut-model leaves during evaluation and for the multi-class label combination.

The Main Loop¶

The run() method iterates n times (the tree index) and trains one tree per target value on every pass:

def run(self):
    logger.info("start training")
    SympyToSqlHelper.columns = self.dc.feature_names + [self.dc.target.feature_name]
    for n in range(self.settings.n):
        for cat in self.dc.target.values:
            if self.pilz_exists(target=cat, n=n):
                logger.info(f"tree {n} for label {cat} already exists")
                continue
            logger.info(f"train tree {n} for label {cat}")
            train_res = Pilz(
                spore=self.train_pilz(
                    target_filter=self.gen_target_filter(cat=cat),
                    path_filter=[],
                ),
                target=cat,
            )
            self.save_pilz(pilz=train_res, n=n)
    self.write_label_stats()

The train_pilz Function¶

This is the core recursive function that builds a single tree. It finds the winning feature combination, merges its observed bin combinations into branches via get_cell_branches(spore_diff) and recurses once per branch (.<i>), plus the residual .n branch for the combinations never observed in the count sample:

def _train_pilz_branches(self, target_filter, path_filter, depth, parent_features, train_df):
    best_feat, winner_features = self._find_winner(train_df=train_df)

    if best_feat is None:
        return self.make_spore(path_filter=path_filter, depth=depth,
                               train_df=train_df, feature_names=parent_features)

    branches, residual = best_feat.get_cell_branches(
        spore_diff=self.settings.spore_diff
    )
    if not branches:
        return self.make_spore(path_filter=path_filter, depth=depth,
                               train_df=train_df, feature_names=parent_features)

    spores = []
    for i, (branch_filter, _) in enumerate(branches):
        spores += self.train_pilz(
            target_filter=target_filter,
            path_filter=path_filter + [branch_filter],
            depth=_child_depth(str(i)),  # "0", "1.0", ...
            parent_features=winner_features,
        )
    if residual is not None and residual.combine is not TRUE:
        spores += self.train_pilz(
            target_filter=target_filter,
            path_filter=path_filter + [residual],
            depth=_child_depth("n"),  # residual branch
            parent_features=winner_features,
        )
    return spores

Each recursive call: 1. Reads a fresh balanced sample for this node via read_akt_train() (see Downsampling) 2. Stops if too few samples remain or max depth reached 3. Categorizes features via cater() (see Feature Categorization) 4. Finds the best split via _find_winner() (see Multi-Dimensional Splits) 5. Recurse on every branch of the winning split (see Splitting Nodes into Branches)

The cater Function¶

Categorizes all features into n_cat bins each. The numerical features go through one native batch call (pilz_rs.cater_cont_batch_arrow, wrapped by cater_cont_batch); the categorical features use the single-call path (pilz_rs.cater_cat, wrapped by cater_cat_reps/feat_cater_fused). Every categorized feature is kept; features that differentiate too poorly are flagged in feature_mask:

def cater(self, train_df: TrainDataframes):
    reps = list(train_df)
    results = {}
    cont_features = [
        feat for feat in self.dc.train_features
        if feat.statistical == FeatureType.NUMERICAL
    ]
    if cont_features:
        results.update(self.cater_cont_batch(
            features=cont_features, reps=reps, n_cat=self.settings.n_cat,
        ))
    for feat in self.dc.train_features:
        if feat.statistical == FeatureType.CATEGORIAL:
            results[feat.name] = self.cater_cat_reps(
                feat=feat, reps=reps, n_cat=self.settings.n_cat,
            )
    for feat in self.dc.train_features:
        median_feature, rep_features = results[feat.name]
        train_df.train_features.append(median_feature)
        train_df.train_features_by_rep.append(rep_features)
        train_df.feature_mask.append(not median_feature.is_diff_to_low())

Features that don't differentiate well enough are excluded by is_diff_to_low():

def is_diff_to_low(self, threshold: float = 0.90) -> bool:
    max_wert = max(self._corr.max_proportion)
    for proportion, proportion_right, max_proportion in zip(
        self._corr.proportion,
        self._corr.proportion_right,
        self._corr.max_proportion,
    ):
        if max_proportion == max_wert:
            return min(proportion, proportion_right) > threshold
    return False

Repetition handling (n_rep)¶

Each feature is categorized over n_rep independent random splits of the data into count and group sets to reduce statistical fluctuation. Unlike the early single-threaded implementation, Pilz no longer keeps a feat_rep/feat_reps helper: the categorization and the per-repetition alignment are handled natively.

TrainDataframes stores n_rep independent random splits. The group frames carry the weight/target_weight columns and drive the binning; the count frames supply the target/non-target bin codes that form the correlation tables:

def __init__(self, target_df, non_target_df, frac_eval_cat, min_size, n_rep):
    self.target_df_size = target_df.height
    self.non_target_df_size = non_target_df.height
    self.min_size = min_size
    self.n_count_target, _ = self._calc_split(target_df.height, frac_eval_cat)
    self.n_count_non_target, _ = self._calc_split(non_target_df.height, frac_eval_cat)

    for _ in range(n_rep):
        perm_target = np.random.permutation(target_df.height)
        shuffled_target = target_df[perm_target]
        self.target_df_count.append(shuffled_target.head(self.n_count_target))
        target_group = shuffled_target.tail(-self.n_count_target)

        perm_non_target = np.random.permutation(non_target_df.height)
        shuffled_non_target = non_target_df[perm_non_target]
        self.non_target_df_count.append(
            shuffled_non_target.head(self.n_count_non_target)
        )
        non_target_group = shuffled_non_target.tail(-self.n_count_non_target)

        w_target = 0.5 / max(1, target_group.height)
        w_non_target = 0.5 / max(1, non_target_group.height)
        group = pl.concat([target_group, non_target_group]).hstack(
            [
                pl.Series("weight", np.concatenate([
                    np.full(target_group.height, w_target),
                    np.full(non_target_group.height, w_non_target),
                ])),
                pl.Series("target_weight", np.concatenate([
                    np.full(target_group.height, 2 * w_target),
                    np.zeros(non_target_group.height),
                ])),
            ]
        )
        self.df_group.append(group)

cater() categorizes every feature once per repetition. The numerical features use one native batch call (pilz_rs.cater_cont_batch_arrow), the categorical features the pilz_rs.cater_cat path; each repetition is scored and the median repetition per feature is selected.
The split search passes all n_reps to pilz_rs.counter_batch, which evaluates every combination once per repetition (only rows of the same repetition are combined — row alignment is preserved) and picks the median repetition for the winner.

The number of repetitions is controlled by the n_rep setting:

Value	Effect
1	No repetition — fast but noisy
5	Default — good balance of speed and stability
10+	Very stable cuts, slower training

The Split Search (`_find_winner`)¶

Finds the best feature or combination to split on. It scores every categorized feature by calc_diff(), sorts them descending and takes the best usable one as the baseline. For n_dims >= 2 the combination search runs natively in pilz_rs.counter_batch: it evaluates the dimension stages 2..=n_dims one after another, scores every candidate over all repetitions and returns the winner plus the updated feature mask:

def _find_winner(self, train_df: TrainDataframes):
    sorted_train_feats = sorted(
        zip(
            train_df.train_features,
            train_df.train_features_by_rep,
            train_df.feature_mask,
        ),
        key=lambda x: x[0].calc_diff(),
        reverse=True,
    )
    feature_mask = [usable for _, _, usable in sorted_train_feats]
    ...
    res = pilz_rs.counter_batch(
        target_codes, non_target_codes, label_counts,
        n_features, n_reps, dim1_scores,
        self.settings.n_dims, self.settings.calcs_per_dim,
        self.settings.min_split_improvement, self.settings.score_method,
        self.settings.smoothing, feature_mask,
        self.settings.pair_corr_filter, self.settings.pair_corr_min_main,
        self.settings.pair_corr_max_features,
    )
    ...
    return best_feat, winner_features

The legacy counter() method is a parity helper only: it delegates to _find_winner() and derives the old three-way l/n/r filters from the winner, and is no longer part of the training path.

The 2D Pair-Correlation Filter¶

In every dimension stage each evaluated candidate checks its joint correlation table for redundant features: when both classes put more than pair_corr_min_main (default 0.9) of their mass into the per-cell dominant bin of a candidate position, the 1D-weakest feature of the candidate is masked immediately. The triggering candidate is excluded and every later combination of the node containing the masked feature is skipped — without consuming calcs_per_dim budget, so the budget stretches further for the meaningful combinations. The mask lives only in the current node; the next node starts fresh. The masking decisions are returned on the CounterBatchResult and retained on the trainer for inspection via Train._debug_masked_pairs. See Pair-Correlation Filter for the full algorithm and Settings Reference for the tuning parameters.

Leaf Creation¶

Recursion stops when not enough samples remain or max depth is reached:

def is_final_size(self) -> bool:
    return (
        self.target_df_size < self.min_size
        or self.non_target_df_size < self.min_size
    )

A leaf is created with its score. Fresh models store the path as structured Cond objects (rendered to SQL only on demand via Spore.get_cuts()), never as pre-rendered cut strings, and record the winning feature combination of its parent node in feature_names:

def make_spore(self, path_filter, depth, train_df, feature_names=None):
    score = train_df.score()
    return [
        Spore(
            cond=[Cond.from_expr(fil.combine) for fil in path_filter],
            score=score,
            depth=depth,
            feature_names=feature_names,
        )
    ]

The score is the signed difference between target and non-target rows at the leaf, (target_size - non_target_size) / n, scaled by a weakening factor so that sparsely populated leaves contribute a smaller magnitude:

def score(self) -> float:
    sum = self.non_target_df_size + self.target_df_size
    if sum == 0:
        return 0.0
    if sum == 1:
        factor = 0.5
    else:
        factor = 1.0 - math.log(sum) / sum
    diff = self.target_df_size - self.non_target_df_size
    return factor * diff / sum

A positive score means more target than non-target rows, 0.0 means a perfect balance. The magnitude grows with the number of rows in the leaf and only approaches 1.0 for large leaves.

Data Caching¶

Darkwing provides caching to avoid reloading data. The cached training frame is randomly shuffled once at load time, so the first rows matching a per-node filter are already a random sample (no per-query ORDER BY RANDOM()):

def get_cached_train_df(self) -> pl.DataFrame:
    if self.df_train_cach is None:
        self.df_train_cach = self.get_df_from_files(self.train_files).sample(
            fraction=1.0, shuffle=True
        )
    return self.df_train_cach

flowchart LR subgraph "First Call" F1[Request] --> L[Load from CSV] L --> C[Cache in memory] end subgraph "Subsequent Calls" S1[Request] --> H[Check Cache] H -->|"Hit"| R1[Return cached] H -->|"Miss"| L end style C fill:#ccffcc style R1 fill:#ccffcc

Summary¶

Component	File	Description
`run()`	`train.py`	Main loop: for n trees, for each target
`train_pilz()`	`train.py`	Recursive tree building (read → cater → find winner → recurse)
`cater()`	`train.py`	Categorizes features into n_cat bins (numeric batch / categorical)
`feat_cater()`	`train.py`	Legacy/test-only dispatcher to numerical or categorical binning
`_find_winner()`	`train.py`	Finds the best split via the native `counter_batch` search
`counter()`	`train.py`	Legacy parity helper; delegates to `_find_winner()`
`make_spore()`	`train.py`	Creates a leaf with structured `cond` and score
`is_final_size()`	`dataframes.py`	Stopping criterion for recursion
`score()`	`dataframes.py`	Signed `(target - non_target) / n` at a leaf
`get_cached_train_df()`	`darkwing.py`	Data caching layer (shuffles once at load)

Next Steps¶

SQL Rules for Deployment — Deploy models to production
Settings Reference — All parameters
Troubleshooting — Common issues