Skip to content

Feature Categorization

Feature categorization is the first step in Pilz's algorithm. Each feature is binned into n_cat categories to enable multi-dimensional correlation analysis.

Why Categorization?

Before we can build correlation tables for feature combinations, we need discrete bins:

flowchart TB subgraph Raw_Data R[Continuous values: 1.2, 3.5, 7.8, 12.4, ...] end subgraph Binning B[Bin into n_cat categories] end subgraph Correlation_Ready C[Category 0, 1, 2, ...] end R --> B B --> C style B fill:#e0f0ff

Two Types of Binning

The training path is cater() → feat_cater_fused(), which calls the native pilz_rs.cater_cont/cater_cont_batch_arrow for a numerical feature and pilz_rs.cater_cat for a categorical one. The legacy, test-only feat_cater() helper still dispatches to the pure-Python cont_cater_impl()/cat_cater_impl() implementations by feature type:

def feat_cater(self, feat, df_group, n_cat) -> CatCats | ContCats:
    match feat.statistical:
        case FeatureType.CATEGORIAL:
            return self.cat_cater_impl(feat=feat, df_group=df_group, n_cat=n_cat)
        case FeatureType.NUMERICAL:
            return self.cont_cater_impl(feat=feat, df_group=df_group, n_cat=n_cat)

Numerical Features: Quantile Binning

For continuous values, cont_cater_impl() creates bins by interpolating quantile boundaries from the cumulative group weights. Null and NaN rows are excluded from the boundary computation and receive a dedicated "missing" bin; when no rows remain, the missing bin becomes the only label:

def cont_cater_impl(self, feat, df_group, n_cat) -> ContCats:
    is_float = df_group[feat.name].dtype.is_float()
    has_nan = is_float and df_group[feat.name].is_nan().any()
    has_null = df_group[feat.name].has_nulls() or has_nan
    filter_expr = pl.col(feat.name).is_not_null()
    if is_float:
        filter_expr = filter_expr & pl.col(feat.name).is_not_nan()
    df_sorted = df_group.filter(filter_expr)[
        [feat.name, "weight"]
    ].sort(feat.name)

    if df_sorted.height == 0:
        return ContCats(
            cuts=[],
            labels=["0"],
            feat_name=feat.name,
            missing_label="1" if has_null else None,
        )

    cumulative_weights = df_sorted["weight"].cum_sum() / df_sorted["weight"].sum()

    res = []
    label = ["0"]
    idx = 0
    for i in range(1, n_cat):
        quantile = i / n_cat
        index = (cumulative_weights >= quantile).arg_true()[0]
        # interpolate the boundary value between the neighbours
        ...
        idx += 1
        label.append(f"{idx}")
        res.append(value)

    return ContCats(
        cuts=res,
        labels=label,
        feat_name=feat.name,
        missing_label=str(len(label)) if has_null else None,
    )
flowchart TB subgraph Input I[Values: 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] end subgraph Sorted S[1, 2, 3, 5, 8, 13, 21, 34, 55, 89] end subgraph Cumulative_fraction C[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0] end subgraph Bins_n_cat_2 M1["Quantile 0.5 reached at 8: cut = 8"] B1["Bin 0: \u2264 8 (5 values)"] B2["Bin 1: > 8 (5 values)"] end I --> S S --> C C --> M1 M1 --> B1 M1 --> B2

The bin boundaries are stored in the ContCats class, which uses pl.Series.cut() to apply them to new data:

class ContCats(CatsMixin):
    cuts: list[float]
    labels: list[str]
    feat_name: str
    missing_label: str | None = None

    def cut(self, df: pl.DataFrame) -> pl.Series:
        res = df[self.feat_name].cut(self.cuts, labels=self.labels).cast(pl.Utf8)
        if self.missing_label is None:
            return res
        return res.fill_null(self.missing_label)

Categorical Features: Target Rate Grouping

For categorical values, cat_cater_impl() assigns each original category to a bin based on its weight and target rate:

  1. Large categories first: Categories with weight >= 1/(n_cat - i) of the remaining weight get their own bin; the remaining weights are renormalized
  2. Small categories grouped: Remaining categories are sorted by target rate and grouped by cumulative-weight quantiles
def cat_cater_impl(self, feat, df_group, n_cat) -> CatCats:
    has_null = df_group[feat.name].has_nulls()
    df = (
        df_group.with_columns(
            pl.col(feat.name).cast(pl.Utf8).fill_null(MISSING_SENTINEL)
        )
        .group_by(feat.name)
        .agg(pl.col("weight").sum(), pl.col("target_weight").sum())
        .sort(pl.col("weight"), descending=True)
    )

    # Phase 1: large categories get their own bin
    for i in range(n_cat):
        if df.height == 0:
            break
        val = df.item(0, "weight")
        if val < 1.0 / (n_cat - i):
            break
        cat = df.item(0, feat.name)
        if cat == MISSING_SENTINEL:
            missing_label = str(i)
        else:
            map[cat] = str(i)
        df = df.tail(-1).with_columns(pl.col("weight").truediv((1.0 - val)))

    # Phase 2: remaining categories grouped by target-rate quantiles
    df = df.with_columns(
        pl.col("target_weight").truediv(pl.col("weight")).alias("target_rate")
    ).sort("target_rate", descending=False)

    df = df.with_columns(pl.col("weight").cum_sum().alias("cum_weight"))
    labels = [str(i) for i in range(offset, n_cat)]
    breaks = [i / len(labels) for i in range(1, len(labels))]
    df = df.with_columns(
        pl.col("cum_weight")
        .cut(breaks=breaks, labels=labels, left_closed=True)
        .alias("bin")
    )
    ...
    return CatCats(mapping=map, default=..., feat_name=feat.name, missing_label=...)
flowchart TB subgraph "Step 1: Calculate Target Rate" T1[Category: admin, 100 samples, 80 target] --> TR1[80%] T2[Category: technician, 50 samples, 25 target] --> TR2[50%] T3[Category: blue-collar, 100 samples, 10 target] --> TR3[10%] end subgraph "Step 2: Sort by Rate" S1[Sort: 10%, 50%, 80%] end subgraph "Step 3: Create n_cat Bins" B1["Bin 2: admin (80%)"] B2["Bin 1: technician (50%)"] B3["Bin 0: blue-collar (10%)"] end TR1 --> S1 TR2 --> S1 TR3 --> S1 S1 --> B1 S1 --> B2 S1 --> B3 style S1 fill:#e0f0ff

The mapping is stored in the CatCats class, which applies it using replace_strict:

class CatCats(CatsMixin):
    mapping: dict[str | int, str]
    default: str
    feat_name: str
    missing_label: str | None = None

    def cut(self, df: pl.DataFrame) -> pl.Series:
        if self.missing_label is None:
            return df[self.feat_name].cast(pl.Utf8).replace_strict(
                self.mapping, default=self.default
            )
        return (
            df.with_columns(
                pl.when(pl.col(self.feat_name).is_null())
                .then(pl.lit(self.missing_label))
                .otherwise(
                    pl.col(self.feat_name)
                    .cast(pl.Utf8)
                    .replace_strict(self.mapping, default=self.default)
                )
                .alias(self.feat_name)
            )
            .select(self.feat_name)
            .to_series()
        )

The Code Path

The entry point is cater(). It iterates over all features in the datacard: the numerical features are categorized in one native batch call (pilz_rs.cater_cont_batch_arrow, wrapped by cater_cont_batch), the categorical features via pilz_rs.cater_cat (wrapped by cater_cat_reps/feat_cater_fused). Every categorized feature is kept; features that differentiate too poorly are flagged in feature_mask so the split search never sees them:

def cater(self, train_df: TrainDataframes):
    reps = list(train_df)
    results = {}
    cont_features = [f for f in self.dc.train_features
                     if f.statistical == FeatureType.NUMERICAL]
    if cont_features:
        results.update(self.cater_cont_batch(
            features=cont_features, reps=reps, n_cat=self.settings.n_cat,
        ))
    for feat in self.dc.train_features:
        if feat.statistical == FeatureType.CATEGORIAL:
            results[feat.name] = self.cater_cat_reps(
                feat=feat, reps=reps, n_cat=self.settings.n_cat,
            )
    for feat in self.dc.train_features:
        median_feature, rep_features = results[feat.name]
        train_df.train_features.append(median_feature)
        train_df.train_features_by_rep.append(rep_features)
        train_df.feature_mask.append(not median_feature.is_diff_to_low())

Each categorized feature is scored via calc_diff(), delegated to the native pilz_rs.score. The scoring method is selected with the score_method training setting:

  • "diff_all_<exp>" — dimension-neutral sum 0.5 * sum(|p - q| ** exp).
  • "diff_dir_<exp>" (default "diff_dir_1.5") — dominant-side directional sum of |p - q| ** exp.
  • "lift_dir_<exp>" — dominant-side, evidence-weighted sum of the log-likelihood lift n * |ln(n / E)| ** exp (E the expected count under independence).
  • "hellinger" — Hellinger distance 0.5 * sum((sqrt(p) - sqrt(q)) ** 2).

With smoothing > 0 the proportions and counts used by all scoring methods are Laplace-smoothed:

def calc_diff(self) -> float:
    if self._score is None:
        self._score = pilz_rs.score(self._corr, self.score_method)
    return self._score

Features that don't differentiate well enough are filtered out by is_diff_to_low():

def is_diff_to_low(self, threshold: float = 0.90) -> bool:
    max_wert = max(self._corr.max_proportion)
    for proportion, proportion_right, max_proportion in zip(
        self._corr.proportion,
        self._corr.proportion_right,
        self._corr.max_proportion,
    ):
        if max_proportion == max_wert:
            return min(proportion, proportion_right) > threshold
    return False

The 2D Pair-Correlation Filter

The 1D check looks at one feature in isolation. The pair_corr_filter training setting extends the same idea to feature combinations and runs in every dimension stage (dim 2..=n_dims) of the split search: for every evaluated candidate the joint correlation table is checked for redundant features. For every candidate position the cells are grouped by the remaining features and the share of each class inside the dominant bin of that position is summed:

S_class = 1 - sum over cells(max share of one bin of the candidate feature)

When BOTH classes put more than pair_corr_min_main (default 0.9) of their mass into the dominant bins, one feature of the combination is redundant given the rest — knowing the remaining features determines it for both classes. The 1D-weakest feature of the candidate is masked immediately:

  • the triggering candidate itself is excluded as a split,
  • every later combination of the same node containing the masked feature is skipped (without consuming calcs_per_dim budget),
  • combinations evaluated before the trigger keep their candidacy,
  • the mask lives only in the current node — the next node starts fresh.

The check runs on the candidate's median-repetition table, so it costs nothing extra: the table is computed for scoring anyway. With n_dims: 1 no pair filtering happens, and the 1D-best feature can never be masked (it is never the weakest member of a combination). The masking decisions are surfaced on the CounterBatchResult and stored on the trainer as Train._debug_masked_pairs for inspection; no log line is emitted.

flowchart TB A[Evaluate candidate combination] --> B{Both classes > pair_corr_min_main in dominant bins?} B -- no --> C[Normal candidate] B -- yes --> D[Mask 1D-weakest feature of the candidate] D --> E[Later combinations with this feature are skipped]

Building Correlation Tables

After binning, Pilz builds the joint contingency table for a feature combination natively with pilz_rs.corr_table, from the target/non-target bin codes of the combined features (the raw series are not available on the fused path — target_sr/non_target_sr are None):

class CombinedCategorizedFeature(CategorizedFeatureMixin):
    def __init__(self, train_features, non_target_size, target_size, score_method="diff_dir_1.5", smoothing=0.0):
        self._train_features = train_features
        self.cut_list = [train.cuts for train in train_features]
        self._corr = pilz_rs.corr_table(
            [train._target_codes for train in train_features],
            [train._non_target_codes for train in train_features],
            [
                getattr(train, "_n_labels", None) or len(train.cuts.all_labels)
                for train in train_features
            ],
            smoothing,
        )
        self._score = None
        self._diff_df = None
flowchart TB subgraph Binned_Features F1[X: Bin 0, Bin 1] F2[Y: Bin 0, Bin 1] end subgraph Correlation_Table T1["X=0, Y=0: T=15, NT=85"] T2["X=0, Y=1: T=45, NT=55"] T3["X=1, Y=0: T=60, NT=40"] T4["X=1, Y=1: T=80, NT=20"] end F1 --> T1 & T2 F2 --> T1 & T3 style Correlation_Table fill:#ccffcc

Configuration

The n_cat parameter controls the granularity of binning:

n_cat: int = Field(
    description="Number of categories in one featerue",
    default=3,
)

Summary

Feature Type Binning Method Key Function
Numerical Quantile binning with cumulative-weight interpolation cont_cater_impl() (legacy/test-only); native cater_cont on the fused path
Categorical Weight-based own bins + target-rate quantiles cat_cater_impl() (legacy/test-only); native cater_cat on the fused path
Scoring diff_*, lift_dir_* or hellinger score of the correlation table calc_diff()
1D Filtering Masks features with insufficient differentiation is_diff_to_low()
2D Filtering Masks the weakest feature of correlated combinations counter_batch() in rust/src/lib.rs

Next Steps