Skip to content

Downsampling

At every node, Pilz independently samples target and non-target rows. This ensures training data stays manageable regardless of the original dataset size.

How It Works

Training data is read via two separate Polars operations — one for target rows, one for non-target rows — each limited to max_eval_fit rows:

def read_akt_train(self, targer_filter, train_settings, akt_filters):
    full_filters_target_list = [targer_filter.combine] + [f.combine for f in akt_filters]
    full_filter_target = make_and(full_filters_target_list)

    if full_filter_target is FALSE:
        # The path filters exclude every row of the target class.
        return TrainDataframes(target_df=pl.DataFrame(), non_target_df=pl.DataFrame(), ...)

    target_df = self._get_pl_train_df(
        full_filters=full_filter_target,
        max_eval_fit=train_settings.max_eval_fit,
    )

    full_filters_non_target_list = [Not(targer_filter.combine)] + [f.combine for f in akt_filters]
    full_filters_non_target = make_and(full_filters_non_target_list)

    non_target_df = self._get_pl_train_df(
        full_filters=full_filters_non_target,
        max_eval_fit=train_settings.max_eval_fit,
    )

    return TrainDataframes(
        target_df=target_df,
        non_target_df=non_target_df,
        frac_eval_cat=train_settings.frac_eval_cat,
        min_size=train_settings.min_eval_fit,
        n_rep=train_settings.n_rep,
    )

The cached training frame is shuffled once at load time; a per-node query is then a lazy filter plus .head(max_eval_fit), which takes the first matching rows of the already-shuffled cache:

def _get_pl_train_df(self, full_filters, max_eval_fit):
    cached = self.get_cached_train_df()
    expr = expr_to_polars_expr(
        full_filters, schema=cached.schema, cache=self._pl_expr_cache
    )
    return cached.lazy().filter(expr).head(max_eval_fit).collect()

Split into Count and Group Sets

Each side is further split into two parts controlled by frac_eval_cat:

  • Group set (1 - frac_eval_cat): carries the weight and target_weight columns and drives the binning (the cuts of each feature).
  • Count set (frac_eval_cat): supplies the encoded codes used to build the correlation tables.

The split uses np.random.permutation with head/tail, repeated n_rep times:

class TrainDataframes:
    def __init__(self, target_df, non_target_df, frac_eval_cat, min_size, n_rep):
        self.target_df_size = target_df.height
        self.non_target_df_size = non_target_df.height

        self.n_count_target, _ = self._calc_split(target_df.height, frac_eval_cat)
        self.n_count_non_target, _ = self._calc_split(non_target_df.height, frac_eval_cat)

        self.target_df_count = []
        self.non_target_df_count = []
        self.df_group = []

        for _ in range(n_rep):
            perm_target = np.random.permutation(target_df.height)
            shuffled_target = target_df[perm_target]
            self.target_df_count.append(shuffled_target.head(self.n_count_target))
            target_group = shuffled_target.tail(-self.n_count_target)

            perm_non_target = np.random.permutation(non_target_df.height)
            shuffled_non_target = non_target_df[perm_non_target]
            self.non_target_df_count.append(
                shuffled_non_target.head(self.n_count_non_target)
            )
            non_target_group = shuffled_non_target.tail(-self.n_count_non_target)

            # Each class half carries total weight 0.5; the target rows are the
            # first block of the concatenated group frame and get 2 * w_target.
            w_target = 0.5 / max(1, target_group.height)
            w_non_target = 0.5 / max(1, non_target_group.height)
            group = pl.concat([target_group, non_target_group]).hstack(
                [
                    pl.Series(
                        "weight",
                        np.concatenate(
                            [
                                np.full(target_group.height, w_target),
                                np.full(non_target_group.height, w_non_target),
                            ]
                        ),
                    ),
                    pl.Series(
                        "target_weight",
                        np.concatenate(
                            [
                                np.full(target_group.height, 2 * w_target),
                                np.zeros(non_target_group.height),
                            ]
                        ),
                    ),
                ]
            )
            self.df_group.append(group)
flowchart LR subgraph Original O1[100K rows, Target: 10K, Non-target: 90K] end subgraph Downsampled D1[10K rows, Target: 5K, Non-target: 5K] end O1 -->|"Balance target"| D1 O1 -->|"Balance non-target"| D1 style D1 fill:#ffff99

Configuration

The downsampling behavior is configured through three settings:

n_rep: int = Field(
    description="Wie oft soll jedes Feature pro spore angepasst werden.", default=5
)
max_eval_fit: int = Field(
    description="Wieviel sollen max für training gleichzeitig benutzt werden",
    default=1000,
)
frac_eval_cat: float = Field(
    description="Wie groß ist der anteil der für eval benutzt werden soll",
    default=0.5,
)
  • max_eval_fit: Limits how many rows are sampled per node. Lower values = faster training but less precision.
  • frac_eval_cat: How much of the sampled data goes to the count set (correlation-table codes) vs the weighted group set that drives the binning.
  • n_rep: How many times the random split into count and group sets is repeated. Each repetition generates different cuts; the median result is selected to avoid statistical fluctuations. For multi-dimensional candidates, matching features from the same repetition are combined before the median is selected.

Summary

Concept Description
Independent sampling Target and non-target queried separately with .head(max_eval_fit)
Random ordering The cache is shuffled once at load time
Two-part split Group set (weighted) drives binning, count set supplies correlation-table codes
Per-node sampling Each tree node filters the pre-shuffled cache and takes the first matching rows

Next Steps