Skip to content

Downsampling

At every node, Pilz independently samples target and non-target rows. This ensures training data stays manageable regardless of the original dataset size.

How It Works

Training data is read via two separate Polars operations — one for target rows, one for non-target rows — each limited to max_eval_fit rows:

def read_akt_train(self, targer_filter, train_settings, akt_filters):
    full_filters_target_list = [targer_filter.combine] + [f.combine for f in akt_filters]
    full_filter_target = make_and(full_filters_target_list)

    if full_filter_target is FALSE:
        # The path filters exclude every row of the target class.
        return TrainDataframes(target_df=pl.DataFrame(), non_target_df=pl.DataFrame(), ...)

    target_df = self._get_pl_train_df(
        full_filters=full_filter_target,
        max_eval_fit=train_settings.max_eval_fit,
    )

    full_filters_non_target_list = [Not(targer_filter.combine)] + [f.combine for f in akt_filters]
    full_filters_non_target = make_and(full_filters_non_target_list)

    non_target_df = self._get_pl_train_df(
        full_filters=full_filters_non_target,
        max_eval_fit=train_settings.max_eval_fit,
    )

    return TrainDataframes(
        target_df=target_df,
        non_target_df=non_target_df,
        frac_eval_cat=train_settings.frac_eval_cat,
        min_size=train_settings.min_eval_fit,
        n_rep=train_settings.n_rep,
    )

The cached training frame is shuffled once at load time; a per-node query is then a lazy filter plus .head(max_eval_fit), which takes the first matching rows of the already-shuffled cache:

# inside read_akt_train, once per node; shared by both class reads
expr_cache = {}
target_df = self._get_pl_train_df(full_filter_target, max_eval_fit, expr_cache)
non_target_df = self._get_pl_train_df(full_filter_non_target, max_eval_fit, expr_cache)

def _get_pl_train_df(self, full_filters, max_eval_fit, expr_cache=None):
    cached = self.get_cached_train_df()
    expr_cache = {} if expr_cache is None else expr_cache
    expr = expr_to_polars_expr(
        full_filters,
        schema=cached.schema,
        cache=expr_cache,
        cache_limit=4096,
        cache_max_nodes=128,
    )
    return cached.lazy().filter(expr).head(max_eval_fit).collect()

Only small reusable subexpressions are retained; the cache is discarded after the node's two class reads, so unique residual predicates do not accumulate over the recursive tree.

Split into Count and Group Sets

Each side is further split into two parts controlled by frac_eval_cat:

  • Group set (1 - frac_eval_cat): carries the weight and target_weight columns and drives the binning (the cuts of each feature).
  • Count set (frac_eval_cat): supplies the encoded codes used to build the correlation tables.

The split uses np.random.permutation with head/tail, repeated n_rep times:

class TrainDataframes:
    def __init__(self, target_df, non_target_df, frac_eval_cat, min_size, n_rep):
        self.target_df_size = target_df.height
        self.non_target_df_size = non_target_df.height

        self.n_count_target, _ = self._calc_split(target_df.height, frac_eval_cat)
        self.n_count_non_target, _ = self._calc_split(non_target_df.height, frac_eval_cat)

        self.target_df_count = []
        self.non_target_df_count = []
        self.df_group = []

        for _ in range(n_rep):
            perm_target = np.random.permutation(target_df.height)
            shuffled_target = target_df[perm_target]
            self.target_df_count.append(shuffled_target.head(self.n_count_target))
            target_group = shuffled_target.tail(-self.n_count_target)

            perm_non_target = np.random.permutation(non_target_df.height)
            shuffled_non_target = non_target_df[perm_non_target]
            self.non_target_df_count.append(
                shuffled_non_target.head(self.n_count_non_target)
            )
            non_target_group = shuffled_non_target.tail(-self.n_count_non_target)

            # Each class half carries total weight 0.5; the target rows are the
            # first block of the concatenated group frame and get 2 * w_target.
            w_target = 0.5 / max(1, target_group.height)
            w_non_target = 0.5 / max(1, non_target_group.height)
            group = pl.concat([target_group, non_target_group]).hstack(
                [
                    pl.Series(
                        "weight",
                        np.concatenate(
                            [
                                np.full(target_group.height, w_target),
                                np.full(non_target_group.height, w_non_target),
                            ]
                        ),
                    ),
                    pl.Series(
                        "target_weight",
                        np.concatenate(
                            [
                                np.full(target_group.height, 2 * w_target),
                                np.zeros(non_target_group.height),
                            ]
                        ),
                    ),
                ]
            )
            self.df_group.append(group)
flowchart LR subgraph Original O1[100K rows, Target: 10K, Non-target: 90K] end subgraph Downsampled D1[10K rows, Target: 5K, Non-target: 5K] end O1 -->|"Balance target"| D1 O1 -->|"Balance non-target"| D1 style D1 fill:#ffff99

Configuration

The downsampling behavior is configured through three settings:

n_rep: int = Field(
    description="Wie oft soll jedes Feature pro spore angepasst werden.", default=5
)
max_eval_fit: int = Field(
    description="Wieviel sollen max für training gleichzeitig benutzt werden",
    default=1000,
)
frac_eval_cat: float = Field(
    description="Wie groß ist der anteil der für eval benutzt werden soll",
    default=0.5,
)
  • max_eval_fit: Limits how many rows are sampled per node. Lower values = faster training but less precision.
  • frac_eval_cat: How much of the sampled data goes to the count set (correlation-table codes) vs the weighted group set that drives the binning.
  • n_rep: How many times the random split into count and group sets is repeated. Each repetition generates different cuts; the median result is selected to avoid statistical fluctuations. For multi-dimensional candidates, matching features from the same repetition are combined before the median is selected.

Summary

Concept Description
Independent sampling Target and non-target queried separately with .head(max_eval_fit)
Random ordering The cache is shuffled once at load time
Two-part split Group set (weighted) drives binning, count set supplies correlation-table codes
Per-node sampling Each tree node filters the pre-shuffled cache and takes the first matching rows

Next Steps