Skip to content

Pair-Correlation Filter (Feature Masking)

The n_dims search evaluates combinations of features, but combinations are only useful if the features bring independent information. When two features carry (nearly) the same signal, every combination that includes both wastes calcs_per_dim budget on a redundant candidate — and the resulting split is no better than splitting on either feature alone.

The pair-correlation filter detects this redundancy during the search and masks the redundant feature so it is not used again in the rest of the node. It is the multi-dimensional counterpart of the 1D is_diff_to_low() check.

The idea in one sentence

Whenever a candidate combination's joint table shows that both classes concentrate on a dominant bin (given the other features), one feature of that combination is redundant — so its 1D-weakest member is masked immediately for the rest of the node.

How the filter works

The filter runs inside pilz_rs.counter_batch, the native split search, for every dimension stage 2..=n_dims:

  1. For each evaluated candidate, take its median repetition joint correlation table (the table is already computed for scoring, so the check costs nothing extra).
  2. For every candidate position, group the cells by the remaining features and record, per group, the fraction of each class that falls into that position's dominant bin.
  3. Sum those per-group maxima over all groups.
  4. If both classes put more than pair_corr_min_main (default 0.9) of their mass into the dominant bins, the combination is flagged as correlated:
  5. the candidate itself is excluded from the current best-split race,
  6. the 1D-weakest feature of the candidate is masked immediately.

A masked feature is removed from the feature pool for the rest of that node: every later combination of the node that contains it is skipped without consuming calcs_per_dim budget.

Node-local masking

The mask is computed per node and reset at the start of the next node. A feature masked in one subtree remains fully available in every other subtree. This matters for MNIST-like data: a pixel that is redundant given another pixel in one branch may carry new information in a different branch.

The 1D-best feature can never be masked

Features are sorted by their 1D score descending, and the weakest member of a combination is always its last position. The single best feature (the first position) can therefore never be the weakest member of any combination — so the filter can never mask the strongest feature of the node.

n_dims: 1

With n_dims: 1 there are no combination stages, so no pair filtering happens.

Configuration

Setting Default Description
pair_corr_filter true Enable the per-node correlation filter
pair_corr_min_main 0.9 Minimum share of both classes inside the dominant bin (per remaining-feature cell, summed over all cells) for a candidate to count as correlated. Mirrors the 1D is_diff_to_low threshold
pair_corr_max_features None Restrict the whole dim-2 combination generation to the top-K features by 1D score (the candidate participants, not just the concentration check). Useful for wide datasets (e.g. 784 pixel features) where the number of pairs grows quadratically

Effect of pair_corr_min_main

Higher values only prune near-deterministic dependencies; lower values prune more aggressively (and risk masking features that are only mildly redundant).

Value Effect
0.95 Conservative — only prune nearly identical features
0.9 (default) Prune features that concentrate with the same dominant bin
0.7 Aggressive — prune features that merely trend together

What the filter does — and does not — buy

Benefits

  • calcs_per_dim only counts combinations that are actually evaluated. Combinations containing a masked feature are skipped without consuming budget, so the budget stretches further for the meaningful candidates.
  • Reduces redundant splits that would otherwise chase the same correlation again in a different subtree of the same node.
  • Costs nothing extra: the check reuses the correlation table already built for scoring.

Trade-offs

  • Masking is a hard, greedy decision within a node: once a feature is masked it never returns in that node, even if a later combination would use it differently.
  • A too-low pair_corr_min_main can mask features that are only mildly redundant, shrinking the pool too far.

Verification on MNIST

On the 10 000-row MNIST test set, the filter ran during every multi-node training pass and reduced both the number of surviving spores and the number of evaluated candidates without hurting accuracy. The 9% speed figure is a measured observation from local benchmark runs, not a value encoded in the source or the checked-in bench_fidelity.csv: in those runs the filter was about 9% faster than the same config with the filter disabled, with an insignificant accuracy change. The effect compounds on wide, highly correlated data (such as the 784 pixel features of MNIST): without the filter the search spends most of its budget re-testing pixel pairs that carry the same stroke.

Where it lives in the code

  • Split search + filter driver: pilz_rs.counter_batch (rust/src/lib.rs)
  • Concentration computation: concentration_positions — a mixed-radix group index over the candidate positions (no hashing), so the per-node filter adds essentially no cost to the correlation scoring path.
  • Masked-pair results are surfaced to Python on the CounterBatchResult (masked_pairs) and retained on the trainer for inspection via Train._debug_masked_pairs.

See also