Ten Thousand Rows Were Never the Data. The Weight Was.

Is your training pipeline expanding data it already counted?

I agree to the Privacy Policy Discuss Your ML Pipeline

Your warehouse already counted the payment attempts. Logistic regression can use those counts directly.

If your warehouse gives you counts, do not expand them into millions of duplicate rows just to fit logistic regression. Keep two weighted rows per group.

I tested both versions.

• Expanded: 4,872,300 event rows
• Weighted: 1,600 rows
• Model result: the same, within floating-point noise
• Local fit time: 2.48 seconds versus 5.9 milliseconds
• Design-matrix memory: 446.1 MB versus 162.5 KB

That is the whole result. The rest of this article shows why it works, how I checked it twice, and when you should not use it.

Start with the row you actually have

Most logistic-regression examples assume one row is one event:

merchant=A, device=desktop, country=IT, approved=1

Payment warehouses often give you a tally instead:

merchant=A, device=desktop, country=IT, attempts=10,102, approved=8,407

That second row is not unfinished event data. It is grouped data.

For one merchant context, you can represent the same information in three ways:

The warehouse gives you the aggregate. The modeling choice is whether to explode it into event rows or convert it into two weighted rows.

Checkpoint: 10,102 attempts do not automatically require 10,102 training rows.

First check: is the group valid?

Take the example above: 10,102 attempts and 8,407 approvals. The observed authorization rate is 83.2%.¹

You may model the approval count as:

C_i ~ Binomial(n_i, p_i)

That requires two assumptions:

1. The attempts in the group share the same probability p_i.
2. The attempts are independent.

Real payment data can break both.

A card retried seconds after a decline is correlated with the first attempt. A fraud rule changed halfway through the day means the probability was not constant across the group.

If either happens, the group may not be one binomial trial. Weighting cannot repair a badly defined group. Expanding it into rows cannot repair it either.

Checkpoint: before choosing a representation, verify that the bucket describes repeated draws from one stable probability.

Why the count becomes a weight

Let:

• n_i = number of attempts
• c_i = number of approvals
• p_i = approval probability predicted by the model

One group’s negative log-likelihood is:

L_i = −c_i · log(p_i) − (n_i − c_i) · log(1 − p_i)

Now define the observed approval rate:

r_i = c_i / n_i

Substitute it into the loss:

L_i = −n_i · [r_i · log(p_i) + (1 − r_i) · log(1 − p_i)]

The expression inside the brackets is ordinary binary cross-entropy. The value n_i in front tells the model how much evidence produced that rate.

A rate measured from 200 attempts and the same rate measured from 50,000 attempts are equal as percentages. They are not equal as evidence.

The gradient makes the idea even simpler:

∂L_i/∂z_i = n_i · p_i − c_i
            predicted approvals − observed approvals

The optimizer asks one question for each group:

How many approvals did I predict, and how many happened?

Event-level training asks that question one row at a time and adds the answers. Weighted training receives the answer already added up.

Checkpoint: the weight is the amount of evidence, not a workaround.

Local experiment: 4.87 million rows versus 1,600

I generated 800 merchant contexts across four dimensions:

• merchant segment
• device
• country
• payment method

Each group had between 200 and 50,000 attempts. The complete dataset contained:

The outcomes came from real Binomial(n_i, p_i) draws using a fixed seed.¹

I then fit the same unregularized logistic regression in two ways.

Version A: expand every attempt

# Destination: local expanded-fit experiment
X_exp = np.vstack([
    np.repeat(X[i:i+1], n[i], axis=0)
    for i in range(len(n))
])
y_exp = np.concatenate([
    np.r_[np.ones(c[i]), np.zeros(n[i] - c[i])]
    for i in range(len(n))
])

LogisticRegression(penalty=None).fit(X_exp, y_exp)

Version B: keep two weighted rows

# Destination: local weighted-fit experiment
X_w = np.repeat(X, 2, axis=0)
y_w = np.tile([1.0, 0.0], len(n))
w_w = np.ravel(np.c_[c, n - c])

LogisticRegression(penalty=None).fit(
    X_w,
    y_w,
    sample_weight=w_w,
)

Result

The largest difference across all eleven fitted coefficients was 5.4 × 10⁻¹⁴. Both fits recovered the coefficients used to generate the data within 0.007.²

I also evaluated both loss functions at the same coefficients:

This is not an approximation. It is the same objective evaluated in two representations.

Win: the weighted fit removed 4,870,700 rows without removing statistical information.

Spark check: the result survives distribution

A local process agreeing with itself is useful, but limited. I repeated the experiment on a Docker Compose Spark cluster:

• one Spark master
• two workers
• 2 cores and 2 GiB RAM per worker
• the same 800 groups
• the same 4,872,300 expanded rows
• weightCol for the weighted fit³

Spark’s StringIndexer chose a different reference category from pandas, so raw Spark coefficients cannot be compared directly with the local coefficients.

The two Spark fits can be compared with each other because they share one fitted encoding pipeline.

The expanded and weighted coefficients agreed within 1.306 × 10⁻⁸.

Both models also recovered the true generating probabilities:

• mean absolute error: 0.0007393
• correlation: 0.9999833

The independently recomputed losses agreed too:

• aggregate: 3,022,092.844745
• expanded: 3,022,092.844739
• relative difference: 1.862 × 10⁻¹²

The honest slow number

Building the 1,600-row weighted dataset took 2.504 seconds. Exploding 4.87 million rows took only 1.948 seconds.

That does not reverse the result. At this small group count, Spark scheduling overhead dominates the preparation step. There is almost no weighted data left to process.

The fit itself still took 2.461 seconds weighted versus 8.369 seconds expanded. More importantly, the weighted representation grows with the number of groups. The expanded representation grows with the number of attempts.

Checkpoint: a smaller representation does not win every isolated timing. Its scaling behavior is the reason to use it.

When should you keep event rows?

Use weighting when all of these are true:

1. Every group represents repeated independent trials at one stable probability.
2. The model can learn everything it needs from the group features and counts.
3. You are fitting a compatible objective, such as grouped logistic regression.

Keep event-level rows when the individual event contains information the aggregate lost.

Examples:

• A tree needs transaction value to split on a threshold.
• A neural network needs per-event inputs that are absent from the group.
• Time order matters because the probability changes during the window.
• Retries create dependence between attempts.

The rule is simple:

If events differ in a way the model needs, keep them. If they are repeated evidence for the same probability, weight them.

Why this matters in production

The cost is not one expanded fit. It is every repeated fit:

• every backtest
• every hyperparameter sweep
• every rolling-window retrain
• every failed job caused by worker memory

In the local experiment, expansion added 446 MB to the design matrix without adding information. At millions of groups, that choice can decide whether the job fits in memory.

Spark exposes weightCol, and scikit-learn exposes sample_weight, because multiplicity belongs in the objective. You do not need to materialize every repeated trial to make the optimizer count it.³

The one question to ask first

When the warehouse hands you counts, pause before expanding them.

Ask:

Are these genuinely different events the model needs to see, or repeated evidence for one underlying probability?

If they are repeated evidence, the count already knows its weight.

References

1. scikit-learn, LogisticRegression — sample_weight for the local weighted fit
2. NumPy, numpy.random.Generator.binomial — binomial outcome generation
3. Apache Spark, ml.classification.LogisticRegression — weightCol for the distributed fit
4. McCullagh & Nelder, Generalized Linear Models (2nd ed.) — grouped binomial data and weighted-Bernoulli equivalence
5. Dan Stativa, aggregated-payment-events — code, generated data, Docker Compose cluster, and committed results

Reproducibility notes

Keep exploring

The experiment repository contains the local fits, Spark cluster, and committed results. For another example of checking what a performance number actually measures, read The Memory Hierarchy Doesn’t Break Where You Think It Does.

Hero photo: Vagaro on Unsplash. Article photo: Crissy Jarvis on Unsplash.

Is your training pipeline expanding data it already counted?

I agree to the Privacy Policy Discuss Your ML Pipeline