Published on

Why Batch Size Matters More Than Learning Rate

Authors

You know what is funny? We spent a decade obsessing over learning rate schedules - cosine annealing, cyclic rates, warmup periods, one-cycle policies - like we were tuning a Formula 1 car. Meanwhile, batch size has been sitting there controlling gradient noise, memory throughput, and generalization capacity, and most people treat it as "whatever fits in my GPU."

That is like optimizing your tire pressure while ignoring that you're driving on three wheels.

Let me be clear: learning rate matters. But batch size determines the character of your optimization. It decides whether you're doing stochastic gradient descent (noisy, explorative, generalizes) or gradient descent (smooth, deterministic, overfits). And we have been treating it as a hardware constraint instead of a fundamental algorithmic choice.

Time to fix that.

The Math Nobody Wants to Think About

Here is what is actually happening when you compute a gradient.

The true gradient of your loss L\mathcal{L} over the entire dataset D\mathcal{D} is:

∇θL=1∣D∣∑x∈D∇θℓ(x,θ)\nabla_{\theta} \mathcal{L} = \frac{1}{|\mathcal{D}|} \sum_{x \in \mathcal{D}} \nabla_{\theta} \ell(x, \theta)

But computing this is expensive, so we approximate it with a minibatch B\mathcal{B}:

∇θLB=1∣B∣∑x∈B∇θℓ(x,θ)\nabla_{\theta} \mathcal{L}_{\mathcal{B}} = \frac{1}{|\mathcal{B}|} \sum_{x \in \mathcal{B}} \nabla_{\theta} \ell(x, \theta)

This estimator is unbiased (good), but it has variance:

Var[∇θLB]∝σ2∣B∣\text{Var}[\nabla_{\theta} \mathcal{L}_{\mathcal{B}}] \propto \frac{\sigma^2}{|\mathcal{B}|}

where σ2\sigma^2 is the variance of individual gradients.

Notice something? Variance scales inversely with batch size. Double your batch, halve your gradient noise. Use the full dataset, variance goes to zero.

Most people read this and think "great, let's use huge batches for stable gradients." That's exactly backwards.

Gradient Noise Is a Feature, Not a Bug

The noise in your gradients isn't something to minimize. It's your exploration mechanism.

Think about it: if your gradient estimator has zero variance, you're computing the exact direction toward the nearest local minimum. You'll march straight there, deterministically, and get stuck. Classic gradient descent behavior.

But with high-variance gradients (small batches), your updates are stochastic. You bounce around the loss surface. Sometimes you overshoot. Sometimes you climb up briefly. This randomness helps you:

  1. Escape shallow minima: A noisy gradient can kick you out of a local minimum that a smooth gradient would trap you in.
  2. Find flat minima: Random perturbations naturally favor wide valleys over sharp peaks (more on this later).
  3. Explore the solution space: You're not just descending, you're sampling different trajectories.

This is why small-batch SGD generalizes better than large-batch GD. The noise isn't corrupting your optimization - it's enabling it.

The Generalization Mystery

Here's a dirty secret about deep learning: nobody really knows why it generalizes. The models are overparameterized. They can fit random labels. Classical learning theory says they should overfit catastrophically.

But they don't. And batch size is a huge part of why.

Several papers (Keskar et al., 2017; Hoffer et al., 2017) have shown that small-batch training finds flatter minima. A flat minimum means small changes to the weights don't change the loss much. This is exactly what you want for generalization: a solution that is robust to perturbations is a solution that will work on new data.

Large batches, meanwhile, converge to sharp minima. The loss is low, but it's sitting on a knife edge. Perturb the weights slightly (which is what happens when you evaluate on test data with different statistics), and your loss spikes.

Why does small-batch training find flat minima? Because the gradient noise acts like implicit regularization. It's essentially doing Langevin dynamics - you're not just following the gradient, you're doing a random walk biased toward lower loss. This naturally favors wide valleys where the noise doesn't kick you out.

Important caveat: The relationship between sharpness and generalization is more complex than originally thought. Later work has shown that sharpness measures are sensitive to reparameterization, and sharp minima can sometimes generalize well. The connection between batch size and generalization likely involves multiple mechanisms beyond just minima sharpness.

The Hardware Trap

Here's why we got this wrong: GPUs love large batches.

Modern accelerators are designed for massive parallel throughput. Feed them a batch of 4096 samples and they will crush it. Feed them a batch of 32 and they will sit there twiddling their thumbs while you shuffle data.

So the default advice became: "Use the largest batch size that fits in memory."

This is optimizing for hardware utilization, not model quality. It's like writing code that is fast to compile but slow to run.

The irony is that larger batches aren't even that much faster in practice. Sure, they have higher FLOP utilization. But you need more total gradient steps to reach the same validation loss. So your wall-clock time often ends up similar, except your test accuracy is worse.

What the Evidence Actually Says

Let's look at what actually works in practice.

ImageNet baselines: ResNet-50 trained with batch size 256 is a standard baseline. Papers that push to batch size 8192+ require careful optimization techniques (LARS optimizer, extensive warmup, layer-wise adaptive rates) to match the baseline performance. (You et al., 2017) These aren't just workarounds. They are genuine algorithmic innovations that address the fundamental challenge of maintaining gradient noise properties while scaling batch size. However, they do add complexity and require careful tuning.

Language models: BERT was trained with batch size 256 sequences (roughly 128K tokens per batch). GPT-2's exact batch size isn't documented in the original paper, though estimates based on the GPT-3 paper suggest approximately 0.5M tokens (roughly 512 sequences of 1024 tokens) for similar model sizes. GPT-3 scaled up to 3.2 million tokens per batch, but only by carefully engineering their learning rate schedule and combining it with gradient accumulation. The pattern is clear: you can use large batches if you're willing to do a ton of extra work to compensate for the downsides.

CIFAR-10 records: Most state-of-the-art results use batch sizes in the 64-256 range, not 1024+.

Recent developments: Interestingly, very recent work (2024-2025) has shown that with proper hyperparameter tuning (especially careful adjustment of the second moment decay β2\beta_2 in Adam), even batch size 1 can match the performance of standard large-batch training. This suggests the story is even more nuanced than "small batches are always better." The real key is maintaining appropriate gradient noise characteristics, which can be achieved through batch size, optimizer configuration, or both.

The Memory Efficiency Myth

"But larger batches are more memory efficient!"

Are they though?

Sure, a single forward/backward pass with batch size 1024 uses memory more efficiently than 32 separate passes with batch size 32. But you're also:

  1. Storing more activations: Memory usage grows linearly with batch size.
  2. Needing more total steps: You have to train longer to compensate for the reduced stochasticity.
  3. Potentially using a worse final model: Which means you might need a bigger network to hit the same accuracy.

The real memory efficiency question is: what batch size gives you the best test accuracy per GPU-hour? And the answer is usually "smaller than you think."

When Large Batches Actually Make Sense

Look, I'm not saying large batches are always wrong. There are legitimate use cases:

Data parallelism at scale: If you're training across 1024 GPUs, you need a large global batch size for communication efficiency. But even then, the best approach is often to use a moderate per-GPU batch (say, 32-64) and scale by adding more GPUs, not by cranking up the per-device batch size.

Second-order methods: If you're using an optimizer that explicitly models the curvature (K-FAC, natural gradient, etc.), larger batches give you better curvature estimates. But you're also paying for more expensive updates.

Fine-tuning with stable targets: If your model is already good and you're doing light fine-tuning, large batches with small learning rates can work fine. The exploration matters less when you're near a good solution.

But for training from scratch? For finding a solution that generalizes? Small batches win, at least when using standard optimizers with standard hyperparameters.

The Practical Recommendations

So what should you actually do?

Start small: Begin with batch sizes in the 32-128 range. Yeah, it feels wasteful on a GPU. Do it anyway.

Scale learning rate with batch size: If you double your batch size, you roughly double your effective learning rate (you're averaging over more samples). So scale α∝∣B∣\alpha \propto |\mathcal{B}| initially, then tune. (Goyal et al., 2017) This linear scaling rule has theoretical justification and works well in practice up to a point.

Monitor gradient variance: If your gradients are super noisy (high variance), you might need a slightly larger batch or lower learning rate. If they are too smooth (low variance), consider reducing batch size to add exploration.

Measure what matters: Don't optimize for GPU utilization. Optimize for test accuracy per unit of training time. That's the actual metric.

Use gradient accumulation if you must: If you're memory-constrained and can't fit a small batch, accumulate gradients over several micro-batches. You get the memory profile of a large batch with the stochastic properties of a small one. Best of both worlds.

Consider optimizer adjustments: If you need larger batches for practical reasons, explore optimizer modifications like adjusting β2\beta_2 in Adam or using techniques like LAMB (You et al., 2019) that are specifically designed to handle large-batch training more gracefully.

The Bigger Picture

Here's the thing that bugs me: we have built an entire infrastructure (GPUs, frameworks, training practices) that pushes us toward large batches. And then we are surprised when our models don't generalize as well.

The hardware is optimized for throughput. The algorithms are optimized for generalization. These are fundamentally different objectives, and we have been defaulting to hardware convenience.

Batch size isn't a hyperparameter you set once and forget. It's a core part of your optimization algorithm. It determines whether your gradient descent is stochastic (good) or deterministic (bad). It controls your implicit regularization. It decides whether you'll find a flat minimum or a sharp one.

Treating it as "whatever fits in memory" is like treating learning rate as "whatever doesn't NaN." Technically you can do it, but you're leaving performance on the table.

The next time someone tells you to maximize your batch size, ask them: are we optimizing for GPU utilization or model quality? Because if the answer is "both," they are probably getting neither.

Further Reading:

  • Keskar et al., "On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima" (2017) - arXiv:1609.04836
  • Smith et al., "Don't Decay the Learning Rate, Increase the Batch Size" (2017) - arXiv:1711.00489
  • Hoffer et al., "Train longer, generalize better: closing the generalization gap in large batch training of neural networks" (2017) - arXiv:1705.08741
  • Masters & Luschi, "Revisiting Small Batch Training for Deep Neural Networks" (2018) - arXiv:1804.07612
  • Goyal et al., "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour" (2017) - arXiv:1706.02677
  • You et al., "ImageNet Training in Minutes" (2017) - arXiv:1709.05011
  • You et al., "Large Batch Optimization for Deep Learning: Training BERT in 76 minutes" (2019) - arXiv:1904.00962
  • Malladi et al., "Small Batch Size Training for Language Models" (2024) - arXiv:2507.07101
Share: