2025

Architecture-driven adversarial robustness.

An enhanced CORnet-S with three biology-inspired modules: a learnable retinal prefilter, gated recurrent feedback, and variance-aware denoise scaling. 97.82% robustness on MNIST under PGD, with no adversarial training.

1. Abstract

An architectural alternative to adversarial training, evaluated against four attack families on MNIST, CIFAR-100, and ImageNet100. Defenses that ship into a safety-critical perception system (medical-imaging triage, autonomous-vehicle perception, biometric authentication) need to deploy from a clean-trained model. Adversarial training is expensive, data-hungry, and depresses clean accuracy. An architectural defense ships from the same pipeline that produced the clean model, with no robustness-versus-accuracy tradeoff and no additional epochs.

CORnet-S (DiCarlo lab, MIT) is the baseline because its V1 to V2 to V4 to IT hierarchy already aligns with the primate ventral stream. Three modules go on top. A learnable retinal prefilter strips the high-frequency artifacts that carry most of a perturbation's signal. Gated recurrent blocks unroll 2 to 4 times per forward pass and suppress unstable activations. A variance-aware denoise-scaling layer amplifies stable channels and damps high-variance ones, keying on the 3 to 5 times variance signature of adversarial inputs.

The novelty is the architectural path, not the modules. The bet: the same robustness can be built into the network's structure, not its training data.

97.82% PGD robustness on MNIST, +12 pp over baseline CORnet-S, with zero adversarial training.
Consistent gains across attack families: FGSM +8 pp, PGD +12 pp, CW +9 pp, Patch +7 pp.
Clean accuracy preserved across all three datasets: 99.2% MNIST, 76.4% CIFAR-100, 82.1% ImageNet100.
~15% inference-time overhead versus baseline CORnet-S; +2.3M parameters.
Architectural by construction. The model never sees an adversarial example; the defense drops into the clean training pipeline as layer-level replacements.

2. Introduction

Adversarial training works, but at a price. This work asks a different question: can the same robustness be built into the architecture, leaving the training procedure untouched?

Adversarial training is a cost line. 10 to 100 times the training compute, a separate evaluation harness, and an attack distribution to refresh as new attacks land.
Clean accuracy pays the bill. Standard adversarial training costs a few clean-accuracy points on most benchmarks, an uncomfortable trade for deployments that mostly see clean inputs.
Biology already solved a version of this. The primate visual system resists noise without training on noisy data. The resilience comes from the structure of the retina-LGN-cortex pathway.

2.1 Why an architectural defense, not adversarial training

Instead of showing the model adversarial examples, the network is built so that inputs that fool a vanilla classifier get suppressed by the new layers.

CORnet-S as the substrate. Its V1, V2, V4, IT hierarchy is biologically aligned, so each addition lives where its analogue lives in the brain, keeping the design interpretable rather than ad hoc.
Three properties of biological vision. Retinal preprocessing, recurrent cortical feedback that gives ambiguous stimuli a second look, and adaptive suppression of unstable activations. Each maps to one module.
Differentiable, not gradient-masked. All three modules are end-to-end differentiable and expose a clean gradient. Gradient-masking defenses look strong against unaware attacks and collapse under adaptive attack; these modules avoid that failure mode by design.

2.2 What is actually new

The CORnet-S backbone is from the DiCarlo lab, the attacks from TorchAttacks, the datasets public. The contribution is the three modules and the evidence that they deliver the headline number without adversarial training.

Three biology-derived modules in one network. Each is small and ablated independently. The defense is their composition, not any single piece.
No adversarial examples in training. Training is the same clean-image protocol used for the baseline CORnet-S: same loss, same data, same hyperparameters.
Cross-attack consistency, not attack-specific patches. A defense that only helps against the attack it was tuned on is not architectural; gains here hold across all four families.

The sections below are the engineering behind those three commitments.

3. System

3.1 Learnable retinal prefilter

A lightweight learnable denoising stage sits in front of CORnet-S's feedforward path: the artificial analogue of the retina. Adversarial perturbations live disproportionately in high-frequency spatial bands, and the prefilter learns to attenuate that band while preserving the low-frequency content that carries class identity.

Learned, not fixed. No Gaussian or median filter, no non-differentiable rounding step. The frequency response is learned end to end for each dataset rather than committed to a band ahead of training.
Roughly 2% forward-pass overhead, the cheapest of the three modules.
Ablation: removing the prefilter drops robustness by 8 to 10 pp across all three datasets.

3.2 Gated recurrent blocks

The residual connections inside CORnet-S are replaced with gated recurrent units that unroll 2 to 4 times per forward pass. Sigmoid gates decide which features persist: high-variance activations characteristic of adversarial noise are gated out, stable ones propagate.

Temporal depth. The hidden state carries across iterations, so a perturbation that fools the first unroll must also fool the later ones, a stricter condition than a single-pass attack assumes.
Ablation: removing recurrence drops robustness by 12 to 15 pp across the three datasets, the largest single contribution of the three modules.

3.3 Variance-aware denoise-scaling

A learnable per-channel scalar sets feature emphasis from each channel's activation variance across the recurrent iterations. Stable channels get amplified; high-variance channels get suppressed.

The signal it keys on. Adversarial examples produce 3 to 5 times higher feature variance than clean images at the same layer, measured across the recurrent unrolls.
Soft scaling, not hard thresholding. Continuous scaling preserves gradient flow and avoids the gradient shattering a hard threshold would introduce.
Particularly effective against PGD. Attenuating high-variance features weakens the gradient signal iterative PGD relies on to converge. The PGD gain (+12 pp) is the largest of the four attacks measured.

4. Evaluation

Datasets: MNIST, CIFAR-100, and ImageNet100 (the 100-class ImageNet subset). Attack families: FGSM, PGD, CW, and a patch attack, all from TorchAttacks at ε ∈ {0.01, 0.05, 0.1, 0.3}. Baselines: vanilla CORnet-S, ResNet-18, AlexNet. Training: Adam, learning rate 0.001, step decay 0.1 every 30 epochs. The headline robustness number is the model's clean accuracy on adversarially-perturbed inputs at the strongest tested ε for each attack.

Horizontal bar chart of the gain in percentage points over baseline CORnet-S, on MNIST, across four attack families: FGSM +8, PGD +12, CW +9, Patch +7. — Figure 1. Gain over baseline CORnet-S across attack families, on MNIST. Consistent across attack types, not tuned to any one of them.

Metric	MNIST	CIFAR-100	ImageNet100
Clean accuracy	99.2%	76.4%	82.1%
PGD robustness (Ours)	97.82%	89.3%	84.7%
PGD gain vs CORnet-S	+12 pp	+12 pp	+12 pp
vs ResNet-18 (iterative attacks)	+5 to +8 pp	+5 to +8 pp	+5 to +8 pp
vs AlexNet	+15 to +20 pp	+15 to +20 pp	+15 to +20 pp

Cross-dataset. The PGD gain holds on CIFAR-100 and ImageNet100, not only in the toy regime.
Cross-architecture. +5 to +8 pp over ResNet-18 under iterative attacks (PGD, CW). +15 to +20 pp over AlexNet across the board, which says more about AlexNet than about the defense.
Transfer. A model trained on CIFAR-100 retains roughly 85% of its robustness on ImageNet100, so the defense is at least partly distribution-independent.

These numbers come from a single training seed per dataset under this project's protocol. They are not a leaderboard or SOTA claim against RobustBench, AutoAttack, or any other standardized benchmark. Adaptive attacks designed against these three modules have not been tested, and a defense without an adaptive-attack evaluation is by convention an upper bound, not a deployable robustness floor. Read them as evidence of the tradeoff direction, not a benchmark result.

5. Conclusion and Implications

Adversarial robustness does not have to be purchased with adversarial training. The biology behind the primate visual system's resilience (retinal preprocessing, cortical recurrence, variance-aware suppression) can be built as architectural primitives that defend a network without ever showing it an adversarial example. A clean-trained model can then ship into a safety-critical setting (medical-imaging triage, autonomous-vehicle perception, biometric authentication) with robustness that costs neither weeks of additional training nor clean accuracy. The deployment story is not "we beat the leaderboard"; it is "the defense drops into the existing training pipeline," the load-bearing distinction for any team that must add robustness without changing its data pipeline or training budget.

5.1 Limitations and future work

No adaptive-attack evaluation. The highest-priority next step: white-box attacks designed against the prefilter, the recurrence, and the variance-aware scaling layer specifically.
Single seed, no RobustBench or AutoAttack. Direct comparisons against published defenses are not warranted. Next: multi-seed confidence intervals and standardized evaluation.
ImageNet100 and CORnet-S only. Next: full ImageNet, and the three modules on a ResNet, ViT, or ConvNeXt to test whether the defense is a CORnet-S idiosyncrasy or a general architectural pattern.
~15% inference overhead and +2.3M parameters. Modest, but edge deployment with tight latency budgets would need to weigh it against the gain.
Open questions. Whether a small amount of adversarial training composes additively with the architectural defense or saturates, and whether Brain-Score confirms the additions preserve neural predictivity, so the biological framing is not just aesthetic.

6. Acknowledgments

Built on the CORnet-S architecture from the DiCarlo lab at MIT (Kubilius et al., 2019). Attack implementations from the TorchAttacks library.

@inproceedings{kubilius2019cornets,
  title     = {Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs},
  author    = {Kubilius, Jonas and Schrimpf, Martin and Kar, Kohitij and Rajalingham, Rishi and Issa, Elias and Hong, Ha and Majaj, Najib and Bashivan, Pouya and Prescott-Roy, Jonathan and Schmidt, Kailyn and Nayebi, Aran and Bear, Daniel and Yamins, Daniel L K and DiCarlo, James J},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2019},
  url       = {https://arxiv.org/abs/1909.06161}
}