End of Year 2 Project (PFA2)
Beylessen Jendoubi & Maram Bahri
🩷 Read more about it on my Blog: From Adversarial Attacks to Defensive Design: A Red Teaming Approach to AI Security
This project is an end-to-end AI red-teaming exercise against an image-based malware classification pipeline. It asks one practical question:
The project runs the full offense/defense cycle:
- Recon & baseline : train two independently architected classifiers to simulate a defender's production model and an attacker's surrogate.
- White-box attacks : DeepFool and PGD, evaluated for minimal-perturbation evasion and targeted misclassification.
- Black-box attacks : transferability studies, plus a conditional GAN (AdvGAN-style) trained against a surrogate ensemble, simulating an attacker with zero access to the victim model's weights.
- Threat modeling : every technique mapped to MITRE ATLAS, because knowing what you're defending, and from whom, has to come before you defend it.
- Defense : curriculum-scheduled PGD adversarial training applied to the victim model, then re-measured against every attack above, including the ones it never saw during training.
- Source: Malware-to-image dataset (byte sequences rendered as 299×299 RGB images), 8 malware families + benign samples — built on the Malware Benign Image Classification Dataset.
- Classes (8):
adware,backdoor,benign,downloader,spyware,trojan,virus,worm. - Split: 70% train / 20% validation / 10% test (stratified per class).
| Split | adware | backdoor | benign | downloader | spyware | trojan | virus | worm | Total |
|---|---|---|---|---|---|---|---|---|---|
| Train | 1,390 | 471 | 6,043 | 1,749 | 662 | 2,497 | 1,674 | 949 | 15,435 |
| Val | 397 | 135 | 1,727 | 500 | 189 | 714 | 478 | 272 | 4,412 |
| Test | 199 | 68 | 864 | 250 | 95 | 357 | 240 | 136 | 2,209 |
Preprocessing: images resized to 224×224, normalized with ImageNet statistics (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) to match pretrained backbones.
Two independently trained classifiers stand in for a production model and an attacker's local copy / different-vendor model : the standard setup for measuring cross-architecture transferability.
| Model A | Model B | |
|---|---|---|
| Backbone | VGG16 (frozen features + custom head) | ResNet-50 (frozen features + custom FC head) |
| Head | Linear(4096→n_classes) |
Linear(2048→1000) → ReLU → Linear(1000→n_classes) |
| Val accuracy | ~97.98% | ~97.01% |
| Test accuracy | ~99.05% (clean, later run) | ~96.92% / 99.23% (clean, later run) |
| Role | White-box victim (attacker has full gradient access) | Black-box transfer target and hardened defense target |
Both models are serialized as TorchScript (torch.jit.script) for deployment-realistic inference in the attack scripts.
Minimal-norm evasion. Implemented from scratch.
| Metric | Value |
|---|---|
| Attack Success Rate | 100.0% (2,209/2,209) |
| Avg L2 Perturbation | 0.0499 |
| Avg L∞ Perturbation | 0.0027 |
| Avg Iterations | 5.9 |
| Confidence Drop | 0.961 → 0.484 |
Common misclassifications: benign→trojan (561×), downloader→benign (250×), virus→adware (215×)
L∞-bounded iterative FGSM.
Configuration: ε=0.005, α=0.001, 10 steps
| Metric | Value |
|---|---|
| Clean Accuracy | 96.97% |
| Adversarial Accuracy | 0.00% |
| Attack Success Rate | 100.0% |
Forcing worm → benign evasion (security-critical scenario).
Configuration: ε=8/255, α=2/255, 40 steps, random start
| Metric | Value |
|---|---|
| Targeted Success Rate | 136/136 = 100.0% |
| Benign Confidence | ~100.00% |
Generated white-box against one model, replayed black-box against the other.
| Attack | Source | Target | Transfer Rate |
|---|---|---|---|
| DeepFool (untargeted) | Model B → Model A | 30 samples | 20.0% |
PGD targeted (worm→benign) |
Model A → Model B | 30 samples | 3.3% |
Key insight: Untargeted attacks transfer moderately well. Targeted evasion barely transfers.
Zero access to victim models. Trains a conditional perturbation generator against a locally-built surrogate ensemble.
Architecture:
- U-Net style generator (encoder-decoder + class embeddings)
- PatchGAN discriminator
- C&W attack loss + L∞ perturbation budget + feature matching
Surrogate Ensemble: VGG16 (0.7) + ResNet18 (0.3)
Training: 30 epochs, ε = 0.1, feature loss enabled
| Target | Targeted ASR | Untargeted ASR |
|---|---|---|
| Surrogate VGG16 | — | — |
| Model A (VGG16, white-box) | 0.23% | 46.49% |
| Model B (ResNet-50, black-box) | 1.99% | 51.83% |
Bottom line: Training AdvGAN against a surrogate ensemble substantially improves transferability over iterative attacks. Without ever querying the target model, the attack achieves 51.83% untargeted and 1.99% targeted success on the black-box ResNet-50 victim, demonstrating that learned, transferable perturbations pose a realistic threat even in zero-query settings.
Simple idea: train the model on adversarial examples.
Smarter implementation: ramp perturbation budget gradually (ε: 2/255 → 8/255 over 4 epochs) so the model doesn't collapse early. Each batch split 50/50 clean/adversarial (7-step PGD).
| Model | Clean | PGD-20 | DeepFool | AdvGAN |
|---|---|---|---|---|
| Standard ResNet-50 | 99.28% | 0.00% | 0.27% | 49.75% |
| Curriculum-PGD ResNet-50 | 98.55% | 95.30% | 98.55% | 98.51% |
| Gain | -0.73 pp | +95.30 pp | +98.28 pp | +48.76 pp |
Why it matters: Defending against PGD alone also hardens against:
- DeepFool (geometrically different attack)
- AdvGAN (generative, black-box attack)
The defense generalizes beyond what it trained on.
-
Attack findings: Even without model access, a blind attacker can meaningfully disrupt predictions
-
Defense findings: Adversarial training recovers most of the lost ground without sacrificing clean accuracy.
-
Threat modeling perspective: The model is only one piece of a much larger attack surface.
-
Targeted black-box transfer never exceeded 2%
-
Surrogate ensemble helped, but two architectures is still a thin approximation
-
More architectural diversity in surrogate ensembles
-
Input transformation strategies during training
-
Cross-domain transferability (testing if adversarial examples generalize across datasets and problem domains entirely)
-
Adaptive attack evaluations (stress-testing whether robustness reflects genuine hardening vs. gradient masking)
- Moosavi-Dezfooli, S-M., Fawzi, A., Frossard, P. (2016). DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks. CVPR.
- Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A. (2018). Towards Deep Learning Models Resistant to Adversarial Attacks. ICLR.
- Xiao, C., Li, B., Zhu, J-Y., He, W., Liu, M., Song, D. (2018). Generating Adversarial Examples with Adversarial Networks (AdvGAN). IJCAI.
- Cai, Q-Z., Liu, C., Song, D. (2018). Curriculum Adversarial Training. ICLR Workshop.
- Sitawarin, C., Chakraborty, S., Wagner, D. (2021). SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing. AISec.
- MITRE ATLAS — Adversarial Threat Landscape for Artificial-Intelligence Systems. https://atlas.mitre.org/
- OWASP — Machine Learning Security Top 10. https://owasp.org/www-project-machine-learning-security-top-10/