Adversarial Machine Learning: Key Terms

As machine learning moves from research into production, security teams are being asked to assess systems described in unfamiliar language. Adversarial machine learning, the study of how ML models are attacked and defended, has its own vocabulary, and conversations stall when half the room does not share it. This is a plain-English glossary of the terms that matter most, with why each one should concern anyone deploying ML.

The two moments an ML system can be attacked

Almost every attack fits into one of two phases:

• Training time. The attacker influences the model while it is being built, by tampering with the data it learns from.
• Inference time. The model is already trained and deployed, and the attacker manipulates the inputs it receives or studies its outputs.

Keep that split in mind; most of the terms below hang off it.

Core attack types

Evasion attack. An inference-time attack where the input is crafted to make the model produce the wrong output while looking normal to a human. The classic example: an image altered in ways invisible to people but that cause a classifier to mislabel it entirely. Evasion is the most common practical attack against deployed models.

Adversarial example. The malicious input itself in an evasion attack, a sample deliberately perturbed to fool the model. The unsettling part is how small the change can be: a few pixels, a few characters, a slightly reworded sentence.

Data poisoning. A training-time attack. The attacker injects corrupted or misleading examples into the training data so the resulting model behaves badly, either broadly degraded or subtly wrong on specific inputs. If you train on data you do not fully control, scraped from the web, crowdsourced, or user-submitted, poisoning is a real exposure.

Backdoor (trojan) attack. A targeted form of poisoning where the model learns a hidden trigger. It behaves normally almost always, but when it sees the attacker’s secret pattern, it produces the attacker’s chosen output. Dangerous because it is nearly invisible in ordinary testing; the model looks fine until the trigger appears.

Attacks on confidentiality

Some attacks do not aim to break the model’s behavior; they aim to steal from it.

Model extraction (model stealing). By querying a model enough and studying its responses, an attacker reconstructs a functional copy, stealing the intellectual property and expensive training that went into it. A concern for any model exposed through a public API.

Model inversion. The attacker uses the model’s outputs to reconstruct characteristics of its training data, potentially recovering sensitive information about the people or records it was trained on.

Membership inference. A narrower privacy attack: determining whether a specific record was part of the training set. If your model was trained on medical, financial, or otherwise sensitive records, confirming that a particular individual’s data was used can itself be a serious privacy breach.

Terms specific to large language models

The generative AI era added vocabulary of its own:

Prompt injection. Manipulating a language model’s behavior through crafted input that overrides its intended instructions, either directly from the user or indirectly through content the model reads.

Jailbreaking. Coaxing a model past its safety guardrails to produce content or behavior it was configured to refuse.

Hallucination. The model generating confident, fluent output that is simply false. Not an attack in itself, but a reliability failure attackers can exploit, and a risk wherever output is trusted without verification.

Why the vocabulary matters

This is not academic. Each term names a concrete way a production ML system can fail:

• Deploying a public model API? Model extraction and evasion are on the table.
• Training on data you did not fully vet? Poisoning and backdoors are risks.
• Training on sensitive records? Inversion and membership inference threaten privacy.
• Shipping an LLM feature? Prompt injection and jailbreaking come with the territory.

Sharing this language lets security and engineering teams reason about ML risk the way they already reason about web and network risk, and it is the starting point for testing AI systems meaningfully. You cannot assess a threat you cannot name. If you are deploying machine learning and want to understand your real exposure, scope an assessment that maps these attack classes onto your specific system. Our AI/ML penetration testing service is built around exactly this threat model.