Glossary
Every important AI and ML term, explained in one breath. Search or browse.
53 terms
- Activation Function
- A nonlinear function (like ReLU or sigmoid) applied to a neuron's output. Without it, stacked layers would collapse into a single linear model.
- Deep Learning
- Agent
- A system that perceives its environment and takes actions to achieve goals. The unifying abstraction of AI — from thermostats to LLMs with tools.
- Artificial Intelligence
- AGI (Artificial General Intelligence)
- A hypothetical AI with human-level flexibility across arbitrary domains, as opposed to today's narrow systems that excel only at specific tasks.
- Artificial Intelligence
- Attention
- A mechanism letting every element of a sequence directly look at every other and pull in relevant information via query-key-value lookups. The core of transformers.
- Deep Learning
- AUC (Area Under the ROC Curve)
- A threshold-free score of a classifier's ranking quality: the probability that a random positive example scores higher than a random negative one. 0.5 = coin flip, 1.0 = perfect.
- Machine Learning
- Backpropagation
- The algorithm that computes the gradient of the loss for every weight in a network by applying the chain rule backwards through the layers.
- Deep Learning
- Bias (model parameter)
- The constant term b in a model like y = wx + b, shifting the output independently of inputs. Not to be confused with statistical or societal bias.
- Mathematics
- Bias–Variance Trade-off
- The tension between models that are too simple (high bias: systematically wrong) and too flexible (high variance: memorize noise). Generalization lives in between.
- Machine Learning
- Classification
- Supervised learning where the output is a category — spam/not-spam, cat/dog/horse — rather than a number.
- Machine Learning
- Clustering
- Unsupervised learning that groups similar examples together without labels, e.g. discovering customer segments with k-means.
- Machine Learning
- CNN (Convolutional Neural Network)
- A network that slides small learned filters over images to detect patterns, exploiting locality and translation invariance. The architecture that conquered vision.
- Deep Learning
- Cross-Entropy Loss
- The standard loss for classification: it measures how surprised the model is by the true labels, punishing confident wrong predictions hardest.
- Machine Learning
- Cross-Validation
- Evaluating a model by splitting data into k folds and training k times, each holding out a different fold — every point serves as validation once.
- Machine Learning
- Data Leakage
- When information unavailable at prediction time sneaks into training (e.g. a feature written after the outcome). Produces inflated test scores and production failures.
- Machine Learning
- Diffusion Model
- A generative model trained to remove noise step by step; generation starts from pure noise and denoises until an image crystallizes. Powers Stable Diffusion and DALL·E.
- Deep Learning
- Dot Product
- Multiply matching components of two vectors and sum. Measures alignment/similarity — the single most executed operation in AI.
- Mathematics
- Dropout
- A regularization technique that randomly silences neurons during training so the network learns redundant, robust features.
- Deep Learning
- Embedding
- A learned dense vector representing a word, image, user or any object, positioned so that geometric closeness encodes semantic similarity.
- Machine Learning
- Epoch
- One full pass of the training algorithm over the entire training dataset. Training typically runs for many epochs.
- Deep Learning
- Feature
- One measurable input property of an example — a house's size, an email's word counts. Features form the input vector x.
- Machine Learning
- Fine-Tuning
- Continuing to train a pretrained model on a smaller, task-specific dataset — adapting general knowledge to a particular job.
- Deep Learning
- GAN (Generative Adversarial Network)
- Two networks trained against each other: a generator forging samples and a discriminator detecting fakes. Produced the first photorealistic synthetic faces.
- Deep Learning
- Gradient
- The vector of all partial derivatives of a function — a compass pointing in the direction of steepest increase of the loss.
- Mathematics
- Gradient Descent
- The optimization algorithm behind nearly all model training: repeatedly step parameters in the direction that decreases the loss.
- Mathematics
- Hallucination
- When a generative model produces fluent but fabricated content — invented facts, citations or APIs — stated with unwarranted confidence.
- Artificial Intelligence
- Hyperparameter
- A knob set by the practitioner rather than learned — learning rate, tree depth, k in k-means — tuned using the validation set.
- Machine Learning
- Inference
- Using a trained model to make predictions on new data, as opposed to training. Also: the reasoning step in logic-based AI.
- Machine Learning
- Label
- The correct answer attached to a training example — the price of a house, the 'spam' tag on an email. What supervised models learn to predict.
- Machine Learning
- Learning Rate
- The step size of gradient descent. Too small: training crawls. Too large: the loss oscillates or explodes. The most important hyperparameter.
- Mathematics
- LLM (Large Language Model)
- A transformer with billions of parameters trained to predict the next token over internet-scale text, then aligned into an assistant (e.g. GPT, Claude).
- Artificial Intelligence
- Loss Function
- A formula measuring how wrong a model's predictions are. Training = minimizing it. Examples: mean squared error, cross-entropy.
- Mathematics
- Matrix
- A rectangular grid of numbers: simultaneously a table of data and a transformation of vectors. Neural network weights are matrices.
- Mathematics
- Neural Network
- Layers of simple units (weighted sums + nonlinearities) composed into one large trainable function. The foundation of deep learning.
- Deep Learning
- Overfitting
- When a model memorizes training data's noise instead of learning the pattern — great training scores, poor real-world performance.
- Machine Learning
- Parameter
- A number inside a model that training adjusts — the weights and biases. GPT-class models have hundreds of billions.
- Machine Learning
- Precision & Recall
- Precision: of everything flagged positive, how much truly was? Recall: of all true positives, how many were caught? They trade off via the decision threshold.
- Machine Learning
- Prompt Engineering
- Crafting the input text to an LLM — instructions, examples, output format — to reliably elicit the desired behavior.
- Artificial Intelligence
- RAG (Retrieval-Augmented Generation)
- Grounding an LLM by vector-searching relevant documents for each query and including them in the prompt, so answers cite real sources.
- Artificial Intelligence
- Regression
- Supervised learning where the output is a continuous number — a price, a temperature — rather than a category.
- Machine Learning
- Regularization
- Deliberately constraining a model (weight penalties, dropout, early stopping) to trade a little training fit for much better generalization.
- Machine Learning
- Reinforcement Learning
- Learning by acting: an agent tries actions, receives rewards, and improves its policy to maximize long-term reward. Trained AlphaGo; aligns LLMs via RLHF.
- Artificial Intelligence
- ReLU
- max(0, x) — the simple activation whose non-fading gradient helped unlock deep networks. The default hidden-layer nonlinearity.
- Deep Learning
- RLHF
- Reinforcement learning from human feedback: humans rank model outputs, a reward model learns those preferences, and RL optimizes the LLM against it.
- Artificial Intelligence
- Softmax
- A function turning a vector of raw scores into probabilities that sum to 1. The standard final layer of classifiers and language models.
- Machine Learning
- Supervised Learning
- Learning from examples labeled with correct answers — the dominant ML paradigm, covering classification and regression.
- Machine Learning
- Token
- The unit LLMs read and write: a learned subword chunk of text. Tokenization explains many LLM quirks, including weaker performance in underrepresented languages.
- Artificial Intelligence
- Training / Test / Validation Split
- The discipline of separating data: fit on training, tune on validation, and measure once on a locked-away test set.
- Machine Learning
- Transfer Learning
- Reusing a model pretrained on a large dataset for a new task — freeze the body, retrain the head. The blueprint of the foundation-model era.
- Deep Learning
- Transformer
- The architecture of modern AI: stacked attention + MLP blocks with skip connections, processing all tokens in parallel. Powers GPT, BERT and vision transformers.
- Deep Learning
- Underfitting
- When a model is too simple to capture the pattern at all — poor performance even on training data. The opposite of overfitting.
- Machine Learning
- Unsupervised Learning
- Finding structure in unlabeled data: clustering, dimensionality reduction, anomaly detection.
- Machine Learning
- Vector
- An ordered list of numbers — the universal container of ML. Every input, output and embedding is a vector.
- Mathematics
- Weight
- A learnable multiplier on an input or connection — the synapse strength of artificial neurons. Training means adjusting weights.
- Deep Learning