The Artificial Neuron

A pinch of biology

Your brain contains ~86 billion neurons. Each collects electrical signals from thousands of others through connections called synapses; if the combined stimulation crosses a threshold, the neuron fires, passing a signal onward. Crucially, synapses strengthen and weaken with experience — that's what learning physically is. Artificial neural networks borrow exactly this sketch (and, honestly, not much more — planes don't flap wings).

The artificial neuron

An artificial neuron does three small things:

1. Weigh the inputs: multiply each input $x_i$ by a weight $w_i$ — the synapse strength. Positive weights excite, negative inhibit, near-zero ignore.
2. Sum plus bias: $z = \mathbf{w} \cdot \mathbf{x} + b$. Recognize it? A dot product — the neuron computes exactly the linear model from the ML path. The bias $b$ shifts the firing threshold.
3. Activate: pass $z$ through a nonlinear activation function $a = g(z)$.

$a = g(\mathbf{w} \cdot \mathbf{x} + b)$

One neuron = logistic regression (when $g$ is sigmoid). Deep learning's bet: what one neuron can't do, billions of cooperating ones can.

Why the nonlinearity is everything

Suppose you stack layers of linear-only neurons. Composing linear functions gives… another linear function (matrix multiplication — math path). A 100-layer linear network collapses into one line. All the power of depth comes from inserting a nonlinearity between layers.

The classic choices:

• Sigmoid $\sigma(z) = \frac{1}{1+e^{-z}}$ — squashes to (0,1). Historically central, now mostly reserved for output probabilities: in deep stacks its flat tails kill gradients (you'll see why in the backprop lesson).
• Tanh — like sigmoid but (−1, 1), zero-centered.
• ReLU $\;g(z) = \max(0, z)$ — outputs the input if positive, else zero. Embarrassingly simple, computationally free, and its gradient doesn't fade for positive inputs. ReLU's adoption (~2011) is one of the small changes that unlocked the deep learning era. Modern variants (Leaky ReLU, GELU) tweak the negative side.

Default advice you'll hear everywhere: ReLU-family in hidden layers; sigmoid/softmax at the output when you need probabilities.

What can one neuron do?

Geometrically, a single neuron draws one line (hyperplane) and answers "which side?". It can learn AND and OR. Famously, it cannot learn XOR ("exactly one of the two") — no single line separates XOR's classes. This 1969 observation (Minsky & Papert) froze neural-network research for years. The resolution: stack neurons into layers, and XOR falls immediately — two neurons carve two lines, a third combines them. From this tiny example flows the whole story of depth: combining simple boundaries yields arbitrarily complex ones.

Key takeaways

• Neuron = dot product + bias + nonlinearity; one neuron ≈ logistic regression.
• Without nonlinear activations, any deep stack collapses to a single linear model.
• One neuron draws one line; networks of them draw anything — XOR is the classic gateway example.