What is a neuron

A neural network is built from one small part, repeated up to billions of times. The part is a neuron: it takes a few input values, weighs each one, adds them up, and passes the total through a simple shaping function to produce its output. Every computation a network performs builds from this part, and so does every way it can turn singular. The clearest way to understand a neuron is to operate one.

Each knob is one weight, scaling its input before the sum; the bias shifts the total on its own. The activation turns that total into an output, and the line in the decision space is the boundary the neuron draws between its two answers. Train it on OR or AND and the line settles into place. Switch to XOR and it never does: the two positive cases sit on opposite corners, and a single straight line can never separate them. One neuron draws one cut, and that is its ceiling.

A neuron over an image

Nothing about a neuron limits it to two inputs. Give it a small image with one weight per pixel, and those sixteen weights become a picture of their own, the template the neuron is looking for.

Training sets the sixteen weights together, and the weight grid organises into a detector for whatever you asked it to find: top against bottom, an X against an O. The score is simply how well the input lines up with the template. The same ceiling is still here, though. The vertical-against-horizontal task has no single template that scores every vertical line above every horizontal one, so the accuracy stalls partway, the sixteen-input echo of XOR.

Stacking neurons into a network

To get past that ceiling you feed several neurons into another one. That is a network, and the neurons in between let the boundary bend.

With a hidden layer doing the early work, the once-straight boundary curves around the data, and the network solves the XOR that stopped the single neuron. This is the whole idea of depth: simple parts composed into something that can carve up its input in far richer ways. A working network carries thousands or billions of these weights, and no task pins every one of them down; that slack is where the story turns.

From knobs to a landscape

There is one more way to see what those knobs do. Every setting of the weights produces some total error on the data, so plotting that error against the weights draws a landscape: each point is one setting of the weights, and its height is how badly that setting fits. Turning a knob slides you across the terrain, and training is the act of rolling downhill toward lower ground.

For a single neuron fitting ordinary data the terrain is a round bowl with one lowest point, so there is a single best setting and training rolls straight to it. Hold on to that picture of the bottom as one point, because the twist is what becomes of it in a real network.

Many weights, one output

Stack even two neurons and the bottom of that bowl stops being a single point. Take the smallest possible chain, where the output is the input multiplied by two weights in series. To hit a given target you only need the product of the two weights to come out right, and endlessly many pairs do.

Every weight pair on the gold curve computes the identical output, so the network fits the data equally well all along it. The best fit is a whole curve of weight settings, each as good as the others. Slide along the curve and the weights change while the function stays put: you have found a direction the output simply cannot see. This is your first meeting with a dead direction, in its most extreme form. The family is named for a sensitivity that dies: the general member is a direction along which the output responds ever more weakly as training carries the weights toward a special point, and this one is the limiting case, dead from birth, because the output never responded to it at all. A real network is laced with both kinds, and learning to tell them apart is much of what this section does. What it means for a model when its best fits form a region of weight space rather than a single point is the subject of why neural networks are singular#2.