What Is a Neural Network?

Local AI 101

Lesson 6: What Is a Neural Network?

Level: Beginner → Intermediate
Estimated reading time: 70–90 minutes
Prerequisites: Lessons 1–5
Next lesson: Lesson 7: How Neural Networks Learn

What You Will Learn

By the end of this lesson, you should be able to:

• Explain what an artificial neural network is.
• Understand why neural networks are inspired by biological nervous systems without being literal copies of brains.
• Explain the idea of an artificial neuron.
• Understand inputs, weights, bias, and outputs.
• Understand why multiplication and addition appear everywhere inside neural networks.
• Explain activation functions and why they matter.
• Understand layers and why stacking layers creates deeper models.
• Explain forward propagation.
• Understand what a tensor is at a practical level.
• Understand how a network transforms numerical input into numerical output.
• Understand why nonlinear activation functions are essential.
• Understand the difference between parameters and hyperparameters.
• Understand what makes a network “deep.”
• Understand how networks can represent increasingly complicated patterns.
• Connect neural networks to modern language models.
• Understand why a language model is not simply one giant neuron.
• Understand where training enters the neural-network picture.
• Understand how inference flows through a trained network.
• Develop the mental model needed for the next lesson on how neural networks learn.

1. Opening the Black Box

In Lesson 4, we introduced the idea of an AI model. We learned that a model contains numerical parameters.

In Lesson 5, we learned that training changes those parameters, while inference normally uses them without permanently changing them.

But there is a major question we have not answered yet:

What are those parameters actually connected to?

What is the thing that contains them? What does a neural network actually look like? How can a collection of numbers produce something as complicated as language generation?

The answer begins with a surprisingly simple idea:

A neural network is a mathematical function made from many interconnected computational units and parameters.

At its most basic level, a neural network takes numbers as input, performs mathematical operations on them, and produces numbers as output.

That sounds almost absurdly simple.

And yet, by combining enormous numbers of these operations in carefully structured ways, neural networks can model extremely complicated relationships.

Modern AI systems are built on this basic idea. The language models used for Local AI are not magic boxes.

Underneath all the interfaces, model files, prompts, tokenizers, inference runtimes, and applications is a large mathematical system.

That system is composed of layers of numerical operations. The goal of this lesson is to make that system understandable. We are going to start with one artificial neuron.

Then two. Then a layer. Then multiple layers.

Then a deep network. Then we will connect that network to the models you use in Loco.

2. First: What Is a Biological Neuron?

The phrase “neural network” comes from biology. Your brain contains biological neurons.

A biological neuron receives signals through structures called dendrites, processes those signals, and can transmit signals through an axon to other neurons.

A simplified biological picture looks something like:

Biological neuroscience is enormously more complicated than this diagram. Real neurons involve:

• electrochemical processes,
• membrane potentials,
• ion channels,
• neurotransmitters,
• complex connectivity,
• timing,
• feedback,
• different cell types,
• and many other mechanisms.

Artificial neural networks are not realistic simulations of individual biological neurons. The biological analogy is useful because it gives us the idea of:

Many interconnected units collectively performing computation.

That is the important connection.

3. Artificial Neurons Are Mathematical Objects

An artificial neuron is not a tiny digital brain cell. It is better understood as a mathematical operation. A simplified artificial neuron receives several input values.

Each input is multiplied by a weight. The weighted values are added together. A bias is added.

Then an activation function may transform the result. Conceptually:

Input 1 ──× Weight 1 ──┐ Input 2 ──× Weight 2 ──┤ Input 3 ──× Weight 3 ──┤ ├──> Sum + Bias ──> Activation ──> Output Input 4 ──× Weight 4 ──┤ Input 5 ──× Weight 5 ──┘

That is an artificial neuron at a very simplified level. The basic equation is:

Then:

where:

• x values are inputs,
• w values are weights,
• b is a bias,
• f is an activation function.

This tiny equation is one of the foundations of modern neural networks.

4. Why Multiplication?

You might ask:

Why multiply each input by a weight?

Because the weight allows the network to control how strongly that input contributes to the result. Imagine a simple fictional prediction system. Suppose the inputs are:

The model might assign different weights to them. For example:

These numbers are purely illustrative. The weighted contributions become:

hours studied × 0.8 hours slept × 0.3 practice tests × 1.1

The network combines those contributions. The weights determine how much influence each input has. During training, the network learns useful values for those weights.

5. A Tiny Numerical Example

Suppose a neuron receives:

And has:

The neuron calculates:

So:

Then an activation function processes 1.7. If the activation function simply returned the same value, the output would be:

1.7

But many neural networks use nonlinear activation functions. We will get to those shortly. The important point is that the neuron is performing ordinary mathematical operations.

There is no mysterious process happening inside the individual neuron. The complexity comes from combining enormous numbers of simple operations.

6. What Is a Weight?

A weight is a numerical parameter that controls the contribution of an input to a computation. Weights are parameters. That means they can be changed during training.

Imagine:

If the weight is large and positive, the input may have a strong positive influence. If the weight is near zero, its direct influence may be small. If the weight is negative, the contribution can move in the opposite direction.

Again, this is a simplification.

In modern neural networks, weights participate in large matrix operations rather than existing as isolated little knobs.

But the underlying idea remains:

Weights are learned numerical parameters that control how information is transformed.

7. What Is a Bias?

A neuron can also have a bias. The bias is another learned parameter. In the simple equation:

the bias shifts the result. A useful analogy is an adjustable baseline. Imagine a thermostat.

The input signals might influence temperature, while the bias establishes a baseline tendency. This is only an analogy. Mathematically, the bias gives the network another degree of freedom when learning a function.

Weights determine how inputs contribute. The bias allows the output to be shifted. Both can be learned during training.

8. Why One Neuron Is Not Enough

One artificial neuron can represent relatively simple mathematical relationships. But real-world problems are much more complicated. Consider language.

Understanding even a simple sentence can involve relationships between:

• words,
• grammar,
• syntax,
• meaning,
• context,
• entities,
• relationships,
• and previous information.

One mathematical operation is nowhere near enough. So we connect many neurons. This gives us a layer.

9. What Is a Layer?

A layer is a collection of computational units operating together. Imagine three inputs connected to four neurons:

Inputs Layer

x₁ ────────────────> [N₁] \ [N₂] \───────────> [N₃] x₂ ────────────────> [N₄] x₃ ────────────────/

In a real dense layer, many or all inputs may connect to many neurons. Conceptually:

Each neuron can learn different combinations of the inputs. This is where things start becoming interesting.

10. Why Multiple Neurons Help

Suppose one neuron learns one useful pattern. Another neuron learns a different pattern. Another learns another.

The next layer can combine those patterns. Conceptually:

This idea becomes incredibly powerful when repeated across many layers. A network can build increasingly complicated representations from simpler ones.

11. The Input Layer

A neural network needs some representation of its input. For a simple numerical problem, the input could be:

[2.4, 7.1, 0.8, 3.2]

Each number represents a feature. For an image, the input could be pixel values. For text, the network does not directly consume letters as characters.

Text is transformed into numerical representations. For modern language models, token IDs and learned representations are involved. We will explore tokens in Lesson 10.

For now, remember:

Neural networks operate on numerical representations.

12. Neural Networks Do Not Directly Understand Words

This is another important misconception. A neural network does not receive the English word:

cat

as some magical semantic object. The text first goes through a process that turns language into numerical representations. For a language model, the rough pipeline is:

This is one reason tokenization is so important. The neural network operates mathematically. It needs numbers.

13. Hidden Layers

Layers between the input and output are often called hidden layers. Why “hidden”? Because they are internal computational representations that are not directly exposed as the model's final answer.

A simple network might look like:

A deeper network might have many more:

The word deep in deep learning refers broadly to neural networks with many layers of learned transformations.

14. Why Depth Matters

A single layer can perform a certain mathematical transformation. Multiple layers can compose transformations. Suppose:

Layer 1 learns simple pattern A. Layer 2 combines A with pattern B. Layer 3 combines those into pattern C. Layer 4 uses C to identify pattern D.

The network can construct increasingly complex representations. A simple analogy is language. Letters can form words.

Words can form phrases. Phrases can form sentences. Sentences can form paragraphs.

The analogy is not a literal description of how neural networks internally work. But it captures the general intuition that complex structures can be built from simpler representations.

15. The Importance of Nonlinearity

This is one of the most important technical ideas in neural networks. Suppose every layer only performed linear operations. You might have:

Layer 1: y = W₁x

Layer 2: z = W₂y

Layer 3: output = W₃z

Substituting one into another:

The composition is still effectively a linear transformation.

Stacking many linear layers without nonlinear activation functions does not give you the expressive power you might expect from a deep network.

This is why neural networks use nonlinear operations.

16. Activation Functions

An activation function transforms a neuron's intermediate value. Conceptually:

Activation functions introduce nonlinearity. Some common activation functions include:

• ReLU,
• sigmoid,
• tanh,
• GELU,
• SiLU,
• and others.

Different architectures use different functions.

Modern transformer models often use activation functions such as GELU or related functions within their feed-forward components.

The exact choice matters to architecture design and training behavior.

17. ReLU

One of the most famous activation functions is ReLU, or Rectified Linear Unit. It can be described as:

So:

x = -3 → 0 x = -1 → 0 x = 0 → 0 x = 2 → 2 x = 5 → 5

Graphically, it behaves roughly like:

Output | | / | / | / | / |__/______ Input 0

ReLU is simple, computationally efficient, and historically important.

18. Sigmoid

Another classic activation function is the sigmoid. Its output is between 0 and 1. Conceptually:

very negative → near 0 0 → 0.5 very positive → near 1

Its curve resembles an S. Sigmoid functions have historically been important in neural networks and remain useful in some contexts. However, different architectures use different activation functions for different reasons.

19. GELU

Modern transformer-based language models commonly use activation functions from the GELU family in their feed-forward components.

GELU stands for Gaussian Error Linear Unit. You do not need to memorize the full equation yet. The important concept is:

Neural networks need nonlinear transformations, and modern architectures use carefully chosen nonlinear functions to help the network represent complex relationships.

The specific activation function is an architectural detail.

20. Forward Propagation

Now we can define forward propagation, also called the forward pass. This is the process of sending input through the network to calculate an output. Imagine:

The information flows forward. During inference, this is the central operation. During training, the forward pass is followed by loss calculation and backpropagation.

You saw this distinction in Lesson 5.

21. A Tiny Forward Pass

Suppose we have:

Input: x = [2, 3]

The first layer performs:

Then:

The second layer does:

Then another activation. Eventually:

Even a tiny network can involve many calculations. A modern neural network performs enormous numbers of them.

22. Matrices Enter the Picture

You may have noticed that the previous equations use W. That usually represents a matrix of weights.

Instead of thinking about every neuron individually, modern neural-network software performs operations on arrays of numbers.

For example: Input vector:

[ x₁ ] [ x₂ ] [ x₃ ]

Weight matrix:

[ w₁₁ w₁₂ w₁₃ ] [ w₂₁ w₂₂ w₂₃ ] [ w₃₁ w₃₂ w₃₃ ]

Matrix multiplication combines these efficiently. This is one reason GPUs are so useful for neural networks. Large matrix operations can be performed in parallel.

23. Why GPUs Love Neural Networks

A huge portion of neural-network computation involves operations that can be expressed as matrix or tensor operations. For example:

Matrix A × Matrix B = Matrix C

If the matrices are large, there can be millions or billions of individual arithmetic operations. Many of these operations can be executed simultaneously. GPUs are highly optimized for this kind of parallel computation.

This is one of the fundamental reasons GPUs became so important to modern AI.

24. What Is a Tensor?

You will encounter the word tensor constantly in AI. A practical beginner definition is:

A tensor is a multidimensional array of numerical values.

Examples: A single number:

5

can be considered a scalar. A list:

[1, 2, 3]

is a one-dimensional array, often called a vector. A grid:

[ [1, 2], [3, 4] ]

is a two-dimensional array, often called a matrix. Add more dimensions and you get tensors. For example:

[batch, sequence, features]

might describe a three-dimensional tensor. The exact dimensions depend on the model.

25. Why Tensors Matter for Local AI

When you run a model locally, the inference engine is performing huge numbers of tensor operations. Conceptually:

This is what your hardware is actually doing.

The CPU or GPU is not literally “thinking.”

It is performing numerical operations. The resulting behavior emerges from the structure of the model and its learned parameters.

26. Parameters vs Activations

This is an important distinction.

Parameters

Learned during training. Examples:

• weights,
• biases.

Activations

Intermediate values produced while the network processes an input. For example:

Parameters are relatively persistent components of the trained model. Activations are generated during a particular computation. This distinction becomes important when discussing memory usage.

27. Parameters Are Stored

When you download a model file, the file contains numerical information representing the model's learned parameters and other model metadata.

The inference system loads these values. They are then used in computations. For example:

The model file is therefore much more than a text document. It is a numerical representation of a trained computational system.

28. Activations Are Created During Inference

Suppose the model receives a prompt. The input enters the network. The first layers produce intermediate values.

Those values are passed forward. More intermediate values are created. Eventually the model produces output-related values.

Those intermediate values are activations. Conceptually:

The exact memory requirements depend on architecture and inference strategy. But activations are one reason inference requires memory beyond simply storing the model file.

29. Parameters and Hyperparameters

Another important distinction:

Parameters

Learned by the model. Examples:

• weights,
• biases.

Hyperparameters

Chosen by humans or training systems. Examples include:

• learning rate,
• batch size,
• number of layers,
• hidden dimension,
• training schedule,
• optimizer settings.

The distinction is:

Parameters are learned values inside the model. Hyperparameters are configuration choices that influence how the model is built or trained.

Not every architectural setting is traditionally called a hyperparameter in exactly the same way, but the distinction is useful.

30. The Architecture

A neural network's architecture describes how its computational components are organized. It can include:

• number of layers,
• type of layers,
• dimensions,
• connections,
• activation functions,
• attention mechanisms,
• normalization components,
• positional mechanisms,
• and other structural choices.

Think of architecture as the design of the machine. The parameters are the learned numerical settings inside that design. This is similar to the distinction from Lesson 4:

31. Architecture vs Parameters

Imagine two networks with the same architecture. They can still behave very differently if their parameters differ. Conceptually:

Same structure. Different learned values. Therefore:

Same architecture ≠ Same model behavior

Likewise:

Different architecture ≠ Same capabilities

Architecture and learned parameters interact.

32. Why Neural Networks Can Represent Complex Functions

A neural network can be viewed as a function approximator. It receives inputs. It transforms them.

It produces outputs. With enough capacity and appropriate training, neural networks can approximate extremely complicated relationships. For example, a network could learn a relationship between:

This idea is foundational to machine learning. The network does not need a human programmer to manually specify every relationship. Training can discover useful parameter configurations.

33. The Universal Approximation Idea

You may eventually encounter the universal approximation theorem.

Very roughly, certain neural-network architectures can approximate a wide range of continuous functions given sufficient capacity and appropriate activation functions.

This does not mean:

“Any neural network can solve any problem.”

It does not guarantee:

• easy training,
• good generalization,
• practical efficiency,
• useful data,
• or a good solution.

It is a mathematical result about representational capacity. The practical lesson is:

Neural networks can represent remarkably complicated functions.

34. More Layers Do Not Automatically Mean Better

It would be a mistake to conclude:

“Just add more layers and the model becomes smarter.”

Not necessarily. Increasing depth can introduce:

• optimization challenges,
• computational costs,
• memory requirements,
• latency,
• instability,
• and diminishing returns.

Architecture matters. Training matters. Data matters.

Parameter count matters. Optimization matters. The relationship between model size and capability is complex.

35. Width vs Depth

Two useful concepts are:

Depth

How many computational stages or layers the network has.

Width

How many units or dimensions exist within a layer. A simplified picture: Narrow network:

[N] [N] [N]

Wide network:

[N][N][N][N][N][N][N][N]

A network can be:

• deep and narrow,
• shallow and wide,
• deep and wide,
• or something more complicated.

Modern architectures often use dimensions that are not easily described as simply “number of neurons.”

Still, depth and width are useful conceptual tools.

36. From Neurons to Modern Architectures

The simple artificial neuron is the foundation.

But modern language models do not consist of billions of isolated perceptrons arranged in a simplistic feed-forward stack.

Modern architectures use sophisticated components. For example, transformer-based language models can contain:

• token embeddings,
• attention mechanisms,
• feed-forward networks,
• normalization,
• positional information,
• residual connections,
• and many repeated transformer blocks.

We will study transformers in detail later. For now, understand the progression:

The modern model is built from increasingly sophisticated combinations of basic mathematical ideas.

37. What Is a Perceptron?

One of the historical building blocks of neural networks is the perceptron. A simplified perceptron takes inputs, applies weights, sums them, and applies a decision function. Conceptually:

Perceptrons are historically important because they demonstrate how simple learned numerical weights can produce useful decision boundaries.

Modern neural networks are vastly more sophisticated. But understanding the perceptron makes the origin of the artificial-neuron concept easier to appreciate.

38. Decision Boundaries

Imagine two groups of points:

Group A: ● ● ● ●

Group B: ○ ○ ○ ○

A simple model might learn a boundary separating them. For example:

● ● ● | ○ ○ ● ● ● | ○ ○ ● ● ● | ○ ○

The model is effectively learning a mathematical rule for separating the groups. A single linear neuron can represent certain simple boundaries. Multiple layers and nonlinearities allow networks to represent much more complicated boundaries.

39. Why Nonlinear Layers Are Powerful

Imagine trying to classify shapes. A straight line might separate some simple cases. But suppose the classes are arranged like this:

○ ○ ○ ○ ○ ○ ● ● ○ ○ ● ● ○ ○ ○ ○

A single linear boundary may not solve the problem. Multiple nonlinear transformations can create more complicated decision regions. This is a simplified illustration, but it demonstrates why depth and nonlinearity matter.

40. Representation Learning

One of the most important ideas in modern deep learning is representation learning. Instead of humans manually designing every feature, the network can learn useful representations from data. For example, in image processing, earlier layers might learn representations related to:

• edges,
• textures,
• shapes.

Later layers can combine those into more complex structures. In language, learned representations can capture relationships involving:

• words,
• syntax,
• semantic patterns,
• entities,
• context,
• and other structures.

The exact internal representations are complex and cannot always be described cleanly in human language. But the central idea is:

The network learns useful numerical representations as part of the training process.

41. Why This Was Such a Big Deal

Older machine-learning systems often relied more heavily on manually engineered features. Humans would decide:

“These particular measurements are important.”

Deep learning shifted much of this feature-discovery burden toward the model. Instead of:

deep learning can often look more like:

This ability to learn representations contributed heavily to the success of deep learning.

42. The Layer-by-Layer Intuition

Imagine a fictional language network.

We should not interpret this as a literal description of what individual neurons “mean,” but it can be useful as a conceptual illustration.

A lower-level representation might respond to patterns related to:

word forms

A later representation might capture:

syntactic relationships

A later representation might capture:

semantic relationships

A later representation might help with:

task-specific output

Real transformer models are much more complicated than this simple hierarchy. But the general idea of layered transformations is fundamental.

43. Residual Connections

Modern deep networks often use residual connections. A residual connection allows information to bypass a transformation and be combined with the transformed result. Conceptually:

Mathematically, a simplified version might look like:

where F(x) represents a learned transformation.

Residual connections became extremely important in deep neural-network architecture because they help information and gradients flow through deep networks.

Transformers use residual pathways extensively.

44. Normalization

Modern neural networks also commonly use normalization techniques. Normalization can help keep internal numerical behavior in useful ranges and improve training stability. You may encounter terms such as:

• LayerNorm,
• RMSNorm,
• BatchNorm.

Different architectures use different approaches. The exact mathematics can become quite technical. At this stage, remember:

Neural networks contain many supporting components beyond neurons and weights.

The simple neuron is the foundation, not the complete modern architecture.

45. Neural Networks Are Mathematical Pipelines

A useful way to think about a neural network is:

The network does not contain a tiny person reading the input. It performs numerical transformations. The remarkable part is that training can discover parameter values that make those transformations useful.

46. Where “Knowledge” Comes From

When a language model answers:

“The capital of Japan is Tokyo.”

Where did that information come from? The model's learned parameters contain representations resulting from training. Those parameters allow the network to produce the appropriate sequence of tokens under suitable input conditions.

But there is no simple guarantee that the model is consulting a database row. The information is represented through distributed numerical relationships. This is one reason understanding neural networks is necessary for understanding both their power and their limitations.

47. Distributed Representations

Modern neural networks often use distributed representations.

Instead of representing a concept with one number or one neuron, information can be distributed across many dimensions and components.

For example, a concept might correspond to a pattern across a large vector. Conceptually:

That vector is not a simple dictionary definition. Its meaning emerges from its relationships with other representations and transformations in the network. This distributed structure is one reason neural networks can represent complex relationships.

48. Embeddings

An embedding is a numerical representation of some object. For language, an embedding might represent a token or other linguistic unit as a vector. Conceptually:

Another token:

The actual values are model-specific and illustrative here. The important idea is that semantically related things can occupy related regions of representation space. Embeddings are central to modern AI systems.

They are also extremely important for retrieval systems, which we will study later.

49. Vector Space Intuition

Imagine a coordinate system. Instead of two dimensions:

imagine hundreds or thousands of dimensions. A concept can be represented as a point in that space. Other concepts occupy other positions.

Relationships between those points can encode useful information. Humans cannot easily visualize spaces with hundreds or thousands of dimensions. But the mathematics works perfectly well.

This is one reason AI can operate on representations that are difficult for humans to interpret directly.

50. The Model's Internal Representations Are Not Human-Readable

A common misconception is that researchers can simply inspect the model and read its knowledge. Not usually. You can inspect:

• weights,
• activations,
• attention patterns,
• embeddings,
• gradients,
• and other quantities.

But interpreting exactly what a model has learned can be difficult. This is part of the broader field of interpretability and mechanistic interpretability. Researchers investigate how internal components contribute to model behavior.

This remains an active area of research.

51. Neural Networks Can Be Extremely Large

A simple network might have:

10 parameters

A larger one:

10,000 parameters

A modern neural model can contain:

Millions Billions or more

parameters. Remember:

Parameter count is not a direct intelligence score.

A model's capabilities depend on many factors, including:

• architecture,
• training data,
• training process,
• parameter count,
• optimization,
• post-training,
• context handling,
• inference configuration,
• and task.

Still, scale is an important part of modern deep learning.

52. Why Billions of Parameters?

Because complex functions may require substantial representational capacity. Imagine trying to describe a complicated landscape with only three adjustable numbers. You would have very limited flexibility.

Give yourself thousands of adjustable values and you can represent much more complexity. The analogy is imperfect, but useful. More parameters provide more capacity.

That does not guarantee better results. The parameters must be trained effectively.

53. The Relationship Between Data and Parameters

A model with billions of parameters still needs useful training data. Imagine having a huge blank notebook. The notebook has enormous capacity.

But if you never write anything useful in it, capacity alone does not create knowledge. Training data provides the examples from which parameters are adjusted. So a simplified relationship is:

All four matter.

54. Why Data Quality Matters

More data is not automatically better. Poor-quality data can introduce:

• noise,
• duplication,
• errors,
• unwanted biases,
• irrelevant content,
• and other problems.

Training systems therefore often involve data processing and filtering. The quality and composition of training data influence what the model learns. This is another reason a model is not simply defined by its parameter count.

55. What Happens During Training to the Network?

Now connect this lesson to Lesson 5. Suppose the network contains:

Weights Biases Other learned parameters

The model processes a training example. It produces a prediction. Loss is calculated.

Backpropagation calculates gradients. The optimizer updates parameters. Conceptually:

The apostrophe means:

a slightly updated version of the network.

Repeat this enough times and the network can become highly capable.

56. Training Is Finding Useful Parameter Values

This gives us a powerful way to understand training. Training is not primarily:

“Writing facts into a neural network.”

It is:

Searching for parameter values that cause the network to perform well on its training objective.

This is a much more accurate mental model. The learned model is the result of optimization.

57. Inference Is Running the Network Forward

Once training is complete, inference looks more like:

The parameters are loaded. The network performs its transformations. The output is produced.

For a language model, that output is generally represented as probabilities over possible next tokens. A token is selected. The process repeats.

That is how a trained neural network becomes a text-generating system.

58. The Output of a Language Model

Suppose the prompt is:

The sky is

The model may calculate probabilities for many possible next tokens. Conceptually:

blue 0.42 clear 0.12 dark 0.08 falling 0.01 ...

These values are illustrative. The model chooses a token according to its generation strategy. Then it processes the expanded sequence.

For example:

Each step is part of inference.

59. The Neural Network Is Not Producing Words Directly

The model ultimately manipulates numbers. The generated token is selected from a probability distribution. The token is then converted back into text through the tokenizer.

So the complete process is roughly:

We will explore tokens and logits in much greater detail later.

60. What Are Logits?

A logit is an unnormalized numerical score produced by a model before probabilities are calculated. Suppose the model considers several possible next tokens. It may produce scores like:

blue 4.2 green 2.1 yellow 1.4 purple 0.3

These are not necessarily probabilities. A transformation such as softmax can convert them into a probability distribution. Conceptually:

This is one of the final stages involved in token generation.

61. Softmax

Softmax converts a set of numerical scores into values that sum to 1. Conceptually:

The exact mathematical formula is:

You do not need to memorize it yet. The key concept is:

Softmax turns relative scores into a probability distribution.

This becomes important when we study how language models generate text.

62. One Network, Many Tasks

A sufficiently capable neural network can sometimes perform multiple tasks because the same learned representations can be reused.

For example:

This flexibility is one of the reasons foundation models became so important. The model does not necessarily contain one isolated neural network for every sentence it can generate. Instead, a shared set of learned parameters can support many behaviors.

63. The Transformer Changes the Story

At this point you might ask:

“So are modern language models just giant stacks of ordinary neurons?”

Not exactly. Modern language models commonly use the Transformer architecture or architectures derived from transformer ideas. Transformers introduced a powerful mechanism called attention.

Attention allows the model to dynamically consider relationships between different parts of the input sequence. This became a major breakthrough in natural-language processing. We will study attention and transformers later.

For now, remember:

64. Why Attention Matters

Consider:

The dog chased the ball because it was moving.

What does “it” refer to? Understanding the sentence requires relationships between different tokens. Attention mechanisms allow transformer models to calculate relationships between token representations.

This gives the architecture a powerful way of incorporating contextual information. Attention is not the entire transformer. But it is one of the defining ideas.

Lesson 13 will cover attention in depth.

65. Neural Networks and Context

The neural network itself does not simply contain a fixed English dictionary. It receives numerical representations representing the current input. The resulting computations depend on that input.

Therefore, the same learned parameters can produce different outputs depending on the context. Conceptually:

The parameters are shared. The input changes. The resulting computation changes.

66. Why Prompt Wording Matters

Suppose you ask:

Explain quantum mechanics.

Then:

Explain quantum mechanics to a ten-year-old using a simple analogy.

The underlying model parameters are the same. But the input context differs. The network processes different input tokens.

Therefore, its internal activations and resulting output can differ substantially. This is why prompt engineering can change model behavior without changing the underlying model.

67. The Network Does Not “Look Up” the Prompt

The model is not generally searching a dictionary inside itself for each word. It processes the numerical representations through the network. Each layer transforms those representations.

The resulting representation is used to calculate output scores. This is a fundamentally different computational process from a traditional search engine or database query.

68. Why Neural Networks Can Feel Intelligent

Consider what happens when billions of learned parameters interact with:

• rich representations,
• attention,
• nonlinear transformations,
• contextual input,
• large training datasets,
• and powerful inference hardware.

The resulting behavior can be remarkably sophisticated. The individual operations are relatively simple. The overall system is not.

This is an important lesson in complex systems:

Simple operations can produce extremely complex behavior when arranged at sufficient scale and structure.

69. Emergent Behavior

You may hear the term emergence in AI discussions.

It can refer to capabilities or behavioral patterns that become apparent as models scale or as systems become more complex.

The term is used in different ways in research and popular discussion, and not every claim about emergence is equally well established.

The useful point for now is:

A model's overall behavior can be far more complicated than the behavior of any individual component.

You should not expect to understand a language model by inspecting one neuron. The capability comes from interactions across the network.

70. The Network as a Computational Graph

Another useful concept is the computational graph. A computational graph represents operations and dependencies. For example:

A modern neural network is effectively a huge computational graph. During inference, values flow through the graph. During training, gradients flow backward through the graph.

This connects directly to the forward and backward passes discussed earlier.

71. Forward Pass vs Backward Pass

Let's make the distinction extremely clear.

Forward pass

Backward pass

Inference primarily needs the forward pass. Training requires the forward pass plus the machinery needed to calculate and apply updates. This is one of the clearest ways to understand why training is more demanding.

72. Inference Through a Local Model

Imagine Loco has loaded a trained local model. You provide a prompt. The inference engine performs the forward computations.

Conceptually:

Your CPU, GPU, or other accelerator performs these calculations. The model's parameters are used. No ordinary training loop is happening.

73. Why Memory Matters

The neural network's parameters must be accessible during inference. If the model requires more memory than your system can provide efficiently, problems can occur. Potential symptoms include:

• failure to load,
• swapping,
• extremely slow performance,
• memory exhaustion,
• or reduced usability.

This is why model size matters so much for Local AI. You are not just downloading a file. You are preparing to execute a large numerical computation.

74. RAM and VRAM

You will study hardware in detail later, but we can introduce the distinction.

RAM

General-purpose system memory.

VRAM

Memory associated with a GPU. Depending on the inference system, some or all model data may be placed in GPU memory, system memory, or a combination. This is often called offloading when portions of computation or model data are moved between memory domains.

The details depend on the inference runtime and hardware.

75. Why Memory Bandwidth Matters

Imagine that your processor is extremely fast at calculations. But the model's data cannot be supplied to it quickly enough. The processor may spend time waiting for data.

This is why memory bandwidth matters. Local AI performance is therefore not simply:

CPU/GPU speed

It can involve:

Compute + Memory capacity + Memory bandwidth + Model architecture + Implementation

This is one reason two computers with similar-looking specifications can perform differently on the same model.

76. The Neural Network and Quantization

Quantization changes how the model's numerical parameters are represented. Suppose a weight is stored using a higher numerical precision. Quantization may represent it using fewer bits.

Conceptually:

The inference engine reconstructs or interprets the numerical values needed for computation. The goal is to reduce memory and improve efficiency while preserving as much useful model behavior as practical. Quantization will receive an entire lesson later.

77. Why Quantized Models Are Still Neural Networks

Quantization does not turn the model into something fundamentally unrelated. It is still the same general learned computational system. The numerical representation has changed.

Think:

The model's architecture and learned behavior are related, but the numerical approximation is different. This is why different quantizations can have slightly different behavior.

78. What Is a Neural Network Layer Doing?

At a high level, a layer performs a transformation. You can think of it as:

The transformation may involve:

• matrix multiplication,
• bias addition,
• activation functions,
• normalization,
• attention,
• gating,
• or other operations.

A modern transformer block is much more sophisticated than a single neuron. But the core principle remains:

Take numerical representations and transform them into new numerical representations.

79. Repeated Transformation

Suppose:

x₀ = input

x₁ = Layer₁(x₀)

x₂ = Layer₂(x₁)

x₃ = Layer₃(x₂)

...

xₙ = Layerₙ(xₙ₋₁)

The network repeatedly transforms the representation. Each stage can alter the information in useful ways. By the final layers, the representation may contain information useful for predicting the output.

This is the essence of deep computation.

80. A Conceptual Example With Images

Although our main focus is language models, images provide an intuitive example. Imagine a network processing a photograph. A simplified conceptual hierarchy might be:

This is not a literal map of every modern vision model. But it illustrates hierarchical representation learning.

Language models also learn layered representations, although transformer architectures organize the computation differently.

81. A Conceptual Example With Language

For language, a simplified conceptual hierarchy might look like:

Again, do not interpret each layer as having one clean human-readable job. Modern language models distribute information across many components. The hierarchy is useful as intuition.

82. Why Neural Networks Need Training

If you randomly initialize all the weights in a neural network, the network is not automatically useful. It needs learned parameter values. Training provides those values.

Imagine:

Architecture + Random/initial parameters = Untrained network

Then:

The architecture provides the structure. Training supplies the learned numerical configuration.

83. Random Initialization

Neural networks often begin training with parameter values generated according to carefully chosen initialization strategies.

The values are not necessarily literally arbitrary random numbers. Initialization methods are designed to help optimization begin effectively. The important concept is:

The network begins without the final learned parameter configuration.

Training gradually moves it toward a useful solution.

84. Why the Starting Point Matters

Optimization does not magically search every possible parameter configuration. Modern neural networks have enormous parameter spaces. Training methods use numerical optimization to move through that space.

Initialization can therefore influence training behavior. So can:

• learning rate,
• optimizer,
• data ordering,
• architecture,
• regularization,
• batch size,
• and many other choices.

Training a neural network is a complicated engineering process.

85. The Parameter Space

Imagine a model with only two parameters:

w₁ w₂

You could visualize possible configurations as a 2D landscape. Each point represents one possible pair of parameter values. The loss function gives each point a score.

You might imagine valleys:

Training attempts to move toward lower-loss regions. A real model has millions or billions of parameters, so this landscape exists in a vastly higher-dimensional space. We cannot visualize it directly.

But the mathematical idea is similar.

86. Why “Learning” Is a Useful Metaphor

The model is not conscious. It does not sit there thinking:

“I should increase this weight because I now understand grammar.”

Instead, mathematical optimization changes the parameters. We call this learning because the model's behavior improves as a result. This is why technical AI education should distinguish:

Human learning

from:

Machine learning

They are related metaphors, not identical processes.

87. Neural Networks Do Not Have to Be Language Models

Neural networks are a broad family of computational systems. They can be used for:

• image processing,
• speech recognition,
• recommendation systems,
• forecasting,
• classification,
• robotics,
• scientific modeling,
• computer vision,
• language processing,
• and many other tasks.

A language model is one type of neural-network-based system. Local AI includes more than text generation. Loco's long-term ecosystem can potentially involve many types of models.

88. Different Architectures for Different Problems

Different problems benefit from different architectural ideas. Historically and currently, you may encounter architectures such as:

• feed-forward networks,
• convolutional neural networks,
• recurrent neural networks,
• transformers,
• autoencoders,
• diffusion models,
• graph neural networks,
• mixture-of-experts systems,
• and others.

These are not all interchangeable. They make different design choices. The transformer became particularly important for modern language modeling.

We will study the historical progression in later lessons.

89. Feed-Forward Networks

A feed-forward network generally moves information forward through a sequence of transformations. Conceptually:

There is no simple recurrent loop in the basic architecture. This is one of the easiest neural-network structures to understand. Many more advanced architectures build additional mechanisms on top of this basic idea.

90. Convolutional Neural Networks

Convolutional neural networks, or CNNs, became highly influential in computer vision. They use convolution operations that can detect spatial patterns. For an image:

CNNs demonstrate another important principle:

Architecture can encode useful assumptions about the structure of the data.

For images, spatial locality matters. For language, sequence relationships matter. Different architectures exploit different structures.

91. Recurrent Neural Networks

Before transformers became dominant in many language tasks, recurrent neural networks were an important approach to sequence processing.

A simplified recurrent network passes information from one time step to the next. Conceptually:

This allows previous information to influence later processing.

RNNs were important historically and remain relevant in some applications, but modern language models are heavily associated with transformer-based architectures.

92. Transformers

Transformers introduced a different approach to sequence processing based heavily on attention mechanisms.

Instead of relying primarily on a sequential recurrent state, transformers can calculate relationships among tokens through attention.

A simplified transformer block contains components such as:

Real architectures vary. But this is a useful conceptual picture.

93. Why Transformers Matter to Loco

Many modern local language models use transformer-derived architectures.

Therefore, when you download a language model for Loco, you are often dealing with a sophisticated neural network containing many repeated computational blocks.

The model file stores learned numerical information. The inference runtime executes the architecture. Your prompt provides the input.

The output is generated through repeated numerical computation. This connects all six lessons so far.

94. The Full Stack So Far

You can now visualize the system as:

And the neural network itself came from:

And Loco sits around the inference process:

This is the beginning of the complete Local AI mental model.

95. A Single Neuron vs a Modern Language Model

Let's compare the two.

Single artificial neuron

Modern language model

The second system is vastly more complicated. But the foundational idea is still:

Transform numerical information through learned mathematical operations.

96. Why “Artificial Neural Network” Is a Useful Name

The name tells you two things.

Artificial

It is a computational construct created by humans.

Neural network

It consists of interconnected computational structures loosely inspired by biological neural systems. The name does not mean:

“A computer brain made of digital neurons that work exactly like human neurons.”

That would be misleading. The biological inspiration is historical and conceptual. The mathematics is its own field.

97. What a Neural Network Is Not

A neural network is not:

• a database,
• a collection of hard-coded rules,
• a conscious mind,
• a literal human brain,
• a search engine,
• a magical knowledge container,
• or a single algorithm.

It is a parameterized computational function implemented using numerical operations. Its behavior emerges from:

Architecture + Parameters + Input + Computation

And for trained models:

Architecture + Learned parameters + Input/context + Inference computation

98. Why Neural Networks Can Make Mistakes

A network's output is the result of learned computation. There is no guarantee that the result is correct. The network can:

• misunderstand context,
• generate an incorrect association,
• produce a plausible but false statement,
• fail on unfamiliar inputs,
• or behave unpredictably in edge cases.

The network is not checking every answer against objective reality. Its parameters represent learned patterns. This is why model evaluation matters.

99. Why Bigger Networks Can Still Fail

More parameters provide more capacity. But capacity is not truth. A huge model can still produce:

“The moon is made of cheese.”

if the conditions produce that output. Larger models can become more capable in many tasks, but size does not provide an absolute guarantee of correctness. This is another reason we need to distinguish:

Capability

from:

Truth

100. Neural Networks and Generalization

A network is useful partly because it can generalize beyond exact training examples. Suppose the model sees: The dog chased the ball.

It can potentially process: The cat chased the toy. even if that exact sentence was not part of training.

The network has learned patterns that can be reused. This is one of the core strengths of neural networks. But generalization has limits.

The model can still fail when inputs are sufficiently unfamiliar, ambiguous, contradictory, or outside the distribution represented by its training.

101. Why This Matters for Local AI

When you run a local model, you are not merely running a “chatbot.”

You are executing a neural network. That means you are dealing with:

• tensor computation,
• numerical precision,
• memory,
• compute,
• model architecture,
• tokenization,
• context,
• inference algorithms,
• and output sampling.

This is why Local AI becomes much easier to understand once you stop thinking of it as mysterious AI software and start thinking of it as a computational system.

102. The Hardware Perspective

Your computer is essentially being asked to execute a huge mathematical function. Imagine:

The CPU or GPU does not know that it is answering:

“What is photosynthesis?”

It is executing instructions on numerical data.

The meaning comes from the model's learned structure and the relationship between the input representations and output behavior.

This is one of the most useful conceptual shifts in understanding AI.

103. Why AI Feels Like Magic

If you only see:

Prompt → Answer

the system looks mysterious. If you see:

it becomes a computational pipeline. It is still extraordinarily sophisticated. But it is no longer magic.

That is the goal of Loco Learn.

104. The Complete Neural Network Mental Model

Here is the mental model you should carry forward. A neural network is:

A structured collection of mathematical transformations containing learned parameters that can transform numerical inputs into useful numerical outputs.

During training:

During inference:

For a language model:

For Local AI:

That is the foundation.

105. Knowledge Check

Try to answer these before reading the answers.

Question 1

What is an artificial neuron?

Question 2

What is a weight?

Question 3

What is a bias?

Question 4

Why are activation functions important?

Question 5

What does a neural-network layer contain?

Question 6

What does “deep” mean in deep learning?

Question 7

What is forward propagation?

Question 8

What is a tensor?

Question 9

What is the difference between a parameter and a hyperparameter?

Question 10

Why are nonlinear transformations important?

Question 11

What is an embedding?

Question 12

What is the relationship between a neural network and a transformer?

Question 13

Why are GPUs useful for neural-network computation?

Question 14

Does one neuron contain the entire intelligence of a modern language model?

Question 15

What changes during training?

106. Knowledge Check Answers

Answer 1

An artificial neuron is a mathematical computational unit that combines inputs using learned weights and a bias, then applies an activation function.

Answer 2

A weight is a learned numerical parameter controlling how strongly an input contributes to a computation.

Answer 3

A bias is another learned parameter that shifts the result of a weighted computation.

Answer 4

Activation functions introduce nonlinearity, allowing stacked layers to represent much more complex functions than purely linear transformations.

Answer 5

A layer contains computational operations and learned parameters that transform one representation into another.

Answer 6

Deep generally refers to using multiple layers of learned transformations in a neural network.

Answer 7

Forward propagation is the process of passing input through the network to calculate an output.

Answer 8

A tensor is a multidimensional array of numerical values.

Answer 9

Parameters are learned values such as weights and biases. Hyperparameters are configuration choices that influence architecture or training.

Answer 10

Without nonlinearities, stacking ordinary linear transformations would collapse into another linear transformation and greatly limit the network's expressive power.

Answer 11

An embedding is a numerical representation of an object such as a token, word, document, or other entity.

Answer 12

A transformer is a neural-network architecture built from specialized components including attention and feed-forward transformations. Modern language models commonly use transformer-derived architectures.

Answer 13

Neural networks perform large numbers of parallel numerical operations, especially matrix and tensor operations, which GPUs can execute efficiently.

Answer 14

No. Modern models contain enormous numbers of interacting parameters and computational components. Capability emerges from the network as a whole.

Answer 15

Learned parameters are updated during training.

107. Practical Exercise: Build a Neuron on Paper

You can understand a surprising amount of neural-network mathematics with a calculator. Create a fictional neuron. Use:

x₁ = 2 x₂ = -1 x₃ = 4

w₁ = 0.5 w₂ = 0.8 w₃ = -0.2

b = 0.1

Calculate:

Work it out:

z = (2 × 0.5) + (-1 × 0.8) + (4 × -0.2) + 0.1

Therefore:

Now apply ReLU:

The output is:

0

You have just performed the essential mathematical structure of an artificial neuron. Modern AI simply scales this idea enormously and adds sophisticated architectures around it.

108. Practical Exercise: Change One Weight

Using the previous example, change:

to:

Now calculate again.

z = (2 × 0.5) + (-1 × 0.8) + (4 × 0.5) + 0.1

You should get:

ReLU gives:

2.3

One parameter changed. The output changed substantially. Now imagine billions of parameters interacting.

This gives you a small intuition for why the learned parameter configuration matters so much.

109. Practical Exercise: Think Like an Inference Engine

Imagine the model has already been trained. Its parameters are fixed. You provide:

Input A

The model produces:

Output A

Then you provide:

Input B

The same parameters produce:

Output B

Ask yourself:

Did the model need to retrain between A and B?

No. It simply performed another forward computation. This is the essence of inference.

110. Connecting Lessons 1–6

You now have a chain of concepts.

Lesson 1

AI is a broad field involving systems that perform tasks associated with aspects of intelligence.

Lesson 2

Machine learning is a way of creating systems that learn patterns from data, and deep learning uses neural networks.

Lesson 3

Generative AI produces new outputs based on learned models and inputs.

Lesson 4

An AI model contains architecture and learned numerical parameters.

Lesson 5

Training changes those parameters; inference uses them.

Lesson 6

Those parameters belong to a neural computational system made from layers of mathematical transformations. This is a major milestone. You now have the basic conceptual architecture of modern AI.

111. What Is Still Missing?

We still have several enormous questions. We know what a neural network is. But:

How does it actually learn useful values?

We know that training updates parameters. But:

How do gradients tell the model what to change?

We know the model processes numerical representations. But:

How does text become tokens and embeddings?

We know transformers use attention. But:

How does attention actually work?

We know language models predict tokens. But:

Why can predicting tokens produce surprisingly broad capabilities?

These questions lead directly into the next lessons.

112. Preview of Lesson 7

Lesson 7: How Neural Networks Learn

The next lesson will go deeper into the learning process itself. You will learn:

• how a network makes a prediction,
• how the prediction is compared with a target,
• what loss really means,
• what gradients represent,
• how backpropagation works conceptually,
• how the chain rule appears in neural-network training,
• how optimizers update parameters,
• what gradient descent actually does,
• why learning rates matter,
• what overfitting means,
• what underfitting means,
• why validation data matters,
• why training can fail,
• and how repeated tiny updates can create a powerful model.

We will take the simple neuron from this lesson and teach it how to learn.

Sources and Further Reading

The following resources provide reliable background for the concepts covered in this lesson.

Deep Learning Textbook

Deep Learning
Ian Goodfellow, Yoshua Bengio, Aaron Courville

https://www.deeplearningbook.org/

This is one of the foundational technical references for neural networks, optimization, representation learning, and deep learning.

Google Machine Learning Crash Course

https://developers.google.com/machine-learning/crash-course

Provides practical explanations of neural networks, loss, optimization, embeddings, and other machine-learning concepts.

Google Machine Learning Glossary

https://developers.google.com/machine-learning/glossary

Useful reference for terminology including parameters, activation functions, embeddings, tensors, and related concepts.

Attention Is All You Need

Vaswani et al., 2017

https://arxiv.org/abs/1706.03762

The research paper introducing the Transformer architecture.

Hugging Face Course

https://huggingface.co/learn

Provides practical and technical educational material covering modern machine learning and language models.

Hugging Face Transformers Documentation

https://huggingface.co/docs/transformers/

Technical documentation covering transformer architectures and their implementation.

llama.cpp

https://github.com/ggml-org/llama.cpp

An open-source project focused on efficient local inference for supported language and vision-language models. It provides useful technical context for understanding how trained neural-network models can be executed locally.

NIST AI Risk Management Framework

https://www.nist.gov/itl/ai-risk-management-framework

Provides broader background on AI systems, evaluation, risk, and responsible development.

Final Mental Model

If you remember only one diagram from this lesson, remember this:

The model is a mathematical machine. Training determines its learned parameters. Inference runs those parameters.

Everything else in modern AI builds on top of this foundation.

Completed Lesson

You have completed:

Local AI 101 — Lesson 6: What Is a Neural Network?

Next Lesson

Lesson 7: How Neural Networks Learn

Course

Local AI 101

A deep educational course by Loco Learn.

Loco Academy: Lesson Status

Completed: Lesson 6: What Is a Neural Network?

Next: How Neural Networks Learn

Course: Local AI 101