Local AI 101

Lesson 7: How Neural Networks Learn

Level: Beginner → Intermediate
Estimated reading time: 75–90 minutes
Prerequisites: Lessons 1–6
Next lesson: Lesson 8: What Are Parameters and Weights?

What You Will Learn

By the end of this lesson, you should be able to:

• Explain what it means for a neural network to learn.
• Understand why a network needs a training objective.
• Explain the relationship between predictions, targets, and loss.
• Understand what a loss function measures.
• Understand gradients conceptually.
• Understand why the chain rule is important to neural-network training.
• Explain backpropagation without treating it as magic.
• Understand gradient descent.
• Understand learning rate.
• Understand epochs and batches.
• Understand stochastic gradient descent at a conceptual level.
• Understand why optimizers exist.
• Understand why training can become unstable.
• Explain overfitting and underfitting.
• Understand training, validation, and test data.
• Understand why a model can perform well on training data but poorly on new data.
• Understand regularization at a conceptual level.
• Understand why training data quality matters.
• Understand how a network gradually changes its parameters.
• Connect the mathematics of training to the model files used in Local AI.
• Understand what happens before a model becomes available for inference in Loco.

1. The Question We Need to Answer

In Lesson 6, we opened the neural network. We learned that a neural network is a collection of mathematical transformations containing learned parameters. We saw:

• inputs,
• weights,
• biases,
• layers,
• activation functions,
• tensors,
• embeddings,
• and transformer components.

But that creates a much bigger question:

How does the network figure out what all those numbers should be?

A modern neural network may contain millions or billions of learned parameters. Nobody sits down and manually types:

Weight 1 = 0.173 Weight 2 = -0.482 Weight 3 = 0.911 ...

That would be impossible at scale. Instead, training algorithms repeatedly adjust the parameters. But how?

How does the network know whether it is improving? How does it know which parameters should change? How much should they change?

How does the system avoid making the model worse? How does a network learn from millions or billions of examples? The answer involves a chain of concepts:

This loop is at the heart of neural-network training. Once you understand it, the phrase “the model learned from the data” becomes much less mysterious.

2. Learning Is Parameter Optimization

The simplest useful definition of neural-network learning is:

Learning is the process of adjusting model parameters so that the model performs better according to a chosen objective.

Remember the distinction from Lesson 5. The parameters are things such as:

• weights,
• biases,
• and other learned numerical values.

The architecture determines how those parameters are connected and used. Training changes the parameters. A simplified picture is:

This is optimization. The network is searching through an enormous space of possible parameter configurations.

3. A Tiny Example

Let's forget about language models for a moment. Imagine we want a tiny model to predict house prices. Suppose the only input is:

House size

And the desired output is:

Price

We give the model examples:

100 m² → R1,000,000 150 m² → R1,500,000 200 m² → R2,000,000

A very simple model might start with a completely wrong relationship. Maybe it predicts:

100 m² → R200,000

The model is wrong. We need a way to measure how wrong. That measurement is the role of the loss function.

4. Prediction vs Target

Two words will appear constantly in machine learning:

Prediction

What the model produced.

Target

What the training process considers the desired answer. For our example:

Input: 100 m²

Target: R1,000,000

Model prediction: R200,000

The model is far from the target. Training needs a numerical way to represent that difference. That is where loss comes in.

5. What Is Loss?

A loss function takes a model's prediction and compares it with the target according to the training objective. It produces a numerical value. Very roughly:

This is a simplification because different loss functions behave differently. But the core idea is:

Loss tells the training system how undesirable the current prediction is according to the objective.

Without a loss signal, the optimizer would not know what direction to move the parameters.

6. Loss Is Not the Same as “Truth”

This distinction matters. A loss function does not necessarily measure whether the model's answer is universally true. It measures performance according to a defined objective.

For example, if the training task is next-token prediction, the loss measures how well the model predicts the target token distribution.

A model can therefore have low training loss without being a perfect representation of reality. The objective determines what the system is optimizing.

7. A Simple Squared Error

For a simple numerical prediction problem, we might use squared error. A basic version is:

Suppose:

Then:

If the prediction is:

then:

The closer the prediction is to the target, the smaller the loss. This is just an illustrative loss function. Real language models use different objectives, commonly based on probability assigned to target tokens.

8. Why Squaring?

In a squared-error example, squaring has useful properties. It:

• makes negative and positive errors positive,
• penalizes larger errors more strongly,
• produces a smooth mathematical function.

But neural networks can use many different loss functions. Examples include:

• mean squared error,
• cross-entropy,
• negative log-likelihood,
• binary cross-entropy,
• contrastive objectives,
• and many specialized objectives.

The loss function depends on the problem.

9. Language Models Have a Different Kind of Target

For a language model, imagine the input:

The capital of France is

The training target might be the next token:

Paris

The model produces scores or probabilities for many possible tokens. Conceptually:

Paris 0.10 London 0.08 Berlin 0.06 Tokyo 0.04 ...

The actual target token receives some probability. If the model assigns a high probability to the correct token, the loss can be relatively low. If it assigns a tiny probability, the loss is higher.

This is one of the foundations of language-model pretraining.

10. Cross-Entropy

A common training objective for classification and language modeling is related to cross-entropy. You do not need to memorize the complete mathematical derivation yet. The important intuition is:

If the model assigns high probability to the correct answer, loss is lower. If it assigns low probability to the correct answer, loss is higher.

For a target token with predicted probability p, a simplified expression is:

If:

the loss is relatively small. If:

the loss is much larger. This gives the training process a strong incentive to assign more probability to correct targets.

11. The Model Is Not Told “Be Smarter”

Training does not normally contain a magical instruction such as:

“Become intelligent.”

Instead, the model is optimized against specific objectives. For a language model, one major objective can be:

Predict the next token accurately.

Repeated across enormous amounts of data, this objective can produce surprisingly broad capabilities. This is one of the most important ideas in modern AI. A seemingly simple objective can lead to a sophisticated learned system when applied at enormous scale.

12. From Loss to Action

Now we have a loss value. Suppose:

How does the model know what to change? This is the central problem. A model can contain millions or billions of parameters.

Imagine:

θ₁ θ₂ θ₃ ... θ₁,000,000,000

Which parameters contributed to the loss? How much did each one contribute? Should each parameter increase or decrease?

And by how much? This is where gradients enter.

13. What Is a Gradient?

A gradient tells us how a function changes with respect to its inputs. In neural-network training, we care about how the loss changes when the model's parameters change. Imagine a simple function:

L(w)

where L is loss and w is a parameter. The derivative:

dL/dw

tells us how the loss changes as w changes. If the derivative is positive, increasing w may increase the loss locally. If it is negative, increasing w may decrease the loss locally.

This information gives us a direction for improvement.

14. The Hill Analogy

Imagine you are standing on a mountain. You want to reach a low valley. You cannot see the entire landscape.

But you can feel the slope beneath your feet. If the ground slopes upward in one direction, you know that moving in the opposite direction goes downhill. The gradient tells you the direction of steepest increase.

Therefore, the negative gradient points toward the direction of steepest local decrease. Conceptually:

This is the basic intuition behind gradient descent.

15. A Two-Parameter Landscape

Imagine a model with only two parameters:

w₁ w₂

You can imagine every possible combination as a point on a two-dimensional landscape. Each point has a loss. Some areas have high loss.

Some areas have low loss. Conceptually:

High loss
                /\
               /  \
        ______/    \______
       /                  \
      /                    \
     /       LOW LOSS      \
    /          ↓            \
   /__________________________\

Training tries to move through this parameter landscape toward lower-loss regions. A real model may have billions of parameters. The landscape therefore exists in a vastly higher-dimensional space.

We cannot visualize it directly, but the mathematics still works.

16. Gradient Descent

Gradient descent is a family of optimization methods based on moving parameters in a direction that reduces loss.

A simplified update rule is:

new parameter

old parameter

learning rate × gradient

Mathematically:

Where:

• θ is the parameter,
• η is the learning rate,
• ∇L is the gradient of the loss.

This equation captures the central idea of neural-network learning.

17. What Does the Minus Sign Mean?

The gradient points toward increasing loss. We want to move in the opposite direction. Therefore:

• gradient

points toward decreasing loss. The optimizer uses that direction to update parameters. Again, this is a local approximation.

The loss landscape can be extremely complicated. But repeated small updates can still lead to useful parameter configurations.

18. Learning Rate

The learning rate controls how large an update is. Imagine you are walking downhill. If every step is:

1 mm

you may eventually reach the valley, but it could take an extremely long time. If every step is:

100 km

you might overshoot the valley completely. The learning rate is analogous to step size.

Small learning rate → small updates

Large learning rate → large updates

Neither extreme is automatically good. Training requires appropriate optimization settings.

19. What Happens If the Learning Rate Is Too Large?

Suppose the loss landscape looks like a valley. You want to move toward the bottom. If your steps are too large:

\ / \ / \ / \ / \/

You may jump from one side to the other. The loss might bounce around rather than decrease smoothly. In extreme cases, training can become unstable.

The model may fail to converge effectively.

20. What Happens If the Learning Rate Is Too Small?

Now imagine tiny steps. The model may improve extremely slowly. Training could require much more computation to reach a useful region.

This is why learning-rate selection is one of the important parts of training engineering. Modern training systems may also use learning-rate schedules that change the learning rate over time.

21. Learning Rate Schedules

A training process might use:

The reasoning is intuitive. Early in training, large movements can help the model move toward useful regions. Later, smaller updates can allow more precise adjustment.

Real training schedules can be much more sophisticated. Common approaches include:

• warmup,
• decay,
• cosine schedules,
• step schedules,
• and other strategies.

The exact schedule depends on the model and training setup.

22. What Is Backpropagation?

Now we can answer one of the most famous questions in neural networks.

Backpropagation is a method for efficiently calculating gradients of the loss with respect to the model's parameters.

The key word is:

gradients.

Backpropagation tells us how the loss changes with respect to parameters throughout a computational graph. It uses the chain rule of calculus.

23. Why Do We Need the Chain Rule?

Suppose:

x → a → b → loss

The loss depends on b. b depends on a. a depends on x.

If we want to know how the loss changes when x changes, we can multiply the relevant derivatives. Conceptually:

dLoss/dx

dLoss/db × db/da × da/dx

This is the chain rule. A neural network contains huge chains of operations. Backpropagation efficiently applies this principle across the entire network.

24. A Simple Chain Example

Imagine:

The loss depends on the output. The output depends on the activation. The activation depends on z.

z depends on w. To determine how changing w affects the loss, we trace those dependencies backward. Conceptually:

That is the core intuition behind backpropagation.

25. Forward Pass

During training, the model first performs a forward pass.

The system stores or otherwise makes available information needed for gradient computation. Then it calculates the loss.

26. Backward Pass

Now the system works backward.

The gradients tell the optimizer how parameter changes could affect the loss. Then the optimizer updates the parameters.

27. The Full Training Loop

Now we can combine everything.

This loop can execute an enormous number of times. That repetition is how the model gradually changes.

28. What Is an Optimizer?

An optimizer is an algorithm that uses gradient information to update model parameters. Gradient descent is the simplest conceptual example. Modern deep-learning training often uses more sophisticated optimizers.

One widely used family includes methods such as:

• SGD,
• Momentum,
• Adam,
• AdamW,
• and others.

The optimizer can incorporate information beyond the raw gradient. For example, momentum-like methods can use information from previous updates.

29. Momentum

Imagine rolling a ball downhill. It has momentum.

Instead of reacting only to the exact slope at the current point, the optimization process can maintain information about previous movement.

This can help navigate complicated landscapes. A simplified intuition is:

Current gradient + Previous update information = More informed parameter update

Momentum can help optimization move more efficiently in certain landscapes.

30. Adam and AdamW

Adam is an optimization algorithm that combines ideas involving adaptive learning rates and momentum-like estimates. AdamW is a related optimizer that handles weight decay in a way that separates it from the gradient update.

These details become increasingly technical. For Local AI users, the important point is:

The optimizer is the mechanism that turns gradient information into parameter updates during training.

You do not need to memorize every optimizer.

31. Batch Training

Now imagine a dataset with millions of examples. It would be inefficient to process the entire dataset as one giant operation for every parameter update. Instead, training commonly uses batches.

Suppose we have:

1,000,000 examples

A batch might contain:

256 examples

The model processes that batch. It calculates a loss and gradient. The parameters are updated.

Then the next batch is processed. Conceptually:

And so on.

32. Why Batches?

Batches provide several benefits. They can:

• make computation more manageable,
• enable efficient parallel processing,
• improve hardware utilization,
• provide useful noisy gradient estimates,
• and reduce the memory requirement compared with processing the entire dataset at once.

Batch size is an important training hyperparameter.

33. Stochastic Gradient Descent

The phrase stochastic gradient descent, or SGD, refers broadly to gradient-based optimization using individual examples or small batches rather than the entire dataset for every update.

Instead of calculating the exact gradient over the whole dataset:

we may estimate it using a sample or batch:

The estimate is noisier. But that noise can be useful and makes training computationally practical.

34. Why Noisy Gradients Can Help

Imagine hiking toward a valley. You do not have a complete map. You only have local information.

Your direction estimate may be imperfect. But repeated movement can still take you toward useful regions. The noise in batch-based gradients can sometimes help optimization avoid becoming stuck in certain undesirable regions.

Again, the analogy is imperfect. But it captures why stochastic optimization can work extremely well.

35. Epochs

An epoch is approximately one full pass through the training dataset. Suppose there are:

1,000,000 examples

and your training process eventually processes all of them. That is one epoch. A second pass is another epoch.

Conceptually:

Epoch 1: ████████████████████

Epoch 2: ████████████████████

Epoch 3: ████████████████████

Large modern model training can involve complicated data pipelines where the simple “epoch” concept is less central than it is in small educational datasets.

Still, it remains a useful concept.

36. Why One Pass Is Not Enough

One example gives the model limited information. A dataset contains many patterns. The model may need to encounter huge numbers of examples before its parameters become useful.

Repeated optimization allows the network to gradually adjust. The goal is not to memorize one example. The goal is to learn useful statistical structure across the dataset.

37. Training Example by Training Example

Imagine a tiny dataset:

Input A → Target A Input B → Target B Input C → Target C Input D → Target D

The model processes A. Its parameters update. Then B.

More updates. Then C. More updates.

Then D. More updates. After many cycles, the parameters may represent patterns shared across the examples.

This is the basic mechanism scaled up.

38. The Model's Parameters Move Gradually

Imagine one parameter begins at:

After one update:

Then:

Then:

Then:

Real training involves enormous numbers of parameters changing together. There is no guarantee that every parameter moves monotonically toward some obvious “correct” value. The network is optimizing the overall objective.

39. A Parameter Does Not Have a Simple Job

Do not assume:

“Weight #12 is responsible for grammar.”

Modern neural networks distribute information across many parameters. A single parameter can influence many outputs. A concept can be represented across many parameters.

This distributed nature is one reason neural networks are difficult to interpret.

40. The Model Learns Relationships

Suppose the network repeatedly sees patterns involving:

cat dog animal pet fur

It can learn numerical relationships useful for representing these concepts. The network is not necessarily creating a human-readable rule:

Instead, many learned numerical transformations collectively produce behavior consistent with those relationships. This is representation learning.

41. Training a Language Model

Now let's apply everything to a language model. Imagine a sequence:

The quick brown fox jumps over the lazy dog.

The tokenizer converts it into tokens. The model receives part of the sequence. Its task can be to predict the next token.

For example:

The quick brown fox jumps over the

Target:

lazy

The model produces logits. The logits become probabilities. The target token has a probability.

The loss measures how well the model predicted it. Backpropagation calculates gradients. The optimizer updates parameters.

Then the process repeats across many sequences.

42. One Tiny Prediction

Suppose the model produces:

lazy → 0.15 fast → 0.10 big → 0.08 ...

The target is:

lazy

The model assigned only 15% probability to the correct token. The loss is relatively high compared with a situation where the model assigned 90%. Training nudges the parameters so that similar contexts can produce better predictions in the future.

That is the core idea.

43. Why Predicting Tokens Teaches More Than Facts

This is one of the most fascinating aspects of language-model training. To predict the next token well, the model can benefit from understanding many aspects of its training data. For example:

Grammar Syntax Word meanings Common facts Writing styles Programming patterns Mathematical structures Long-range relationships

If the model repeatedly encounters enough structured data, predicting the next token can require learning rich internal representations.

This is one reason the next-token prediction objective can produce broad capabilities.

44. Why the Objective Is Still Limited

However, next-token prediction is not equivalent to:

“Learn everything about the universe.”

The model optimizes the objective it is given. This means its behavior reflects:

• its data,
• its architecture,
• its objective,
• its optimization,
• its post-training,
• and its context.

The model can therefore be highly capable while still having major limitations.

45. Training Loss vs Real-World Performance

A model can have lower training loss than another model and still perform worse on a particular real-world task. Why? Because training loss measures performance according to the training objective and data.

Real-world performance depends on:

• generalization,
• task alignment,
• evaluation design,
• data distribution,
• prompting,
• and many other factors.

This is why AI development uses evaluation datasets and benchmarks.

46. Generalization

Generalization means performing well on data that was not directly used for the specific parameter updates that produced the model.

Suppose a model trains on:

Examples A–Z

Then we test it on:

Examples AA–AZ

If it performs well, it has generalized. The goal of machine learning is generally not merely to memorize training examples. It is to learn patterns that transfer to new examples.

47. Overfitting

Overfitting occurs when a model becomes too specialized to its training data and does not generalize well.

Imagine a student who memorizes every answer in a practice test. They get:

Practice test: 100%

Then the teacher changes the questions slightly. The student struggles. That is analogous to overfitting.

The model has learned the training examples too specifically instead of learning patterns that transfer well.

48. Underfitting

The opposite problem is underfitting. The model is not capable of representing the training patterns well enough. A student who barely studies might perform poorly on:

Practice test

and:

Real test

The model has not learned enough. Conceptually:

Underfitting: Training performance → poor Validation performance → poor

Overfitting often looks more like:

Training performance → excellent Validation performance → poor

The goal is useful generalization.

49. Training, Validation, and Test Data

Machine-learning workflows often divide data into different sets.

Training data

Used to update model parameters.

Validation data

Used to evaluate choices during development.

Test data

Used for final evaluation under a defined methodology. The exact workflow varies. The key idea is:

We need data that was not used to directly optimize the model so we can estimate how well it generalizes.

50. Why Validation Matters

Suppose you only measure training loss. You might see:

Epoch 1 → high Epoch 2 → lower Epoch 3 → lower Epoch 4 → lower Epoch 5 → lower

Everything looks great. But validation performance might behave like:

Epoch 1 → improving Epoch 2 → improving Epoch 3 → improving Epoch 4 → stable Epoch 5 → getting worse

This could indicate overfitting. Validation data provides another perspective.

51. Regularization

Regularization refers broadly to methods that encourage useful generalization rather than simply fitting training data as aggressively as possible.

Methods can include:

• weight decay,
• dropout,
• data augmentation,
• early stopping,
• noise injection,
• and architectural choices.

The exact methods depend heavily on the model and training stage. The basic idea is:

Prevent the model from becoming unnecessarily specialized to the training data.

52. Weight Decay

Weight decay is a regularization technique that discourages parameters from becoming excessively large. Conceptually, the training objective may include an additional penalty related to parameter magnitude. Very roughly:

Total objective

Task loss + Regularization penalty

This encourages the optimizer to consider both:

Fit the data

and:

Avoid unnecessarily extreme parameter values

The details depend on the optimizer and implementation.

53. Dropout

Dropout is another regularization method.

During certain training configurations, some activations are temporarily omitted or “dropped.”

Conceptually:

Neuron 1 → active Neuron 2 → inactive Neuron 3 → active Neuron 4 → inactive Neuron 5 → active

The exact implementation varies. The intuition is that the network is discouraged from becoming overly dependent on a narrow set of pathways.

Dropout is historically important, although modern large-model training uses many different techniques and architectures.

54. Data Quality and Training

Optimization cannot rescue fundamentally poor data indefinitely. Imagine training on a dataset filled with:

• incorrect labels,
• duplicated examples,
• broken text,
• irrelevant content,
• contradictory information,
• or maliciously manipulated data.

The model receives those examples as training signals. Therefore:

Training quality depends heavily on data quality.

This is why large AI training pipelines often involve extensive data processing.

55. Data Filtering

Training pipelines may perform operations such as:

• deduplication,
• filtering,
• normalization,
• quality scoring,
• language identification,
• removal of unwanted content,
• and other processing.

The exact pipeline depends on the model. Data preparation is an enormous field of AI engineering.

56. Why Duplicate Data Can Matter

Suppose the same training example appears thousands of times. The optimizer sees it repeatedly. That can change how strongly the model is influenced by that example.

Therefore, dataset composition matters. Large training corpora are not simply:

“Throw every piece of text available into a folder.”

Data selection is a major part of model development.

57. Why Training Can Be Expensive

Now the computational picture becomes clearer. For each batch, the training system may need to:

And this happens across huge datasets. The network may contain billions of parameters. The operations are performed repeatedly.

This creates enormous computational demand.

58. Training Requires More Than Parameter Storage

During inference, you primarily need to execute the model. During training, additional memory can be needed for:

• gradients,
• optimizer state,
• activations,
• batches,
• temporary buffers,
• and other intermediate information.

Therefore:

Training memory

can be substantially greater than:

Inference memory

for the same model. This is one of the reasons a model that is practical to run locally may still be impractical to train locally.

59. Mixed Precision

Modern AI training often uses numerical formats that are smaller than traditional high-precision floating-point representations.

This can reduce memory requirements and improve computational efficiency. You may encounter terms such as:

• FP32,
• FP16,
• BF16,
• FP8,
• and various other numerical formats.

The exact format depends on hardware and training methods. The broad principle is:

Different numerical precisions trade accuracy, range, memory, and computational efficiency.

This becomes increasingly important in both training and inference.

60. Numerical Stability

Lower numerical precision can introduce numerical challenges. For example:

• underflow,
• overflow,
• rounding errors,
• loss of precision.

Modern training systems use techniques designed to maintain stable optimization. This is another reason AI engineering is not simply:

“Multiply some numbers together.”

At scale, numerical behavior matters enormously.

61. Gradient Explosion

Gradients can sometimes become extremely large. This is called exploding gradients. If the gradients become huge, parameter updates can become unstable.

Conceptually:

Normal gradient: small update

Exploding gradient: massive update

This can make training diverge.

62. Vanishing Gradients

The opposite problem is vanishing gradients. Gradients can become extremely small as they propagate through certain deep networks. Then earlier layers receive very weak learning signals.

Conceptually:

Gradient at output: 0.8

Earlier: 0.2

Earlier: 0.03

Earlier: 0.001

Earlier: 0.000001

If the gradient becomes tiny, parameter updates can become extremely small.

Architectural innovations, activation functions, normalization, residual connections, initialization strategies, and optimization methods can all help address training challenges.

63. Why Residual Connections Help

You encountered residual connections in Lesson 6. They provide pathways that allow information and gradients to move more directly through deep networks. Conceptually:

The simplified equation:

allows the network to learn a transformation around the identity path. This architecture became highly influential in deep learning. Transformers rely heavily on residual pathways.

64. Why Normalization Helps

Normalization techniques can help keep the numerical behavior of activations more controlled. This can make optimization easier. Modern architectures use various normalization strategies.

The exact mathematical details depend on the architecture. The larger lesson is:

Deep neural networks require carefully designed components to make optimization practical.

65. Training Is Not Guaranteed to Work

Even with a good architecture and large dataset, training can fail. Possible problems include:

• bad learning rate,
• unstable gradients,
• poor data,
• implementation bugs,
• insufficient compute,
• unsuitable initialization,
• poor hyperparameters,
• objective mismatch,
• overfitting,
• underfitting,
• numerical instability,
• or unexpected interactions between components.

AI training is an engineering discipline. It requires experimentation and evaluation.

66. Hyperparameter Search

Training involves many configuration choices. For example:

Learning rate Batch size Optimizer Weight decay Number of layers Hidden dimension Sequence length Training schedule

Developers may experiment with different configurations. This process is often called hyperparameter tuning or search. The goal is to find configurations that produce useful training behavior and strong evaluation results.

67. Why Training Runs Are Reproducibility Challenges

Two training runs may differ because of:

• random initialization,
• data ordering,
• hardware,
• numerical precision,
• software versions,
• optimizer settings,
• random seeds,
• and other factors.

This is why reproducibility is an important issue in machine learning research. A model is not simply:

Dataset → guaranteed identical result

There can be many implementation details between those endpoints.

68. The Training Pipeline

A large model training project can involve a pipeline like:

The final model that users download is the product of many stages.

69. Checkpoints

Training can take a long time. It would be dangerous to keep everything only in memory. Training systems therefore often save checkpoints.

A checkpoint is a saved state of the model and potentially other training information. For example:

Checkpoint 1 Checkpoint 2 Checkpoint 3 Checkpoint 4

If training fails, a later run may be able to resume from a checkpoint. Checkpoints can also be used for evaluation.

70. The Model Release

Eventually, developers may select a trained model checkpoint for release.

The released model can then be converted, packaged, optimized, or quantized into formats appropriate for different environments.

This creates the bridge from training to Local AI. Conceptually:

Not every model follows exactly this pipeline. But the conceptual lifecycle is useful.

71. From Training to Loco

Now imagine the completed model reaches a model repository. A user discovers it. They select an appropriate version.

They download it.

Loco can then manage that model and make it available for local inference, depending on compatibility and supported formats.

The user is now at the other end of the lifecycle. The original training may have been enormously complex. The user's experience can be:

That simplicity is possible because the difficult training process has already happened.

72. Why This Is Important for Loco Learn

A beginner might think:

“Loco runs AI.”

That is true, but incomplete. A better understanding is:

Loco provides an environment for discovering, managing, configuring, and using trained AI models locally.

The trained model is the result of an upstream development process. Loco helps bring that model into a user's local environment. This distinction matters because it explains why model choice is so important.

73. Training a Model vs Training an Application

Another important distinction: You can build an AI application without training a new foundation model. For example, an application can combine:

Existing model + Prompt + Knowledge base + Tools + Workflow + User interface

That can produce a highly useful AI product.

The application itself can have sophisticated behavior without owning the entire training pipeline for the underlying model.

This is a major principle in modern AI engineering.

74. Fine-Tuning Revisited

Suppose an existing model is already capable. You want to specialize its behavior. Instead of training from scratch, you may perform additional training.

Conceptually:

The same training concepts still apply:

• forward pass,
• loss,
• gradients,
• backpropagation,
• optimizer,
• parameter updates.

The starting point is simply an already-trained model rather than an untrained one.

75. LoRA Revisited

Parameter-efficient fine-tuning methods such as LoRA change the amount and structure of what is trained.

Instead of updating every original parameter in the same way, an adaptation can be represented through additional trainable components.

Conceptually:

The adapter itself is learned through optimization. So LoRA is still training. It is not simply a prompt.

76. Prompting vs Training Again

Let's make this distinction extremely clear. Suppose you tell a model:

“Always answer in three bullet points.”

The model may follow that instruction during inference. No parameter update is required. Now suppose you create 100,000 examples of responses and train the model so that it consistently produces that format.

Now you are performing training. The difference is:

Prompting: Input changes.

Training: Parameters change.

This distinction is foundational.

77. Retrieval vs Training Again

Suppose your business has:

10,000 documents

You want an AI assistant to answer questions about them. You could potentially use retrieval. The system finds relevant information and puts it into context.

The base model remains unchanged. Alternatively, you could train or fine-tune a model using examples derived from those documents. Now the parameters can change.

Both approaches can be useful. They solve different problems.

78. Why Changing Information Often Belongs Outside the Model

Imagine a company changes its pricing every week. If the pricing information is stored in an external knowledge system:

the information can be updated without changing the base model. If pricing were embedded into model parameters through training, updating it could require another training process. This is one reason modern AI applications often separate:

Model capability

from:

Current information

79. Evaluation

How do we know whether training worked? We evaluate the model. Evaluation can involve:

• held-out datasets,
• task-specific tests,
• human evaluation,
• automated metrics,
• safety evaluations,
• coding tests,
• mathematical tasks,
• language understanding tasks,
• and many other methods.

A single metric is rarely enough to describe a modern AI model completely.

80. Why One Benchmark Is Not Enough

Imagine Model A performs better on:

mathematics

while Model B performs better on:

creative writing

A single score cannot capture every useful difference. This is why model evaluation should consider:

• task,
• dataset,
• methodology,
• prompting,
• hardware,
• generation settings,
• and other conditions.

A benchmark result is meaningful only in context.

81. Training Loss Is Not a Complete Model Score

A lower training loss does not automatically mean:

“This model is better at everything.”

Training loss measures a particular objective. Real-world usefulness can depend on many additional factors. This distinction matters when comparing local models.

When Loco's model library eventually presents benchmark information, benchmark methodology and evaluation context matter just as much as the numbers.

82. Why a Model Can Be Brilliant and Still Wrong

Suppose training produces excellent predictive performance. The model may still generate an incorrect fact. Why?

Because:

Low loss ≠ Perfect truth

The training objective is not necessarily a universal truth detector. This is one of the reasons AI systems need evaluation, verification, and appropriate human oversight.

83. Training Is About Statistics

At a deep level, much of machine learning involves discovering statistical structure. The model learns relationships that help it predict outputs. This does not make the system “just statistics” in a way that makes it trivial.

Large neural networks can represent incredibly complicated functions. But the mathematical foundation remains statistical modeling and optimization.

84. Why Statistical Learning Can Produce Reasoning-Like Behavior

This is a major philosophical and technical question. A model trained to predict language can develop internal representations that support:

• multi-step patterns,
• symbolic-like structures,
• code generation,
• mathematical transformations,
• planning-like behavior,
• and other capabilities.

Researchers continue to study exactly how these abilities arise. The safe mental model is:

The network learns statistical representations and transformations that can support surprisingly sophisticated behavior.

We do not need to assume a human-like internal mind to understand the mechanism.

85. Training as Compression

Another useful perspective is that training can be viewed partly as a form of learned compression. The model processes enormous amounts of information. The resulting parameters encode useful statistical structure in a compact computational representation.

This does not mean the model is simply compressing the training data losslessly. It is learning a representation that supports prediction.

Still, the compression analogy can help explain why a model can capture patterns from enormous datasets without containing a literal copy of everything it saw.

86. The Model Is a Learned Function

A powerful mathematical perspective is:

Training tries to find parameters that make this function useful. Inference evaluates the function. For a simplified model:

Training finds useful θ. Inference uses that θ. This connects Lesson 4, Lesson 5, Lesson 6, and this lesson.

87. The Four-Part Mental Model

You can now think about a trained neural model as four major pieces:

1. Architecture

How the computations are organized.

2. Parameters

The learned numerical values.

3. Training objective

What the model was optimized to do.

4. Data

The examples that shaped the learned parameters. Together:

Then:

This is one of the most important mental models in the entire course.

88. What Your Computer Does During Local Inference

When you use a local model, your computer is executing the learned function. It is not repeating the original training process. It is doing something more like:

This is why local inference can be dramatically more accessible than model training.

89. Why Local AI Hardware Still Matters

Even though you are not training the model, inference still requires computation. Your hardware must:

• store model data,
• move numerical values,
• perform matrix/tensor operations,
• maintain context,
• and generate output.

This is why:

Model size + Quantization + RAM + VRAM + Compute + Memory bandwidth

all matter. The next part of Local AI 101 will focus heavily on these hardware concepts.

90. A Complete Example

Let's trace a fictional language model from beginning to end.

Stage 1: Architecture

Developers choose an architecture.

Transformer-based architecture

Stage 2: Initialization

Parameters receive initial values.

Weights Biases Other parameters

Stage 3: Training Data

The system prepares a large dataset.

Text Code Other data

Stage 4: Training

The model predicts tokens.

This repeats many times.

Stage 5: Evaluation

The model is tested.

Validation Evaluation Analysis

Stage 6: Post-training

Additional training or adaptation may occur.

Stage 7: Release

The model is packaged and distributed.

Stage 8: Local AI

A user downloads an appropriate version.

Stage 9: Loco

The user loads it into Loco.

Stage 10: Inference

You have now traced the entire lifecycle.

91. Why the Lifecycle Matters

Without understanding the lifecycle, Local AI can seem like:

“Download a model and it magically works.”

With the lifecycle, you understand:

That is the Local AI ecosystem.

92. Common Misconceptions

Misconception 1: “The model learns every time I use it.”

Normally false. Inference does not normally update model parameters.

Misconception 2: “Loss is a score of intelligence.”

No. Loss measures performance according to a particular objective.

Misconception 3: “Gradient descent means the model literally rolls downhill.”

No. That is an analogy for mathematical optimization.

Misconception 4: “Backpropagation tells the model the correct answer.”

Not exactly. The target and loss define the training signal. Backpropagation calculates how parameter changes affect the loss.

Misconception 5: “More training always makes the model better.”

Not necessarily. Training can overfit, become unstable, or optimize the wrong objective.

Misconception 6: “More parameters automatically means a better model.”

Not automatically. Architecture, data, optimization, training quality, and other factors matter.

Misconception 7: “Fine-tuning and prompting are the same.”

They are not. Fine-tuning changes learned parameters. Prompting changes the input/context.

93. Practical Exercise: Follow One Parameter

Imagine a tiny network has one parameter:

Suppose the gradient is:

and the learning rate is:

Using:

we get:

The parameter moves from:

0.50

to:

0.48

That is one tiny parameter update. Now imagine billions of parameters undergoing related updates across enormous numbers of training steps. That is neural-network learning.

94. Practical Exercise: Reverse the Gradient

Suppose:

Then:

Notice what happened. A negative gradient caused the parameter to increase. The sign of the gradient matters.

That is how the optimization process knows which direction to move.

95. Practical Exercise: Learning Rate

Suppose:

With learning rate:

0.1

the update is:

0.02

With learning rate:

0.01

the update becomes:

0.002

Same gradient. Different learning rate. Different step size.

This is why learning-rate selection matters.

96. Practical Exercise: Identify Overfitting

Imagine a model has:

Training accuracy: 99%

Validation accuracy: 65%

What might this suggest? Potentially overfitting. Now imagine:

Training accuracy: 60%

Validation accuracy: 58%

This could suggest underfitting or another limitation. These numbers are hypothetical and the meaning of “accuracy” depends on the task. The important lesson is that training performance and generalization performance are not identical.

97. The Learning Loop as a Machine

You can think of training as a machine with feedback.

+------------------+
                |                  |
                v                  |
Input → Model → Prediction        |
          |                        |
          v                        |
        Loss                       |
          |                        |
          v                        |
      Backpropagation              |
          |                        |
          v                        |
       Gradients                   |
          |                        |
          v                        |
       Optimizer ------------------+
          |
          v
    Updated parameters

The feedback loop is what makes learning possible. The model predicts. The system evaluates.

The parameters change. The model predicts again.

98. Why Training Is an Optimization Problem

Imagine a model has billions of parameters. Each possible combination of those parameters represents a different function. The training process searches this enormous space for useful configurations.

The loss function defines what “better” means. The gradient provides local directional information. The optimizer decides how to use that information.

The training data provides the examples. The architecture defines the space of functions the model can represent. Everything fits together.

99. A More Formal View

We can define a model:

where:

• x is input,
• θ is parameters,
• ŷ is prediction.

We have target y. The loss is:

L(ŷ, y)

The training objective is to find:

θ*

such that:

For a dataset, the objective may involve the average loss over many examples. The optimizer then attempts to find useful parameter values. This is the mathematical core of learning.

100. Why We Cannot Simply Solve for the Best Parameters

You might wonder:

“Why not just calculate the perfect parameter values directly?”

For small mathematical models, direct solutions can sometimes exist. For huge neural networks, the optimization landscape is far too complicated for a simple closed-form solution. There can be:

• enormous dimensionality,
• nonlinearities,
• complicated interactions,
• huge datasets,
• non-convex loss landscapes,
• and computational constraints.

Iterative optimization is therefore extremely useful.

101. Non-Convex Optimization

Modern neural-network loss landscapes can be highly complex. They are not generally simple bowls where there is one obvious lowest point. There can be:

• valleys,
• plateaus,
• saddle points,
• local structures,
• and many equivalent or near-equivalent solutions.

This is one reason training neural networks is an active area of research. Yet gradient-based methods have proven remarkably effective.

102. Saddle Points

A saddle point is a location that can behave like a minimum in one direction and a maximum in another. A simple mathematical analogy is:

/ / ------X------ \ \

The geometry is more complex than a simple valley. Large neural-network optimization landscapes can contain many such structures. Modern optimizers and high-dimensional geometry make training possible despite this complexity.

103. Plateaus

A plateau is a region where the gradient can be relatively small. If the gradient is tiny:

The parameter changes may be very small. Training can become slow. This is another reason optimization methods and architecture matter.

104. Why Modern Neural Networks Are Trainable

Given all these difficulties, it is remarkable that large neural networks can be trained at all. Several advances contribute:

• improved architectures,
• better initialization,
• activation functions,
• normalization,
• residual connections,
• optimizers,
• hardware acceleration,
• large-scale distributed training,
• better datasets,
• numerical techniques,
• and extensive engineering.

Modern AI is the result of many technological layers working together.

105. Training at Scale

Imagine one training machine is not enough. A large training project can distribute computation across many accelerators. Conceptually:

The exact distributed-training strategy can be extremely complex. But the idea is simple:

Multiple pieces of hardware cooperate on the same training problem.

This is another reason large-scale model training differs so dramatically from running a model locally.

106. Why Inference Is Different Again

Once training is complete, the model no longer needs to calculate parameter gradients for ordinary inference. It simply needs to execute the learned function. Therefore:

Training: Forward + backward + optimizer

Inference: Forward

This is a simplified but extremely useful distinction.

107. What Happens to the Optimizer After Training?

The optimizer's state is part of the training process.

For ordinary model distribution, users typically receive the model's learned parameters rather than the complete training system with all optimizer state.

This is another reason the distributed model can be much more focused on inference. The user does not need the original training machinery just to generate responses.

108. Why Local Models Can Be Distributed

Once the learned parameters are packaged appropriately, they can be distributed to users. Users can execute them locally. The model becomes a reusable computational artifact.

This is one of the foundational ideas behind the Local AI ecosystem.

109. Why Model Formats Matter

The parameters need to be stored in a representation that an inference runtime can understand. Different model ecosystems can use different formats and conventions.

For Local AI, formats such as GGUF are important because they package model information in a form suitable for supported local inference workflows.

You will study GGUF in depth later. The important point here is:

Training produces learned numerical information; model formats package that information for later use.

110. Training Does Not End at Pretraining

A model may undergo multiple training stages. A simplified lifecycle could be:

Not every model follows exactly this pipeline. The terminology and techniques vary. But the broader lesson is:

“Training” can refer to a family of different processes.

111. Preference Training

Some models undergo additional training designed to make outputs more useful according to human or system preferences. This can involve preference datasets and specialized optimization techniques. The exact methodology varies considerably.

The important distinction is that the model's parameters can continue to change after pretraining. Therefore:

Base pretrained model

and:

Instruction-following model

may have undergone different training processes.

112. Why This Matters When Choosing a Local Model

When you browse a model library, you may encounter labels such as:

• base,
• instruct,
• chat,
• fine-tuned,
• reasoning,
• code,
• vision,
• and other descriptors.

These labels can describe differences in training or intended use. A model designed for raw completion may behave differently from one designed to follow user instructions. This is why model metadata matters.

A user should understand what they are downloading.

113. Model Behavior Is a Product of Training

A model's behavior is influenced by:

Architecture + Pretraining data + Pretraining objective + Post-training + Fine-tuning + System configuration + Inference settings + Current context

This is why two models with similar parameter counts can behave very differently. Parameter count alone does not tell the whole story.

114. Why Training Is Hard to Explain With One Analogy

We have used:

• a student,
• a hiker,
• a mountain,
• a cook,
• and a feedback loop.

Each analogy captures one aspect. None is literally correct. The safest technical model is still:

Data → Forward pass → Loss → Gradients → Optimizer → Parameter update → Repeat

Use the analogies to build intuition. Use the mathematics to maintain accuracy.

115. The Most Important Concept in This Lesson

If there is one concept to remember, it is this:

A neural network learns by using feedback from its loss to calculate gradients and repeatedly adjust its parameters.

Everything else is a refinement of that idea.

116. Knowledge Check

Try these before looking at the answers.

Question 1

What does a loss function do?

Question 2

What is the difference between a prediction and a target?

Question 3

What is a gradient?

Question 4

Why does gradient descent use the negative gradient?

Question 5

What does the learning rate control?

Question 6

What is backpropagation?

Question 7

Why is the chain rule important?

Question 8

What does an optimizer do?

Question 9

What is a batch?

Question 10

What is an epoch?

Question 11

What is overfitting?

Question 12

What is underfitting?

Question 13

Why do we use validation data?

Question 14

What is the difference between training and inference?

Question 15

Why can a trained model run without the original optimizer?

117. Knowledge Check Answers

Answer 1

A loss function measures how undesirable the model's prediction is according to the training objective.

Answer 2

A prediction is what the model produces. A target is the desired or reference outcome used by the training objective.

Answer 3

A gradient describes how a function changes with respect to its inputs. In training, it tells us how the loss changes with respect to model parameters.

Answer 4

The gradient points toward increasing loss locally, so the negative gradient points toward decreasing loss.

Answer 5

The learning rate controls the approximate size of parameter updates.

Answer 6

Backpropagation is a method for efficiently calculating gradients throughout a computational graph.

Answer 7

The chain rule allows gradients to be propagated through a sequence of dependent mathematical operations.

Answer 8

An optimizer uses gradient information to update model parameters according to a chosen optimization strategy.

Answer 9

A batch is a group of training examples processed together for an update.

Answer 10

An epoch is approximately one complete pass through the training dataset.

Answer 11

Overfitting occurs when a model becomes too specialized to its training data and generalizes poorly to new data.

Answer 12

Underfitting occurs when a model fails to learn enough useful structure from the training data.

Answer 13

Validation data provides an evaluation signal on data not directly used for the parameter updates, helping assess generalization and development choices.

Answer 14

Training changes model parameters. Inference normally uses those learned parameters without updating them.

Answer 15

The optimizer is part of the training machinery. Ordinary inference only needs the trained model and the computation required to produce outputs.

118. Practical Exercise: The Complete Training Cycle

Take a blank page and draw this without looking:

Now explain each arrow in your own words. If you can do that, you understand the core training loop.

119. Practical Exercise: Explain It to a Beginner

Imagine someone asks:

“How does AI learn?”

A good answer would be:

“The model starts with parameters. It makes predictions on training examples. A loss function measures how well those predictions match the training objective. Backpropagation calculates how the parameters contributed to the error. An optimizer uses that information to adjust the parameters. The process repeats across many examples until the model develops useful patterns.”

That explanation is technically far more useful than:

“The AI reads a lot of data and learns.”

120. Connecting This to Local AI

Now imagine the model has completed training. The parameter values are stored in a model representation. You download that model.

Loco loads it. The model performs inference. The original training loop is not running.

Instead:

This is why understanding training makes Local AI easier to understand.

121. Why Your Local Computer Does Not Need to “Learn the Model”

When you download a trained model, you are receiving the result of the training process. You are not starting from scratch. The learned parameters already exist.

Your computer's job is to execute them. This is analogous to receiving a finished mathematical program rather than rebuilding the mathematics that produced it. Again, the analogy is imperfect, but the distinction is important.

122. Training vs Inference One More Time

Training

Inference

This distinction is so important that it is worth repeating.

Training changes the model.

Inference uses the model.

123. Why the Next Lesson Matters

We now understand:

• what a neural network is,
• what its parameters are,
• and how those parameters are learned.

But there is still one major piece we need to understand deeply:

What exactly are these parameters and weights?

We have used the words repeatedly. But we have not yet explored:

• why there can be billions of them,
• how parameter count is calculated,
• how parameters relate to model size,
• why “7B” means roughly seven billion parameters,
• why parameter count does not equal file size,
• how weights are represented numerically,
• how biases fit into parameter counts,
• how transformer layers contribute parameters,
• and why parameters matter for Local AI hardware.

That is the subject of Lesson 8.

124. Key Takeaways

Remember these points:

1. Neural networks learn by adjusting parameters.
2. A prediction is compared against a target through a loss function.
3. Loss provides a numerical training signal.
4. Gradients describe how loss changes with respect to parameters.
5. Backpropagation calculates those gradients efficiently.
6. The chain rule makes gradient propagation possible.
7. Optimizers use gradients to update parameters.
8. Gradient descent moves parameters toward lower loss.
9. Learning rate controls update size.
10. Batches allow training to process manageable groups of examples.
11. An epoch represents approximately one complete pass through a dataset.
12. Overfitting means learning training data too specifically.
13. Underfitting means failing to learn enough useful structure.
14. Validation data helps evaluate generalization.
15. Training loss is not the same thing as universal truth or intelligence.
16. Large-scale training can require enormous computation and memory.
17. Inference does not normally require gradients or an optimizer.
18. A released model contains the result of a long training process.
19. Loco users generally perform inference rather than training.
20. Understanding optimization is essential to understanding modern AI.

125. Final Mental Model

Keep this diagram:

The model learns because its parameters are repeatedly adjusted according to feedback from the training objective. That is the engine underneath modern deep learning.

Completed Lesson

You have completed:

Local AI 101 — Lesson 7: How Neural Networks Learn

Next Lesson

Lesson 8: What Are Parameters and Weights?

Course

Local AI 101

A deep educational course by Loco Learn.

Loco Academy: Lesson Status

Completed: Lesson 7: How Neural Networks Learn

Next: What Are Parameters and Weights?

Course: Local AI 101