What Is an AI Model?

What You Will Learn

By the end of this lesson, you should be able to:

• Explain what an AI model actually is.
• Understand the difference between a model, an application, and an inference runtime.
• Understand what model parameters and weights are.
• Understand what a model architecture is.
• Understand why model size is often described in billions of parameters.
• Understand why parameter count does not directly equal intelligence.
• Understand what a model file contains.
• Understand the difference between weights and metadata.
• Understand why the same model can exist in different numerical formats.
• Understand why quantization changes model size.
• Understand why a model's file size is not always the same as its total memory requirement.
• Understand what happens when Loco loads a local model.
• Understand the relationship between a model and the hardware running it.
• Understand why context length and runtime overhead affect memory.
• Understand the basic difference between a base model and an instruction-tuned model.
• Understand why model architecture matters when choosing a local model.

4.1 We Are Finally Opening the Black Box

So far, we have deliberately stayed at a relatively high level. You learned what artificial intelligence is. You learned how machine learning fits inside AI.

You learned what deep learning and neural networks are. You learned what generative AI means. Now we are going to start looking at the actual object at the centre of modern AI:

the model.

If you have ever downloaded a local AI model, you may have seen files that are several gigabytes in size. You may have seen names containing things like:

7B 8B 14B 32B 70B Q4 Q5 Q8 GGUF Instruct Base MoE

At first glance, this can look like a completely different language. It isn't. These labels are describing characteristics of the model.

Once you understand what they mean, choosing local models becomes much easier. More importantly, you'll understand what is actually happening when Loco loads one.

4.2 The Simplest Definition of an AI Model

Let's start with a deliberately simple definition:

An AI model is a mathematical system whose parameters have been learned from data and which can be used to produce predictions or outputs from new inputs.

That definition is much more useful than saying:

"An AI model is the AI."

The model is a learned mathematical representation. For a neural network, the model contains a large collection of numerical values called parameters. Those parameters were adjusted during training.

When the model is later used, the parameters are used to perform calculations. Conceptually:

This means that a model isn't a mysterious digital brain hidden inside a file. It is a very large mathematical object. The file is a way of storing that object so that software can load and use it.

4.3 Model vs Application

This distinction is so important that it deserves repeating.

A model is not an application.

Imagine you download a model file. The model file might contain billions of numerical values. But if you simply double-click that file, nothing useful necessarily happens.

Something needs to load it. Something needs to allocate memory. Something needs to perform the mathematical operations.

Something needs to process input. Something needs to generate output. Something needs to display the result.

This is where software such as Loco becomes important. A simplified architecture looks like:

LOCO │ ┌─────────┼─────────┐ │ │ │ Interface Features Settings │ │ └─────────┼─────────┘ │ Inference Runtime │ AI Model │ Hardware

The model is one layer. Loco is the application/platform layer around it. The inference runtime is responsible for executing the model.

The hardware performs the underlying computation. This distinction will become extremely important as you learn how Local AI works.

4.4 What Is Inside a Model?

Let's imagine a simplified model file. It might contain:

Model metadata Tokenizer information Architecture information Vocabulary Model weights Configuration

The exact contents depend on the model format. But the most important component is usually the weights. The weights contain learned numerical values.

These values represent a substantial portion of what the model learned during training. A model can contain millions, billions, or even hundreds of billions of parameters. For example:

1B model ≈ 1 billion parameters

7B model ≈ 7 billion parameters

14B model ≈ 14 billion parameters

70B model ≈ 70 billion parameters

The "B" generally means billion. So when someone says:

"This is an 8B model."

they are referring approximately to the number of parameters.

4.5 What Is a Parameter?

A parameter is a learned numerical value inside a model. This definition sounds almost disappointingly simple. But billions of these values can collectively encode extremely complicated statistical patterns.

Imagine a neural network as an enormous mathematical function. The parameters determine how the function behaves. During training, the model's parameters are adjusted.

Very roughly:

After a huge number of training steps, the resulting parameters contain information about patterns learned from the training data.

The model doesn't store its knowledge in the same way a human stores memories in a brain. And it doesn't necessarily store every training example as a simple database record. Instead, the learned information is distributed across a huge numerical parameter space.

This is one of the central ideas of neural networks.

4.6 Why Are Parameters Called Weights?

You will hear two terms constantly:

parameters

and

weights.

They are closely related, but technically they aren't always perfectly interchangeable.

In many discussions about neural networks, "weights" refers to learned numerical values associated with connections or transformations inside the network.

"Parameters" is the broader term for learned values. A model may contain different types of parameters, depending on its architecture. For beginners, it is useful to think:

Weights are learned numerical values, and parameters are the broader collection of learned values that define the model.

This distinction becomes more important when we discuss model architectures in greater detail. For now, don't get trapped in terminology. The key idea is:

The model's learned parameters are what give the trained model its learned behaviour.

4.7 Why Does a Model Need So Many Parameters?

Because the problems modern models solve are extraordinarily complex. Imagine trying to create a mathematical system capable of handling:

• English
• Spanish
• French
• German
• programming languages
• mathematics
• history
• science
• conversational dialogue
• summarisation
• reasoning patterns
• formatting
• code structures
• different writing styles

That requires representing an enormous number of relationships. The model needs enough capacity to represent those patterns. Parameters provide that capacity.

However, this does not mean:

More parameters = more intelligence.

Parameter count is one property of a model. It is not a universal intelligence score.

4.8 Parameter Count Is Not an Intelligence Meter

Suppose you have:

Model A: 8B parameters Model B: 14B parameters

It would be tempting to conclude:

Model B must be better.

That conclusion is not justified. Model A might have:

• better training data
• better data quality
• better architecture
• better training methods
• better post-training
• better instruction tuning
• better optimisation
• better domain specialisation

The 8B model could outperform the 14B model on some tasks. The reverse could happen on others. A model's capabilities depend on much more than parameter count.

This is one of the reasons Loco's model-selection experience should eventually focus on what the model is good at and what hardware it requires, rather than simply sorting models from largest to smallest.

4.9 What Is Model Architecture?

If parameters are the learned values, what determines how those values are organised and used? That's where architecture comes in. A model architecture describes the structure and computational design of the neural network.

Think of it as the blueprint. A building might contain thousands of bricks. The bricks are not the architecture.

The architecture determines how those bricks are arranged. Similarly:

Two models can have similar parameter counts while using different architectures. They may therefore behave differently and have different hardware requirements.

4.10 Architecture Is More Than a Name

You may encounter model descriptions containing terms such as:

• Transformer
• decoder-only
• encoder-decoder
• mixture of experts
• dense model
• multimodal
• vision-language
• recurrent
• state-space

These terms describe different architectural approaches. For modern local language models, Transformer-based architectures are especially important. We will eventually study Transformers in depth.

For now, think of architecture as:

The mathematical blueprint that determines how the model processes information.

The parameters are learned values inserted into that architecture.

4.11 A Useful Analogy: Blueprint and Building

Imagine a building. The architecture is the blueprint. The building materials are the physical components.

The finished building is the result. For an AI model:

This analogy isn't perfect, but it is useful.

A 7B model isn't just "7 billion random numbers."

Those numbers exist within a specific computational architecture. That architecture determines how those numbers are used.

4.12 What Does "7B" Actually Tell You?

Suppose you see:

Example Model 7B

The 7B generally means approximately seven billion parameters. It tells you something about the scale of the model. But it doesn't tell you:

• how good the model is
• how much RAM it needs
• how fast it will run
• what languages it supports
• what license it uses
• whether it is instruction tuned
• whether it supports images
• what context length it supports
• what quantization it uses

Those are separate properties. This is why a good model library needs much more information than simply:

"Model name + parameter count."

4.13 Model Size vs File Size

This distinction is critical for Local AI. You might hear:

"This is a 7B model."

and then see a file that is:

4.5 GB

It is tempting to think:

"7 billion parameters equals 4.5 GB."

Not exactly. The number of parameters tells you how many learned values exist. The file size depends on how those values are represented.

For example, a parameter might be stored using:

• 32-bit floating point
• 16-bit floating point
• 8-bit integer-like representation
• 4-bit representation
• other numerical formats

The number of bits used per parameter has a huge impact on storage requirements. This is the foundation of quantization.

4.14 The Basic Math of Model Storage

Let's make this concrete. Suppose a hypothetical model contains:

8 billion parameters.

If every parameter were stored using 16 bits:

8,000,000,000 × 16 bits

Convert bits to bytes:

8,000,000,000 × 16 ÷ 8

That equals approximately:

16 GB

before accounting for other information and implementation details. Now imagine representing those parameters at approximately 8 bits per parameter:

8,000,000,000 × 8 ÷ 8

Approximately:

8 GB

At approximately 4 bits per parameter:

8,000,000,000 × 4 ÷ 8

Approximately:

4 GB

These are simplified theoretical calculations.

Real model files contain additional metadata and may use formats whose effective bits-per-parameter are not exactly the simple values above.

But the relationship is extremely important:

Fewer bits per parameter generally means less storage and memory required for the model weights.

That is the basic motivation behind quantization. We will dedicate an entire lesson to it.

4.15 Why Model Files Can Be Huge

Now the reason becomes clearer. Imagine a model containing tens of billions of parameters. Even at reduced precision, billions of numerical values require significant storage.

For example:

7B model ≈ billions of learned values

32B model ≈ tens of billions of learned values

70B model ≈ tens of billions more

This is why local AI model files can be several gigabytes or much larger. You're not downloading a traditional application with a few hundred megabytes of source code. You're downloading a very large numerical representation of a trained neural network.

4.16 What Is a Model File?

A model file is a file containing the information required for software to reconstruct and execute a model. Depending on the format, this can include:

• weights
• architecture metadata
• tensor information
• tokenizer information
• vocabulary
• configuration
• special tokens
• quantization information
• other metadata

Different model formats organise this information differently. This is why model files have different extensions and conventions. For Local AI, one particularly important format is:

GGUF

You will receive a dedicated lesson on GGUF later.

For now, simply understand that GGUF is a model file format designed for efficient model storage and loading, particularly within the ecosystem of local inference software.

4.17 What Is a Tensor?

You will encounter another word constantly when dealing with modern AI:

tensor.

A tensor is, broadly, a generalisation of arrays to multiple dimensions. For example: A single number can be considered a scalar.

A list of numbers is a vector. A grid of numbers is a matrix. A higher-dimensional collection of numbers can be represented as a tensor.

Conceptually:

Neural networks perform enormous numbers of operations on tensors. Model weights are commonly stored as tensors. This is why model formats often contain collections of tensors.

You don't need advanced linear algebra to understand Local AI. But eventually, understanding tensors will help you understand what the hardware is actually calculating.

4.18 Why GPUs Are So Important

Remember from Lesson 2 that deep learning involves enormous amounts of mathematical computation. Many of these operations can be performed in parallel. GPUs were originally designed around highly parallel computation for graphics.

That architecture turned out to be extremely useful for neural-network workloads. Imagine needing to perform millions of similar mathematical operations. A CPU might have a relatively small number of powerful general-purpose cores.

A GPU can have a much larger number of parallel processing units designed for workloads that can be split into many simultaneous operations.

This is one reason GPUs became so important to AI. But GPUs aren't mandatory for all local AI. A model can also be executed on a CPU.

The trade-off is often performance, although the exact result depends heavily on the model, runtime, CPU architecture, memory bandwidth, and workload.

We'll explore hardware in much greater depth later.

4.19 What Happens When Loco Loads a Model?

This is where all of this becomes practical. Imagine you select a local model in Loco. You press:

Load Model

A simplified conceptual sequence might look like this:

The exact implementation can vary. But the important thing is that the model must be loaded into memory in a form that the inference system can use. This is why model size matters.

Your computer needs enough available resources to load the model and perform the required computations.

4.20 Why "The Model Is 4 GB" Doesn't Mean You Only Need 4 GB of RAM

This is one of the most common mistakes people make. Suppose a model file is approximately:

4 GB

You might assume:

"My computer has 4 GB of RAM, so it should run."

Not necessarily. The system needs memory for other things. For example:

Operating system + Loco + Inference runtime + Model weights + Context / KV cache + Temporary buffers + Other application memory = Total memory requirement

This is why the model's file size is only part of the story. The actual memory requirement can be significantly higher.

4.21 The Context Also Uses Memory

Remember the context window from the earlier lessons?

When you interact with a language model, the system has to maintain information about the current sequence being processed.

This involves additional memory. One important component is commonly referred to as the KV cache, or key-value cache. You don't need to understand its mathematics yet.

Conceptually, think of it as memory used during inference to retain information needed for efficient processing of the current context.

As context grows, memory requirements can grow. That means:

Model weights + Context memory + Runtime overhead = Actual working memory

This is one reason a model that appears small enough based purely on file size can still struggle on a machine with limited RAM or VRAM.

4.22 Why Context Length Matters

Imagine a model supports a context window of:

8,000 tokens

Another supports:

128,000 tokens

The second model can potentially process much larger amounts of information within one context. But larger context capacity can also require additional memory and computational resources. This creates a trade-off.

A user might think:

"More context is always better."

Not necessarily. If your hardware is limited, extremely large contexts can create significant memory pressure. The practical question is:

How much context does your workload actually need?

For a simple chatbot, perhaps not much. For analysing hundreds of pages of documentation, substantially more may be useful. This becomes particularly important for local knowledge systems.

4.23 What Is a Base Model?

A base model is a model trained primarily to model its underlying training objective, without necessarily being specifically tuned for following conversational instructions.

For a language model, a base model may be very good at continuing text. You might give it:

The chemical reaction begins when

and it continues the sequence. But that does not necessarily mean it has been optimised to behave like a helpful assistant. This distinction leads to another important category:

instruction-tuned models.

4.24 What Is an Instruction-Tuned Model?

An instruction-tuned model has undergone additional training designed to improve its ability to follow instructions. Instead of merely predicting text continuation, it is further trained to respond usefully to requests. For example:

User: Explain photosynthesis.

Model: Photosynthesis is...

The instruction-tuned model has been trained to behave more like an assistant. This is why local model names often contain terms such as:

Instruct Chat Instruction Assistant

These names can indicate that additional post-training has been performed for instruction following. The exact training method varies between model families.

4.25 Base vs Instruct: A Simple Analogy

Imagine two extremely knowledgeable people. Person A has spent years reading books but has never been trained to work as a teacher. Person B has similar knowledge but has additionally been trained in:

• answering questions
• following instructions
• explaining concepts
• formatting responses
• interacting with users

Both may know similar things. But Person B is likely to be easier to use as an assistant. That is a rough analogy for the difference between a base model and an instruction-tuned model.

It is not technically exact, but it captures the practical distinction.

4.26 What Is a Model Family?

A model family is a group of related models developed using a common architecture, training philosophy, or development lineage.

A family might contain variants such as:

Model Family │ ├── Small ├── Medium ├── Large ├── Base ├── Instruct ├── Vision ├── Coding └── Specialised variants

The exact structure varies. A model family can provide users with choices for different hardware and workloads. This is particularly useful for local AI.

Someone with limited RAM may need a smaller variant. Someone with a powerful GPU may choose a larger model. Someone working with images may need a multimodal variant.

Someone writing code may prefer a model optimised for coding tasks.

4.27 What Is an MoE Model?

You may encounter the abbreviation:

MoE

It stands for Mixture of Experts.

This is an architectural approach where a model contains multiple specialised components, often referred to as experts, while a routing mechanism determines which experts are activated for a particular input.

A simplified conceptual diagram:

One of the important ideas is that a model can have a very large total parameter count while only activating a subset of parameters for a particular token or computation.

This can change the relationship between:

• total parameters
• active parameters
• memory requirements
• compute requirements

MoE models are therefore an important concept when evaluating modern local models. We'll return to them later.

4.28 What Does "Active Parameters" Mean?

Suppose a model is described as:

Total parameters: 100B Active parameters: 20B

That means the model may contain approximately 100 billion parameters in total, while a particular inference step may activate approximately 20 billion parameters through its routing mechanism.

This is not the same as saying the model only contains 20B parameters. The total model still has to be represented and may still require substantial storage and memory. However, the computational workload can differ from a dense model with the same total parameter count.

This is why model specifications need to be read carefully.

4.29 What Is a Model's Context Window?

A model's context window is the maximum amount of tokenised information it can process within a given context. That may include:

• system instructions
• conversation history
• user prompts
• retrieved documents
• tool results
• generated text

Imagine a model with a context capacity of 32,000 tokens. That capacity isn't necessarily:

"32,000 words."

Tokens are not equivalent to words. The exact relationship varies by language and tokenizer.

The important idea is that the context window defines how much tokenised information can be considered within one inference context.

This becomes extremely important for RAG, document analysis, and long conversations.

4.30 Why Model Files Can Have Different Quantizations

Now we return to something you saw earlier:

Q4 Q5 Q8

These labels often relate to quantization. Quantization reduces the numerical precision used to represent model parameters. For example, a model might be represented approximately using:

16-bit values 8-bit values 5-bit values 4-bit values

The exact implementation is more sophisticated than simply replacing every number with an integer of that size. But the basic objective is:

Reduce the amount of memory required while preserving as much model quality as practical.

This is one of the key technologies that makes large language models practical on consumer hardware. We'll have an entire lesson dedicated to quantization.

4.31 Why Quantization Matters for Loco

Imagine you have:

8 GB of RAM

You want to run an 8B model. A high-precision representation might require too much memory. A quantized representation might make the model much more practical.

This creates a powerful concept:

Quantization can make models accessible to hardware that could not reasonably run the full-precision representation. There is usually a trade-off. Lower precision can affect model quality or numerical behaviour.

The goal is not:

"Use the lowest possible precision."

The goal is:

Find an appropriate balance between quality, memory usage, and performance.

4.32 Model Quality vs Model Size vs Hardware

When selecting a local model, you are effectively solving a multi-variable problem. You have:

Model capability ↕ Model size ↕ Memory requirements ↕ Inference speed ↕ Hardware limitations ↕ Task requirements

Suppose you want a coding assistant. You could choose a massive model. But if it generates one token every few seconds on your computer, it may be frustrating to use.

A smaller model might generate much faster while being sufficiently capable for your task. Therefore:

The best local model is not necessarily the largest model you can technically load.

It is the model that provides the best practical trade-off for your specific workload and hardware.

4.33 What Happens During Model Loading?

Let's examine this at a slightly deeper level. Suppose you select a GGUF model in Loco. A simplified sequence could look like:

Step 1: Open the model file

The runtime accesses the model file from storage.

Step 2: Read metadata

The runtime determines things such as:

• architecture
• tensor information
• vocabulary
• context-related settings
• quantization information

Step 3: Allocate memory

Memory is reserved for the model and required runtime structures.

Step 4: Load tensors

The model's numerical tensors are loaded.

Step 5: Initialise execution structures

The inference runtime prepares the operations required by the model architecture.

Step 6: Initialise the context

Memory structures needed for the current inference context are created.

Step 7: Model becomes available

The model is now ready to process input. The exact implementation depends on the model, runtime, hardware, and software configuration.

But conceptually, this is what is happening when the application appears to be "loading an AI model."

4.34 Why Model Loading Can Take Time

A model may be several gigabytes. That data has to move from storage into memory. If the model is stored on a relatively slow drive, loading can take longer.

A faster SSD can reduce loading time. The system may also need to perform additional initialisation. This means there are several different performance metrics to distinguish.

Load time

How long it takes to make the model ready.

Time to first token

How long it takes before generated output begins.

Prompt processing speed

How quickly the model processes input tokens.

Generation speed

How quickly the model generates output tokens. These are not the same thing. We'll explore them in detail later.

4.35 The Model Does Not "Run From the Hard Drive"

Another common misconception:

"The model runs from my SSD."

Technically, the model file is stored on the SSD. But inference requires active computation using memory and processing resources. Conceptually:

The storage device provides persistent storage. RAM and VRAM provide working memory. The CPU and/or GPU performs the computations.

This distinction is fundamental to understanding local hardware.

4.36 RAM vs VRAM

You will eventually receive entire lessons on both. For now, remember:

RAM is the main system memory used by your computer.

VRAM is memory attached to or associated with the GPU.

When running a model, the model weights and other data may be placed in:

• system RAM
• GPU VRAM
• a combination of both

The exact behaviour depends on the runtime and hardware. If a model doesn't fit comfortably into GPU memory, some systems can offload portions to system RAM. This can make a model runnable but may affect performance.

This is why:

"The model fits in my RAM"

does not necessarily mean:

"The model will run quickly."

4.37 What Does "Offloading" Mean?

Offloading generally means moving some computation or model data to another processing or memory resource. For example:

GPU VRAM + System RAM

A model might keep some components in VRAM while other components remain in system memory. This can allow larger models to run on hardware with limited VRAM. But moving data between memory systems can introduce overhead.

That means a model can be technically runnable but significantly slower. This leads to a critical Local AI lesson:

Compatibility and performance are different questions.

4.38 A Model Is More Than Its Parameter Count

When evaluating a model, you should eventually look at:

• model family
• architecture
• parameter count
• active parameter count
• context length
• modalities
• supported languages
• training approach
• instruction tuning
• license
• quantization
• memory requirements
• runtime compatibility
• hardware requirements
• benchmark results
• intended use cases
• known limitations

This is why Loco's model library should eventually provide meaningful model pages rather than forcing users to decode filenames.

The goal should be:

"Help me choose a model."

not:

"Give me 4,000 model files and make me figure it out."

4.39 A Practical Model Example

Imagine Loco's model library shows:

Example Model 8B Instruct

Parameters: 8B

Architecture: Transformer

Type: Instruction-tuned language model

Context: 32K tokens

Quantization: Q4

Approximate file size: 5 GB

Recommended RAM: 8–16 GB

Primary uses: General assistant Writing Coding Research

Modalities: Text

That tells you far more than:

ExampleModel-8B-Q4.gguf

The first description helps you make a decision. The second requires you to already understand the terminology. This is the philosophy behind a good educational model library.

4.40 Why Loco Needs to Explain Models Properly

Local AI has traditionally had a steep learning curve. A technically experienced user might look at:

ModelName-8B-Instruct-Q4_K_M.gguf

and understand most of it. A beginner may see:

"ModelName-8B-Instruct-Q4_K_M"

and have absolutely no idea what it means. A good local AI platform should reduce that friction.

Instead of forcing users to understand everything before they can use AI, Loco can make the technology approachable while still giving advanced users the technical information they need.

That means model information should eventually answer questions such as:

• What is this model?
• What can it do?
• How big is it?
• What hardware does it need?
• How much RAM should I have?
• Does it support images?
• What languages does it support?
• Is it instruction tuned?
• What is the context length?
• What license does it use?
• Which quantization should I download?
• Is there a smaller version?
• Is it suitable for my computer?

Those are much more useful questions than simply:

"How many parameters does it have?"

4.41 The Model Lifecycle

It is useful to think about a model as having a lifecycle.

You are usually entering this lifecycle near the end. The model developer has already done most of the expensive training. Loco then helps make the released model usable locally.

Understanding this lifecycle helps explain why different files and variants exist.

4.42 What Is Model Conversion?

A model may be trained and originally stored using formats designed for training infrastructure. That isn't necessarily the most convenient format for local inference. The model can therefore be converted into another representation.

The conversion process may involve:

• changing tensor representation
• reorganising data
• adding metadata
• converting architecture-specific structures
• preparing the model for an inference runtime

Quantization may then be applied. This is one reason you can find multiple versions of what is fundamentally the same model.

4.43 One Model Can Have Many Files

Suppose a developer releases:

Example Model 8B

You might eventually encounter:

The underlying trained model family is related. But the files have different numerical representations. The smaller versions may be much more practical for consumer hardware.

This is why model libraries can contain many files for one model family. It is not necessarily because there are hundreds of completely different models. Sometimes they are different representations or variants of the same underlying model.

4.44 What Is a Checkpoint?

You may encounter another term:

checkpoint.

A checkpoint is a saved state of a model during training. A training process can take enormous amounts of time. Researchers therefore periodically save model states.

If training stops unexpectedly, they can resume from a checkpoint. A released model may ultimately be derived from a particular checkpoint or final trained state.

For local users, however, the important distinction is that you are generally downloading a released model intended for inference rather than a training checkpoint that you are expected to continue training.

4.45 What Is a Model Card?

A model card is documentation describing a model. A useful model card can contain information such as:

• intended use
• limitations
• training information
• evaluation
• risks
• license
• supported languages
• modalities
• architecture
• release information

When choosing a model, documentation is extremely valuable. A model's filename tells you only a fraction of the story. Good model documentation tells you what the model is actually intended to do.

Loco's model pages should therefore complement technical model sources rather than pretending a filename is sufficient documentation.

4.46 The Difference Between a Model and Its Quantized Version

This distinction is subtle. Suppose a developer releases a model. You download a quantized Q4 representation.

It is reasonable to call it the same model family. But technically, you are not using exactly the same numerical representation as the original high-precision model. Quantization changes the numerical representation of the parameters.

This can introduce some loss of information.

The goal is to keep that loss small enough that the model remains useful while significantly reducing resource requirements.

This is why quantization is a trade-off.

4.47 The Model Is Not "Thinking" While Loaded

Another conceptual point. When you load a model into Loco, the model doesn't suddenly become conscious. It is now available for computation.

When you send input, the inference process performs mathematical operations using the model parameters. When generation stops, there isn't necessarily some ongoing internal thought process continuing in the background. This distinction is especially useful when discussing local AI.

The model is a mathematical system being executed by software on hardware. That may produce incredibly sophisticated behaviour. But the underlying mechanism remains computation.

4.48 The Complete Local AI Picture

We can now combine everything we've learned.

This is the foundation of Local AI. And we're only four lessons into the course.

4.49 The Most Important Mental Model

Remember this:

An AI model is a learned mathematical system represented by parameters and organised according to an architecture.

A model file stores those learned values and associated information. Software loads the model. An inference runtime executes it.

Hardware performs the calculations. Loco provides the application environment that allows a user to interact with the local AI system. The model itself is not the entire AI application.

4.50 Key Takeaways

1. An AI model is a mathematical system learned from data.

Its learned parameters encode patterns acquired during training.

2. Parameters are learned numerical values.

They form a major part of the model's learned representation.

3. Architecture defines how the model is structured.

The architecture determines how parameters are organised and used.

4. "7B" generally means approximately seven billion parameters.

It does not mean the model is automatically better than a smaller model.

5. Model size and file size are different concepts.

The same model can have different file sizes depending on numerical representation and quantization.

6. Model files contain more than weights.

They can contain metadata, tokenizer information, architecture information, and other required data.

7. File size isn't the same as total memory requirement.

Inference also requires memory for context, caches, runtime structures, and other processes.

8. Base and instruction-tuned models are different.

Instruction tuning helps models behave more effectively as assistants.

9. MoE models use routing between expert components.

Total parameters and active parameters can therefore be different.

10. Loading a model is not the same as training a model.

When you use a downloaded model locally, you are generally performing inference.

11. Storage, memory, and compute are different resources.

The model is stored on storage, loaded into memory, and executed by processing hardware.

12. Loco is the application/platform around the local model.

Loco can manage and expose local AI capabilities while the model and inference technology perform the underlying computation.

4.51 Knowledge Check

Question 1

What is an AI model?

Answer:
A learned mathematical system whose parameters have been adjusted during training and which can be used to process new inputs and produce outputs.

Question 2

What are model parameters?

Answer:
Learned numerical values that form a major part of the model's internal representation.

Question 3

What does 8B mean in a model name?

Answer:
It generally indicates approximately eight billion parameters.

Question 4

Does 14B automatically mean a model is better than 8B?

Answer:
No. Model quality depends on many factors, including architecture, data, training, post-training, optimisation, and task suitability.

Question 5

Why can two versions of the same model have different file sizes?

Answer:
They may use different numerical representations or quantization levels.

Question 6

Why isn't a 4 GB model necessarily usable on a computer with only 4 GB of RAM?

Answer:
The computer also needs memory for the operating system, application, inference runtime, context, caches, temporary buffers, and other processes.

Question 7

What is model architecture?

Answer:
The structural and computational design of the neural network that determines how its parameters are organised and used.

Question 8

What is an instruction-tuned model?

Answer:
A model that has undergone additional training intended to improve its ability to follow instructions and interact with users.

Question 9

What is quantization?

Answer:
A technique for representing model parameters using reduced numerical precision in order to decrease memory and storage requirements, usually with some trade-off in quality or numerical fidelity.

Question 10

What is the difference between a model and Loco?

Answer:
The model is the learned mathematical system. Loco is the application/platform that can provide the environment and functionality for managing and using local AI models.

4.52 A Preview of the Next Lesson

We now understand what a model is. But there is still a major mystery:

How does a model actually learn?

Where do those billions of parameters come from? How does the model know that certain words tend to appear together? How does it learn grammar?

How does it learn patterns in code? How does it learn visual concepts? What happens during the enormous computational process called training?

And why is training dramatically more expensive than simply running a trained model? The next lesson answers those questions. We will examine:

Training vs Inference

and follow the journey from raw training data to a trained model, then from that trained model to the response generated on your own computer.

That distinction is one of the most important concepts in all of Local AI.

Sources and Further Reading

• NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0): https://www.nist.gov/itl/ai-risk-management-framework
• Google for Developers, Machine Learning Glossary: https://developers.google.com/machine-learning/glossary
• Vaswani et al., Attention Is All You Need: https://arxiv.org/abs/1706.03762
• Hugging Face, Transformers Documentation: https://huggingface.co/docs/transformers/index
• Hugging Face, Quantization Documentation: https://huggingface.co/docs/transformers/main/en/quantization/overview
• llama.cpp, Open-source local inference framework: https://github.com/ggml-org/llama.cpp

Loco Academy: Lesson Status

Completed: Lesson 4: What Is an AI Model?

Next: Lesson 5: Training vs Inference

Course: Local AI 101