Author Watermark

taking AI off the cloud and bringing it directly to local hardware

Ollama is a free, open-source application that allows us to download and run Large Language Models (LLMs) locally on our own computer.

By serving as a local package manager and inference engine for AI, it removes the need for cloud infrastructure, internet connectivity, or paid subscription fees to use advanced AI. We can download the software directly from the official Ollama Website.

Core Benefits

Key Features

Popular Supported Models

Ollama hosts a massive library of pre-packaged, optimized open-weights models:

Ecosystem Integrations

Because Ollama exposes a standard API, it plugs directly into hundreds of popular developer tools and consumer interfaces:

An LLM (Large Language Model) is a type of artificial intelligence trained on massive amounts of text data to understand, generate, and manipulate human language. Ollama is needed because it solves the privacy, cost, and complexity barriers of running these massive models directly on our own computer instead of relying on big tech cloud servers.
Here is a direct breakdown of what LLMs are and why Ollama is a game-changer for using them.

Lets try to solve some of the questions those may be in our mind as a fresher.

What is an LLM?

An LLM (Large Language Model) is a type of artificial intelligence trained on massive amounts of text data to understand, generate, and manipulate human language. An LLM is a foundational AI model that acts like a highly sophisticated autocomplete engine. It predicts the most logical next word in a sequence based on patterns it learned during training.

Why We Need Ollama

Traditionally, running a powerful AI model required renting expensive cloud servers or having a computer science degree to configure complex code libraries. Ollama acts as the bridge that brings this technology to our everyday computers.

We need Ollama for three primary reasons:

1. It Democratizes Open-Source AI

Before Ollama, if we wanted to run a model like Meta's Llama locally, we had to manually download massive files, manage Python environments, configure complex hardware drivers, and write code just to chat with it.

2. It Solves Data Privacy & Cloud Dependency

When we use cloud-based AIs (like ChatGPT or Claude), our private documents, financial data, or proprietary code are sent over the internet to external corporate servers.

3. It Eliminates Token Fees and Subscriptions

Cloud AI APIs charge us for every single word (token) the AI reads or writes. Premium chat apps charge $20+/month per user.

4. It Optimizes Our Computer's Hardware

AI models are massive and sluggish if run purely on a standard computer processor (CPU). They need Graphics Cards (GPUs) to run at usable speeds.

How to determine if a model will run smoothly ?

To determine if a model will run smoothly, we can easily figure it out just by looking at the model's name on the website or by checking its file size.

There is no complex diagnostic software needed. The deciding factor is how much VRAM (Video RAM on Windows/Linux graphics cards) or Unified Memory (on Mac computers) we have. Ollama loads the entire model into this memory; if it fits, it runs lightning fast. If it overflows, it slows down drastically.

Method 1: Look at the Model Name (The "B" Rule)

When browsing the Ollama Model Library, we will see numbers like 1B, 3B, 8B, or 32B in the tags or model names. The "B" stands for Billion parameters (the size of the AI's brain).

As a universal rule of thumb, we can map the "B" size directly to our computer's RAM/VRAM requirements:

Model Parameter Size Required VRAM (Windows/Linux) or RAM (Mac) Smoothly Runs On...
1B to 3B (e.g., Llama 3.2 3B) 4 GB to 6 GB Most standard laptops & older PCs
7B to 9B (e.g., Llama 3 8B) 8 GB to 12 GB Entry-level gaming PCs & base Macs
14B to 32B (e.g., Qwen 32B) 16 GB to 24 GB High-end gaming PCs & upgraded Macs
70B+ (e.g., Llama 3 70B) 64 GB+ Advanced workstations only

Method 2: Check the Model File Size

If we look closely at the specific model tags on the Ollama Website, it will show us the exact file size (e.g., 4.7GB).

Method 3: Check via the Command Line

If we have already downloaded a model and want to know how it is behaving on our system, we can open our terminal or command prompt while chatting with it and type:

CODE 
ollama ps

This built-in command will print out exactly how much of the model is fitting into our graphics card (GPU) versus our computer processor (CPU). If it says 100% GPU, it is running perfectly smoothly.

CODE 
ollama ps

NAME               ID              SIZE     PROCESSOR    CONTEXT    UNTIL              
llama3.2:latest    a80c4f17acd5    18 GB    100% GPU     131072     4 minutes from now    
subratakumardas@Mac ~ %

Here is the breakdown of our terminal output, explaining exactly what each column means and why our 2 GB model is currently occupying 18 GB of memory.

Part 1: What Each Part of the Output Means

Part 2: Why is a 2 GB Model Taking 18 GB of Memory?

Our model file only takes up 2 GB of space on our hard disk, but it expands to 18 GB in our RAM because of the CONTEXT size (131072).
When an AI model runs, its total memory usage is calculated using a simple equation:

CODE 
Total Memory = Base Model Size + Context Window Memory(KV Cache)

Here is how that breaks down for our session:

  1. The Base Model (2 GB): Ollama loads the core 2 GB file from our disk into our RAM. This is the permanent "brain" of the AI.
  2. The Context Window Allocation (+16 GB): Because our context window is set to a massive 131,072 tokens, Ollama must pre-allocate a huge pool of temporary memory called the Key-Value (KV) Cache. The AI uses this space to calculate and store the mathematical relationships between every single word in a long conversation.
  3. The Result (18 GB): Even if we have only typed a short sentence, Ollama prepares our system for a massive, book-length conversation by reserving that 16 GB of extra RAM right from the start. Since our Mac has plenty of available Unified Memory, Ollama claims this space immediately to guarantee that our chat won't slow down as the conversation grows.

Lets create a Shrunk Model

Lets create a custom clone of llama3.2 that defaults to a lower memory footprint.

  1. Create an empty text file on our Mac desktop and name it Modelfile (no file extension).
  2. Open it in a basic text editor and paste these two lines:
CODE 
FROM llama3.2
PARAMETER num_ctx 2048

or in terminal

CODE 
echo 'FROM llama3.2\nPARAMETER num_ctx 2048' > ModelFile
  1. Open our Mac terminal, navigate to our desktop, and build the custom version:

cd ~/Desktop
ollama create llama3.2-small -f Modelfile

  1. Now we can run our lightweight model permanently using:
CODE 
ollama run llama3.2-small:latest
CODE 
ollama ps
NAME                     ID              SIZE      PROCESSOR    CONTEXT    UNTIL              
llama3.2-small:latest    bb1c85d7e477    2.3 GB    100% GPU     2048       4 minutes from now

What is a Context window ?

A context window is the short-term memory of an AI model. It represents the total amount of text (both our prompts and the AI's previous responses) that the model can process, read, and remember at any single point in time during a conversation.

Think of it like a digital whiteboard.
As we chat, everything we and the AI say gets written on the whiteboard. The AI reads the entire whiteboard every time we press enter to figure out what to say next.

How the Context Window Works

How an Enterprise Application Handles This

When we build an enterprise application, we use a framework like LangChain, LlamaIndex, or an API wrapper to sit between our users and Ollama.

Since Ollama only processes exactly what is sent to its /api/generate endpoint, our wrapper code is responsible for tracking tokens. When it notices our conversation is getting too close to the memory limit, our application code pauses, tells a model to summarize the past 20 messages, and packages that fresh summary with the newest prompt.

By the time the request hits Ollama, the context is already "compacted" by our application layer, keeping Ollama's memory usage tight, predictable, and cost-effective.

We may have seen context compacting in ChatGpt, Claude and Open WebUI chat applications.

Why Does It Affect Our Computer's RAM?

The context window directly impacts our computer hardware because of how AI processes information:

  1. Massive Calculations: To remember context, the AI doesn't just read text like a document; it has to calculate mathematical relationships between every single word on that whiteboard simultaneously.
  2. The "KV Cache": To avoid recalculating those word relationships over and over again with every new prompt, Ollama saves those calculations into a temporary data structure in our RAM called the Key-Value (KV) Cache.
  3. The RAM Spike: A context window of 2,048 tokens requires a tiny whiteboard—so the KV Cache is small, and our model runs at its bare-minimum size (~2 GB to 3 GB). A context window of 131,072 tokens requires an absolutely massive whiteboard, forcing Ollama to pre-allocate an extra 15+ GB of RAM just to hold the math equations for our future conversation.

When should we use a large or small context window?

The Enterprise UseCase : managing infrastructure costs.

Controlling context solves, One of the most critical challenges in modern enterprise software engineering: managing infrastructure costs.

When building a cloud-based enterprise application that uses AI, memory management is no longer just a technical detail—it directly dictates our monthly cloud bill. Controlling our context window sizes and optimizing how models sit in memory is vital for keeping an enterprise profitable.

Here is exactly why managing this memory is so critical when moving from our local Mac to a cloud environment:

1. Cloud RAM and VRAM are Extremely Expensive

On our Mac, our Unified Memory is a one-time purchase. On the cloud (AWS, Google Cloud, or Azure), we rent specialized AI hardware—specifically NVIDIA GPUs (like the A100 or H100)—and we are billed by the minute or hour.

2. The Context Window Cost Multiplier (The "Quadratic" Problem)

As we discovered with our ollama ps output, a model that takes up 2 GB on a disk can swell to 18 GB in memory when the context window is maxed out.
In AI engineering, the memory required for the context window (the KV Cache) scales heavily as the conversation gets longer. If we leave the context window unmanaged at 131k tokens for every user:

3. Server Efficiency: Maximize "Concurrancy"

In an enterprise setup, our goal is to fit as many user requests onto a single server as possible before needing to launch a second one.

Summary for our Enterprise Architecture:

When pitching or designing an enterprise AI system, we can explain it using this core financial principle:

"We must aggressively manage our AI runtime memory. By tailoring our model context windows to the specific task—using small context windows for quick transactions and reserving large context windows only for massive document processing—we drastically maximize our hardware efficiency, prevent cloud server waste, and directly control our operational expenses (OpEx)."

Page # 1 subraatakumar | @subraatakumar