Run Large Language Models Locally from Scratch

How to Run Large Language Models Locally from Scratch?

Running Large Language Models (LLMs) locally is becoming more accessible — giving you privacy, speed, and control over your AI workflows without relying on cloud APIs.


In this guide, we'll set up everything from scratch using:

• Conda for environment management
• Pip for installing Python dependencies
• Ollama for running local LLMs like Mistral-7B and DeepSeek
  
  

1. Why Run LLMs Locally?

• Privacy – Keep sensitive data on your machine.
• Offline Capability – No internet needed after setup.
• Cost Savings – Avoid API subscription costs.
• Customizability – Fine-tune and experiment without restrictions.
  
  

2. Install Conda

If you don’t have Conda yet, download Miniconda (lightweight) or Anaconda.

After installation, verify:

conda --version

3. Create a virtual environment

We'll be creating a virtual environment for our LLM project.

conda create --name local-llm
conda activate local-llm

4. Install Ollama

Ollama is a runtime for local models which is available on macOS, Linux, and Windows.

In macOS:

brew install ollama

In Linux:

curl -fsSL https://ollama.com/install.sh | sh

In Windows:
Download from Ollama's [official website] (https://ollama.com/download)

5. Verify Ollama Installation

ollama --version

If installed correctly, you should see the Ollama version number.

6. Download Mistral-7B and DeepSeek Models

Now we'll be downloading two models locally. Ollama makes it simple to pull models:

ollama pull mistral
ollama pull deepseek

7. Useful Ollama commands

Here are some list of useful Ollama commands:

ollama list # list all available models
ollama run <model_name> # run any specific model interactively
ollama serve # serve ollama api so that other apps can connect
ollama rm <model_name> # remove model from your system

8. Integrating Ollama with Python

Install the python client:

pip install ollama

Example script:

import ollama
response = ollama.chat(
    model="mistral",
    messages=[{"role": "user", "content": "Write a one line description of Australia"}],
)
print(response["message"]["content"])

Output from the model:

Australia, the world's largest island and sixth-largest country, is a vibrant and
diverse nation known for its unique wildlife, vast outback landscapes, bustling cities,
and beautiful beaches.

Here's a fun fact: Did you know that Australia has more types of terrestrial mammals
than any other continent? This includes the famous kangaroo, wallaby, koala, and Tasmanian
devil!