Run Large Language Models Locally from Scratch

Running Large Language Models (LLMs) locally is becoming more accessible — giving you privacy, speed, and control over your AI workflows without relying on cloud APIs. In this guide, we'll set up everything from scratch using:

• Conda for environment management
• Pip for installing Python dependencies
• Ollama for running local LLMs like Mistral-7B and DeepSeek

1. Why Run LLMs Locally?#

• Privacy – Keep sensitive data on your machine.
• Offline Capability – No internet needed after setup.
• Cost Savings – Avoid API subscription costs.
• Customizability – Fine-tune and experiment without restrictions.

2. Install Conda#

If you don’t have Conda yet, download Miniconda (lightweight) or Anaconda.

After installation, verify:

conda --version

3. Create a virtual environment#

We'll be creating a virtual environment for our LLM project.

conda create --name local-llm
conda activate local-llm

4. Install Ollama#

Ollama is a runtime for local models which is available on macOS, Linux, and Windows.

In macOS:

brew install ollama

In Linux:

curl -fsSL https://ollama.com/install.sh | sh

In Windows: Download from Ollama's [official website] (https://ollama.com/download)

5. Verify Ollama Installation#

ollama --version

If installed correctly, you should see the Ollama version number.

6. Download Mistral-7B and DeepSeek Models#

Now we'll be downloading two models locally. Ollama makes it simple to pull models:

ollama pull mistral
ollama pull deepseek

7. Useful Ollama commands#

Here are some list of useful Ollama commands:

ollama list # list all available models
ollama run <model_name> # run any specific model interactively
ollama serve # serve ollama api so that other apps can connect
ollama rm <model_name> # remove model from your system

8. Integrating Ollama with Python#

Install the python client:

pip install ollama

Example script:

import ollama
response = ollama.chat(
    model="mistral",
    messages=[{"role": "user", "content": "Write a one line description of Australia"}],
)
print(response["message"]["content"])

Output from the model:

Australia, the world's largest island and sixth-largest country, is a vibrant and
diverse nation known for its unique wildlife, vast outback landscapes, bustling cities,
and beautiful beaches.
 
Here's a fun fact: Did you know that Australia has more types of terrestrial mammals
than any other continent? This includes the famous kangaroo, wallaby, koala, and Tasmanian
devil!

Happy Coding 💻