How to Run Ollama on a CPU-Only VPS as a Private LLM API

Ollama runs open language models on your own server and exposes them through a simple HTTP API. On a VPS without a GPU, it runs small, quantised models on the CPU. That is slower than a graphics card, but it keeps the data on your server, costs nothing per request, and is enough for many internal jobs: summarising support tickets, classifying messages, drafting text, or powering a private chat for a small team.

This guide sets up Ollama on Ubuntu 24.04 on a CPU-only server. It covers the install, choosing a model that fits in memory, the settings that control memory and concurrency, calling the API, and the part most guides skip: protecting the API. Ollama binds to the local machine by default, and the API has no user accounts, so you must put an authenticated proxy in front of it before anyone else can reach it.

Before you choose a server#

A CPU model needs RAM, not a graphics card. Memory decides what you can run, and the number of CPU cores decides how fast it responds. Work out the requirement before you buy a plan:

• RAM. A quantised model must fit in memory with room for the system and for the context the request uses. Small models of a few billion parameters run in a few gigabytes. Larger models need more RAM, and on a small server they become too slow to be useful.
• Cores. More cores speed up generation, up to the point where memory bandwidth becomes the limit.
• Disk. Each model is a file of several gigabytes. Check the free space before you pull a model.

Start with a model that fits comfortably, measure it with real prompts, and move up only if the quality is worth the wait. Use the numbers you measure, not the numbers on a model’s listing page.

Step 1: Check the server and install Ollama#

free -h
nproc
df -h /

Check the output against the requirement you worked out. Then install Ollama. Download the install script first and read it before you run it, rather than piping it straight into a shell:

curl -fsSLo /tmp/ollama-install.sh https://ollama.com/install.sh
less /tmp/ollama-install.sh
sudo sh /tmp/ollama-install.sh

Expected: the script reports that it installed Ollama and created the ollama service. Check the version and the service:

ollama --version
systemctl status ollama --no-pager | head -5

The installer creates a system user called ollama and runs the server as that user, not as root. Keep it that way.

Step 2: Pull a model and run one prompt#

Pull a small, quantised model and send it a prompt from the command line. Choose a model name from the Ollama library, and check its size before you pull:

ollama pull llama3.2
ollama list
ollama run llama3.2 "In one sentence, what does a reverse proxy do?"

Expected: ollama list shows the model with its size on disk, and the prompt returns a short answer. The first answer takes longer, because the model is loaded into memory. Run ollama ps while the model is loaded: it shows how much of the model is in memory and whether it runs on the CPU or a GPU. On a CPU-only server, it should say the CPU.

Step 3: Set the memory and concurrency limits#

Ollama’s behaviour is set with environment variables, and the two that matter most for a small server are how many requests a model handles at once, and how many models stay loaded. The Ollama documentation gives these defaults:

• OLLAMA_NUM_PARALLEL is the number of requests each model processes at the same time. The default is 1. The memory needed grows with this number and with the context length, so leave it at 1 on a small server, and let other requests wait in the queue.
• OLLAMA_MAX_LOADED_MODELS is how many models can be in memory at once. The default for CPU inference is 3. On a small server, set it to 1, so a second model does not push the first out of memory.

Set them with a systemd override, which survives updates and reboots:

sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_LOADED_MODELS=1"

Save, then reload and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

If you store models on a different disk, set OLLAMA_MODELS in the same override to that folder. The ollama user must be able to read and write it, so check the folder’s owner after you change it.

Step 4: Call the API locally#

Ollama listens on 127.0.0.1:11434. Send a request with curl. Setting stream to false returns the whole answer in one response:

curl -s http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Summarise this ticket in one line: The checkout page times out after the last update.",
  "stream": false
}'

Expected: a JSON object with a response field holding the answer, and timing fields for the load and the generation. List the installed models with curl -s http://127.0.0.1:11434/api/tags.

From your application, the same call goes to your own endpoint, and the code is the same. Keep the timeout generous, because a CPU model takes longer than a hosted one, and a request that waits for a queue slot takes longer still.

Step 5: Protect the API before anyone else uses it#

Put Nginx in front of Ollama, on a separate hostname, with HTTPS and basic authentication. First, create the password file with the Apache password tool, which is in the apache2-utils package:

sudo apt install -y nginx apache2-utils
sudo htpasswd -c /etc/nginx/.ollama-htpasswd api-user

Then the server block, which proxies only to the local Ollama port and requires the password:

server {
    listen 443 ssl;
    server_name llm.example.com;

    auth_basic "Private LLM";
    auth_basic_user_file /etc/nginx/.ollama-htpasswd;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_buffering off;
        proxy_read_timeout 300s;
    }
}

The proxy_buffering off line lets streamed answers reach the client as they are generated. proxy_read_timeout is set long, because a CPU answer takes time. Add the TLS certificate with how to set up free SSL on Nginx with Certbot, then test and reload:

sudo nginx -t && sudo systemctl reload nginx

Test the protection from another computer. Without the password, the answer must be 401:

curl -s -o /dev/null -w "%{http_code}\n" https://llm.example.com/api/tags

Expected: 401. With the password, the same request returns 200. Check also that the raw port is closed to the internet: nc -vz your-server-ip 11434 from outside should fail.

Step 6: Watch memory and time under real load#

Run a few prompts that look like your real ones, and watch the server while they run:

watch -n 2 free -h

Watch the available column, not free. Linux uses spare memory for caches, so free looks low even on a healthy server. A falling available figure while a model answers is the sign you are near the limit. Check how the model is placed while the answer is generating:

ollama ps

If the available memory drops below about a fifth of the total during a normal prompt, use a smaller model or a larger plan. Keep headroom for the operating system and the other services on the server, because the kernel stops processes that run out of memory, and that can include your database.

Common mistakes#

• Binding Ollama to a public address. Setting OLLAMA_HOST to 0.0.0.0 without a proxy and password exposes the API to anyone who finds the port.
• Judging a model by its size on disk alone. The memory during a request is larger than the file, and it grows with the context length.
• Allowing several parallel requests on a small server. Each one needs its own share of memory, so the server starts swapping. Keep OLLAMA_NUM_PARALLEL=1 and queue the work.
• Running the install script without reading it. Download it, read it, then run it, as shown in Step 1.
• Expecting hosted-model speed. A CPU model is slower. Build the feature around that, with a clear loading state, instead of a response in the same second.

Frequently asked questions#

Can Ollama run without a GPU?#

Yes. It runs on the CPU, which is slower but works. ollama ps shows where the model runs.

Which model should I start with?#

Start with a small, quantised model that fits in memory with headroom, then test it on prompts from your real use. Move to a larger model only if the answers are not good enough for the job, and only if the server still responds in a time your users accept.

Is it safe to send customer data to a model on my own server?#

The data stays on your server, which is the main advantage over a hosted API. You are still responsible for who can reach the API, the logs you keep, and the protection described in Step 5.

Can I use it from a website?#

Call it from your server-side code, not from the browser. A browser that calls the API directly would need the password, and anyone could read it from the page.

How do I update Ollama and the models?#

Re-run the install script you read in Step 1 to update Ollama, then restart the service. Pull a newer model tag with ollama pull, and remove old ones with ollama rm once you have tested the new one.

Ollama gives you a private language model on a plain Linux server. Keep it on localhost, limit it to one request at a time, choose a model that fits with room to spare, and publish it only through a proxy that asks for a password and uses HTTPS. Then measure it with real prompts, and let the measurements decide whether it is good enough for the job.