How to Run Qwen3.5-27B Locally: Complete Step by Step Guide

Running large language models locally gives you complete control over your AI infrastructure. Instead of relying on expensive API calls to OpenAI or Anthropic, you can deploy powerful open-weight models like Qwen3.5-27B on your own GPU hardware. This complete tutorial shows you how to set up a remote H100 SXM instance on Vast.ai, serve Qwen3.5-27B with vLLM, connect it to OpenCode, and test its agentic coding ability on a real FastAPI project.

Why Choose vLLM Over llama.cpp for Qwen3.5-27B

While llama.cpp is excellent for many local inference setups, running a quantized 27B model through it often comes with trade-offs: lower output quality, reduced coding performance, and more setup friction. For larger models like Qwen3.5-27B, quantization can noticeably hurt reliability, and local deployments through llama.cpp can become difficult to manage if you want smooth, production-like agent behavior.

Instead, this tutorial uses vLLM on a rented H100 SXM GPU. That gives you a more stable OpenAI-compatible endpoint, better performance from the full model, and a much cleaner setup for agentic coding workflows. You get very high tokens-per-second throughput, a large context length, and a much smoother overall coding experience.

Prerequisites

Before you begin, make sure you have the following:

Step 1: Launch and Access Your Vast.ai Instance

We are using Vast.ai because it is a GPU marketplace that usually gives you far more flexibility on price and hardware than a traditional single-vendor cloud. You can rent anything from community-hosted machines to Secure Cloud / datacenter offers, which Vast marks with a blue datacenter label. For a setup like this, we strongly recommend choosing one of those blue-labeled datacenter machines for better reliability and a smoother experience.

Step 2: Install vLLM and Serve Qwen3.5

In this step, you will create an isolated Python environment, install vLLM, and launch Qwen3.5-27B as an OpenAI-compatible API that OpenCode can talk to. vLLM is a high-throughput inference engine for large language models, designed to serve models efficiently through an API.

This gives you a dedicated Python 3.12 environment so the model server dependencies stay isolated from the rest of the machine.

After the model downloads and loads into GPU memory, you should see a message showing that the server startup is complete. This launches Qwen3.5-27B locally on port 8000 and protects the endpoint with the API key local-dev-key.

Step 3: Install and Configure OpenCode

In this step, you will open a second terminal, install OpenCode, and connect it to the local vLLM endpoint you started in Step 2. OpenCode is an open-source coding agent that can run in the terminal and connect to local models through its provider configuration.

Then refresh your shell:

This reloads your shell so the opencode command is available right away.

This tells OpenCode to use your local vLLM endpoint at http://127.0.0.1:8000/v1 instead of a hosted API. It also uses the same API key you set when launching the vLLM server in Step 2, so OpenCode can authenticate successfully.

Step 4: Connect OpenCode to the Local Model

Now, create a project directory for testing the coding agent:

This launches OpenCode inside the investment-api folder and uses your local Qwen3.5 model as its backend. From here, you can begin asking it to inspect files, explain code, or generate new project components using the model running on your rented GPU.

Step 5: Test Qwen3.5 on a Real Coding Task

Now it is time to test the full setup on a real coding task. Start with a very simple prompt just to make sure everything is working. Within a second, you should get a reply back, which is a good sign that the model is connected properly.

Next, give it a more realistic agentic coding task. For example:

After around a minute of reasoning, the model will start asking useful follow-up questions. It will ask which data source to use, what kind of storage or database should be used, and which stock tickers to include. This is a good sign because it shows that the model is not just jumping into code blindly. It is trying to clarify the requirements first.

Once that is done, it creates a plan and starts working through the task step by step. It breaks the problem into smaller tasks, adds them to its to-do list, and then completes each part one by one. In less than five minutes, it can create the full project structure and generate a mostly working FastAPI backend, which is very fast for a task like this.

After that, ask it to test the API and run a smoke test. You will get an almost-working financial API. There may still be a few issues to fix, but for a short run like this, the performance is very impressive.

Performance Advantages of This Setup

Compared with trying to run a heavily quantized 27B model locally through llama.cpp, this setup is often more stable and better suited for serious coding work. You avoid many of the performance trade-offs, memory limitations, and setup issues that can show up when pushing larger models into a purely local environment.