Qwen and other open-weight models

Open-weight models don’t connect to MCP servers by themselves — the agent framework running them does. Two routes work well; both need a model that supports tool calling.

Qwen with Qwen-Agent

Qwen-Agent is the Qwen team’s agent framework, with MCP built in. It runs Qwen through Alibaba Cloud (DashScope) or any OpenAI-compatible server — vLLM, or Ollama on your machine.

pip install -U "qwen-agent[mcp]"

import os

from qwen_agent.agents import Assistant

llm_cfg = {
    # A local or self-hosted Qwen behind an OpenAI-compatible API (vLLM, Ollama, …)
    "model": "qwen3",
    "model_server": "http://localhost:11434/v1",
    "api_key": "EMPTY",
    # Or Qwen on Alibaba Cloud instead:
    # "model_type": "qwen_dashscope", "model": "qwen-max-latest",  (uses DASHSCOPE_API_KEY)
}

tools = [{
    "mcpServers": {
        "jitprompt": {
            "url": "https://app.jitprompt.com/mcp",
            "type": "streamable-http",   # required: without it Qwen-Agent assumes SSE
            "headers": {"Authorization": f"Bearer {os.environ['JITPROMPT_KEY']}"},
        }
    }
}]

bot = Assistant(
    llm=llm_cfg,
    function_list=tools,
    system_message=(
        "Your operating instructions are in the jitprompt tools: call get_toc first, and "
        "search_instructions before each new task. Follow what they return."
    ),
)

messages = [{"role": "user", "content": "Can I get a refund on my annual plan?"}]
for response in bot.run(messages=messages):
    pass  # streams partial responses; the last one is complete
print(response[-1]["content"])

Any local model with Ollama and LangChain

Ollama serves open-weight models on your machine, and the LangChain setup drives them like any other model. Pull a model with tool calling, then point MODEL at it:

ollama pull qwen3        # or: gpt-oss, llama3.3, mistral-small, …
pip install "langchain[mcp]" langchain-ollama

MODEL = "ollama:qwen3"   # or "ollama:gpt-oss" (OpenAI's open-weight model), "ollama:llama3.3", …

Everything else — the jitprompt connection, the tools, the system prompt — is the code on the LangChain page, unchanged.

Tips for smaller models

• Keep the system-prompt line that tells the model when to call search_instructions; smaller models follow tool descriptions less reliably than frontier ones.
• Models with a short context window benefit most from jitprompt: they read the few sections that apply instead of a whole policy. Leave room for them — a search returns up to about 4,000 tokens.
• Test phrasings in the search sandbox to see what the model will get back.