Harbor

What is Harbor about?

Harbor provides a unified CLI and companion desktop app that automatically provisions, connects, and configures a wide range of LLM back‑ends (Ollama, llama.cpp, vLLM, etc.), front‑ends (Open WebUI, LibreChat, etc.), and auxiliary services (SearXNG, Speaches, ComfyUI, Boost, and many satellites). All components are wired together out‑of‑the‑box, so users can focus on interacting with models rather than managing Docker files or network routes.

How to use Harbor?

# Start the default stack (Open WebUI + llama.cpp)
harbor up

# Add additional services, e.g., web search and voice chat
harbor up searxng speaches

# Launch a specific backend and bind it to a coding tool
harbor launch --backend ollama --model qwen3.5:4b codex

# Expose a service to the internet (use with caution)
harbor tunnel vllm

# Save and switch configuration profiles
harbor profile save llama4
harbor profile use default

The CLI also offers shortcuts for model management, logs, shell access, QR‑code generation for mobile access, and an eject command that outputs a standalone docker‑compose.yml for the selected services.

Key Features

• One‑command stack: harbor up pulls and starts a fully wired LLM stack.
• Extensive service catalog: >200 front‑ends, back‑ends, and satellites (e.g., Boost, ComfyUI, LangFlow, n8n, etc.).
• Cross‑service connectivity: Automatic Open WebUI integration, Traefik reverse‑proxy, and shared model caches.
• Host‑native inference: macOS Metal (DMR, MLX, oMLX) and GPU support without containers.
• Boost workflow engine: Chain web research, auditing, and style modules for coding agents.
• Portable configuration: Profiles, env‑file imports, and harbor eject for custom Docker setups.
• Convenient access: QR codes, direct harbor open <service>, and built‑in tunneling.
• CLI ergonomics: Command scrambling tolerance, history, and unified harbor config management.

Use Cases

• Local model experimentation – quickly test different inference back‑ends and model versions.
• Agentic coding – run Codex, Claude, or other agents through Boost‑enhanced workflows.
• Web RAG and chat – combine SearXNG with Open WebUI for internet‑augmented responses.
• Image generation – launch ComfyUI + Flux for AI‑driven graphics.
• Voice assistants – use Speaches for speech‑to‑text / text‑to‑speech integrated with chat UI.
• Workflow automation – pipe n8n, Flowise, or LangFlow into LLM‑driven pipelines.
• Educational sandbox – provide students a ready‑to‑run LLM stack without manual Docker composition.

FAQ

Q: Do I need Docker installed? A: Yes, Docker (and Docker‑Compose) are required for containerized services. Host‑native back‑ends on macOS run without containers.

Q: Can I run multiple back‑ends simultaneously? A: Yes, you can start several (harbor up vllm llamacpp tgi …). The CLI handles port allocation and cross‑service routing.

Q: How are models cached? A: A shared cache directory is used across services; downloading a model once makes it available to all compatible back‑ends.

Q: Is there a way to run Harbor without the GUI app? A: The CLI works fully independently; the desktop app is optional for visual configuration.

Q: How do I expose my stack to the internet safely? A: Use harbor tunnel <service> which runs an internal Traefik reverse‑proxy with optional authentication. Exposing services publicly is recommended only with proper auth and network security.

Harbor

https://github.com/user-attachments/assets/e4897391-c5a8-4391-93c3-9f8b76155f11

Setup your local LLM stack effortlessly.

# Starts fully configured Open WebUI and llama.cpp
harbor up

# Now, Open WebUI can do Web RAG and TTS/STT
harbor up searxng speaches

Harbor is a CLI and companion app that lets you spin up a complete local LLM stack—backends like Ollama, llama.cpp, or vLLM, frontends like Open WebUI, plus supporting services like SearXNG for web search, Speaches for voice chat, and ComfyUI for image generation—all pre-wired to work together with a single harbor up command. No manual setup: just pick the services you want and Harbor handles the Docker Compose orchestration, configuration, and cross-service connectivity so you can focus on actually using your models.

News

• v0.5.19 - harbor up no longer flags Harbor's own loopback-published ports as conflicts or fails on init sidecars that finish early, Hermes starts with images that ship no PyYAML, AutoGPT runs the current AutoGPT Platform, and Boost's Anthropic API works with OpenAI gpt-5 and o-series models
• v0.5.18 - Boost security fix: the discussurl module's URL matching can no longer be stalled by crafted input, and similar slow pattern matching was removed from autocheck and web page reading
• v0.5.17 - Harbor AppImages install and launch correctly, and the desktop app opens reliably on Linux
• v0.5.16 - Two new services: rummage, local multimodal semantic search over your files with EmbeddingGemma 2 on CPU, NVIDIA or AMD, and Harbor Framework agent evaluations under the harbor handle
• v0.5.14 - Eight new services (Latitude, AutoGen Studio, LobeHub, R2R, Caddy, RAGFlow, ClaraVerse, Qwen Code), a Dockerized Harbor CLI runner, external Ollama support for Boost and Open WebUI, opt-in CLI command wrappers, and fixes for ROCm detection, Perplexica on LAN, AnythingLLM settings persistence and the Harbor App on dot-prefixed homes
• v0.5.13 - harbor models pull no longer reports success when a model download actually failed, and the PrismML backend documents its two Bonsai 2 quants with measured speed and accuracy
• v0.5.12 - New PrismML llama.cpp backend that runs Ternary Bonsai 2 GGUFs on CPU, NVIDIA, and ROCm with the same model workflow and cross-service integrations as llama.cpp

Documentation

• Installing Harbor Guides to install Harbor CLI and App, including the agent install prompt for Claude Code, Codex, and similar tools
• Harbor User Guide High-level overview of working with Harbor
• Harbor App Overview and manual for the Harbor companion application
• Harbor Services Catalog of services available in Harbor
• Harbor CLI Reference Read more about Harbor CLI commands and options. Read about supported services and the ways to configure them.
• Harbor Guides Practical guides for local LLM stack setup, local web RAG, coding agents, OpenAI-compatible backends, and self-hosted AI stack planning.
• Join our Discord Get help, share your experience, and contribute to the project.

What can Harbor do?

✦ Local LLMs

Run LLMs and related services locally, with no or minimal configuration, typically in a single command or click.

# All backends are pre-connected to Open WebUI
harbor up ollama
harbor up llamacpp
harbor up vllm

# Host-native Metal/GPU inference on macOS
harbor up dmr
harbor up mlx
harbor up omlx

# Set and remember args for llama.cpp
harbor llamacpp args -ngl 32

Local Coding Tools

Use Harbor backends from installed coding and agent CLIs without hand-editing each tool's provider config. harbor launch starts or detects a Harbor OpenAI-compatible backend, wires the selected model into the host tool, and leaves the tool running in your current project directory.

# Run Codex against an explicit Harbor backend and model
harbor launch --backend ollama --model qwen3.5:4b codex

# Add Harbor Boost web search and URL-reading tools
harbor launch --web --backend ollama --model qwen3.5:4b codex

# Write adapter config without starting the tool
harbor launch --config opencode

# Force the container service when a name also has a host adapter
harbor launch --service opencode --help

Supported host tools include claude, codex, copilot, droid, grok, hermes, mi, openclaw, opencode, pi, pool, and vscode.

Agentic coding with Boost

Harbor Boost chains agentic modules—web research, deliverable audits, scope checks, and style passes—into custom workflows. harbor launch --workflow routes coding agents through a single Boost module instead of the raw backend model.

# Two-hop research before answering
harbor launch --workflow deephop --backend ollama --model qwen3.5:4b codex

# Fast web research before answering
harbor launch --workflow quickhop --backend ollama --model qwen3.5:4b codex

# Deliverable audit before the final answer
harbor launch --workflow autocheck --backend ollama --model qwen3.5:4b codex

Modules quickhop and deephop auto-start SearXNG. Define multi-step workflows with HARBOR_BOOST_WORKFLOWS or workflows.yaml, and mount your project with boost.workspace so agents can verify real file paths—see Boost configuration and the modules reference.

Cutting Edge Inference

Harbor supports most of the major inference engines as well as a few of the lesser-known ones. On macOS, Docker Model Runner, MLX, and oMLX run inference on the host with Metal acceleration — no containers needed for the compute.

# We sincerely hope you'll never try to run all of them at once
harbor up vllm llamacpp tgi litellm tabbyapi aphrodite sglang ktransformers mistralrs airllm

# Host-native inference (macOS Metal / Linux GPU)
harbor up dmr mlx omlx

Tool Use

Enjoy the benefits of MCP ecosystem, extend it to your use-cases.

# Manage MCPs with a convenient Web UI
harbor up metamcp

# Connect MCPs to Open WebUI
harbor up metamcp mcpo

Generate Images

Harbor includes ComfyUI + Flux + Open WebUI integration.

# Use FLUX in Open WebUI in one command
harbor up comfyui

Local Web RAG / Deep Research

Harbor includes SearXNG that is pre-connected to a lot of services out of the box: Perplexica, ChatUI, Morphic, Local Deep Research and more.

# SearXNG is pre-connected to Open WebUI
harbor up searxng

# And to many other services
harbor up searxng chatui
harbor up searxng morphic
harbor up searxng perplexica
harbor up searxng ldr

LLM Workflows

Harbor includes multiple services for build LLM-based data and chat workflows: Dify, LitLytics, n8n, Open WebUI Pipelines, FloWise, LangFlow

# Use Dify in Open WebUI
harbor up dify

Talk to your LLM

Setup voice chats with your LLM in a single command. Open WebUI + Speaches

# Speaches includes OpenAI-compatible SST and TTS
# and connected to Open WebUI out of the box
harbor up speaches

Chat from the phone

You can access Harbor services from your phone with a QR code. Easily get links for local, LAN or Docker access.

# Print a QR code to open the service on your phone
harbor qr
# Print a link to open the service on your phone
harbor url webui

Chat from anywhere

Harbor includes a built-in tunneling service to expose your Harbor to the internet.

[!WARN] Be careful exposing your computer to the Internet, it's not safe.

# Expose default UI to the internet
harbor tunnel

# Expose a specific service to the internet
# ⚠️ Ensure to configure authentication for the service
harbor tunnel vllm

# Harbor comes with traefik built-in and pre-configured
# for all included services
harbor up traefik

LLM Scripting

Harbor Boost allows you to easily script workflows and interactions with downstream LLMs.

# Use Harbor Boost to script LLM workflows
harbor up boost

Config Profiles

Save and manage configuration profiles for different scenarios. For example - save llama.cpp args for different models and contexts and switch between them easily.

# Save and use config profiles
harbor profile save llama4
harbor profile use default

# Import profiles from a URL
harbor profile use https://example.com/path/to/harbor-profile.env

Command History

Harbor keeps a local-only history of recent commands. Look up and re-run easily, standalone from the system shell history.

# Lookup recently used harbor commands
harbor history

Eject

Ready to move to your own setup? Harbor will give you a docker-compose file replicating your setup.

# Eject from Harbor into a standalone Docker Compose setup
# Will export related services and variables into a standalone file.
harbor eject searxng llamacpp > docker-compose.harbor.yml

---

Services

UIs

AnythingLLM ⦁︎ AstrBot ⦁︎ BionicGPT ⦁︎ Chat Nio ComfyUI ⦁︎ Hollama ⦁︎ HuggingFace ChatUI ⦁︎ KoboldCpp LibreChat ⦁︎ Lobe Chat ⦁︎ Mikupad ⦁︎ ol1 Omnichain ⦁︎ Onyx ⦁︎ Open WebUI ⦁︎ oterm Parllama ⦁︎ SillyTavern ⦁︎ Voicebox

Backends

AirLLM ⦁︎ Aphrodite ⦁︎ Chandra ⦁︎ faster-whisper-server ⦁︎ ik_llama.cpp ⦁︎ PrismML llama.cpp KoboldCpp ⦁︎ KTransformers ⦁︎ Lemonade ⦁︎ llama.cpp lmdeploy ⦁︎ mistral.rs ⦁︎ Modular MAX ⦁︎ Needle Nexa SDK ⦁︎ Ollama ⦁︎ openedai-speech ⦁︎ Parler Docker Model Runner ⦁︎ MLX ⦁︎ oMLX ⦁︎ SGLang ⦁︎ Speaches TabbyAPI ⦁︎ Text Generation Inference ⦁︎ Text Embeddings Inference ⦁︎ vLLM

Satellites

Activepieces ⦁︎ Agent Zero ⦁︎ aichat ⦁︎ Aider Airweave ⦁︎ AstrBot ⦁︎ autogpt ⦁︎ Beszel Bifrost ⦁︎ Bolt.new ⦁︎ Browser Use ⦁︎ cloudflared cmdh ⦁︎ Cognee ⦁︎ DBHub ⦁︎ DeerFlow Dify ⦁︎ Docling ⦁︎ Drawio ⦁︎ Fabric facts ⦁︎ Flowise ⦁︎ gptme ⦁︎ Harbor Bench Harbor Boost ⦁︎ Hermes Agent ⦁︎ Home Assistant ⦁︎ JupyterLab K6 ⦁︎ Karakeep ⦁︎ Khoj ⦁︎ KoboldCpp Kotaemon ⦁︎ LangFlow ⦁︎ LangFuse ⦁︎ Latent Scope LibreTranslate ⦁︎ LiteLLM ⦁︎ LitLytics ⦁︎ llama-swap lm-evaluation-harness ⦁︎ Local Deep Research ⦁︎ LocalAI ⦁︎ MCP Forge mcpo ⦁︎ MetaMCP ⦁︎ mi ⦁︎ MindsDB ML Intern ⦁︎ Morphic ⦁︎ n8n ⦁︎ nanobot Netdata ⦁︎ npcsh ⦁︎ OmniParser ⦁︎ Open Design Open Interpreter ⦁︎ Open Notebook ⦁︎ Open Terminal ⦁︎ Open WebUI Pipelines OpenClaw ⦁︎ OpenCode ⦁︎ OpenFang ⦁︎ OpenHands OptiLLM ⦁︎ Perplexica ⦁︎ PhotoPrism ⦁︎ Plandex Postiz ⦁︎ Presenton ⦁︎ Promptfoo ⦁︎ Qdrant RAGLite ⦁︎ Repopack ⦁︎ Resume Matcher ⦁︎ ROS MCP Server SearXNG ⦁︎ Sim Studio ⦁︎ Solo CLI ⦁︎ SQL Chat SuperGateway ⦁︎ SurfSense ⦁︎ TextGrad ⦁︎ tokscale Traefik ⦁︎ txtai RAG ⦁︎ Unsloth ⦁︎ Unsloth Studio Webtop ⦁︎ Windmill ⦁︎ LightRAG ⦁︎ Paperless-ngx ⦁︎ Paperless-GPT ⦁︎ Whishper ⦁︎ Linkwarden

See services documentation for a brief overview of each.

CLI Tour

# Run Harbor with default services:
# Open WebUI and llama.cpp
harbor up

# Run Harbor with additional services
# Running SearXNG automatically enables Web RAG in Open WebUI
harbor up searxng

# Speaches includes OpenAI-compatible SST and TTS
# and connected to Open WebUI out of the box
harbor up speaches

# Run additional/alternative LLM Inference backends
# Open Webui is automatically connected to them.
harbor up llamacpp tgi litellm vllm tabbyapi aphrodite sglang ktransformers

# Host-native backends for macOS (Metal acceleration, no containers)
harbor up dmr
harbor up mlx
harbor up omlx

# Run different Frontends
harbor up librechat chatui bionicgpt hollama

# Get a free quality boost with
# built-in optimizing proxy
harbor up boost

# Use FLUX in Open WebUI in one command
harbor up comfyui

# Use custom models for supported backends
harbor llamacpp model https://huggingface.co/user/repo/model.gguf

# Manage models for host-native backends
harbor dmr pull ai/mistral
harbor dmr ls
harbor mlx pull mlx-community/Qwen3.5-4B-4bit
harbor omlx pull mlx-community/Qwen3.5-4B-4bit

# Access service CLIs without installing them
# Caches are shared between services where possible
harbor hf scan-cache
harbor hf download google/gemma-2-2b-it
harbor ollama list

# Shortcut to HF Hub to find the models
harbor hf find gguf gemma-2
# Use HFDownloader and official HF CLI to download models
harbor hf dl -m google/gemma-2-2b-it -c 10 -s ./hf
harbor hf download google/gemma-2-2b-it

# Where possible, cache is shared between the services
harbor tgi model google/gemma-2-2b-it
harbor vllm model google/gemma-2-2b-it
harbor aphrodite model google/gemma-2-2b-it
harbor tabbyapi model google/gemma-2-2b-it-exl2
harbor mistralrs model google/gemma-2-2b-it
harbor opint model google/gemma-2-2b-it
harbor sglang model google/gemma-2-2b-it

# Convenience tools for docker setup
harbor logs llamacpp
harbor exec llamacpp ./scripts/llama-bench --help
harbor shell vllm

# Tell your shell exactly what you think about it
harbor opint
harbor aider
harbor aichat
harbor cmdh

# Use fabric to LLM-ify your linux pipes
cat ./file.md | harbor fabric --pattern extract_extraordinary_claims | grep "LK99"

# Open services from the CLI
harbor open webui
harbor open llamacpp
# Print yourself a QR to quickly open the
# service on your phone
harbor qr
# Feeling adventurous? Expose your Harbor
# to the internet
harbor tunnel

# Config management
harbor config list
harbor config set webui.host.port 8080

# Create and manage config profiles
harbor profile save l370b
harbor profile use default
# Import profile from a URL
harbor profile use https://example.com/path/to/harbor-profile.env

# Lookup recently used harbor commands
harbor history

# Eject from Harbor into a standalone Docker Compose setup
# Will export related services and variables into a standalone file.
harbor eject searxng llamacpp > docker-compose.harbor.yml

# Run a built-in LLM benchmark with
# your own tasks
harbor bench run

# Gimmick/Fun Area

# Argument scrambling, below commands are all the same as above
# Harbor doesn't care if it's "vllm model" or "model vllm", it'll
# figure it out.
harbor model vllm
harbor vllm model

harbor config get webui.name
harbor get config webui_name

harbor tabbyapi shell
harbor shell tabbyapi

# 50% gimmick, 50% useful
# Ask harbor about itself
harbor how to ping ollama container from the webui?

Harbor App Demo

https://github.com/user-attachments/assets/a5cd2ef1-3208-400a-8866-7abd85808503

In the demo, Harbor App is used to launch a default stack with Ollama and Open WebUI services. Later, SearXNG is also started, and WebUI can connect to it for the Web RAG right out of the box. After that, Harbor Boost is also started and connected to the WebUI automatically to induce more creative outputs. As a final step, Harbor config is adjusted in the App for the klmbr module in the Harbor Boost, which makes the output unparsable for the LLM (yet still undetstandable for humans).

Why?

• If you're comfortable with Docker and Linux administration - you likely don't need Harbor to manage your local LLM environment. However, while growing it - you're also likely to eventually arrive to a similar solution. I know this for a fact, since that's exactly how Harbor came to be.
• Harbor is not designed as a deployment solution, but rather as a helper for the local LLM development environment. It's a good starting point for experimenting with LLMs and related services.
• Workflow/setup centralisation - you can be sure where to find a specific config or service, logs, data and configuration files.
• Convenience factor - single CLI with a lot of services and features, accessible from anywhere on your host.

Supporters