Bodhi App is an open-source local LLM inference solution that takes a different and simpler approach. Instead of re-inventing the wheel, it leverages existing, tried and tested ecosystem and solutions:

## Technical Architecture:

- llama.cpp as inference engine

- Rust/Axum backend for type-safe API layer

- Tauri for multiplatform builds

- HuggingFace integration

- YAML based configurations and update at runtime (no restarts required)

- OpenAI/Ollama API compatibility layer

## Key Technical Decisions:

1. No proprietary model format - directly use of GGUF files from HuggingFace

2. Opt-in Authentication, provides RBAC for team access

3. API design with proper authentication/authorization

4. Built-in Swagger UI with complete OpenAPI specs

5. Built-in User guide

# What Sets It Apart:

Designed with non-technical users in mind. So it comes a basic Web-based user interface, allowing users to get started quickly with their first AI-assistant conversation.

## Setup Wizard:

- App displays a setup wizard when run for first time

- Allows user to download popular models in a user friendly way

## Built-in Chat UI:

- Ships with a complete Chat UI

- Chat UI is simple enough for non-technical users to get started with their first AI-conversation

- Adapts to power users by providing complete control over request settings

- Supports realtime streaming response, markdown rendering, code rendering with syntax highlights

- Displays chat stats, request tokens, response tokens, token speed

- Allow copying of the AI-response etc.

## Built-in UI for Model + App Management + API access:

- Manage complete Model lifecycle from the UI

- Downloading models, deleting models

- Configuring models, request + inference server configurations using Model Alias yaml files

- Allows configuring for parallel processing of requests

- Configuring App Settings - chosing betwen CPU/GPU, server idle time etc.

- API tokens for authenticated/authorized access to APIs by 3rd party

## Tech for UI:

- Uses Nextjs, Tailwindcss, Shadcn to build powerful, responsive and user friendly UI

- Supports Dark/Light mode

- Exported using config `output: "export"` to export the entire frontend as static html + javascript

- Served by the backend as static asset

- Thus no packaged nodejs server, reducing app size, complexity and compute

# Links

Try it out: https://getbodhi.app/

Source: https://github.com/BodhiSearch/BodhiApp

Looking forward to technical feedback and discussions.