Add NexusAI: universal MCP service (TTS/STT/image/video/music/web-search)
Monorepo-lite with shared @nexusai/core and @nexusai/mcp-server (stdio). Provider-agnostic tools with explicit provider selection; NordRouter adapter (media generate/poll/download, estimate, upload, models) and search via perplexity/sonar. Local STT via faster-whisper uv sidecar. SQLite journal of every generation for usage stats and future admin. 10 MCP tools.
This commit is contained in:
OpenCode
@@ -0,0 +1,35 @@
|
||||
# --- NordRouter ---
|
||||
# API key from your NordRouter account (starts with sk-)
|
||||
NORDROUTER_API_KEY=sk-your-key-here
|
||||
NORDROUTER_BASE_URL=https://nordrouter.com
|
||||
|
||||
# --- NexusAI general ---
|
||||
NEXUS_DEFAULT_PROVIDER=nordrouter
|
||||
# Where generated media files and the SQLite journal are stored
|
||||
NEXUS_MEDIA_DIR=./media-output
|
||||
# Optional: custom SQLite path (defaults to <media dir>/nexus.sqlite)
|
||||
# NEXUS_DB_PATH=./media-output/nexus.sqlite
|
||||
# Max bytes for embedding image/audio directly in MCP results (default 10MB)
|
||||
NEXUS_EMBED_MAX_BYTES=10485760
|
||||
# Log level: debug | info | warn | error (logs go to stderr)
|
||||
NEXUS_LOG_LEVEL=info
|
||||
|
||||
# --- Local STT (faster-whisper via uv sidecar) ---
|
||||
# Launcher: "uv" (recommended, auto-installs deps) or a python interpreter path
|
||||
NEXUS_STT_PYTHON=uv
|
||||
# Whisper size (tiny|base|small|medium|large-v3) OR an absolute path to a
|
||||
# pre-downloaded CTranslate2 model directory
|
||||
NEXUS_STT_MODEL=small
|
||||
NEXUS_STT_DEVICE=auto # auto | cpu | cuda
|
||||
NEXUS_STT_COMPUTE=auto # auto | int8 | float16
|
||||
# Offline mode: require a locally present model (no downloads)
|
||||
# NEXUS_STT_LOCAL_ONLY=1
|
||||
# Cache dir for downloaded models
|
||||
# NEXUS_STT_DOWNLOAD_ROOT=./models
|
||||
# Optional custom sidecar path
|
||||
# NEXUS_STT_SIDECAR=/abs/path/to/sidecar/stt_server.py
|
||||
# HuggingFace mirror (for restricted networks)
|
||||
# HF_ENDPOINT=https://hf-mirror.com
|
||||
|
||||
# --- Proxy (optional) ---
|
||||
# NEXUS_HTTP_PROXY=http://proxy:8080
|
||||
@@ -0,0 +1,25 @@
|
||||
|
||||
# secrets
|
||||
api-sec.txt
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
|
||||
# node
|
||||
node_modules/
|
||||
**/dist/
|
||||
*.tsbuildinfo
|
||||
|
||||
# data / media
|
||||
media-output/
|
||||
*.sqlite
|
||||
*.sqlite-journal
|
||||
*.db
|
||||
|
||||
# python
|
||||
sidecar/.venv/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
|
||||
# os
|
||||
.DS_Store
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"$schema": "https://opencode.ai/config.json",
|
||||
"mcp": {
|
||||
"nexusai": {
|
||||
"type": "local",
|
||||
"command": [
|
||||
"node",
|
||||
"/home/user/moder/mcp/NexusAI/packages/mcp-server/dist/index.js"
|
||||
],
|
||||
"env": {
|
||||
"NORDROUTER_API_KEY": "sk-your-key-here",
|
||||
"NEXUS_MEDIA_DIR": "/home/user/moder/mcp/NexusAI/media-output",
|
||||
"NEXUS_STT_MODEL": "small"
|
||||
},
|
||||
"enabled": true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
# NexusAI — универсальный MCP-сервис для агентов
|
||||
|
||||
Единый MCP-сервер (TypeScript, stdio), дающий агентам провайдер-абстрагированный доступ к:
|
||||
|
||||
- **TTS** — озвучка текста
|
||||
- **STT** — распознавание речи (локально, faster-whisper)
|
||||
- **Изображения** — генерация и правка
|
||||
- **Видео** — text→video / image→video
|
||||
- **Музыка** — генерация треков
|
||||
- **Веб-поиск** — ответ с цитатами (search-augmented модели)
|
||||
|
||||
Первый провайдер — **NordRouter**. Архитектура расширяемая: RouterAI, GPTunnel, OpenRouter и облачные STT добавляются как новые адаптеры без изменения MCP-инструментов. Каждая операция логируется в SQLite (провайдер, модель, стоимость, статус, файл) — основа для будущей веб-админки.
|
||||
|
||||
## Структура
|
||||
|
||||
```
|
||||
NexusAI/
|
||||
├── packages/
|
||||
│ ├── core/ # провайдеры, media-storage, journal, типы, registry
|
||||
│ └── mcp-server/ # stdio MCP-сервер (10 инструментов)
|
||||
├── sidecar/ # faster-whisper STT (PEP 723, запуск через uv)
|
||||
├── .env.example
|
||||
└── .opencode/opencode.json
|
||||
```
|
||||
|
||||
## Установка
|
||||
|
||||
```bash
|
||||
cd NexusAI
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
> `better-sqlite3` собирается нативно. Если сборка прервалась: `npm rebuild better-sqlite3`.
|
||||
|
||||
Для STT нужен [`uv`](https://docs.astral.sh/uv/) и `ffmpeg`. Зависимости whisper `uv` ставит автоматически при первом запуске.
|
||||
|
||||
## Подключение к opencode
|
||||
|
||||
`.opencode/opencode.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcp": {
|
||||
"nexusai": {
|
||||
"type": "local",
|
||||
"command": ["node", "/абс/путь/NexusAI/packages/mcp-server/dist/index.js"],
|
||||
"env": {
|
||||
"NORDROUTER_API_KEY": "sk-...",
|
||||
"NEXUS_MEDIA_DIR": "/абс/путь/NexusAI/media-output",
|
||||
"NEXUS_STT_MODEL": "small"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Переменные окружения — см. `.env.example`.
|
||||
|
||||
## Инструменты
|
||||
|
||||
| Инструмент | Назначение |
|
||||
|---|---|
|
||||
| `nexus_tts` | текст → речь (`audio/elevenlabs-tts`, `audio/gemini-3.1-flash-tts`, …) |
|
||||
| `nexus_image` | генерация/правка изображений (`image/*`) |
|
||||
| `nexus_video` | видео из текста/фото (`video/*`) |
|
||||
| `nexus_music` | музыка (`music/*`) |
|
||||
| `nexus_stt` | речь → текст (локально) |
|
||||
| `nexus_web_search` | веб-поиск с цитатами (`perplexity/sonar*`, `search/*`) |
|
||||
| `nexus_media_models` | список моделей провайдера (id, тип, цена) |
|
||||
| `nexus_media_estimate` | оценка стоимости до запуска |
|
||||
| `nexus_media_upload` | загрузка исходника (фото/аудио/видео) |
|
||||
| `nexus_usage_stats` | траты и история из журнала |
|
||||
|
||||
### Единый формат вызова
|
||||
|
||||
```json
|
||||
{
|
||||
"provider": "nordrouter",
|
||||
"model": "audio/elevenlabs-tts",
|
||||
"input": { "text": "Привет, мир", "voice": "James · Husky, Engaging and Bold" },
|
||||
"output": { "save": true, "embed": false }
|
||||
}
|
||||
```
|
||||
|
||||
- `provider` опционален (default из `NEXUS_DEFAULT_PROVIDER`).
|
||||
- `input` пропускает любые модель-специфичные поля (`aspect_ratio`, `resolution`, `duration`, `sound`, …).
|
||||
- `output.save` — сохранять файл локально (по умолчанию да).
|
||||
- `output.embed` — встроить image/audio прямо в ответ MCP (с лимитом размера).
|
||||
|
||||
### Единый формат ответа
|
||||
|
||||
```json
|
||||
{
|
||||
"provider": "nordrouter",
|
||||
"capability": "tts",
|
||||
"model": "audio/elevenlabs-tts",
|
||||
"status": "done",
|
||||
"file_path": "/.../media-output/audio-...wav",
|
||||
"mime_type": "audio/wav",
|
||||
"result_url": "https://nordrouter.com/media/file/...",
|
||||
"result_url_2": null,
|
||||
"cost_usd": 0.0049
|
||||
}
|
||||
```
|
||||
|
||||
`result_url_2` заполняется для музыкальных моделей, отдающих два трека (Suno).
|
||||
|
||||
## Как это работает
|
||||
|
||||
- **Media** асинхронно: `POST /media/generate` → polling `GET /media/job/:id` → скачивание `result_url` в `NEXUS_MEDIA_DIR`. Таймаут зависит от типа (картинки — быстро, видео/музыка — до минут).
|
||||
- **Web search** через OpenAI-совместимый `/v1/chat/completions` с sonar/`search/*`; цитаты берутся из `message.annotations[].url_citation`.
|
||||
- **STT** локально: TS запускает `uv run sidecar/stt_server.py`; faster-whisper декодирует аудио (mp3/wav/ogg/m4a/webm) и возвращает `{text, language, segments}`.
|
||||
- **Журнал** SQLite (`nexus.sqlite` в media-dir) пишет каждую операцию; `nexus_usage_stats` агрегирует траты.
|
||||
- MCP работает по stdio — **все логи идут в stderr**.
|
||||
|
||||
## STT в ограниченных сетях
|
||||
|
||||
faster-whisper скачивает модель с HuggingFace. Если `huggingface.co` недоступен:
|
||||
|
||||
- задать зеркало: `HF_ENDPOINT=https://hf-mirror.com`
|
||||
- либо предзагрузить модель и указать путь: `NEXUS_STT_MODEL=/abs/model-dir` + `NEXUS_STT_LOCAL_ONLY=1`
|
||||
- либо кэш-директорию: `NEXUS_STT_DOWNLOAD_ROOT=./models`
|
||||
|
||||
## Расширение: новый провайдер
|
||||
|
||||
1. Реализовать `MediaProvider` / `SearchProvider` / `SttProvider` из `@nexusai/core`.
|
||||
2. Зарегистрировать в `packages/mcp-server/src/context.ts`.
|
||||
|
||||
Инструменты и схемы менять не нужно — они провайдер-агностичны.
|
||||
|
||||
## Roadmap
|
||||
|
||||
- Фаза 2: веб-админка (отдельный пакет `admin`, Hono/Fastify + лёгкий фронт) на общем `core`: галерея контента, дашборд трат, управление провайдерами.
|
||||
- Доп. провайдеры: RouterAI, GPTunnel, OpenRouter; облачный STT.
|
||||
Generated
+1563
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"name": "nexusai",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"description": "Universal MCP service for agents: TTS, STT, image/video/music generation and web search",
|
||||
"workspaces": [
|
||||
"packages/core",
|
||||
"packages/mcp-server"
|
||||
],
|
||||
"scripts": {
|
||||
"build": "npm run build --workspace @nexusai/core && npm run build --workspace @nexusai/mcp-server",
|
||||
"build:core": "npm run build --workspace @nexusai/core",
|
||||
"build:mcp": "npm run build --workspace @nexusai/mcp-server",
|
||||
"start": "node packages/mcp-server/dist/index.js",
|
||||
"dev": "npm run build && node packages/mcp-server/dist/index.js"
|
||||
},
|
||||
"license": "MIT"
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"name": "@nexusai/core",
|
||||
"version": "0.1.0",
|
||||
"description": "NexusAI shared core: providers, media storage, journal, types",
|
||||
"main": "dist/index.js",
|
||||
"types": "dist/index.d.ts",
|
||||
"type": "commonjs",
|
||||
"scripts": {
|
||||
"build": "tsc",
|
||||
"dev": "tsc --watch"
|
||||
},
|
||||
"dependencies": {
|
||||
"better-sqlite3": "^11.8.0",
|
||||
"undici": "^6.27.0",
|
||||
"zod": "^3.25.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/better-sqlite3": "^7.6.0",
|
||||
"@types/node": "^22.15.0",
|
||||
"typescript": "^5.8.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
|
||||
export interface NordRouterConfig {
|
||||
apiKey?: string;
|
||||
baseUrl: string;
|
||||
}
|
||||
|
||||
export interface SttConfig {
|
||||
/** Path to python interpreter or "uv" launcher command. */
|
||||
pythonCmd: string;
|
||||
sidecarPath?: string;
|
||||
model: string; // whisper size: tiny/base/small/medium/large-v3
|
||||
device: string; // cpu / cuda / auto
|
||||
computeType: string; // int8 / float16 / auto
|
||||
}
|
||||
|
||||
export interface NexusConfig {
|
||||
defaultProvider: string;
|
||||
mediaDir: string;
|
||||
dbPath: string;
|
||||
embedMaxBytes: number;
|
||||
nordrouter: NordRouterConfig;
|
||||
stt: SttConfig;
|
||||
}
|
||||
|
||||
function envInt(name: string, fallback: number): number {
|
||||
const raw = process.env[name];
|
||||
if (!raw) return fallback;
|
||||
const n = Number.parseInt(raw, 10);
|
||||
return Number.isFinite(n) ? n : fallback;
|
||||
}
|
||||
|
||||
export function loadConfig(): NexusConfig {
|
||||
const mediaDir =
|
||||
process.env.NEXUS_MEDIA_DIR || path.join(process.cwd(), 'media-output');
|
||||
const dbPath = process.env.NEXUS_DB_PATH || path.join(mediaDir, 'nexus.sqlite');
|
||||
|
||||
return {
|
||||
defaultProvider: process.env.NEXUS_DEFAULT_PROVIDER || 'nordrouter',
|
||||
mediaDir,
|
||||
dbPath,
|
||||
embedMaxBytes: envInt('NEXUS_EMBED_MAX_BYTES', 10 * 1024 * 1024),
|
||||
nordrouter: {
|
||||
apiKey: process.env.NORDROUTER_API_KEY,
|
||||
baseUrl: (process.env.NORDROUTER_BASE_URL || 'https://nordrouter.com').replace(/\/$/, ''),
|
||||
},
|
||||
stt: {
|
||||
pythonCmd: process.env.NEXUS_STT_PYTHON || 'uv',
|
||||
sidecarPath: process.env.NEXUS_STT_SIDECAR,
|
||||
model: process.env.NEXUS_STT_MODEL || 'small',
|
||||
device: process.env.NEXUS_STT_DEVICE || 'auto',
|
||||
computeType: process.env.NEXUS_STT_COMPUTE || 'auto',
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export const HOME_DIR = os.homedir();
|
||||
@@ -0,0 +1,141 @@
|
||||
import Database from 'better-sqlite3';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { createLogger } from '../shared/logger.js';
|
||||
|
||||
const log = createLogger('journal');
|
||||
|
||||
export interface GenerationRecord {
|
||||
id: string;
|
||||
provider: string;
|
||||
capability: string;
|
||||
model: string;
|
||||
inputSummary: string;
|
||||
status: 'done' | 'failed';
|
||||
costUsd: number | null;
|
||||
filePath: string | null;
|
||||
resultUrl: string | null;
|
||||
error: string | null;
|
||||
durationMs: number | null;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export interface UsageStats {
|
||||
totalCostUsd: number;
|
||||
totalCount: number;
|
||||
byProvider: Array<{ provider: string; count: number; costUsd: number }>;
|
||||
byCapability: Array<{ capability: string; count: number; costUsd: number }>;
|
||||
byDay: Array<{ day: string; count: number; costUsd: number }>;
|
||||
}
|
||||
|
||||
/**
|
||||
* SQLite-backed journal of every generation. Enables usage stats now and a
|
||||
* shared data source for the future admin package.
|
||||
*/
|
||||
export class Journal {
|
||||
private readonly db: Database.Database;
|
||||
|
||||
constructor(dbPath: string) {
|
||||
fs.mkdirSync(path.dirname(dbPath), { recursive: true });
|
||||
this.db = new Database(dbPath);
|
||||
this.db.pragma('journal_mode = WAL');
|
||||
this.init();
|
||||
}
|
||||
|
||||
private init(): void {
|
||||
this.db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS generations (
|
||||
id TEXT PRIMARY KEY,
|
||||
provider TEXT NOT NULL,
|
||||
capability TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
input_summary TEXT NOT NULL DEFAULT '',
|
||||
status TEXT NOT NULL,
|
||||
cost_usd REAL,
|
||||
file_path TEXT,
|
||||
result_url TEXT,
|
||||
error TEXT,
|
||||
duration_ms INTEGER,
|
||||
created_at TEXT NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_gen_created ON generations(created_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_gen_provider ON generations(provider);
|
||||
CREATE INDEX IF NOT EXISTS idx_gen_capability ON generations(capability);
|
||||
`);
|
||||
}
|
||||
|
||||
record(rec: GenerationRecord): void {
|
||||
try {
|
||||
this.db
|
||||
.prepare(
|
||||
`INSERT INTO generations
|
||||
(id, provider, capability, model, input_summary, status, cost_usd, file_path, result_url, error, duration_ms, created_at)
|
||||
VALUES (@id, @provider, @capability, @model, @inputSummary, @status, @costUsd, @filePath, @resultUrl, @error, @durationMs, @createdAt)`
|
||||
)
|
||||
.run(rec);
|
||||
} catch (err) {
|
||||
// Journaling must never break a successful generation.
|
||||
log.warn('failed to record generation', err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
}
|
||||
|
||||
list(limit = 50, offset = 0, capability?: string): GenerationRecord[] {
|
||||
const where = capability ? 'WHERE capability = ?' : '';
|
||||
const params = capability ? [capability, limit, offset] : [limit, offset];
|
||||
const rows = this.db
|
||||
.prepare(
|
||||
`SELECT id, provider, capability, model, input_summary as inputSummary, status,
|
||||
cost_usd as costUsd, file_path as filePath, result_url as resultUrl,
|
||||
error, duration_ms as durationMs, created_at as createdAt
|
||||
FROM generations ${where}
|
||||
ORDER BY created_at DESC LIMIT ? OFFSET ?`
|
||||
)
|
||||
.all(...params);
|
||||
return rows as GenerationRecord[];
|
||||
}
|
||||
|
||||
stats(sinceDays?: number): UsageStats {
|
||||
const filter = sinceDays
|
||||
? `WHERE created_at >= datetime('now', '-${Math.max(1, Math.floor(sinceDays))} days')`
|
||||
: '';
|
||||
|
||||
const total = this.db
|
||||
.prepare(
|
||||
`SELECT COALESCE(SUM(cost_usd),0) as cost, COUNT(*) as cnt FROM generations ${filter}`
|
||||
)
|
||||
.get() as { cost: number; cnt: number };
|
||||
|
||||
const byProvider = this.db
|
||||
.prepare(
|
||||
`SELECT provider, COUNT(*) as count, COALESCE(SUM(cost_usd),0) as costUsd
|
||||
FROM generations ${filter} GROUP BY provider ORDER BY costUsd DESC`
|
||||
)
|
||||
.all() as UsageStats['byProvider'];
|
||||
|
||||
const byCapability = this.db
|
||||
.prepare(
|
||||
`SELECT capability, COUNT(*) as count, COALESCE(SUM(cost_usd),0) as costUsd
|
||||
FROM generations ${filter} GROUP BY capability ORDER BY costUsd DESC`
|
||||
)
|
||||
.all() as UsageStats['byCapability'];
|
||||
|
||||
const byDay = this.db
|
||||
.prepare(
|
||||
`SELECT substr(created_at,1,10) as day, COUNT(*) as count, COALESCE(SUM(cost_usd),0) as costUsd
|
||||
FROM generations ${filter} GROUP BY day ORDER BY day DESC LIMIT 30`
|
||||
)
|
||||
.all() as UsageStats['byDay'];
|
||||
|
||||
return {
|
||||
totalCostUsd: total.cost,
|
||||
totalCount: total.cnt,
|
||||
byProvider,
|
||||
byCapability,
|
||||
byDay,
|
||||
};
|
||||
}
|
||||
|
||||
close(): void {
|
||||
this.db.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
export * from './config.js';
|
||||
export * from './shared/errors.js';
|
||||
export * from './shared/logger.js';
|
||||
export * from './shared/http.js';
|
||||
export * from './shared/polling.js';
|
||||
export * from './db/journal.js';
|
||||
export * from './media/storage.js';
|
||||
export * from './media/content.js';
|
||||
export * from './providers/types.js';
|
||||
export * from './providers/registry.js';
|
||||
export * from './providers/nordrouter/index.js';
|
||||
export * from './providers/local-stt/whisper.js';
|
||||
@@ -0,0 +1,36 @@
|
||||
import * as fs from 'fs';
|
||||
import { extToMime } from './storage.js';
|
||||
import * as path from 'path';
|
||||
|
||||
/**
|
||||
* MCP content block for embedding media directly in a tool result.
|
||||
* The MCP SDK accepts `image` and `audio` blocks with base64 data + mimeType.
|
||||
*/
|
||||
export type EmbeddedContent =
|
||||
| { type: 'image'; data: string; mimeType: string }
|
||||
| { type: 'audio'; data: string; mimeType: string };
|
||||
|
||||
/**
|
||||
* Build an embedded content block from a local file, respecting a byte cap.
|
||||
* Returns null when the file is missing, too large, or not embeddable.
|
||||
*/
|
||||
export function buildEmbeddedContent(
|
||||
filePath: string,
|
||||
mimeType: string | undefined,
|
||||
maxBytes: number
|
||||
): EmbeddedContent | null {
|
||||
try {
|
||||
const st = fs.statSync(filePath);
|
||||
if (!st.isFile() || st.size > maxBytes) return null;
|
||||
|
||||
const mime = mimeType || extToMime(path.extname(filePath));
|
||||
const buf = fs.readFileSync(filePath);
|
||||
const data = buf.toString('base64');
|
||||
|
||||
if (mime.startsWith('image/')) return { type: 'image', data, mimeType: mime };
|
||||
if (mime.startsWith('audio/')) return { type: 'audio', data, mimeType: mime };
|
||||
return null; // video/other are not embeddable as MCP content
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { fetch, ProxyAgent } from 'undici';
|
||||
import { NexusError } from '../shared/errors.js';
|
||||
import { createLogger } from '../shared/logger.js';
|
||||
|
||||
const log = createLogger('storage');
|
||||
|
||||
const EXT_BY_MIME: Record<string, string> = {
|
||||
'audio/mpeg': 'mp3',
|
||||
'audio/mp3': 'mp3',
|
||||
'audio/wav': 'wav',
|
||||
'audio/x-wav': 'wav',
|
||||
'audio/ogg': 'ogg',
|
||||
'image/png': 'png',
|
||||
'image/jpeg': 'jpg',
|
||||
'image/webp': 'webp',
|
||||
'video/mp4': 'mp4',
|
||||
'application/octet-stream': 'bin',
|
||||
};
|
||||
|
||||
const MIME_BY_EXT: Record<string, string> = {
|
||||
mp3: 'audio/mpeg',
|
||||
wav: 'audio/wav',
|
||||
ogg: 'audio/ogg',
|
||||
png: 'image/png',
|
||||
jpg: 'image/jpeg',
|
||||
jpeg: 'image/jpeg',
|
||||
webp: 'image/webp',
|
||||
mp4: 'video/mp4',
|
||||
};
|
||||
|
||||
export function mimeToExt(mime?: string): string {
|
||||
if (!mime) return 'bin';
|
||||
return EXT_BY_MIME[mime.toLowerCase()] || 'bin';
|
||||
}
|
||||
|
||||
export function extToMime(ext: string): string {
|
||||
return MIME_BY_EXT[ext.toLowerCase().replace(/^\./, '')] || 'application/octet-stream';
|
||||
}
|
||||
|
||||
function proxyDispatcher(): ProxyAgent | undefined {
|
||||
const url =
|
||||
process.env.NEXUS_HTTP_PROXY || process.env.HTTPS_PROXY || process.env.HTTP_PROXY;
|
||||
return url ? new ProxyAgent(url) : undefined;
|
||||
}
|
||||
|
||||
export interface SaveResult {
|
||||
filePath: string;
|
||||
mimeType: string;
|
||||
bytes: number;
|
||||
}
|
||||
|
||||
export class MediaStorage {
|
||||
constructor(private readonly baseDir: string) {
|
||||
fs.mkdirSync(baseDir, { recursive: true });
|
||||
}
|
||||
|
||||
private uniqueName(prefix: string, ext: string): string {
|
||||
const stamp = new Date().toISOString().replace(/[:.]/g, '-');
|
||||
const rand = Math.random().toString(36).slice(2, 8);
|
||||
return `${prefix}-${stamp}-${rand}.${ext}`;
|
||||
}
|
||||
|
||||
/** Download a remote URL to a local file. */
|
||||
async downloadUrl(url: string, prefix: string): Promise<SaveResult> {
|
||||
const dispatcher = proxyDispatcher();
|
||||
const res = await fetch(url, dispatcher ? { dispatcher } : {});
|
||||
if (!res.ok) {
|
||||
throw new NexusError('IO', `Failed to download result (${res.status}) from ${url}`);
|
||||
}
|
||||
const mimeType = res.headers.get('content-type')?.split(';')[0]?.trim() || 'application/octet-stream';
|
||||
const urlExt = path.extname(new URL(url).pathname).replace(/^\./, '');
|
||||
const ext = urlExt || mimeToExt(mimeType);
|
||||
const buf = Buffer.from(await res.arrayBuffer());
|
||||
const filePath = path.join(this.baseDir, this.uniqueName(prefix, ext));
|
||||
fs.writeFileSync(filePath, buf);
|
||||
log.debug('saved file', { filePath, bytes: buf.length });
|
||||
return { filePath, mimeType: mimeType === 'application/octet-stream' ? extToMime(ext) : mimeType, bytes: buf.length };
|
||||
}
|
||||
|
||||
/** Save a base64 (optionally data-URL) payload to disk. */
|
||||
saveBase64(data: string, prefix: string, mimeHint?: string): SaveResult {
|
||||
let mimeType = mimeHint || 'application/octet-stream';
|
||||
let b64 = data;
|
||||
const dataUrlMatch = /^data:([^;]+);base64,(.*)$/s.exec(data);
|
||||
if (dataUrlMatch) {
|
||||
mimeType = dataUrlMatch[1];
|
||||
b64 = dataUrlMatch[2];
|
||||
}
|
||||
const buf = Buffer.from(b64, 'base64');
|
||||
const ext = mimeToExt(mimeType);
|
||||
const filePath = path.join(this.baseDir, this.uniqueName(prefix, ext));
|
||||
fs.writeFileSync(filePath, buf);
|
||||
return { filePath, mimeType, bytes: buf.length };
|
||||
}
|
||||
|
||||
/** Remove files older than ttlDays. Best-effort. */
|
||||
cleanup(ttlDays: number): number {
|
||||
let removed = 0;
|
||||
const cutoff = Date.now() - ttlDays * 24 * 60 * 60 * 1000;
|
||||
try {
|
||||
for (const name of fs.readdirSync(this.baseDir)) {
|
||||
const full = path.join(this.baseDir, name);
|
||||
try {
|
||||
const st = fs.statSync(full);
|
||||
if (st.isFile() && st.mtimeMs < cutoff && !name.endsWith('.sqlite') && !name.includes('nexus.sqlite')) {
|
||||
fs.unlinkSync(full);
|
||||
removed += 1;
|
||||
}
|
||||
} catch {
|
||||
/* ignore per-file errors */
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
return removed;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
import { spawn } from 'child_process';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { fetch, ProxyAgent } from 'undici';
|
||||
import type { SttConfig } from '../../config.js';
|
||||
import { NexusError } from '../../shared/errors.js';
|
||||
import { createLogger } from '../../shared/logger.js';
|
||||
import type { SttInput, SttProvider, SttResult } from '../types.js';
|
||||
|
||||
const log = createLogger('local-stt');
|
||||
|
||||
interface SidecarResponse {
|
||||
ok: boolean;
|
||||
text?: string;
|
||||
language?: string;
|
||||
duration?: number;
|
||||
segments?: Array<{ start: number; end: number; text: string }>;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Local speech-to-text via a faster-whisper Python sidecar launched with `uv`.
|
||||
* The sidecar reads a single JSON request on argv and prints a JSON response.
|
||||
* Architecture allows swapping in cloud STT adapters later without MCP changes.
|
||||
*/
|
||||
export class LocalWhisperProvider implements SttProvider {
|
||||
readonly name = 'local';
|
||||
|
||||
constructor(private readonly cfg: SttConfig) {}
|
||||
|
||||
private resolveSidecar(): string {
|
||||
if (this.cfg.sidecarPath) return this.cfg.sidecarPath;
|
||||
// packages/core/dist/providers/local-stt/ -> repo/sidecar/stt_server.py
|
||||
const guess = path.resolve(__dirname, '../../../../../sidecar/stt_server.py');
|
||||
return guess;
|
||||
}
|
||||
|
||||
private async materializeAudio(input: SttInput): Promise<{ file: string; cleanup: boolean }> {
|
||||
if (input.audioPath) {
|
||||
if (!fs.existsSync(input.audioPath)) {
|
||||
throw new NexusError('IO', `Audio file not found: ${input.audioPath}`);
|
||||
}
|
||||
return { file: input.audioPath, cleanup: false };
|
||||
}
|
||||
if (input.audioBase64) {
|
||||
const m = /^data:([^;]+);base64,(.*)$/s.exec(input.audioBase64);
|
||||
const b64 = m ? m[2] : input.audioBase64;
|
||||
const tmp = path.join(os.tmpdir(), `nexus-stt-${Date.now()}-${Math.random().toString(36).slice(2)}.audio`);
|
||||
fs.writeFileSync(tmp, Buffer.from(b64, 'base64'));
|
||||
return { file: tmp, cleanup: true };
|
||||
}
|
||||
if (input.audioUrl) {
|
||||
const proxy = process.env.NEXUS_HTTP_PROXY || process.env.HTTPS_PROXY || process.env.HTTP_PROXY;
|
||||
const res = await fetch(input.audioUrl, proxy ? { dispatcher: new ProxyAgent(proxy) } : {});
|
||||
if (!res.ok) throw new NexusError('IO', `Failed to download audio (${res.status})`);
|
||||
const tmp = path.join(os.tmpdir(), `nexus-stt-${Date.now()}-${Math.random().toString(36).slice(2)}.audio`);
|
||||
fs.writeFileSync(tmp, Buffer.from(await res.arrayBuffer()));
|
||||
return { file: tmp, cleanup: true };
|
||||
}
|
||||
throw new NexusError('BAD_REQUEST', 'Provide one of: audioPath, audioUrl, audioBase64');
|
||||
}
|
||||
|
||||
async transcribe(input: SttInput): Promise<SttResult> {
|
||||
const { file, cleanup } = await this.materializeAudio(input);
|
||||
const sidecar = this.resolveSidecar();
|
||||
|
||||
const request = {
|
||||
audio: file,
|
||||
model: input.model || this.cfg.model,
|
||||
device: this.cfg.device,
|
||||
compute_type: this.cfg.computeType,
|
||||
language: input.language,
|
||||
};
|
||||
|
||||
// With uv: `uv run --with faster-whisper python sidecar/stt_server.py <json>`
|
||||
const usingUv = /(^|\/)uv$/.test(this.cfg.pythonCmd) || this.cfg.pythonCmd === 'uv';
|
||||
const sidecarDir = path.dirname(sidecar);
|
||||
const args = usingUv
|
||||
? ['run', sidecar, JSON.stringify(request)]
|
||||
: [sidecar, JSON.stringify(request)];
|
||||
|
||||
log.debug('spawning stt sidecar', { cmd: this.cfg.pythonCmd, model: request.model });
|
||||
|
||||
try {
|
||||
const response = await this.runSidecar(this.cfg.pythonCmd, args, sidecarDir);
|
||||
if (!response.ok) {
|
||||
throw new NexusError('PROVIDER', response.error || 'STT sidecar failed');
|
||||
}
|
||||
return {
|
||||
provider: this.name,
|
||||
text: response.text || '',
|
||||
language: response.language,
|
||||
durationSec: response.duration,
|
||||
segments: response.segments,
|
||||
};
|
||||
} finally {
|
||||
if (cleanup) {
|
||||
try {
|
||||
fs.unlinkSync(file);
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private runSidecar(cmd: string, args: string[], cwd: string): Promise<SidecarResponse> {
|
||||
return new Promise((resolve, reject) => {
|
||||
// Forward HF mirror / offline settings so the sidecar can locate models
|
||||
// in restricted networks.
|
||||
const env = {
|
||||
...process.env,
|
||||
...(process.env.HF_ENDPOINT ? { HF_ENDPOINT: process.env.HF_ENDPOINT } : {}),
|
||||
};
|
||||
const child = spawn(cmd, args, { cwd, env });
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
child.stdout.on('data', (d) => (stdout += d.toString()));
|
||||
child.stderr.on('data', (d) => (stderr += d.toString()));
|
||||
child.on('error', (err) =>
|
||||
reject(new NexusError('CONFIG', `Failed to launch STT sidecar (${cmd}): ${err.message}`))
|
||||
);
|
||||
child.on('close', (code) => {
|
||||
if (code !== 0 && !stdout.trim()) {
|
||||
reject(new NexusError('PROVIDER', `STT sidecar exited ${code}: ${stderr.slice(-500)}`));
|
||||
return;
|
||||
}
|
||||
// The sidecar prints exactly one JSON line on stdout.
|
||||
const line = stdout.trim().split('\n').filter(Boolean).pop() || '';
|
||||
try {
|
||||
resolve(JSON.parse(line) as SidecarResponse);
|
||||
} catch {
|
||||
reject(new NexusError('PROVIDER', `Invalid STT sidecar output: ${stdout.slice(-500)} | stderr: ${stderr.slice(-300)}`));
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import { HttpClient, RateLimiter } from '../../shared/http.js';
|
||||
import { NexusError } from '../../shared/errors.js';
|
||||
|
||||
/**
|
||||
* Low-level NordRouter HTTP client. Shared by media, search, upload and models.
|
||||
* Auth: `Authorization: Bearer sk-...`. Base: https://nordrouter.com
|
||||
*/
|
||||
export class NordRouterClient {
|
||||
readonly http: HttpClient;
|
||||
readonly baseUrl: string;
|
||||
|
||||
constructor(apiKey: string | undefined, baseUrl: string) {
|
||||
if (!apiKey) {
|
||||
throw new NexusError('CONFIG', 'NORDROUTER_API_KEY is not set');
|
||||
}
|
||||
this.baseUrl = baseUrl;
|
||||
this.http = new HttpClient({
|
||||
baseUrl,
|
||||
defaultHeaders: { Authorization: `Bearer ${apiKey}` },
|
||||
timeoutMs: 60_000,
|
||||
maxRetries: 2,
|
||||
// NordRouter rate-limits per key; keep it modest and let 429 retries handle bursts.
|
||||
rateLimiter: new RateLimiter(30, 10_000),
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import type { NordRouterConfig } from '../../config.js';
|
||||
import type { MediaStorage } from '../../media/storage.js';
|
||||
import { NordRouterClient } from './client.js';
|
||||
import { NordRouterMediaProvider } from './media.js';
|
||||
import { NordRouterSearchProvider } from './search.js';
|
||||
|
||||
export { NordRouterClient } from './client.js';
|
||||
export { NordRouterMediaProvider } from './media.js';
|
||||
export { NordRouterSearchProvider } from './search.js';
|
||||
|
||||
export interface NordRouterFactoryOptions {
|
||||
config: NordRouterConfig;
|
||||
storage: MediaStorage;
|
||||
save: boolean;
|
||||
}
|
||||
|
||||
export function createNordRouterProviders(opts: NordRouterFactoryOptions): {
|
||||
media: NordRouterMediaProvider;
|
||||
search: NordRouterSearchProvider;
|
||||
} {
|
||||
const client = new NordRouterClient(opts.config.apiKey, opts.config.baseUrl);
|
||||
return {
|
||||
media: new NordRouterMediaProvider(client, { save: opts.save, storage: opts.storage }),
|
||||
search: new NordRouterSearchProvider(client),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
import * as fs from 'fs';
|
||||
import { NexusError } from '../../shared/errors.js';
|
||||
import { createLogger } from '../../shared/logger.js';
|
||||
import { pollUntilDone } from '../../shared/polling.js';
|
||||
import { MediaStorage } from '../../media/storage.js';
|
||||
import type {
|
||||
Capability,
|
||||
EstimateInput,
|
||||
MediaProvider,
|
||||
MediaRequest,
|
||||
MediaResult,
|
||||
ModelInfo,
|
||||
UploadInput,
|
||||
UploadResult,
|
||||
} from '../types.js';
|
||||
import { NordRouterClient } from './client.js';
|
||||
|
||||
const log = createLogger('nordrouter:media');
|
||||
|
||||
interface JobCreateResponse {
|
||||
id: string;
|
||||
model: string;
|
||||
status: string;
|
||||
poll?: string;
|
||||
webhook?: string;
|
||||
}
|
||||
|
||||
interface JobStatusResponse {
|
||||
id: string;
|
||||
status: 'processing' | 'done' | 'failed';
|
||||
result_url: string | null;
|
||||
result_url_2?: string | null;
|
||||
results?: Array<{ index: number; name: string; url: string }>;
|
||||
cost_usd?: number | null;
|
||||
error?: string | null;
|
||||
}
|
||||
|
||||
interface ModelsResponse {
|
||||
models: Array<{ id: string; type?: string; label?: string; est_usd?: number; fields?: unknown }>;
|
||||
retention_days?: number;
|
||||
}
|
||||
|
||||
/** Timeout per capability — images are quick, video/music take longer. */
|
||||
const TIMEOUT_MS: Record<Capability, number> = {
|
||||
image: 3 * 60 * 1000,
|
||||
tts: 3 * 60 * 1000,
|
||||
music: 8 * 60 * 1000,
|
||||
video: 15 * 60 * 1000,
|
||||
stt: 3 * 60 * 1000,
|
||||
search: 60 * 1000,
|
||||
};
|
||||
|
||||
const PREFIX: Record<Capability, string> = {
|
||||
image: 'image',
|
||||
tts: 'audio',
|
||||
music: 'music',
|
||||
video: 'video',
|
||||
stt: 'stt',
|
||||
search: 'search',
|
||||
};
|
||||
|
||||
export interface NordRouterMediaOptions {
|
||||
save: boolean;
|
||||
storage?: MediaStorage;
|
||||
pollIntervalMs?: number;
|
||||
}
|
||||
|
||||
export class NordRouterMediaProvider implements MediaProvider {
|
||||
readonly name = 'nordrouter';
|
||||
readonly capabilities: Capability[] = ['tts', 'image', 'video', 'music'];
|
||||
|
||||
private modelsCache?: { at: number; models: ModelInfo[] };
|
||||
private readonly modelsTtlMs = 10 * 60 * 1000;
|
||||
|
||||
constructor(
|
||||
private readonly client: NordRouterClient,
|
||||
private readonly defaults: NordRouterMediaOptions
|
||||
) {}
|
||||
|
||||
async generate(capability: Capability, req: MediaRequest): Promise<MediaResult> {
|
||||
const created = await this.client.http.requestJson<JobCreateResponse>({
|
||||
method: 'POST',
|
||||
path: '/media/generate',
|
||||
body: { model: req.model, input: req.input },
|
||||
});
|
||||
|
||||
if (!created.id) {
|
||||
throw new NexusError('PROVIDER', 'NordRouter did not return a job id', created);
|
||||
}
|
||||
log.debug('job created', { id: created.id, model: req.model });
|
||||
|
||||
const final = await pollUntilDone<JobStatusResponse>({
|
||||
intervalMs: this.defaults.pollIntervalMs ?? 3000,
|
||||
timeoutMs: TIMEOUT_MS[capability],
|
||||
poll: async () => {
|
||||
const s = await this.client.http.requestJson<JobStatusResponse>({
|
||||
method: 'GET',
|
||||
path: `/media/job/${encodeURIComponent(created.id)}`,
|
||||
});
|
||||
if (s.status === 'failed') return { done: false, failed: true, error: s.error || 'job failed' };
|
||||
if (s.status === 'done') return { done: true, value: s };
|
||||
return { done: false };
|
||||
},
|
||||
});
|
||||
|
||||
const result: MediaResult = {
|
||||
provider: this.name,
|
||||
capability,
|
||||
model: req.model,
|
||||
status: 'done',
|
||||
resultUrl: final.result_url ?? undefined,
|
||||
resultUrl2: final.result_url_2 ?? undefined,
|
||||
costUsd: final.cost_usd ?? undefined,
|
||||
raw: final,
|
||||
};
|
||||
|
||||
if (this.defaults.save && this.defaults.storage && final.result_url) {
|
||||
const saved = await this.defaults.storage.downloadUrl(final.result_url, PREFIX[capability]);
|
||||
result.filePath = saved.filePath;
|
||||
result.mimeType = saved.mimeType;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
async estimate(req: EstimateInput): Promise<{ usd: number }> {
|
||||
return this.client.http.requestJson<{ usd: number }>({
|
||||
method: 'POST',
|
||||
path: '/media/estimate',
|
||||
body: { model: req.model, input: req.input },
|
||||
});
|
||||
}
|
||||
|
||||
async upload(input: UploadInput): Promise<UploadResult> {
|
||||
if (!fs.existsSync(input.path)) {
|
||||
throw new NexusError('IO', `Upload file not found: ${input.path}`);
|
||||
}
|
||||
const buf = fs.readFileSync(input.path);
|
||||
const mime = input.mimeType || 'application/octet-stream';
|
||||
const name = input.name || input.path.split('/').pop() || 'file';
|
||||
const dataUrl = `data:${mime};base64,${buf.toString('base64')}`;
|
||||
const res = await this.client.http.requestJson<{ url?: string }>({
|
||||
method: 'POST',
|
||||
path: '/media/upload',
|
||||
body: { data: dataUrl, name },
|
||||
});
|
||||
return { ref: res, url: res.url };
|
||||
}
|
||||
|
||||
async listModels(): Promise<ModelInfo[]> {
|
||||
const now = Date.now();
|
||||
if (this.modelsCache && now - this.modelsCache.at < this.modelsTtlMs) {
|
||||
return this.modelsCache.models;
|
||||
}
|
||||
const res = await this.client.http.requestJson<ModelsResponse>({
|
||||
method: 'GET',
|
||||
path: '/media/models',
|
||||
});
|
||||
const models: ModelInfo[] = (res.models || []).map((m) => ({
|
||||
id: m.id,
|
||||
type: m.type,
|
||||
label: m.label,
|
||||
estUsd: m.est_usd,
|
||||
fields: m.fields,
|
||||
}));
|
||||
this.modelsCache = { at: now, models };
|
||||
return models;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
import { createLogger } from '../../shared/logger.js';
|
||||
import type { SearchInput, SearchProvider, SearchResult } from '../types.js';
|
||||
import { NordRouterClient } from './client.js';
|
||||
|
||||
const log = createLogger('nordrouter:search');
|
||||
|
||||
interface ChatResponse {
|
||||
choices: Array<{
|
||||
message: {
|
||||
content: string;
|
||||
annotations?: Array<{
|
||||
type: string;
|
||||
url_citation?: { url: string; title?: string };
|
||||
}>;
|
||||
};
|
||||
}>;
|
||||
usage?: { total_cost?: number };
|
||||
}
|
||||
|
||||
const DEFAULT_MODEL = 'perplexity/sonar-pro';
|
||||
|
||||
/**
|
||||
* Web search via NordRouter's OpenAI-compatible chat endpoint using
|
||||
* Perplexity sonar / search-augmented models. Citations arrive in
|
||||
* `message.annotations[].url_citation`.
|
||||
*/
|
||||
export class NordRouterSearchProvider implements SearchProvider {
|
||||
readonly name = 'nordrouter';
|
||||
|
||||
constructor(private readonly client: NordRouterClient) {}
|
||||
|
||||
async search(input: SearchInput): Promise<SearchResult> {
|
||||
const model = input.model || DEFAULT_MODEL;
|
||||
let query = input.query;
|
||||
if (input.recency) {
|
||||
query += `\n\n(Focus on information from the last ${input.recency}.)`;
|
||||
}
|
||||
|
||||
const res = await this.client.http.requestJson<ChatResponse>({
|
||||
method: 'POST',
|
||||
path: '/v1/chat/completions',
|
||||
body: { model, messages: [{ role: 'user', content: query }] },
|
||||
timeoutMs: 90_000,
|
||||
});
|
||||
|
||||
const choice = res.choices?.[0]?.message;
|
||||
const answer = choice?.content ?? '';
|
||||
const seen = new Set<string>();
|
||||
const citations: SearchResult['citations'] = [];
|
||||
let idx = 1;
|
||||
for (const a of choice?.annotations ?? []) {
|
||||
const c = a.url_citation;
|
||||
if (c?.url && !seen.has(c.url)) {
|
||||
seen.add(c.url);
|
||||
citations.push({ index: idx++, url: c.url, title: c.title });
|
||||
}
|
||||
}
|
||||
if (input.maxResults && citations.length > input.maxResults) {
|
||||
citations.length = input.maxResults;
|
||||
}
|
||||
|
||||
log.debug('search done', { model, citations: citations.length });
|
||||
return {
|
||||
provider: this.name,
|
||||
model,
|
||||
answer,
|
||||
citations,
|
||||
costUsd: res.usage?.total_cost,
|
||||
raw: res,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
import { NexusError } from '../shared/errors.js';
|
||||
import type { Capability, MediaProvider, SearchProvider, SttProvider } from './types.js';
|
||||
|
||||
/**
|
||||
* Registry maps a provider name + capability to the concrete adapter.
|
||||
* Adding RouterAI / GPTunnel / OpenRouter / cloud STT means registering a new
|
||||
* adapter here — no changes needed in the MCP tool layer.
|
||||
*/
|
||||
export class ProviderRegistry {
|
||||
private media = new Map<string, MediaProvider>();
|
||||
private search = new Map<string, SearchProvider>();
|
||||
private stt = new Map<string, SttProvider>();
|
||||
|
||||
registerMedia(provider: MediaProvider): void {
|
||||
this.media.set(provider.name, provider);
|
||||
}
|
||||
registerSearch(provider: SearchProvider): void {
|
||||
this.search.set(provider.name, provider);
|
||||
}
|
||||
registerStt(provider: SttProvider): void {
|
||||
this.stt.set(provider.name, provider);
|
||||
}
|
||||
|
||||
getMedia(name: string, capability: Capability): MediaProvider {
|
||||
const p = this.media.get(name);
|
||||
if (!p) {
|
||||
throw new NexusError(
|
||||
'UNSUPPORTED',
|
||||
`Unknown media provider "${name}". Available: ${[...this.media.keys()].join(', ') || 'none'}`
|
||||
);
|
||||
}
|
||||
if (!p.capabilities.includes(capability)) {
|
||||
throw new NexusError(
|
||||
'UNSUPPORTED',
|
||||
`Provider "${name}" does not support capability "${capability}"`
|
||||
);
|
||||
}
|
||||
return p;
|
||||
}
|
||||
|
||||
getSearch(name: string): SearchProvider {
|
||||
const p = this.search.get(name);
|
||||
if (!p) {
|
||||
throw new NexusError(
|
||||
'UNSUPPORTED',
|
||||
`Unknown search provider "${name}". Available: ${[...this.search.keys()].join(', ') || 'none'}`
|
||||
);
|
||||
}
|
||||
return p;
|
||||
}
|
||||
|
||||
getStt(name: string): SttProvider {
|
||||
const p = this.stt.get(name);
|
||||
if (!p) {
|
||||
throw new NexusError(
|
||||
'UNSUPPORTED',
|
||||
`Unknown STT provider "${name}". Available: ${[...this.stt.keys()].join(', ') || 'none'}`
|
||||
);
|
||||
}
|
||||
return p;
|
||||
}
|
||||
|
||||
listMediaProviders(): string[] {
|
||||
return [...this.media.keys()];
|
||||
}
|
||||
listSearchProviders(): string[] {
|
||||
return [...this.search.keys()];
|
||||
}
|
||||
listSttProviders(): string[] {
|
||||
return [...this.stt.keys()];
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
/**
|
||||
* Provider abstraction. Adapters (NordRouter, RouterAI, GPTunnel, OpenRouter,
|
||||
* local STT) implement one or more of these capabilities. The MCP layer stays
|
||||
* provider-agnostic and selects an adapter by an explicit `provider` argument.
|
||||
*/
|
||||
|
||||
export type Capability = 'tts' | 'stt' | 'image' | 'video' | 'music' | 'search';
|
||||
|
||||
export interface MediaRequest {
|
||||
model: string;
|
||||
input: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface MediaResult {
|
||||
provider: string;
|
||||
capability: Capability;
|
||||
model: string;
|
||||
status: 'done';
|
||||
/** Local saved file path (when storage enabled). */
|
||||
filePath?: string;
|
||||
mimeType?: string;
|
||||
/** Remote URL from provider (may expire). */
|
||||
resultUrl?: string;
|
||||
/** Second track for music providers (Suno). */
|
||||
resultUrl2?: string;
|
||||
costUsd?: number;
|
||||
raw?: unknown;
|
||||
}
|
||||
|
||||
export interface SttInput {
|
||||
/** One of: local path, remote URL, or base64 data URL. */
|
||||
audioPath?: string;
|
||||
audioUrl?: string;
|
||||
audioBase64?: string;
|
||||
language?: string;
|
||||
model?: string;
|
||||
}
|
||||
|
||||
export interface SttResult {
|
||||
provider: string;
|
||||
text: string;
|
||||
language?: string;
|
||||
durationSec?: number;
|
||||
segments?: Array<{ start: number; end: number; text: string }>;
|
||||
}
|
||||
|
||||
export interface SearchInput {
|
||||
query: string;
|
||||
model?: string;
|
||||
recency?: 'day' | 'week' | 'month' | 'year';
|
||||
maxResults?: number;
|
||||
}
|
||||
|
||||
export interface SearchResult {
|
||||
provider: string;
|
||||
model: string;
|
||||
answer: string;
|
||||
citations: Array<{ index?: number; url: string; title?: string }>;
|
||||
costUsd?: number;
|
||||
raw?: unknown;
|
||||
}
|
||||
|
||||
export interface ModelInfo {
|
||||
id: string;
|
||||
type?: string;
|
||||
label?: string;
|
||||
estUsd?: number;
|
||||
fields?: unknown;
|
||||
}
|
||||
|
||||
export interface EstimateInput {
|
||||
model: string;
|
||||
input: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface UploadInput {
|
||||
/** Local path to read and upload. */
|
||||
path: string;
|
||||
name?: string;
|
||||
mimeType?: string;
|
||||
}
|
||||
|
||||
export interface UploadResult {
|
||||
/** Whatever the provider returns to reference the uploaded asset later. */
|
||||
ref: unknown;
|
||||
url?: string;
|
||||
}
|
||||
|
||||
export interface MediaProvider {
|
||||
readonly name: string;
|
||||
readonly capabilities: Capability[];
|
||||
generate(capability: Capability, req: MediaRequest): Promise<MediaResult>;
|
||||
estimate?(req: EstimateInput): Promise<{ usd: number }>;
|
||||
upload?(input: UploadInput): Promise<UploadResult>;
|
||||
listModels?(): Promise<ModelInfo[]>;
|
||||
}
|
||||
|
||||
export interface SearchProvider {
|
||||
readonly name: string;
|
||||
search(input: SearchInput): Promise<SearchResult>;
|
||||
}
|
||||
|
||||
export interface SttProvider {
|
||||
readonly name: string;
|
||||
transcribe(input: SttInput): Promise<SttResult>;
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
/**
|
||||
* Unified error type for NexusAI. Every provider adapter should throw NexusError
|
||||
* so the MCP layer can render a consistent { code, message } shape.
|
||||
*/
|
||||
export type NexusErrorCode =
|
||||
| 'CONFIG'
|
||||
| 'AUTH'
|
||||
| 'BAD_REQUEST'
|
||||
| 'NOT_FOUND'
|
||||
| 'RATE_LIMIT'
|
||||
| 'INSUFFICIENT_BALANCE'
|
||||
| 'PROVIDER'
|
||||
| 'TIMEOUT'
|
||||
| 'UNSUPPORTED'
|
||||
| 'IO'
|
||||
| 'UNKNOWN';
|
||||
|
||||
export class NexusError extends Error {
|
||||
readonly code: NexusErrorCode;
|
||||
readonly details?: unknown;
|
||||
|
||||
constructor(code: NexusErrorCode, message: string, details?: unknown) {
|
||||
super(message);
|
||||
this.name = 'NexusError';
|
||||
this.code = code;
|
||||
this.details = details;
|
||||
}
|
||||
|
||||
static from(error: unknown, fallbackCode: NexusErrorCode = 'UNKNOWN'): NexusError {
|
||||
if (error instanceof NexusError) return error;
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
return new NexusError(fallbackCode, message);
|
||||
}
|
||||
}
|
||||
|
||||
/** Map an HTTP status code to a NexusErrorCode. */
|
||||
export function codeFromHttpStatus(status: number): NexusErrorCode {
|
||||
switch (status) {
|
||||
case 400:
|
||||
return 'BAD_REQUEST';
|
||||
case 401:
|
||||
case 403:
|
||||
return 'AUTH';
|
||||
case 402:
|
||||
return 'INSUFFICIENT_BALANCE';
|
||||
case 404:
|
||||
return 'NOT_FOUND';
|
||||
case 429:
|
||||
return 'RATE_LIMIT';
|
||||
default:
|
||||
return status >= 500 ? 'PROVIDER' : 'UNKNOWN';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
import { fetch, ProxyAgent, type RequestInit, type Response } from 'undici';
|
||||
import { NexusError, codeFromHttpStatus } from './errors.js';
|
||||
import { sleep } from './polling.js';
|
||||
|
||||
function getProxyUrl(): string | undefined {
|
||||
return (
|
||||
process.env.NEXUS_HTTP_PROXY ||
|
||||
process.env.HTTPS_PROXY ||
|
||||
process.env.https_proxy ||
|
||||
process.env.HTTP_PROXY ||
|
||||
process.env.http_proxy
|
||||
);
|
||||
}
|
||||
|
||||
function isRetryableError(error: unknown): boolean {
|
||||
if (error instanceof Error) {
|
||||
const msg = error.message.toLowerCase();
|
||||
return (
|
||||
msg.includes('timeout') ||
|
||||
msg.includes('econnreset') ||
|
||||
msg.includes('socket') ||
|
||||
msg.includes('fetch failed') ||
|
||||
msg.includes('und_err')
|
||||
);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Simple sliding-window rate limiter. */
|
||||
export class RateLimiter {
|
||||
private timestamps: number[] = [];
|
||||
constructor(private readonly maxRequests: number, private readonly windowMs: number) {}
|
||||
|
||||
async acquire(): Promise<void> {
|
||||
while (true) {
|
||||
const now = Date.now();
|
||||
this.timestamps = this.timestamps.filter((ts) => now - ts < this.windowMs);
|
||||
if (this.timestamps.length < this.maxRequests) {
|
||||
this.timestamps.push(now);
|
||||
return;
|
||||
}
|
||||
const oldest = this.timestamps[0];
|
||||
await sleep(Math.max(0, this.windowMs - (now - oldest)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export interface HttpClientOptions {
|
||||
baseUrl: string;
|
||||
defaultHeaders?: Record<string, string>;
|
||||
timeoutMs?: number;
|
||||
maxRetries?: number;
|
||||
rateLimiter?: RateLimiter;
|
||||
}
|
||||
|
||||
export interface RequestOptions {
|
||||
method?: 'GET' | 'POST' | 'PUT' | 'DELETE';
|
||||
path: string;
|
||||
headers?: Record<string, string>;
|
||||
body?: unknown;
|
||||
timeoutMs?: number;
|
||||
/** Skip JSON parsing and return raw Response (for binary downloads). */
|
||||
raw?: boolean;
|
||||
}
|
||||
|
||||
export class HttpClient {
|
||||
private readonly proxyAgent?: ProxyAgent;
|
||||
|
||||
constructor(private readonly opts: HttpClientOptions) {
|
||||
const proxyUrl = getProxyUrl();
|
||||
if (proxyUrl) this.proxyAgent = new ProxyAgent(proxyUrl);
|
||||
}
|
||||
|
||||
async requestRaw(options: RequestOptions): Promise<Response> {
|
||||
if (this.opts.rateLimiter) await this.opts.rateLimiter.acquire();
|
||||
|
||||
const url = options.path.startsWith('http')
|
||||
? options.path
|
||||
: `${this.opts.baseUrl}${options.path}`;
|
||||
const method = options.method ?? 'GET';
|
||||
const timeoutMs = options.timeoutMs ?? this.opts.timeoutMs ?? 60_000;
|
||||
const maxRetries = this.opts.maxRetries ?? 2;
|
||||
|
||||
const headers: Record<string, string> = { ...this.opts.defaultHeaders, ...options.headers };
|
||||
let bodyStr: string | undefined;
|
||||
if (options.body !== undefined) {
|
||||
bodyStr = typeof options.body === 'string' ? options.body : JSON.stringify(options.body);
|
||||
if (!headers['Content-Type'] && !headers['content-type']) {
|
||||
headers['Content-Type'] = 'application/json';
|
||||
}
|
||||
}
|
||||
|
||||
let lastError: Error | undefined;
|
||||
for (let attempt = 0; attempt <= maxRetries; attempt++) {
|
||||
try {
|
||||
const init: RequestInit & { dispatcher?: ProxyAgent } = {
|
||||
method,
|
||||
headers,
|
||||
body: bodyStr,
|
||||
signal: AbortSignal.timeout(timeoutMs),
|
||||
};
|
||||
if (this.proxyAgent) init.dispatcher = this.proxyAgent;
|
||||
|
||||
const response = await fetch(url, init);
|
||||
|
||||
if ((response.status === 429 || response.status >= 500) && attempt < maxRetries) {
|
||||
await sleep(Math.min(1000 * 2 ** attempt, 8000));
|
||||
continue;
|
||||
}
|
||||
return response;
|
||||
} catch (error: unknown) {
|
||||
lastError = error instanceof Error ? error : new Error(String(error));
|
||||
if (isRetryableError(lastError) && attempt < maxRetries) {
|
||||
await sleep(Math.min(1000 * 2 ** attempt, 8000));
|
||||
continue;
|
||||
}
|
||||
if (lastError.name === 'TimeoutError' || lastError.message.includes('aborted')) {
|
||||
throw new NexusError('TIMEOUT', `Request to ${url} timed out after ${timeoutMs}ms`);
|
||||
}
|
||||
throw NexusError.from(lastError, 'PROVIDER');
|
||||
}
|
||||
}
|
||||
throw NexusError.from(lastError ?? new Error('Request failed'), 'PROVIDER');
|
||||
}
|
||||
|
||||
async requestJson<T = unknown>(options: RequestOptions): Promise<T> {
|
||||
const response = await this.requestRaw(options);
|
||||
const text = await response.text();
|
||||
|
||||
if (!response.ok) {
|
||||
let message = `HTTP ${response.status}`;
|
||||
try {
|
||||
const parsed = JSON.parse(text) as { message?: string; error?: { message?: string }; msg?: string };
|
||||
message = parsed.message || parsed.error?.message || parsed.msg || message;
|
||||
} catch {
|
||||
if (text) message = `${message}: ${text.slice(0, 300)}`;
|
||||
}
|
||||
throw new NexusError(codeFromHttpStatus(response.status), message, { status: response.status });
|
||||
}
|
||||
|
||||
if (!text) return undefined as T;
|
||||
try {
|
||||
return JSON.parse(text) as T;
|
||||
} catch {
|
||||
throw new NexusError('PROVIDER', `Invalid JSON response: ${text.slice(0, 300)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
/**
|
||||
* MCP servers communicate over stdio, so ALL logs must go to stderr.
|
||||
* Never write logs to stdout — it corrupts the JSON-RPC stream.
|
||||
*/
|
||||
type Level = 'debug' | 'info' | 'warn' | 'error';
|
||||
|
||||
const LEVELS: Record<Level, number> = { debug: 10, info: 20, warn: 30, error: 40 };
|
||||
|
||||
function currentThreshold(): number {
|
||||
const env = (process.env.NEXUS_LOG_LEVEL || 'info').toLowerCase() as Level;
|
||||
return LEVELS[env] ?? LEVELS.info;
|
||||
}
|
||||
|
||||
function emit(level: Level, scope: string, message: string, extra?: unknown): void {
|
||||
if (LEVELS[level] < currentThreshold()) return;
|
||||
const ts = new Date().toISOString();
|
||||
let line = `[${ts}] ${level.toUpperCase()} (${scope}) ${message}`;
|
||||
if (extra !== undefined) {
|
||||
try {
|
||||
line += ` ${typeof extra === 'string' ? extra : JSON.stringify(extra)}`;
|
||||
} catch {
|
||||
line += ' [unserializable extra]';
|
||||
}
|
||||
}
|
||||
process.stderr.write(line + '\n');
|
||||
}
|
||||
|
||||
export interface Logger {
|
||||
debug(message: string, extra?: unknown): void;
|
||||
info(message: string, extra?: unknown): void;
|
||||
warn(message: string, extra?: unknown): void;
|
||||
error(message: string, extra?: unknown): void;
|
||||
}
|
||||
|
||||
export function createLogger(scope: string): Logger {
|
||||
return {
|
||||
debug: (m, e) => emit('debug', scope, m, e),
|
||||
info: (m, e) => emit('info', scope, m, e),
|
||||
warn: (m, e) => emit('warn', scope, m, e),
|
||||
error: (m, e) => emit('error', scope, m, e),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
import { NexusError } from './errors.js';
|
||||
|
||||
export function sleep(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
export interface PollOptions<T> {
|
||||
/** Called every interval. Return { done: true, value } to stop. */
|
||||
poll: () => Promise<{ done: boolean; value?: T; failed?: boolean; error?: string }>;
|
||||
intervalMs?: number;
|
||||
timeoutMs?: number;
|
||||
onTick?: (attempt: number, elapsedMs: number) => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Generic polling loop with wall-clock timeout. Used for async media jobs.
|
||||
*/
|
||||
export async function pollUntilDone<T>(options: PollOptions<T>): Promise<T> {
|
||||
const intervalMs = options.intervalMs ?? 3000;
|
||||
const timeoutMs = options.timeoutMs ?? 10 * 60 * 1000;
|
||||
const start = Date.now();
|
||||
let attempt = 0;
|
||||
|
||||
while (true) {
|
||||
attempt += 1;
|
||||
const result = await options.poll();
|
||||
options.onTick?.(attempt, Date.now() - start);
|
||||
|
||||
if (result.failed) {
|
||||
throw new NexusError('PROVIDER', result.error || 'Job failed');
|
||||
}
|
||||
if (result.done) {
|
||||
return result.value as T;
|
||||
}
|
||||
|
||||
if (Date.now() - start > timeoutMs) {
|
||||
throw new NexusError('TIMEOUT', `Polling timed out after ${Math.round(timeoutMs / 1000)}s`);
|
||||
}
|
||||
await sleep(intervalMs);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "CommonJS",
|
||||
"moduleResolution": "node",
|
||||
"declaration": true,
|
||||
"declarationMap": true,
|
||||
"sourceMap": true,
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true
|
||||
},
|
||||
"include": ["src/**/*"]
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"name": "@nexusai/mcp-server",
|
||||
"version": "0.1.0",
|
||||
"description": "NexusAI MCP server (stdio): TTS, STT, image/video/music generation, web search",
|
||||
"main": "dist/index.js",
|
||||
"type": "commonjs",
|
||||
"bin": {
|
||||
"nexusai-mcp": "dist/index.js"
|
||||
},
|
||||
"scripts": {
|
||||
"build": "tsc",
|
||||
"dev": "tsc --watch",
|
||||
"start": "node dist/index.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/sdk": "^1.12.0",
|
||||
"@nexusai/core": "*",
|
||||
"zod": "^3.25.0",
|
||||
"zod-to-json-schema": "^3.25.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^22.15.0",
|
||||
"typescript": "^5.8.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
import {
|
||||
Journal,
|
||||
MediaStorage,
|
||||
ProviderRegistry,
|
||||
createNordRouterProviders,
|
||||
loadConfig,
|
||||
LocalWhisperProvider,
|
||||
createLogger,
|
||||
type NexusConfig,
|
||||
} from '@nexusai/core';
|
||||
|
||||
const log = createLogger('context');
|
||||
|
||||
export interface AppContext {
|
||||
config: NexusConfig;
|
||||
registry: ProviderRegistry;
|
||||
storage: MediaStorage;
|
||||
journal: Journal;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wire up all providers, storage and the journal once at startup.
|
||||
*/
|
||||
export function buildContext(): AppContext {
|
||||
const config = loadConfig();
|
||||
const storage = new MediaStorage(config.mediaDir);
|
||||
const journal = new Journal(config.dbPath);
|
||||
const registry = new ProviderRegistry();
|
||||
|
||||
// NordRouter (media + search). Registered even without a key so error messages
|
||||
// are clear when a tool is actually invoked.
|
||||
try {
|
||||
const nr = createNordRouterProviders({ config: config.nordrouter, storage, save: true });
|
||||
registry.registerMedia(nr.media);
|
||||
registry.registerSearch(nr.search);
|
||||
log.info('registered nordrouter provider (media + search)');
|
||||
} catch (err) {
|
||||
log.warn('nordrouter not registered', err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
|
||||
// Local STT (faster-whisper sidecar).
|
||||
registry.registerStt(new LocalWhisperProvider(config.stt));
|
||||
log.info('registered local STT provider');
|
||||
|
||||
return { config, registry, storage, journal };
|
||||
}
|
||||
@@ -0,0 +1,275 @@
|
||||
import { randomUUID } from 'crypto';
|
||||
import {
|
||||
NexusError,
|
||||
buildEmbeddedContent,
|
||||
createLogger,
|
||||
type Capability,
|
||||
type MediaResult,
|
||||
} from '@nexusai/core';
|
||||
import type { AppContext } from './context.js';
|
||||
import {
|
||||
ImageSchema,
|
||||
MediaEstimateSchema,
|
||||
MediaModelsSchema,
|
||||
MediaUploadSchema,
|
||||
MusicSchema,
|
||||
SttSchema,
|
||||
TtsSchema,
|
||||
UsageStatsSchema,
|
||||
VideoSchema,
|
||||
WebSearchSchema,
|
||||
} from './schemas.js';
|
||||
|
||||
const log = createLogger('handlers');
|
||||
|
||||
type McpContent =
|
||||
| { type: 'text'; text: string }
|
||||
| { type: 'image'; data: string; mimeType: string }
|
||||
| { type: 'audio'; data: string; mimeType: string };
|
||||
|
||||
export interface McpResult {
|
||||
content: McpContent[];
|
||||
isError?: boolean;
|
||||
}
|
||||
|
||||
function jsonBlock(obj: unknown): McpContent {
|
||||
return { type: 'text', text: JSON.stringify(obj, null, 2) };
|
||||
}
|
||||
|
||||
function summarize(input: Record<string, unknown>): string {
|
||||
const s = JSON.stringify(input);
|
||||
return s.length > 200 ? s.slice(0, 197) + '...' : s;
|
||||
}
|
||||
|
||||
export class Handlers {
|
||||
constructor(private readonly ctx: AppContext) {}
|
||||
|
||||
private providerName(explicit?: string): string {
|
||||
return explicit || this.ctx.config.defaultProvider;
|
||||
}
|
||||
|
||||
async handle(name: string, args: Record<string, unknown>): Promise<McpResult> {
|
||||
switch (name) {
|
||||
case 'nexus_tts':
|
||||
return this.media('tts', TtsSchema.parse(args));
|
||||
case 'nexus_image':
|
||||
return this.media('image', ImageSchema.parse(args));
|
||||
case 'nexus_video':
|
||||
return this.media('video', VideoSchema.parse(args));
|
||||
case 'nexus_music':
|
||||
return this.media('music', MusicSchema.parse(args));
|
||||
case 'nexus_stt':
|
||||
return this.stt(SttSchema.parse(args));
|
||||
case 'nexus_web_search':
|
||||
return this.search(WebSearchSchema.parse(args));
|
||||
case 'nexus_media_models':
|
||||
return this.models(MediaModelsSchema.parse(args));
|
||||
case 'nexus_media_estimate':
|
||||
return this.estimate(MediaEstimateSchema.parse(args));
|
||||
case 'nexus_media_upload':
|
||||
return this.upload(MediaUploadSchema.parse(args));
|
||||
case 'nexus_usage_stats':
|
||||
return this.usage(UsageStatsSchema.parse(args));
|
||||
default:
|
||||
throw new NexusError('NOT_FOUND', `Unknown tool: ${name}`);
|
||||
}
|
||||
}
|
||||
|
||||
private async media(
|
||||
capability: Capability,
|
||||
args: { provider?: string; model: string; input: Record<string, unknown>; output?: { save?: boolean; embed?: boolean } }
|
||||
): Promise<McpResult> {
|
||||
const providerName = this.providerName(args.provider);
|
||||
const provider = this.ctx.registry.getMedia(providerName, capability);
|
||||
const started = Date.now();
|
||||
const id = randomUUID();
|
||||
|
||||
try {
|
||||
const result = await provider.generate(capability, { model: args.model, input: args.input });
|
||||
this.ctx.journal.record({
|
||||
id,
|
||||
provider: providerName,
|
||||
capability,
|
||||
model: args.model,
|
||||
inputSummary: summarize(args.input),
|
||||
status: 'done',
|
||||
costUsd: result.costUsd ?? null,
|
||||
filePath: result.filePath ?? null,
|
||||
resultUrl: result.resultUrl ?? null,
|
||||
error: null,
|
||||
durationMs: Date.now() - started,
|
||||
createdAt: new Date().toISOString(),
|
||||
});
|
||||
return this.mediaResult(result, args.output?.embed ?? false);
|
||||
} catch (err) {
|
||||
const e = NexusError.from(err);
|
||||
this.ctx.journal.record({
|
||||
id,
|
||||
provider: providerName,
|
||||
capability,
|
||||
model: args.model,
|
||||
inputSummary: summarize(args.input),
|
||||
status: 'failed',
|
||||
costUsd: null,
|
||||
filePath: null,
|
||||
resultUrl: null,
|
||||
error: `${e.code}: ${e.message}`,
|
||||
durationMs: Date.now() - started,
|
||||
createdAt: new Date().toISOString(),
|
||||
});
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
private mediaResult(result: MediaResult, embed: boolean): McpResult {
|
||||
const content: McpContent[] = [
|
||||
jsonBlock({
|
||||
provider: result.provider,
|
||||
capability: result.capability,
|
||||
model: result.model,
|
||||
status: result.status,
|
||||
file_path: result.filePath,
|
||||
mime_type: result.mimeType,
|
||||
result_url: result.resultUrl,
|
||||
result_url_2: result.resultUrl2,
|
||||
cost_usd: result.costUsd,
|
||||
}),
|
||||
];
|
||||
|
||||
if (embed && result.filePath) {
|
||||
const block = buildEmbeddedContent(result.filePath, result.mimeType, this.ctx.config.embedMaxBytes);
|
||||
if (block) content.push(block);
|
||||
else content.push({ type: 'text', text: '(embed skipped: file too large or not embeddable)' });
|
||||
}
|
||||
return { content };
|
||||
}
|
||||
|
||||
private async stt(args: {
|
||||
provider?: string;
|
||||
input: { audioPath?: string; audioUrl?: string; audioBase64?: string; language?: string; model?: string };
|
||||
}): Promise<McpResult> {
|
||||
const providerName = args.provider || 'local';
|
||||
const provider = this.ctx.registry.getStt(providerName);
|
||||
const started = Date.now();
|
||||
const id = randomUUID();
|
||||
try {
|
||||
const res = await provider.transcribe(args.input);
|
||||
this.ctx.journal.record({
|
||||
id,
|
||||
provider: providerName,
|
||||
capability: 'stt',
|
||||
model: args.input.model || 'default',
|
||||
inputSummary: summarize({ ...args.input, audioBase64: args.input.audioBase64 ? '[base64]' : undefined }),
|
||||
status: 'done',
|
||||
costUsd: 0,
|
||||
filePath: args.input.audioPath ?? null,
|
||||
resultUrl: null,
|
||||
error: null,
|
||||
durationMs: Date.now() - started,
|
||||
createdAt: new Date().toISOString(),
|
||||
});
|
||||
return {
|
||||
content: [
|
||||
jsonBlock({
|
||||
provider: res.provider,
|
||||
language: res.language,
|
||||
duration_sec: res.durationSec,
|
||||
text: res.text,
|
||||
segments: res.segments,
|
||||
}),
|
||||
],
|
||||
};
|
||||
} catch (err) {
|
||||
const e = NexusError.from(err);
|
||||
this.ctx.journal.record({
|
||||
id,
|
||||
provider: providerName,
|
||||
capability: 'stt',
|
||||
model: args.input.model || 'default',
|
||||
inputSummary: 'stt',
|
||||
status: 'failed',
|
||||
costUsd: null,
|
||||
filePath: null,
|
||||
resultUrl: null,
|
||||
error: `${e.code}: ${e.message}`,
|
||||
durationMs: Date.now() - started,
|
||||
createdAt: new Date().toISOString(),
|
||||
});
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
private async search(args: {
|
||||
provider?: string;
|
||||
query: string;
|
||||
model?: string;
|
||||
recency?: 'day' | 'week' | 'month' | 'year';
|
||||
maxResults?: number;
|
||||
}): Promise<McpResult> {
|
||||
const providerName = this.providerName(args.provider);
|
||||
const provider = this.ctx.registry.getSearch(providerName);
|
||||
const started = Date.now();
|
||||
const id = randomUUID();
|
||||
const res = await provider.search({
|
||||
query: args.query,
|
||||
model: args.model,
|
||||
recency: args.recency,
|
||||
maxResults: args.maxResults,
|
||||
});
|
||||
this.ctx.journal.record({
|
||||
id,
|
||||
provider: providerName,
|
||||
capability: 'search',
|
||||
model: res.model,
|
||||
inputSummary: summarize({ query: args.query }),
|
||||
status: 'done',
|
||||
costUsd: res.costUsd ?? null,
|
||||
filePath: null,
|
||||
resultUrl: null,
|
||||
error: null,
|
||||
durationMs: Date.now() - started,
|
||||
createdAt: new Date().toISOString(),
|
||||
});
|
||||
return {
|
||||
content: [jsonBlock({ provider: res.provider, model: res.model, answer: res.answer, citations: res.citations })],
|
||||
};
|
||||
}
|
||||
|
||||
private async models(args: { provider?: string; type?: string; search?: string }): Promise<McpResult> {
|
||||
const providerName = this.providerName(args.provider);
|
||||
// any capability that this provider supports is fine to fetch the catalog
|
||||
const provider = this.ctx.registry.getMedia(providerName, 'image');
|
||||
if (!provider.listModels) throw new NexusError('UNSUPPORTED', `Provider ${providerName} cannot list models`);
|
||||
let models = await provider.listModels();
|
||||
if (args.type) models = models.filter((m) => m.type === args.type);
|
||||
if (args.search) {
|
||||
const q = args.search.toLowerCase();
|
||||
models = models.filter((m) => m.id.toLowerCase().includes(q) || (m.label || '').toLowerCase().includes(q));
|
||||
}
|
||||
return { content: [jsonBlock({ count: models.length, models })] };
|
||||
}
|
||||
|
||||
private async estimate(args: { provider?: string; model: string; input: Record<string, unknown> }): Promise<McpResult> {
|
||||
const providerName = this.providerName(args.provider);
|
||||
const provider = this.ctx.registry.getMedia(providerName, 'image');
|
||||
if (!provider.estimate) throw new NexusError('UNSUPPORTED', `Provider ${providerName} cannot estimate`);
|
||||
const est = await provider.estimate({ model: args.model, input: args.input });
|
||||
return { content: [jsonBlock({ model: args.model, usd: est.usd })] };
|
||||
}
|
||||
|
||||
private async upload(args: { provider?: string; path: string; name?: string; mimeType?: string }): Promise<McpResult> {
|
||||
const providerName = this.providerName(args.provider);
|
||||
const provider = this.ctx.registry.getMedia(providerName, 'image');
|
||||
if (!provider.upload) throw new NexusError('UNSUPPORTED', `Provider ${providerName} cannot upload`);
|
||||
const res = await provider.upload({ path: args.path, name: args.name, mimeType: args.mimeType });
|
||||
return { content: [jsonBlock({ url: res.url, ref: res.ref })] };
|
||||
}
|
||||
|
||||
private async usage(args: { sinceDays?: number; list?: boolean; limit: number }): Promise<McpResult> {
|
||||
const stats = this.ctx.journal.stats(args.sinceDays);
|
||||
const payload: Record<string, unknown> = { stats };
|
||||
if (args.list) payload.recent = this.ctx.journal.list(args.limit);
|
||||
log.debug('usage stats requested');
|
||||
return { content: [jsonBlock(payload)] };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env node
|
||||
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
|
||||
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
|
||||
import {
|
||||
CallToolRequestSchema,
|
||||
ListToolsRequestSchema,
|
||||
type Tool,
|
||||
} from '@modelcontextprotocol/sdk/types.js';
|
||||
import { NexusError, createLogger } from '@nexusai/core';
|
||||
import { buildContext } from './context.js';
|
||||
import { Handlers } from './handlers.js';
|
||||
import { TOOLS } from './tools.js';
|
||||
|
||||
const log = createLogger('server');
|
||||
|
||||
function main(): void {
|
||||
const ctx = buildContext();
|
||||
const handlers = new Handlers(ctx);
|
||||
|
||||
const server = new Server(
|
||||
{ name: 'nexusai-mcp-server', version: '0.1.0' },
|
||||
{ capabilities: { tools: {} } }
|
||||
);
|
||||
|
||||
server.setRequestHandler(ListToolsRequestSchema, async () => ({
|
||||
tools: TOOLS.map((t) => ({
|
||||
name: t.name,
|
||||
description: t.description,
|
||||
inputSchema: t.inputSchema,
|
||||
})) as Tool[],
|
||||
}));
|
||||
|
||||
server.setRequestHandler(CallToolRequestSchema, async (request) => {
|
||||
const { name, arguments: args } = request.params;
|
||||
try {
|
||||
return (await handlers.handle(name, args ?? {})) as never;
|
||||
} catch (error) {
|
||||
const e = NexusError.from(error);
|
||||
log.error(`tool ${name} failed`, `${e.code}: ${e.message}`);
|
||||
return {
|
||||
content: [{ type: 'text', text: `Error [${e.code}]: ${e.message}` }],
|
||||
isError: true,
|
||||
} as never;
|
||||
}
|
||||
});
|
||||
|
||||
const transport = new StdioServerTransport();
|
||||
server.connect(transport).catch((error: unknown) => {
|
||||
log.error('failed to start MCP server', error instanceof Error ? error.message : String(error));
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
log.info(`NexusAI MCP server started with ${TOOLS.length} tools`);
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,138 @@
|
||||
import { z } from 'zod';
|
||||
|
||||
/** Common output options for media tools. */
|
||||
const OutputOptions = z
|
||||
.object({
|
||||
save: z.boolean().default(true).describe('Save the result to a local file.'),
|
||||
embed: z
|
||||
.boolean()
|
||||
.default(false)
|
||||
.describe('Also embed image/audio directly in the MCP result (size-limited).'),
|
||||
})
|
||||
.partial()
|
||||
.default({});
|
||||
|
||||
const provider = z
|
||||
.string()
|
||||
.optional()
|
||||
.describe('Provider name. Defaults to the configured default provider (e.g. "nordrouter").');
|
||||
|
||||
export const TtsSchema = z.object({
|
||||
provider,
|
||||
model: z
|
||||
.string()
|
||||
.default('audio/elevenlabs-tts')
|
||||
.describe('TTS model id, e.g. audio/elevenlabs-tts, audio/elevenlabs-multilingual, audio/gemini-3.1-flash-tts.'),
|
||||
input: z
|
||||
.object({
|
||||
text: z.string().describe('Text to synthesize.'),
|
||||
voice: z.string().optional().describe('Voice name (see nexus_media_models for options).'),
|
||||
voice_id: z.string().optional(),
|
||||
speed: z.union([z.string(), z.number()]).optional().describe('Speed, e.g. "0.8" | "1.0" | "1.2".'),
|
||||
})
|
||||
.passthrough()
|
||||
.describe('Provider input. Extra model-specific fields are passed through.'),
|
||||
output: OutputOptions,
|
||||
});
|
||||
|
||||
export const ImageSchema = z.object({
|
||||
provider,
|
||||
model: z
|
||||
.string()
|
||||
.default('image/nano-banana-2')
|
||||
.describe('Image model id, e.g. image/nano-banana-2, image/flux2-pro, image/gpt-image-2, image/z-image. Use *-edit models for editing.'),
|
||||
input: z
|
||||
.object({
|
||||
prompt: z.string().describe('Image description. For edits also pass "image".'),
|
||||
image: z.string().optional().describe('Source image URL for edit/i2i models.'),
|
||||
aspect_ratio: z.string().optional(),
|
||||
resolution: z.string().optional().describe('1K | 2K | 4K (model dependent).'),
|
||||
})
|
||||
.passthrough(),
|
||||
output: OutputOptions,
|
||||
});
|
||||
|
||||
export const VideoSchema = z.object({
|
||||
provider,
|
||||
model: z
|
||||
.string()
|
||||
.default('video/kling-2.6-t2v')
|
||||
.describe('Video model id, e.g. video/kling-2.6-t2v, video/veo-3.1-fast, video/seedance-2. Use i2v models with an "image".'),
|
||||
input: z
|
||||
.object({
|
||||
prompt: z.string().describe('Video description.'),
|
||||
image: z.string().optional().describe('First-frame image URL for i2v models.'),
|
||||
image2: z.string().optional().describe('Last-frame image URL (transition models).'),
|
||||
duration: z.union([z.string(), z.number()]).optional(),
|
||||
resolution: z.string().optional(),
|
||||
aspect_ratio: z.string().optional(),
|
||||
sound: z.boolean().optional(),
|
||||
})
|
||||
.passthrough(),
|
||||
output: OutputOptions,
|
||||
});
|
||||
|
||||
export const MusicSchema = z.object({
|
||||
provider,
|
||||
model: z
|
||||
.string()
|
||||
.default('music/suno-v5')
|
||||
.describe('Music model id, e.g. music/suno-v5, music/suno-cover, music/suno-sounds.'),
|
||||
input: z
|
||||
.object({
|
||||
title: z.string().optional(),
|
||||
style: z.string().optional(),
|
||||
prompt: z.string().optional().describe('Lyrics/prompt.'),
|
||||
instrumental: z.boolean().optional(),
|
||||
version: z.string().optional().describe('V4 | V4_5 | V4_5PLUS | V4_5ALL | V5 | V5_5.'),
|
||||
})
|
||||
.passthrough(),
|
||||
output: OutputOptions,
|
||||
});
|
||||
|
||||
export const SttSchema = z.object({
|
||||
provider: z.string().optional().describe('STT provider. Defaults to "local" (faster-whisper).'),
|
||||
input: z.object({
|
||||
audioPath: z.string().optional().describe('Local audio file path.'),
|
||||
audioUrl: z.string().optional().describe('Remote audio URL.'),
|
||||
audioBase64: z.string().optional().describe('Base64 or data-URL audio.'),
|
||||
language: z.string().optional().describe('Language hint, e.g. "ru", "en". Auto-detect if omitted.'),
|
||||
model: z.string().optional().describe('Whisper size: tiny|base|small|medium|large-v3.'),
|
||||
}),
|
||||
});
|
||||
|
||||
export const WebSearchSchema = z.object({
|
||||
provider,
|
||||
query: z.string().describe('Search query / question.'),
|
||||
model: z
|
||||
.string()
|
||||
.optional()
|
||||
.describe('Search model, e.g. perplexity/sonar, perplexity/sonar-pro, or search/<model>.'),
|
||||
recency: z.enum(['day', 'week', 'month', 'year']).optional(),
|
||||
maxResults: z.number().int().positive().max(50).optional(),
|
||||
});
|
||||
|
||||
export const MediaModelsSchema = z.object({
|
||||
provider,
|
||||
type: z.enum(['image', 'video', 'music', 'audio', 'upscale']).optional().describe('Filter by model type.'),
|
||||
search: z.string().optional().describe('Substring filter on model id/label.'),
|
||||
});
|
||||
|
||||
export const MediaEstimateSchema = z.object({
|
||||
provider,
|
||||
model: z.string().describe('Model id to estimate.'),
|
||||
input: z.record(z.unknown()).default({}).describe('Same input you would pass to generate.'),
|
||||
});
|
||||
|
||||
export const MediaUploadSchema = z.object({
|
||||
provider,
|
||||
path: z.string().describe('Local file path to upload.'),
|
||||
name: z.string().optional(),
|
||||
mimeType: z.string().optional(),
|
||||
});
|
||||
|
||||
export const UsageStatsSchema = z.object({
|
||||
sinceDays: z.number().int().positive().optional().describe('Limit stats to the last N days.'),
|
||||
list: z.boolean().default(false).describe('Also return recent generations.'),
|
||||
limit: z.number().int().positive().max(200).default(20),
|
||||
});
|
||||
@@ -0,0 +1,78 @@
|
||||
import { zodToJsonSchema } from 'zod-to-json-schema';
|
||||
import {
|
||||
ImageSchema,
|
||||
MediaEstimateSchema,
|
||||
MediaModelsSchema,
|
||||
MediaUploadSchema,
|
||||
MusicSchema,
|
||||
SttSchema,
|
||||
TtsSchema,
|
||||
UsageStatsSchema,
|
||||
VideoSchema,
|
||||
WebSearchSchema,
|
||||
} from './schemas.js';
|
||||
|
||||
export interface ToolDef {
|
||||
name: string;
|
||||
description: string;
|
||||
inputSchema: object;
|
||||
}
|
||||
|
||||
function make(name: string, description: string, schema: unknown): ToolDef {
|
||||
const json = zodToJsonSchema(schema as any, { name, $refStrategy: 'none' }) as any;
|
||||
const def = json.definitions?.[name] ?? json;
|
||||
return { name, description, inputSchema: def };
|
||||
}
|
||||
|
||||
export const TOOLS: ToolDef[] = [
|
||||
make(
|
||||
'nexus_tts',
|
||||
'Text-to-speech. Generates spoken audio from text via a provider. Returns a saved audio file path and optional embedded audio.',
|
||||
TtsSchema
|
||||
),
|
||||
make(
|
||||
'nexus_image',
|
||||
'Generate or edit images from a text prompt (and optional source image URL). Returns a saved image file and optional embedded image.',
|
||||
ImageSchema
|
||||
),
|
||||
make(
|
||||
'nexus_video',
|
||||
'Generate video from text or an image (t2v / i2v). Async, may take minutes. Returns a saved MP4 file path.',
|
||||
VideoSchema
|
||||
),
|
||||
make(
|
||||
'nexus_music',
|
||||
'Generate music from a style/prompt. Returns saved audio file(s); some models produce two tracks.',
|
||||
MusicSchema
|
||||
),
|
||||
make(
|
||||
'nexus_stt',
|
||||
'Speech-to-text. Transcribe local/remote/base64 audio using a local whisper model. Returns text, language and segments.',
|
||||
SttSchema
|
||||
),
|
||||
make(
|
||||
'nexus_web_search',
|
||||
'Web search that returns an answer with cited source URLs (via search-augmented models).',
|
||||
WebSearchSchema
|
||||
),
|
||||
make(
|
||||
'nexus_media_models',
|
||||
'List available media models (id, type, label, estimated price) for a provider. Use to discover valid model ids and parameters.',
|
||||
MediaModelsSchema
|
||||
),
|
||||
make(
|
||||
'nexus_media_estimate',
|
||||
'Estimate the USD cost of a media generation before running it.',
|
||||
MediaEstimateSchema
|
||||
),
|
||||
make(
|
||||
'nexus_media_upload',
|
||||
'Upload a local media file to the provider and get a reference URL for use as input (image/audio/video).',
|
||||
MediaUploadSchema
|
||||
),
|
||||
make(
|
||||
'nexus_usage_stats',
|
||||
'Show usage and spend recorded in the local journal (totals by provider, capability, day) and optionally recent generations.',
|
||||
UsageStatsSchema
|
||||
),
|
||||
];
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "CommonJS",
|
||||
"moduleResolution": "node",
|
||||
"declaration": false,
|
||||
"sourceMap": true,
|
||||
"outDir": "dist",
|
||||
"rootDir": "src",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true
|
||||
},
|
||||
"include": ["src/**/*"]
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# requires-python = ">=3.9"
|
||||
# dependencies = [
|
||||
# "faster-whisper>=1.0.0",
|
||||
# ]
|
||||
# ///
|
||||
"""
|
||||
NexusAI local STT sidecar (faster-whisper).
|
||||
|
||||
Invoked by the TypeScript MCP server as:
|
||||
uv run sidecar/stt_server.py '<json-request>'
|
||||
|
||||
Request JSON:
|
||||
{ "audio": "/path/to/file", "model": "small", "device": "auto",
|
||||
"compute_type": "auto", "language": "ru" }
|
||||
|
||||
Prints exactly ONE JSON line on stdout:
|
||||
{ "ok": true, "text": "...", "language": "ru", "duration": 12.3,
|
||||
"segments": [ { "start": 0.0, "end": 2.1, "text": "..." } ] }
|
||||
On error:
|
||||
{ "ok": false, "error": "message" }
|
||||
|
||||
All diagnostics go to stderr so stdout stays a clean single JSON line.
|
||||
faster-whisper decodes audio via bundled ffmpeg/pyav, so no manual conversion
|
||||
is required for common formats (mp3/wav/ogg/m4a/webm...).
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
print(msg, file=sys.stderr, flush=True)
|
||||
|
||||
|
||||
def resolve_device(device: str, compute_type: str):
|
||||
if device and device != "auto":
|
||||
ct = compute_type if compute_type and compute_type != "auto" else (
|
||||
"float16" if device == "cuda" else "int8"
|
||||
)
|
||||
return device, ct
|
||||
# auto-detect: try cuda, fall back to cpu
|
||||
try:
|
||||
import ctranslate2 # noqa: F401
|
||||
cuda_count = 0
|
||||
try:
|
||||
cuda_count = ctranslate2.get_cuda_device_count()
|
||||
except Exception:
|
||||
cuda_count = 0
|
||||
if cuda_count > 0:
|
||||
return "cuda", (compute_type if compute_type != "auto" else "float16")
|
||||
except Exception:
|
||||
pass
|
||||
return "cpu", (compute_type if compute_type != "auto" else "int8")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if len(sys.argv) < 2:
|
||||
print(json.dumps({"ok": False, "error": "missing request argument"}))
|
||||
return 1
|
||||
|
||||
try:
|
||||
req = json.loads(sys.argv[1])
|
||||
except Exception as exc:
|
||||
print(json.dumps({"ok": False, "error": f"invalid request json: {exc}"}))
|
||||
return 1
|
||||
|
||||
audio = req.get("audio")
|
||||
if not audio:
|
||||
print(json.dumps({"ok": False, "error": "no audio path provided"}))
|
||||
return 1
|
||||
|
||||
model_size = req.get("model") or "small"
|
||||
device, compute_type = resolve_device(req.get("device", "auto"), req.get("compute_type", "auto"))
|
||||
language = req.get("language") or None
|
||||
|
||||
try:
|
||||
from faster_whisper import WhisperModel
|
||||
except Exception as exc:
|
||||
print(json.dumps({"ok": False, "error": f"faster-whisper not available: {exc}"}))
|
||||
return 1
|
||||
|
||||
# Allow pointing at a local model dir (offline) or a mirror via env.
|
||||
# NEXUS_STT_MODEL may be a whisper size name OR an absolute path to a
|
||||
# pre-downloaded CTranslate2 model directory.
|
||||
import os
|
||||
model_ref = model_size
|
||||
local_only = os.environ.get("NEXUS_STT_LOCAL_ONLY", "").lower() in ("1", "true", "yes")
|
||||
download_root = os.environ.get("NEXUS_STT_DOWNLOAD_ROOT") or None
|
||||
|
||||
try:
|
||||
log(f"loading model={model_ref} device={device} compute={compute_type} local_only={local_only}")
|
||||
model = WhisperModel(
|
||||
model_ref,
|
||||
device=device,
|
||||
compute_type=compute_type,
|
||||
download_root=download_root,
|
||||
local_files_only=local_only,
|
||||
)
|
||||
segments_iter, info = model.transcribe(audio, language=language, vad_filter=True)
|
||||
|
||||
segments = []
|
||||
parts = []
|
||||
for seg in segments_iter:
|
||||
parts.append(seg.text)
|
||||
segments.append({"start": round(seg.start, 3), "end": round(seg.end, 3), "text": seg.text.strip()})
|
||||
|
||||
result = {
|
||||
"ok": True,
|
||||
"text": "".join(parts).strip(),
|
||||
"language": info.language,
|
||||
"duration": round(getattr(info, "duration", 0.0) or 0.0, 3),
|
||||
"segments": segments,
|
||||
}
|
||||
print(json.dumps(result, ensure_ascii=False))
|
||||
return 0
|
||||
except Exception as exc:
|
||||
hint = ""
|
||||
msg = str(exc)
|
||||
if "connect" in msg.lower() or "internet" in msg.lower() or "Hub" in msg:
|
||||
hint = (
|
||||
" | Model download failed. Options: set HF_ENDPOINT to a reachable mirror, "
|
||||
"pre-download the model and set NEXUS_STT_MODEL to its local path with "
|
||||
"NEXUS_STT_LOCAL_ONLY=1, or set NEXUS_STT_DOWNLOAD_ROOT to a cache dir."
|
||||
)
|
||||
print(json.dumps({"ok": False, "error": f"transcription failed: {exc}{hint}"}, ensure_ascii=False))
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user