Model serving and LLM gateway

• Serve open-source LLMs, embedding and reranking models on GPUs with vLLM and SGLang, behind OpenAI-compatible endpoints.
• Build the LLM gateway: one API across self-hosted and external models, with routing, fallback, rate limits, quotas, API keys and per-team cost tracking.
• Tune throughput and latency: batching, quantization, KV and prompt caching, autoscaling, GPU capacity planning.

 

Agent runtime and harness

• Offer the agent harness as a platform service: sandboxed execution, session storage, permissions, headless runs in CI/CD and scheduled jobs.
• Extend harnesses with plugins, model adapters and tool registries so teams can run any platform model in them.
• Maintain a shared catalog of tools, MCP servers and skills that teams can reuse, with access control per team.
• Provide templates and an SDK for common agent patterns: single-agent, multi-agent, subagents, human-in-the-loop.

 

Shared NLP and RAG services

• Build retrieval as a service: ingestion, chunking, embedding, hybrid search, reranking, per-tenant indexes.
• Provide reusable NLP components: classification, information extraction, summarization, Vietnamese document processing.

 

Evaluation, observability and governance

• Build the eval platform: benchmark task sets, automated scoring, LLM-as-judge, regression gates before model or prompt releases.
• Provide tracing, logging and session replay for every model call and agent run.
• Enforce guardrails: prompt injection defenses, PII handling, tool permissions, audit logs.

• Bachelor's degree in Computer Science, Artificial Intelligence, Applied Mathematics or equivalent.
• 2+ years of NLP/LLM experience, including at least one LLM or agent system running in production.
• Experience building shared services or internal platforms used by other teams: API design, multi-tenancy, versioning, SLAs.
• Experience serving LLMs with vLLM or SGLang: deploying OpenAI-compatible endpoints on GPUs, tuning batching, quantization and KV cache, and benchmarking throughput and latency.
• Strong Python and backend skills (FastAPI or similar, async, queues); reading and writing TypeScript/Node.js is a big advantage, since many harnesses are written in TypeScript.
• Solid understanding of how agents work: agent loop, tool calling, context window management, prompt caching, structured output.
• Hands-on with at least one agent harness or coding agent (DeepSeek Harness, Codex or similar) at the customization level, not only as an end user.
• Experience building RAG and working with vector databases.
• Strong NLP fundamentals: Transformers, tokenization, embeddings, evaluation metrics.
• Solid infrastructure skills: Docker, Kubernetes, Linux, CI/CD, monitoring.

 

Nice to have

• You have built or operated an internal ML/AI platform or LLM gateway serving several product teams.
• You have written a plugin, MCP server or skill for an agent harness and have a public repo to show.
• GPU cluster operations: scheduling, autoscaling, multi-GPU and multi-node inference.
• Knowledge of agent safety: prompt injection, sandboxing, tool permissions.