Superlinked

Superlinked is an open-source platform designed to provide efficient and cost-effective inference for AI agents. It allows organizations to serve various AI models, including large language models (LLMs), optical character recognition (OCR), vision models, embeddings, and rerankers, directly from a single, self-managed cluster within their existing cloud infrastructure (AWS, GCP, Azure) or air-gapped environments. This approach ensures that sensitive data, such as prompts and documents, remains entirely within the user's control, addressing privacy and compliance concerns.

SIE operates as a Kubernetes inference cluster, featuring a stateless gateway that publishes work to a centralized queue. Worker pods then pull tasks, form full batches, and efficiently share GPU resources across multiple models. The system incorporates advanced features like worker pools, model stacking, and auto-scaling to dynamically manage workloads. A key innovation is its cluster-wide queue, which employs a "POOL-THEN-BATCH" strategy to maximize GPU utilization, achieving significantly higher efficiency compared to traditional router-then-batch methods.

Users can self-host SIE using Docker, Helm, or Terraform, with support for mirrored model snapshots for air-gapped installations. Superlinked also plans to offer a managed service with hosted capacity for selected projects. The platform includes an Agent plugin that integrates with existing agent stacks, offloading document processing tasks like parsing, extraction, summarization, and question answering. This plugin can also redact sensitive data before it reaches third-party models, further enhancing data security for agent-driven workflows.