- Dify
- Agent platform
Dify Enterprise on-premises deployment: Kubernetes architecture and capacity planning in practice
By Apolla delivery team3 min read
Key takeaway
A typical configuration for Dify Enterprise on Kubernetes supporting 3,000 concurrent users uses three node pools: a serving pool of 16C/32G × 4, an async task pool of 32C/64G × 4 and an infrastructure pool of 8C/16G × 2, plus PostgreSQL, Redis, object storage and a vector database. In load testing, the first bottleneck was not model calls but the async task queue for document parsing and the write throughput of the vector database.
Why separate node pools
Dify's components have very different load profiles. API and Web serve online requests and are latency-sensitive. Workers handle document parsing, vectorization and batch jobs, which are CPU- and memory-intensive but can queue. The database and cache need stable IO. Put them all on the same nodes and a single large knowledge base import can take down live conversations.
Configuring the three node pools
- Serving pool, 16C/32G × 4: api, web, plugin-daemon, sandbox; autoscale at 60% CPU
- Async task pool, 32C/64G × 4: worker (parsing, vectorization, batch jobs); scale on queue length
- Infrastructure pool, 8C/16G × 2: Ingress, monitoring, log collection; tainted so business components are never scheduled here
- 3000users
- Target concurrent user scale
- 10nodes
- Across the three node pools
- <2s
- P95 time to first token under load
Apolla delivery measurements (a power group's domestic IT stack environment)
Middleware choices
Deploy PostgreSQL separately with read/write splitting, and reserve connections at 1.5 times api replicas × connection pool limit. Use Redis as the Celery broker and as cache, on separate instances so a task backlog cannot affect session cache. Use an S3-compatible interface for object storage. For the vector database, pgvector is enough below 100,000 documents; migrate to a dedicated vector database beyond that.
Three bottlenecks exposed by load testing
First, a bulk import of 2,000 PDFs left the Worker queue backed up for more than 40 minutes. The cause was a low default concurrency for parsing tasks; raising Worker concurrency from 4 to 16 and isolating the parsing queue brought it down to 6 minutes. Second, vector database write throughput became the bottleneck during concurrent imports; switching to batch writes improved it roughly 5 times. Third, connection reuse on the model gateway was off, exhausting TCP connections under high concurrency; enabling keep-alive made the problem disappear.
The item that gets overlooked
The sandbox component allows outbound network access by default. In production, tighten the network policy to allow only the internal addresses you need. Otherwise the code execution tool becomes a data exfiltration channel.
Extra work for China's domestic IT stack
On Kylin V10 ARM64 + Dameng + containerd (a typical domestic IT stack, or xinchuang, environment in China), every ARM64 image has to be rebuilt, and 3 native modules in the dependency tree must be compiled from source. Dameng's support for JSONB and array types differs from PostgreSQL and needs adaptation at the ORM layer. This work is roughly 30% of the total deployment effort and deserves its own line in the plan.