This repository contains scripts and configuration files to benchmark various AI gateways, supporting the article "AI Gateway Benchmark: Comparing Performance."
This benchmark compares the performance, scalability, and security of popular AI and API gateways that can be used for AI applications. The benchmarks measure key metrics including:
- Throughput (requests per second)
- Average latency (milliseconds)
The benchmark evaluates the following gateways:
- TrustGate (AI-specific gateway)
- KrakenD
- Tyk
- Kong
- Apache APISIX
Each gateway can be benchmarked independently. Make sure Docker is running and stop other gateways before testing a new one.
- Clone TrustGate
cd TrustGate
docker compose up -d redis postgres
./scripts/run_local.shOpen another terminal and run:
sudo chmod +x ./benchmark_trustgate.sh
./benchmark_trustgate.shdocker run -p 8080:8080 -v $PWD:/etc/krakend/ devopsfaith/krakend run --config /gateway/krakend.json
./benchmark_krakend.shgit clone https://github.com/TykTechnologies/tyk-gateway-docker
cd tyk-gateway-docker
docker-compose up -dCreate Proxy Rule
curl -X POST http://localhost:8080/tyk/apis/ \
-H "x-tyk-authorization: foo" \
-H "Content-Type: application/json" \
-d '{
"definition": {
"key": "version",
"location": "header"
},
"name": "Test API",
"api_id": "test-api",
"slug": "test",
"use_keyless": true,
"proxy": {
"listen_path": "/test/",
"target_url": "http://localhost:8080/hello",
"strip_listen_path": true
},
"version_data": {
"not_versioned": true,
"versions": {
"Default": {
"name": "Default"
}
}
},
"active": true
}'
curl -X GET http://localhost:8080/tyk/reload \
-H "x-tyk-authorization: foo"In another terminal, run:
./benchmark_tyk.shcurl -Ls https://get.konghq.com/quickstart | bashConfigure service
curl -i -s -X POST http://localhost:8001/services \
--data name=benchmark \
--data url='http://localhost:8000'Configure route
curl -i -X POST http://localhost:8001/services/benchmark/routes \
--data 'paths[]=/mock'Restart Docker Container
./benchmark_kong.shcurl -sL https://run.api7.ai/apisix/quickstart | shCreate Proxy Rule
curl -X PUT http://127.0.0.1:9180/apisix/admin/routes/1 \
-H "X-API-KEY: $admin_key" \
-H "Content-Type: application/json" \
-d '{
"methods": ["GET"],
"uri": "/hello",
"upstream": {
"type": "roundrobin",
"nodes": {
"127.0.0.1:9180": 1
}
},
"plugins": {
"proxy-rewrite": {
"uri": "/apisix/admin/routes/"
}
}
}'Run benchmark
./benchmark_apisix.shThe benchmarks use the hey HTTP load generator to simulate traffic to each gateway. Each test runs with 50 concurrent users for 30 seconds, measuring throughput and latency.
For detailed benchmark results and analysis, please refer to the accompanying article: "AI Gateway Benchmark: Comparing Performance."
Why isn't LiteLLM/Portkey/Helicone in this benchmark? This benchmark focuses on infrastructure-layer gateways with comparable data-plane architecture (Kong, Tyk, KrakenD, Apache APISIX). LLM-routing gateways solve a different problem — see our comparison articles for that category.
Isn't this biased since NeuralTrust builds TrustGate? The methodology, environment and scripts are public specifically so anyone can re-run it and check. Open an issue if your numbers differ.
How do I reproduce this on my own hardware? See Running the benchmarks — one command, documented test environment above.
MIT — NeuralTrust