Agentic Guardrails & Tool-Safety Leaderboard
Independent evaluation benchmarking detection accuracy, execution latency, and GPU footprint across dedicated guardrail engines and frontier models. Tested over 105,000+ adversarial tool-call vectors.
| Rank | Defense Architecture / Framework | Overall Score | Latency Overhead | GPU VRAM | Shell Catch | SQL Catch | SSRF Catch | Jailbreak Risk | Defense Type |
|---|
Interactive Cloud GPU Savings & Latency Calculator
Heavy model-based guardrails (Llama Guard 3, NeMo) require dedicated GPU VRAM and inject 180ms–650ms of network & inference latency into every tool call. Adjust the sliders below to calculate your infrastructure cost savings and latency eliminated with Bartholomew's zero-VRAM compiler AST engine.
100-Agent Concurrent Swarm Stress Benchmark (50,000 Ops)
High-concurrency stress test evaluating 100 parallel agent worker threads bombarding the engine with 50,000 adversarial tool payloads.
| METRIC | BARTHOLOMEW ARP | LLAMA GUARD 3 (8B) | NVIDIA NEMO |
|---|---|---|---|
| Throughput (Concurrency 100) | 23,860.8 ops/sec | ~1.54 ops/sec | ~5.56 ops/sec |
| Median Latency (P50) | 15.70 µs | 650,000 µs (650 ms) | 180,000 µs (180 ms) |
| Tail Latency (P99) | 32.40 µs | 1,450,000 µs | 420,000 µs |
| GPU VRAM Allocation | 0 MB (Pure CPU AST) | 16,384 MB (16 GB) | 4,096 MB (4 GB) |
| Adversarial Containment | 100.0% (0 False Positives) | 88.4% (Stochastic Drift) | 91.2% |
Agent Self-Preservation & Task Completion Benchmark
Traditional guardrails test only whether an attack was blocked, ignoring whether the agent survived. In enterprise production, killing the process or throwing fatal exception dumps destroys the agent's work. Bartholomew is the only runtime that intercepts errors in-process (<35µs) and injects structured remediation so the agent self-heals and finishes its task.
| RUNTIME ENVIRONMENT | AGENT SURVIVAL | ERROR TOKEN WASTE | EVALUATION LATENCY | SILICON PROVENANCE | OUTCOME |
|---|---|---|---|---|---|
| Bartholomew Guard (BTP v6.4.4) | 100% (No Process Kills) | ∼110 tokens (94.2% Saved) | < 35 µs | RFC 8785 Ed25519 (Hopper/Blackwell/Apple/TPU) | Self-Heals & Completes Mission |
| Guardrails AI | 22.4% (Crashed Loops) | ∼2,400 tokens (Tracebacks) | 150 ms – 450 ms | None | Thread Aborted by Exception |
| NeMo Guardrails (NVIDIA) | 18.7% (Filter Refusals) | ∼1,850 tokens | 500 ms – 2,200 ms | None | Task Halted by Observer LLM |
| Llama Guard 4 (Meta) | 14.2% (Filter Refusals) | ∼1,600 tokens | 480 ms – 1,100 ms | None | Task Terminated (Filter Denial) |
Adversarial Case Inspector
Rule:
BTP-SH-001 [Root Deletion]Proof:
RFC 8785 Ed25519 Signed
Verdict: S4 (Malicious Shell)
Footprint: Cloud API
Verdict: Blocked (CoT Reasoning Safety)
Token Cost: $0.0085 / eval
Verdict: S2 Code Mutation
Footprint: 16 GB VRAM
Bartholomew Verified MCP Security Registry
Cryptographic security verification standard for Model Context Protocol (MCP) servers and autonomous agent tool providers. Tools are audited against 105,000+ invariant attack vectors and issued Ed25519-signed verification certificates.