Performance: Establish Comprehensive Benchmark Suite Against Ansible

[adolago]

Performance: Establish Comprehensive Benchmark Suite Against Ansible

Problem Statement

Rustible claims 5.9x performance improvements over Ansible, but lacks a comprehensive, automated benchmark suite to validate and track these claims. Without consistent benchmarking, performance regressions can go undetected and the claimed speedups cannot be independently verified.

Current State

• README claims 5.9x speedup for simple playbooks, 5.6x for file copies, 5.3x for templates
• Manual benchmarks exist in benches/ directory
• Homelab tests are ignored by default
• No CI/CD integration for performance regression testing
• Performance metrics are not published or tracked over time

Proposed Solution

Phase 1: Benchmark Infrastructure (v0.1.x)

1. Automate existing benchmarks

   • Integrate benches/* into CI/CD pipeline
   • Run benchmarks on every merge to main
   • Store results in benchmarks/results/ with timestamps

2. Create benchmark test matrix

   • Small: 5 hosts, 10 tasks
   • Medium: 25 hosts, 50 tasks
   • Large: 100 hosts, 200 tasks
   • Network conditions: local, LAN, WAN (simulated)

3. Add Ansible comparison runner

   • Automatically run same playbooks with ansible-playbook
   • Compare execution times side-by-side
   • Generate JSON report with speedup metrics

Phase 2: Performance Regression Detection (v0.2.x)

1. Baseline tracking

   • Store baseline metrics in benchmarks/baseline.json
   • Alert on >10% performance degradation
   • Track performance trends across releases

2. Flamegraph profiling

   • Generate flamegraphs for benchmark runs
   • Identify hot spots and optimization opportunities
   • Compare flamegraphs across commits

3. Memory profiling

   • Track memory usage during benchmarks
   • Detect memory leaks
   • Optimize memory allocation patterns

Phase 3: Homelab Validation (v0.2.x)

1. Enable homelab tests by default

   • Remove ignored attribute from homelab tests
   • Create test infrastructure using svr-host, svr-core, svr-nas
   • Document homelab setup in tests/infrastructure/README.md

2. Real-world scenario benchmarks

   • LAMP stack deployment
   • Kubernetes cluster configuration
   • Security hardening playbooks
   • Database cluster setup

3. Publish performance reports

   • Generate performance report as part of release process
   • Include in CHANGELOG
   • Publish to docs/performance/ directory

Expected Outcomes

• Automated benchmark suite running in CI/CD
• Performance regression detection
• Validated 5.9x speedup claim with reproducible evidence
• Performance trends tracked across releases
• Published performance metrics for each release

Success Criteria

• Benchmarks run automatically on every PR
• Performance regression alerts in CI/CD
• Homelab tests run on scheduled basis (daily/weekly)
• Performance report generated for each release
• Benchmark suite covers 80% of common use cases
• Performance metrics tracked over 10+ releases

Related Issues

• feat: Add SSH connection pooling for 11x performance improvement #2: Connection Pooling Optimization
• 🎨 Palette: Add visual indicators to task status output #3: Adaptive Parallelism Implementation
• chore: cleanup temporary documentation and reports #4: Caching System Validation

Additional Notes

This is a P0 (Critical) feature as performance is Rustible's main differentiator. Should be prioritized for v0.1.x release.