Protect benches - #1
Merged
Merged
Conversation
- Connect to template1 when dropping postgres database to avoid 'cannot drop currently open database' error - Replace test tables with integer_plaintext and integer_encrypted tables - Use cs_encrypted_v2 type for encrypted column Co-Authored-By: Warp <agent@warp.dev>
- Add anyhow dependency for better error handling - Create generate_data binary to insert random integers into integer_plaintext table - Uses sqlx for database operations - Requires DATABASE_URL environment variable - Supports configurable NUM_RECORDS (default 10,000) - Batches inserts for efficiency - Fix encrypted column type to eql_v2_encrypted - Add IF NOT EXISTS to CREATE TABLE statements - Remove data loading step from setup-db task - Add psql task to connect to Docker-managed database Co-Authored-By: Warp <agent@warp.dev>
- Updated dependencies in Cargo.toml for enhanced features. - Introduced new Docker Compose setup for PostgreSQL. - Added `encrypt_data` binary for encrypting integers using CipherStash. - Improved `generate_data` binary with better error handling and logging. - Refactored `exact_lookup` benchmark for cleaner code and better readability. - Created a new `lib.rs` for ingest options management.
- Updated `sqlx` version from 0.6 to 0.8 in `Cargo.toml`. - Modified various dependencies in `Cargo.lock` to their latest versions, including `event-listener`, `atoi`, and `base64`. - Added new dependencies such as `concurrent-queue`, `const-oid`, and `flume`. - Updated performance metrics in JSON result files for encryption tests, reflecting changes in execution time and memory usage for different record sizes. - Adjusted timestamps in JSON files to reflect the latest test runs.
…ndling; add preparation script for string_encrypted table
…eparation script for table handling; add SQL index management for integer_encrypted
…ables - Added benchmark results for various operations on encrypted data, including ORE (Order-Revealing Encryption) benchmarks for different data sizes. - Created SQL scripts to drop and create indexes for integer and string encrypted tables with various sizes (10,000 to 10,000,000). - Updated schema to include new tables for integer and string encrypted data. - Modified encryption scripts to dynamically set table names based on an environment variable for better flexibility.
- Updated benchmark result files for MATCH queries with new metrics and improved estimates. - Added a new binary `combine_benchmark` to consolidate hyperfine benchmark results with system metadata. - Enhanced documentation in `encrypt_int`, `encrypt_json_large`, `encrypt_json_small`, and `generate_data` binaries to clarify functionality and environment variable requirements. - Improved comments to reflect the current state of the CipherStash client API and its usage in the encryption processes.
…ng data encryption
coderdan
force-pushed
the
protect-benches
branch
from
January 19, 2026 05:33
52e7da1 to
8369184
Compare
coderdan
added a commit
that referenced
this pull request
Jul 9, 2026
… + field_gt Addresses PR #23 review (James's items #1 and #2): - setup-db-v3 now installs the pinned eql-3.0.0-alpha.3 release bundle instead of building from a local eql_v3 checkout. alpha.3 moved the per-domain types eql_v3.* -> public.* and removed jsonb_query, so the whole v3 surface is migrated to public.<type> (schema_v3.sql, sql/indexes, *_v3 benches, encrypt binaries, src/v3.rs); eql_v3.* FUNCTIONS (eq_term/ord_term/match_term/ord_ope_term/ore_cllw/jsonb_*) are unchanged. - Verified no bench/index/schema calls a moved eql_v3_internal extractor (hmac_256/bloom_filter/ore_block_256/ope_cllw) directly — item #2. JSON bench now queries through the named EQL JSON functions rather than raw jsonb @>/-> operators: - contains/extractor: eql_v3.jsonb_contains(value, $1) on GIN(eql_v3.jsonb_array(value)) - field access: eql_v3.jsonb_path_query_first(value, sel) feeding eql_v3.eq_term / eql_v3.ore_cllw - new scenario json/field_gt/functional: the encrypted `x->'y' > 10` range query -- `>` on public.jsonb_entry inlines to ore_cllw compare, reusing field_order_idx (no new index). Re-ingested + re-ran the JSON tiers 10k-1M-10M against alpha.3 in an isolated DB and regenerated report/V3_COMPARISON.md + report/BENCHMARK_REPORT.md. contains/functional @10m improves 3.36ms -> 1.05ms (jsonb_array recipe). Non-JSON v3 numbers remain the alpha.2-equivalent baseline pending a full re-baseline (flagged by a "Re-baseline status" banner in the report).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Automated Benchmark Reporting System
Overview
This PR adds a comprehensive automated reporting system for benchmark results, improving documentation, and organizing generated artifacts. The changes transform raw benchmark data into readable, shareable reports with visualizations.
🎯 Key Features
• Python-based report generator (report_benchmarks.py) that processes benchmark results⚠️ ) for queries exceeding 100ms
• Generates markdown reports with:
◦ Ingest throughput tables and charts
◦ Query performance tables across multiple data set sizes
◦ SQL queries and parameters for each benchmark
◦ Database index definitions
◦ Performance indicators (
• Optional chart generation using matplotlib
• Created report/ directory for all generated artifacts
• Reports and charts are now tracked in git for historical comparison
• Cleaner repository root with separated source and generated files
• Main README.md: Complete project documentation covering:
◦ How to view benchmark results
◦ Detailed test setup information
◦ Step-by-step instructions for running benchmarks
◦ Troubleshooting guide
◦ Project structure overview
• README_REPORT.md: Detailed documentation for the report generator
• report/README.md: Documentation for the report directory
• MIT License added
The report generator includes:⚠️ ) for slow queries (>100ms)
• SQL Query Display: Shows actual SQL statements used in benchmarks
• Query Parameters: Displays the parameter values passed to each query
• Index Definitions: Includes CREATE INDEX statements from sql/indexes/
• Performance Warnings: Visual indicators (
• Charts: Bar charts for ingest throughput, line charts for query performance
• Table of Contents: Easy navigation within reports
Added mise run report task for easy report generation:
• Fixed inaccurate header comments in all src/bin/ files
• Updated documentation to reflect actual functionality
• Removed outdated TODO comments
• Clarified environment variable usage