[Optimization]: Implement dynamic Context-Window Pruning and Token Truncation Middleware for Semantic Workspaces

[pushtikadia]

🚀 Architectural Optimization Request

Is your feature request related to a problem?

Currently, when the platform runs workspace search routines or parses extensive meeting files for semantic embedding mapping, large structural data aggregates are compiled directly into the LLM context wrapper payload. If a user queries a broad workspace containing sprawling multi-file histories, the raw data injection risks overflowing the downstream provider's maximum token context window capability.

Even if it doesn't overflow, passing repetitive, low-relevance structural data sequences creates context dilution, drives down summary accuracy, and spikes external API compute/token billing overhead.

Proposed Solution

Introduce an abstract context preprocessing utility layer. This middleware will dynamically rank retrieved textual chunks using a lightweight localized score filter (like BM25 or a localized Cosine Similarity threshold) and prune non-essential token sequences before compiling the final payload.

Proposed Implementation Details

1. Dynamic Content Scoring Layer: Implement a preprocessing step that reads structural context fragments and matches them against user intent vectors, assigning relevance scores to individual paragraphs.
2. Hard Token Limit Safeguard: Build a dynamic sliding window truncation loop. If aggregate chunks exceed a configured max boundary parameter (e.g., max_workspace_tokens: 4000), it gracefully drops lowest-scoring sections instead of letting the entire downstream API call crash with a payload error.
3. Debug Logger Tracking: Integrate a system tracking matrix that logs the raw compressed efficiency parameters (e.g., Tokens Reduced: X% | Latency Optimizations: Yms).

Additional Context

Implementing this optimization directly protects the application layer against massive enterprise-scale data inputs, drops average network payload weight, and saves operational API resource budgets.

[pushtikadia]

I have carefully reviewed the workspace parsing architecture in this repository. I would love to build out this structural context pruning middleware to optimize token usage under GSSoC'26. Please assign this issue to me!

[karrisanthoshigayatri]

Hi ,

I would like to work on this issue under GSSoC'26. The problem is well-defined, and I believe implementing a context-window pruning middleware would significantly improve scalability, token efficiency, and response quality for large workspaces.

Proposed Approach

1. Context Relevance Scoring

   • Introduce a preprocessing layer before LLM payload generation.
   • Score retrieved chunks using BM25 or embedding-based cosine similarity against the user query.
   • Rank chunks by relevance and remove low-priority content.

2. Dynamic Token Budget Management

   • Define a configurable token threshold (e.g., max_workspace_tokens).
   • Calculate cumulative token usage while assembling context.
   • Use a sliding-window strategy to retain the highest-ranked chunks and gracefully discard lower-ranked sections when the limit is exceeded.

3. Middleware Integration

   • Keep the solution provider-agnostic by implementing it as an independent middleware layer.
   • Ensure compatibility with existing workspace search and semantic retrieval pipelines.

4. Observability & Debug Metrics

   • Add logging for:

     • Original token count
     • Final token count
     • Percentage reduction
     • Context chunks removed
     • Processing overhead/latency

5. Configuration Support

   • Expose parameters such as token limits, similarity thresholds, and pruning strategies through configuration to allow future tuning.

This approach should prevent context-window overflows, reduce token costs, improve retrieval precision, and provide better performance for large enterprise workspaces.

Please assign this issue to me. I’d be happy to contribute and submit a well-tested implementation.

Thank you!

[pushtikadia]

@karrisanthoshigayatri
I am currently working on this task

[muntasir-396]

Hi @pushtikadia,

I've submitted a PR implementing the context-window pruning middleware described in Issue #278.

The implementation includes:

• Context preprocessing middleware
• Query-aware relevance scoring
• Token-budget enforcement
• Sliding-window pruning
• Debug metrics

I'd appreciate your review whenever you have time. Thank you!