Add hybrid ranking and recency scoring

[dhruvi-16-me]

Summary

This PR improves the RAG retrieval ranking in Ell-ena by introducing hybrid ranking based on semantic similarity, recency, and entity-specific urgency.

Previously, retrieved tasks, tickets, and meetings were primarily ranked using vector similarity. This PR adds a hybrid ranking layer while keeping semantic relevance as the dominant signal.

What Changed

1. Hybrid RAG Ranking

Retrieved candidates are now ranked using:

final_score =
    0.70 × similarity
  + 0.20 × recency_score
  + 0.10 × urgency_score

Semantic similarity remains the strongest ranking signal to ensure that urgent but unrelated items cannot outrank strongly relevant results.

2. Similarity Threshold

A minimum similarity threshold of 0.30 is applied before recency and urgency scoring.

Updated retrieval flow:

Query
  ↓
Generate query embedding
  ↓
HNSW vector search
  ↓
Retrieve larger candidate pool
  ↓
Apply similarity threshold
  ↓
Calculate recency score
  ↓
Calculate urgency/priority score
  ↓
Calculate final hybrid score
  ↓
Sort by final_score
  ↓
Return top-k results

This ensures that recency and urgency only influence results that are already semantically relevant.

3. Recency Scoring

A decay-based recency score has been added:

recency_score = exp(-age_days / 21)

Newer items receive a higher score, while older items gradually lose influence.

The dates used depend on the entity:

• Meetings: meeting_date, with fallback timestamps where available
• Tasks: updated_at / created_at
• Tickets: updated_at / created_at

Future dates are handled safely by preventing negative age values.

4. Task Urgency

Tasks do not have a dedicated priority column, so urgency is derived from existing fields:

• due_date
• status

Higher urgency is given to:

• Overdue active tasks
• Tasks due soon
• Active/in-progress tasks

Completed tasks receive no urgency boost.

No new task priority column was introduced.

5. Ticket Priority

Ticket urgency uses the existing priority and status fields:

high > medium > low

Resolved tickets receive no urgency boost.

6. Meeting Ranking

Meetings are ranked using:

• Semantic similarity
• Recency

No artificial priority or urgency field was introduced for meetings.

7. Candidate Retrieval and Reranking

The existing HNSW cosine similarity search is preserved.

The updated pipeline:

1. Retrieves a larger bounded candidate pool
2. Filters weak matches using the similarity threshold
3. Calculates recency and urgency scores
4. Applies hybrid ranking
5. Returns the requested top-k results

Candidate and match counts are bounded to prevent unnecessarily large retrieval operations.

8. Ranking Metadata

Additional ranking metadata is returned where applicable:

• recency_score
• urgency_score
• final_score

Relevant entity metadata such as dates, status, and priority is also preserved where supported.

Existing return fields remain available for compatibility.

9. Flutter RAG Updates

Updated the RAG result and retrieval layers to support the new ranking system.

Changes include:

• Parsing hybrid ranking metadata in RagResult
• Maintaining backward compatibility with legacy result rows
• Passing the similarity threshold to the RPC
• Applying a client-side similarity filter as an additional safety layer
• Supporting bounded candidate retrieval

The implementation includes coverage for:

• Ranking metadata parsing
• Similarity threshold filtering
• Recency scoring
• Task urgency
• Ticket priority scoring
• Completed task handling
• Candidate and match count bounds
• High semantic relevance vs. high urgency

A key validation ensures that an unrelated but urgent item cannot outrank a strongly semantically relevant result.

✅ Checklist

• I have read the contributing guidelines.
• I have added tests that prove my fix is effective or that my feature works.
• I have added necessary documentation (if applicable).
• Any dependent changes have been merged and published in downstream modules.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (2)

• main
• develop

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: de222a1e-0a3a-458d-807e-f526ecf900b4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}