niemyjski · Oct 14, 2026 · Oct 5, 2026 · Oct 5, 2026 · Oct 6, 2026 · Oct 6, 2026
diff --git a/.agents/skills/cross-repo-validation/SKILL.md b/.agents/skills/cross-repo-validation/SKILL.md
@@ -0,0 +1,56 @@
+---
+name: cross-repo-validation
+description: Validate a Foundatio core change across the provider repos (Redis, AWS, Azure Storage, Azure Service Bus, RabbitMQ, Kafka, Minio, Aliyun, SshNet). Use when changing core abstractions, QueueBase/CacheClient/MessageBus behavior, Foundatio.TestHarness base classes, or shared docs, or when the user asks to build, test, push or update PRs across repos.
+---
+
+# Cross-repo validation
+
+Core changes are only done when every affected provider builds and passes its tests against the local core source. Run this workflow without being asked whenever a change touches `src/Foundatio`, `src/Foundatio.TestHarness`, or provider-facing docs.
+
+## 1. Find the affected repos
+
+- TestHarness base class signature changes (for example a new `GetQueue` parameter) are breaking: every provider override must be updated. Search each sibling repo for the overridden member: `rg -n "override .*GetQueue\(" ../Foundatio.*/tests`.
+- New `virtual` base tests need a `[Fact]`/`[Theory]` override in every provider test class that derives from the base, or they won't run there. Skip only with a concrete reason (`[Fact(Skip = "Legacy mode does not support GroupId")]`).
+
+## 2. Build against local core, once and sequentially
+
+```powershell
+$env:ReferenceFoundatioSource = 'true'
+foreach ($r in 'Foundatio.Redis','Foundatio.AWS','Foundatio.AzureStorage','Foundatio.AzureServiceBus') {
+    Push-Location "../$r"; dotnet build "$r.slnx"; Pop-Location
+}
+```
+
+Provider builds compile core into the same `obj` folders, so never build two provider solutions at the same time.
+
+## 3. Start the emulators and wait until they're healthy
+
+- Run `docker compose up -d` in each provider repo and wait for `docker ps` to report `healthy` (the Service Bus emulator takes 1-2 minutes).
+- After editing `ServiceBus-Emulator/Config/Config.json`, run `docker compose down` and then `docker compose up -d`. The emulator only reads its config at startup.
+- If nearly every test in a repo fails after the same ~5 seconds, the service isn't reachable. Fix the environment before you touch any code.
+- Redis cluster on macOS: start it with `$env:HOST_IP = (ipconfig getifaddr en0); docker compose up -d`. Without `HOST_IP`, the nodes advertise `172.17.0.1`, which a Mac can't reach. `start-services.ps1` can't work out the IP on macOS.
+- These test projects use xUnit v3 on Microsoft.Testing.Platform, where `--filter "FullyQualifiedName~X"` matches nothing. Use `--filter-method "*X*"` instead.
+
+## 4. Run the provider tests in parallel
+
+Give each repo (or pair of small repos) its own subagent so they run at the same time. Tell every subagent to:
+
+- Set `$env:ReferenceFoundatioSource='true'` and run `dotnet test <test csproj> --no-build`.
+- After an edit, rebuild only the projects it changed, with `dotnet build <csproj> --no-dependencies`. It must never rebuild core or a whole `.slnx`.
+- Edit only its own repo, and report core issues back instead of fixing them.
+- Leave git alone: no commit, push, stash, checkout or reset.
+- Return the files it changed, the total/passed/failed/skipped counts, and each remaining failure, re-run once to separate flaky from real.
+
+Meanwhile, run core yourself: `dotnet build Foundatio.slnx` and `dotnet test Foundatio.slnx`.
+
+## 5. Pre-push checks
+
+- Put a blank line between test methods. This finds misses in changed files: `rg -n -U '\}\r?\n[ \t]*\[(Fact|Theory)' <files>`.
+- Check every changed doc statement against the code: defaults on the options class (not the builder), clamping vs. rejecting, and whether a provider actually supports the feature. Re-read `git diff origin/main...HEAD -- docs .agents`.
+- When you change core abstractions, update `.agents/skills/foundatio/SKILL.md`.
+
+## 6. Push and update the PRs
+
+1. Push core first. CI publishes `<version>-preview.<branch>.0.<height>`.
+2. Update the `Foundatio` and `Foundatio.TestHarness` package versions in each provider's `src/*/*.csproj` and `tests/Directory.Build.props` to the new preview.
+3. Push the providers, then update the PR descriptions. Call out breaking changes (TestHarness signatures) and behavior changes.
diff --git a/.agents/skills/foundatio/SKILL.md b/.agents/skills/foundatio/SKILL.md
@@ -111,6 +111,10 @@ await _cache.RemoveByPrefixAsync("user:");
 ```csharp
 await _queue.EnqueueAsync(new OrderWorkItem { OrderId = orderId });
 
+// Multi-tenant: tag messages with a group id (SQS fair queues / FIFO ordering)
+await _queue.EnqueueAsync(workItem, new QueueEntryOptions { GroupId = tenantId });
+// or derive it from the payload once: new SQSQueue<T>(o => o.GroupId(x => x.TenantId))
+
 var entry = await _queue.DequeueAsync(TimeSpan.FromSeconds(5));
 if (entry is not null)
 {
@@ -312,7 +316,9 @@ public class OrderServiceTests : TestLoggerBase
 - **JobContext.RenewLockAsync**: Call in long-running jobs (both `JobBase` and `QueueJobBase`) to prevent lock expiration mid-processing.
 - **Register as singletons**: All infrastructure services (`ICacheClient`, `IMessageBus`, `IQueue<T>`, `IFileStorage`, `ILockProvider`) maintain internal state and connections -- always register as singletons.
 - **CacheLockProvider + IMessageBus**: `IMessageBus` is optional but recommended. Without it, lock release falls back to polling. With it, locks are released instantly via pub/sub notification.
-- **In-memory for tests**: All in-memory implementations are functionally equivalent to production providers. Swap via DI for fast, isolated unit tests with no external dependencies.
+- **In-memory for tests**: In-memory implementations follow the same interfaces as production providers, but some behaviors differ (delivery delay limits, ordering, deduplication, group ids). Swap via DI for fast, isolated unit tests, and check [provider behavioral gaps](https://foundatio.dev/guide/provider-behavioral-gaps) before relying on provider-specific behavior.
+- **Queue `GroupId` is a hint**: It only changes delivery on SQS (fair queues on standard queues, strict ordering on `.fifo` queues) and session-enabled Azure Service Bus queues (ordered per group, one message at a time per group; delayed retries rejoin the back of the group). The other built-in queues store it as metadata and log one debug message. SQS group ids are max 128 characters with no spaces. Empty values are stored as `null`. A non-empty explicit `QueueEntryOptions.GroupId` overrides the queue's `GroupId(...)` resolver; null or empty invokes the resolver. Resolvers should return `null` for no group.
+- **`EnqueueAsync` copies `QueueEntryOptions`**: The caller's instance is never modified and can be reused. `Enqueuing` handlers get the per-message copy, with the resolved `GroupId` and the `Activity.Current` `CorrelationId`/`TraceState` already filled in. `Properties` is copied into a writable `Dictionary` per enqueue, keeping the key comparer of built-in dictionary types (`Dictionary`, `ConcurrentDictionary`, sorted and immutable dictionaries); other implementations use the default comparer.
 
 ## NuGet Packages
 

diff --git a/AGENTS.md b/AGENTS.md
@@ -178,6 +178,7 @@ Before marking work complete, verify:
 7. **Feature documentation**: Add entries to [docs/](docs/) folder for new features or significant changes
 8. **Skill updated**: When changing core abstractions, interfaces, or docs, update [`.agents/skills/foundatio/SKILL.md`](.agents/skills/foundatio/SKILL.md) to keep patterns, gotchas, and the implementation matrix in sync. The skill delegates to context7 for full docs, so only compact patterns and non-obvious gotchas need maintenance.
 9. **Breaking changes flagged**: Clearly identify any breaking changes for review
+10. **Providers validated**: For changes to core abstractions or `Foundatio.TestHarness`, follow [cross-repo-validation](.agents/skills/cross-repo-validation/SKILL.md) to build and test each affected provider repo against local core, using parallel subagents
 
 ### Error Handling
 
@@ -328,6 +329,7 @@ The following skills are available in this repository under `.agents/skills/`:
 | Skill | Description |
 | ----- | ----------- |
 | [foundatio](.agents/skills/foundatio/SKILL.md) | Foundatio infrastructure abstractions — caching, queuing, messaging, file storage, locking, jobs, resilience |
+| [cross-repo-validation](.agents/skills/cross-repo-validation/SKILL.md) | Build and test provider repos against local core in parallel, emulator setup, preview bumps and PR updates. Use automatically for core or TestHarness changes |
 | [analyzing-dotnet-performance](.agents/skills/analyzing-dotnet-performance/SKILL.md) | Scans .NET code for ~50 performance anti-patterns across async, memory, strings, collections, LINQ, regex, serialization, and I/O |
 | [microbenchmarking](.agents/skills/microbenchmarking/SKILL.md) | Creating, running, configuring, or reviewing BenchmarkDotNet benchmarks |
 | [dotnet-trace-collect](.agents/skills/dotnet-trace-collect/SKILL.md) | Capturing diagnostic artifacts to diagnose production .NET performance issues |

diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts
@@ -55,6 +55,7 @@ export default defineConfig({
           items: [
             { text: 'Configuration', link: '/guide/configuration' },
             { text: 'Dependency Injection', link: '/guide/dependency-injection' },
+            { text: 'Provider Behavioral Gaps', link: '/guide/provider-behavioral-gaps' },
             { text: 'Resilience', link: '/guide/resilience' },
             { text: 'Serialization', link: '/guide/serialization' }
           ]

diff --git a/docs/guide/implementations/aws.md b/docs/guide/implementations/aws.md
@@ -117,11 +117,9 @@ AWS SQS queue implementation for reliable work item processing.
 ```csharp
 using Foundatio.Queues;
 
-var queue = new SQSQueue<WorkItem>(o =>
-{
-    o.ConnectionString = connectionString;
-    o.Name = "work-items";
-});
+var queue = new SQSQueue<WorkItem>(o => o
+    .ConnectionString(connectionString)
+    .Name("work-items"));
 
 await queue.EnqueueAsync(new WorkItem { Data = "Hello" });
 var entry = await queue.DequeueAsync();
@@ -131,15 +129,52 @@ var entry = await queue.DequeueAsync();
 
 | Option | Type | Required | Default | Description |
 |--------|------|----------|---------|-------------|
-| `Name` | `string` | ✅ | | Queue name |
+| `Name` | `string` | ✅ | | Queue name (a `.fifo` suffix selects a FIFO queue) |
 | `ConnectionString` | `string` | | | Connection string |
 | `Region` | `RegionEndpoint` | | | AWS region |
+| `Credentials` | `AWSCredentials` | | | AWS credentials |
+| `ServiceUrl` | `string` | | | Custom endpoint (e.g. Floci) |
 | `CanCreateQueue` | `bool` | | `true` | Auto-create queue |
-| `SupportDeadLetter` | `bool` | | `true` | Enable DLQ support |
+| `SupportDeadLetter` | `bool` | | `true` | Create and use a `-deadletter` queue |
 | `ReadQueueTimeout` | `TimeSpan` | | 20s | Long polling timeout |
+| `DequeueInterval` | `TimeSpan` | | 1s | Interval between dequeue attempts when the queue is empty |
+| `RetryDelay` | `Func<int, TimeSpan>` | | Exponential | Visibility delay before an abandoned message is retried |
+| `SqsManagedSseEnabled` | `bool` | | `false` | Enable SQS managed encryption (SSE-SQS) |
+| `KmsMasterKeyId` | `string` | | | KMS key ID for encryption (SSE-KMS) |
+| `KmsDataKeyReusePeriodSeconds` | `int` | | | KMS data key reuse period, in seconds |
 
 For additional options, see [SQSQueueOptions source](https://github.com/FoundatioFx/Foundatio.AWS/blob/main/src/Foundatio.AWS/Queues/SQSQueueOptions.cs).
 
+### Fair Queues and Message Groups
+
+SQS [fair queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html) stop one noisy tenant from delaying everyone else in a shared standard queue. They are enabled by sending a `MessageGroupId` (your tenant or customer id) with each message. Foundatio exposes this as `QueueEntryOptions.GroupId`:
+
+```csharp
+// Derive the tenant from the payload once...
+var queue = new SQSQueue<ImportRequest>(o => o
+    .ConnectionString(connectionString)
+    .Name("imports")
+    .GroupId(r => r.TenantId));
+
+await queue.EnqueueAsync(importRequest);
+
+// ...or set it per call
+await queue.EnqueueAsync(importRequest, new QueueEntryOptions { GroupId = "tenant-123" });
+```
+
+Things to know:
+
+- **A standard queue is enough.** No queue setting and no consumer change are needed, and messages in the same group can still be processed in parallel. `GroupId` on a standard queue does not provide ordering.
+- **FIFO queues use the same field for ordering.** On a `.fifo` queue, messages in a group are delivered one at a time in order, and SQS requires a group id on every message. FIFO queues reject a positive per-message `DeliveryDelay`, and an abandoned message with a `RetryDelay` holds back the rest of its group until it is retried.
+- **Group ids are limited to 128 characters** of alphanumerics and punctuation (no spaces).
+- **Set it on every message.** SQS treats a message without a group id as its own tenant.
+- **Consumer concurrency matters.** SQS marks a tenant as noisy when it holds more than 10% of in-flight messages (with at least 30 in flight) or more than 10% of recent processing time. `SQSQueue` receives one message per worker loop, so run enough workers or instances for the in-flight share to be meaningful.
+- **Monitor with CloudWatch.** Compare `ApproximateNumberOfMessagesVisible` with the `...InQuietGroups` metrics to confirm quiet tenants are not affected.
+
+### Dead Letter Behavior
+
+`SQSQueue` creates a `<name>-deadletter` queue with a native SQS redrive policy, and moves a message there itself once `Retries` is exceeded. FIFO queues use `<name>-deadletter.fifo`, because AWS requires the dead-letter queue of a FIFO queue to be a FIFO queue ([CreateQueue](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_CreateQueue.html)). Without a dead-letter queue, the message is deleted once `Retries` is exceeded. If the dead-letter queue can't be resolved, the error is logged and the message stays on the source queue to be retried. `GetDeadletterItemsAsync` is not supported for SQS; read the dead-letter queue directly.
+
 ## Next Steps
 
 - [File Storage Guide](/guide/storage) - Usage patterns

diff --git a/docs/guide/implementations/azure.md b/docs/guide/implementations/azure.md
@@ -92,7 +92,7 @@ await queue.EnqueueAsync(new WorkItem { Data = "Hello" });
 | Option | Type | Required | Description |
 |--------|------|----------|-------------|
 | `ConnectionString` | `string` | ✅ | Service Bus connection string |
-| `RequiresSession` | `bool?` | | Enable sessions for ordered processing |
+| `RequiresSession` | `bool?` | | Create the queue with sessions enabled. Each message then needs a `GroupId` (sent as `SessionId`), and messages are processed one at a time, in order, per group |
 | `RequiresDuplicateDetection` | `bool?` | | Enable duplicate detection |
 | `EnableDeadLetteringOnMessageExpiration` | `bool?` | | DLQ on expiration |
 

diff --git a/docs/guide/provider-behavioral-gaps.md b/docs/guide/provider-behavioral-gaps.md
@@ -90,9 +90,33 @@ This document catalogs known behavioral differences across Foundatio provider im
 |----------|-----------|-------|
 | InMemory | ✅ | Uses provided ID as the entry ID |
 | Redis | ✅ | Uses provided ID as the entry ID |
-| Azure Service Bus | ✅ | Maps to MessageId |
+| Azure Service Bus | ✅ | Maps to MessageId (deduplicated only when `RequiresDuplicateDetection` is enabled) |
 | Azure Storage Queue | ❌ | Azure assigns its own MessageId |
-| SQS | ❌ | SQS assigns its own MessageId |
+| SQS | ⚠️ | SQS assigns its own MessageId. On FIFO queues the value is sent as `MessageDeduplicationId`; on standard queues it is not sent |
+
+### QueueEntryOptions.GroupId
+
+| Provider | Effect on delivery | Notes |
+|----------|--------------------|-------|
+| InMemory | ❌ | Stored and returned on `entry.GroupId` only |
+| Redis | ❌ | Stored in the payload envelope and returned on `entry.GroupId` only |
+| Azure Storage Queue | ❌ | Stored in the payload envelope (default compatibility mode); not stored in legacy mode |
+| Azure Service Bus | ⚠️ | Sent as `SessionId`. On session-enabled queues (`RequiresSession`), ordered per group, one message at a time per group; ignored otherwise |
+| SQS standard queue | ✅ | Sent as `MessageGroupId` to enable fair queues (noisy neighbor mitigation). No ordering |
+| SQS FIFO queue | ✅ | Sent as `MessageGroupId`. Strict ordering within a group; required by SQS |
+
+Queues that do not use the value for delivery log a single debug message the first time a group id is enqueued. See [Message Groups](/guide/queues#message-groups).
+
+### Queue Ordering Guarantees
+
+| Provider | Ordering | Notes |
+|----------|----------|-------|
+| InMemory | ✅ | FIFO within the queue |
+| Redis | ✅ | FIFO list; retried entries are re-queued |
+| Azure Storage Queue | ⚠️ | Best effort |
+| Azure Service Bus | ⚠️ | Best effort; concurrent workers and retries can reorder processing. On session-enabled queues, ordered per `GroupId`, except delayed retries, which rejoin the back of the group |
+| SQS standard queue | ❌ | Best effort. `GroupId` does not change ordering |
+| SQS FIFO queue | ✅ | Strict ordering within a `GroupId` |
 
 ### Delivery Delay (DelayUntilUtc)
 
@@ -102,7 +126,21 @@ This document catalogs known behavioral differences across Foundatio provider im
 | Redis | ❌ | — | Not supported |
 | Azure Storage Queue | ✅ | Second | `VisibilityTimeout` parameter |
 | Azure Service Bus | ✅ | Second | `ScheduledEnqueueTimeUtc` |
-| SQS | ✅ | Second (0-900s max) | `DelaySeconds` parameter |
+| SQS | ⚠️ | Second | `DelaySeconds`, clamped to 900s. FIFO queues reject a positive delay |
+
+### Retry Delay on Abandon
+
+| Provider | Mechanism | Delivery | Notes |
+|----------|-----------|----------|-------|
+| InMemory | Re-added after a timer | At-least-once | Pending retries are lost if the process exits |
+| Redis | Wait list | At-least-once | Moved back by queue maintenance |
+| Azure Storage Queue | Visibility timeout on the same message | At-least-once | Message ID is unchanged |
+| Azure Service Bus | Scheduled copy of the message | At-least-once when duplicate detection is disabled | Sent before the original is completed, so the original can be redelivered alongside the retry if completion fails. The copy keeps `MessageId`, `CorrelationId`, `SessionId` (`GroupId`) and properties, and joins the back of its session |
+| SQS | Visibility timeout on the same message | At-least-once | Message ID is unchanged. On FIFO queues, later messages in the group wait for the retry |
+
+Handlers should be idempotent with every provider; see [message loss and duplicates](https://learn.microsoft.com/azure/service-bus-messaging/service-bus-message-loss-and-duplicates) for the Service Bus guidance.
+
+Service Bus delayed retries reuse the original `MessageId`. On a queue with [duplicate detection](https://learn.microsoft.com/azure/service-bus-messaging/duplicate-detection) enabled, a scheduled retry sent within the duplicate history window is acknowledged but discarded. The original is then completed, so no retry remains. Sending before completion does not prevent this existing limitation; use immediate abandon (`RetryDelay` returning `TimeSpan.Zero`) when duplicate detection is required.
 
 ---
 
@@ -169,3 +207,5 @@ All tested providers (InMemory, Redis) exhibit fully consistent behavior. No beh
 4. **Don't rely on `GetDeadletterItemsAsync`** — only InMemory supports it. Design deadletter processing around provider-specific mechanisms instead.
 
 5. **Prefer `QueueEntryOptions.UniqueId` only with providers that support it** (InMemory, Redis, Azure Service Bus). On SQS and Azure Storage Queues, the system assigns its own IDs.
+
+6. **Treat `QueueEntryOptions.GroupId` as a hint, not a guarantee.** It only changes delivery on SQS (fair queues on standard queues, ordering on FIFO) and session-enabled Service Bus queues (ordering per group); the other built-in queues store it as metadata.