Skip to content

Archiver Observability#3353

Merged
iancooper merged 20 commits into
masterfrom
arch_obser
Nov 6, 2024
Merged

Archiver Observability#3353
iancooper merged 20 commits into
masterfrom
arch_obser

Conversation

@iancooper

Copy link
Copy Markdown
Member

We want to add observability to the archiver.

  • We call ExternalServiceBus.Archive from the OutboxArchiver. An interesting question here is: should we begin the trace in OutboxArchiver or within ExternalServiceBus. in this case we choose ExternalServiceBus, because it is the last possible moment that the span could begin, but is possible that we could argue to place in OutboxArchiver as the initiator.
  • There are no OTel semantic conventions for this, but we follow the same pattern as elsewhere: create span for the batch; archive batch for each item; use links when available to link the child spans back to the parent.

@iancooper iancooper added 2 - In Progress feature request v10 .NET Pull requests that update .net code labels Oct 15, 2024
@iancooper iancooper self-assigned this Oct 15, 2024

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: OK

Change in average Code Health of affected files: +0.10 (8.09 -> 8.19)

  • Declining Code Health: 1 findings(s) 🚩

View detailed results in CodeScene

CommandProcessorSpanOperation.Publish => "publish",
CommandProcessorSpanOperation.Send => "send",
CommandProcessorSpanOperation.Clear => "clear",
CommandProcessorSpanOperation.Archive => "archive",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ New issue: Complex Method
ToSpanName has a cyclomatic complexity of 9, threshold = 9

Suppress

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: FAILED

Change in average Code Health of affected files: -0.04 (8.15 -> 8.12)

  • Declining Code Health: 4 findings(s) 🚩
  • Improving Code Health: 1 findings(s) ✅
  • Affected Hotspots: 1 files(s) 🔥

View detailed results in CodeScene

Comment on lines +287 to +292

var span = Tracer?.CreateDbSpan(
new OutboxSpanInfo(DbSystem.Brighter, InMemoryAttributes.DbName, OutboxDbOperation.DispatchedMessages, InMemoryAttributes.DbTable),
requestContext?.Span,
options: _instrumentationOptions
);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ New issue: Code Duplication
The module contains 4 functions with similar structure: DispatchedMessages,Get,MarkDispatched,OutstandingMessages

Suppress

IAmABoxTransactionProvider<CommittableTransaction>? transactionProvider = null,
CancellationToken cancellationToken = default)
{
//NOTE: As we call Add, don't create telemetry here

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Getting better: Missing Arguments Abstractions
The average number of function arguments decreases from 4.63 to 4.47, threshold = 4.00

IAmABoxTransactionProvider<CommittableTransaction>? transactionProvider = null,
CancellationToken cancellationToken = default)
{
//NOTE: As we call Add, don't create telemetry here

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ New issue: Primitive Obsession
In this module, 30.3% of all function arguments are primitive types, threshold = 30.0%

Suppress


//assert
_exportedActivities.Count.Should().Be(7);
_exportedActivities.Count.Should().Be(8);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Getting worse: Complex Method
When_Clearing_A_Message_A_Span_Is_Exported increases in cyclomatic complexity from 45 to 49, threshold = 9

Suppress

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: FAILED

Change in average Code Health of affected files: -0.05 (8.15 -> 8.11)

  • Declining Code Health: 5 findings(s) 🚩
  • Improving Code Health: 1 findings(s) ✅
  • Affected Hotspots: 1 files(s) 🔥

View detailed results in CodeScene

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: FAILED

Change in average Code Health of affected files: -0.07 (8.39 -> 8.32)

  • Declining Code Health: 6 findings(s) 🚩
  • Improving Code Health: 1 findings(s) ✅
  • Affected Hotspots: 1 files(s) 🔥

View detailed results in CodeScene


//assert
_exportedActivities.Count.Should().Be(7);
_exportedActivities.Count.Should().Be(8);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Getting worse: Complex Method
When_Clearing_A_Message_A_Span_Is_Exported increases in cyclomatic complexity from 45 to 49, threshold = 9

Suppress

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: FAILED

Change in average Code Health of affected files: -0.05 (8.29 -> 8.24)

  • Declining Code Health: 8 findings(s) 🚩
  • Improving Code Health: 1 findings(s) ✅
  • Affected Hotspots: 2 files(s) 🔥

View detailed results in CodeScene

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: FAILED

Change in average Code Health of affected files: -0.05 (8.29 -> 8.24)

  • Declining Code Health: 8 findings(s) 🚩
  • Improving Code Health: 1 findings(s) ✅
  • Affected Hotspots: 2 files(s) 🔥

View detailed results in CodeScene

@iancooper iancooper marked this pull request as ready for review October 27, 2024 22:59

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: FAILED

Change in average Code Health of affected files: -0.03 (9.17 -> 9.13)

  • Declining Code Health: 9 findings(s) 🚩
  • Improving Code Health: 3 findings(s) ✅
  • Affected Hotspots: 2 files(s) 🔥

View detailed results in CodeScene

Comment on lines +90 to +158
[Fact]
public void When_archiving_from_the_outbox()
{
var parentActivity = new ActivitySource("Paramore.Brighter.Tests").StartActivity("BrighterTracerSpanTests");

var context = new RequestContext();
context.Span = parentActivity;

//add and clear message
var myEvent = new MyEvent();
var myMessage = new MyEventMessageMapper().MapToMessage(myEvent, _publication);
_bus.AddToOutbox(myMessage, context);
_bus.ClearOutbox([myMessage.Id], context);

//se should have an entry in the outbox
_outbox.EntryCount.Should().Be(1);

//allow time to pass
_timeProvider.Advance(TimeSpan.FromSeconds(300));

//archive
var dispatchedSince = TimeSpan.FromSeconds(100);
_archiver.Archive(dispatchedSince, context);

//should be no messages in the outbox
_outbox.EntryCount.Should().Be(0);

parentActivity?.Stop();

_traceProvider.ForceFlush();

//We should have exported matching activities
_exportedActivities.Count.Should().Be(9);

_exportedActivities.Any(a => a.Source.Name == "Paramore.Brighter").Should().BeTrue();

//there should be a n archive create span for the batch
var createActivity = _exportedActivities.Single(a => a.DisplayName == $"{BrighterSemanticConventions.ArchiveMessages} {CommandProcessorSpanOperation.Archive.ToSpanName()}");
createActivity.Should().NotBeNull();
createActivity.ParentId.Should().Be(parentActivity?.Id);

//check for outstanding messages span
var osCheckActivity = _exportedActivities.SingleOrDefault(a =>
a.DisplayName == $"{OutboxDbOperation.DispatchedMessages.ToSpanName()} {InMemoryAttributes.DbName} {InMemoryAttributes.DbTable}");
osCheckActivity.Should().NotBeNull();
osCheckActivity?.ParentId.Should().Be(createActivity.Id);

//check for delete messages span
var deleteActivity = _exportedActivities.SingleOrDefault(a =>
a.DisplayName == $"{OutboxDbOperation.Delete.ToSpanName()} {InMemoryAttributes.DbName} {InMemoryAttributes.DbTable}");
deleteActivity?.Should().NotBeNull();
deleteActivity?.ParentId.Should().Be(createActivity.Id);

//check the tags for the create span
createActivity.TagObjects.Should().Contain(t => t.Key == BrighterSemanticConventions.ArchiveAge && Math.Abs(Convert.ToDouble(t.Value) - dispatchedSince.TotalMilliseconds) < TOLERANCE);

//check the tags for the outstanding messages span
osCheckActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbOperation && t.Value == OutboxDbOperation.DispatchedMessages.ToSpanName()).Should().BeTrue();
osCheckActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbTable && t.Value == InMemoryAttributes.DbTable).Should().BeTrue();
osCheckActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbSystem && t.Value == DbSystem.Brighter.ToDbName()).Should().BeTrue();
osCheckActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbName && t.Value == InMemoryAttributes.DbName).Should().BeTrue();

//check the tages for the delete messages span
deleteActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbOperation && t.Value == OutboxDbOperation.Delete.ToSpanName()).Should().BeTrue();
deleteActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbTable && t.Value == InMemoryAttributes.DbTable).Should().BeTrue();
deleteActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbSystem && t.Value == DbSystem.Brighter.ToDbName()).Should().BeTrue();
deleteActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbName && t.Value == InMemoryAttributes.DbName).Should().BeTrue();

}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ New issue: Complex Method
When_archiving_from_the_outbox has a cyclomatic complexity of 10, threshold = 9

Suppress

Comment on lines +56 to +72
public OutboxArchiver(
IAmAnOutbox outbox,
IAmAnArchiveProvider archiveProvider,
IAmARequestContextFactory? requestContextFactory = null,
int archiveBatchSize = 100,
IAmABrighterTracer? tracer = null,
InstrumentationOptions instrumentationOptions = InstrumentationOptions.All)
{
_archiveProvider = archiveProvider;
_archiveBatchSize = archiveBatchSize;
_tracer = tracer;
_instrumentationOptions = instrumentationOptions;
_requestContextFactory = requestContextFactory ?? new InMemoryRequestContextFactory();

if (outbox is IAmAnOutboxSync<TMessage, TTransaction> syncOutbox) _outBox = syncOutbox;
if (outbox is IAmAnOutboxAsync<TMessage, TTransaction> asyncOutbox) _asyncOutbox = asyncOutbox;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ New issue: Constructor Over-Injection
OutboxArchiver has 6 arguments, threshold = 5

Suppress

var subscription = _replySubscriptions?.FirstOrDefault(s => s.DataType == typeof(TResponse));

if (subscription is null)
throw new ArgumentOutOfRangeException($"No Subscription registered fpr replies of type {typeof(T)}");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Getting worse: Code Duplication
introduced similar code in: ClearOutstandingFromOutbox

Suppress

Comment on lines +92 to +160
[Fact]
public async Task When_archiving_from_the_outbox()
{
var parentActivity = new ActivitySource("Paramore.Brighter.Tests").StartActivity("BrighterTracerSpanTests");

var context = new RequestContext();
context.Span = parentActivity;

//add and clear message
var myEvent = new MyEvent();
var myMessage = new MyEventMessageMapper().MapToMessage(myEvent, _publication);
await _bus.AddToOutboxAsync(myMessage, context);
await _bus.ClearOutboxAsync([myMessage.Id], context);

//se should have an entry in the outbox
_outbox.EntryCount.Should().Be(1);

//allow time to pass
_timeProvider.Advance(TimeSpan.FromSeconds(300));

//archive
var dispatchedSince = TimeSpan.FromSeconds(100);
await _archiver.ArchiveAsync(dispatchedSince, context);

//should be no messages in the outbox
_outbox.EntryCount.Should().Be(0);

parentActivity?.Stop();

_traceProvider.ForceFlush();

//We should have exported matching activities
_exportedActivities.Count.Should().Be(9);

_exportedActivities.Any(a => a.Source.Name == "Paramore.Brighter").Should().BeTrue();

//there should be a n archive create span for the batch
var createActivity = _exportedActivities.Single(a => a.DisplayName == $"{BrighterSemanticConventions.ArchiveMessages} {CommandProcessorSpanOperation.Archive.ToSpanName()}");
createActivity.Should().NotBeNull();
createActivity.ParentId.Should().Be(parentActivity?.Id);

//check for outstanding messages span
var osCheckActivity = _exportedActivities.SingleOrDefault(a =>
a.DisplayName == $"{OutboxDbOperation.DispatchedMessages.ToSpanName()} {InMemoryAttributes.DbName} {InMemoryAttributes.DbTable}");
osCheckActivity.Should().NotBeNull();
osCheckActivity?.ParentId.Should().Be(createActivity.Id);

//check for delete messages span
var deleteActivity = _exportedActivities.SingleOrDefault(a =>
a.DisplayName == $"{OutboxDbOperation.Delete.ToSpanName()} {InMemoryAttributes.DbName} {InMemoryAttributes.DbTable}");
deleteActivity.Should().NotBeNull();
deleteActivity?.ParentId.Should().Be(createActivity.Id);

//check the tags for the create span
createActivity.TagObjects.Should().Contain(t => t.Key == BrighterSemanticConventions.ArchiveAge && Math.Abs(Convert.ToDouble(t.Value) - dispatchedSince.TotalMilliseconds) < TOLERANCE);

//check the tags for the outstanding messages span
osCheckActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbOperation && t.Value == OutboxDbOperation.DispatchedMessages.ToSpanName()).Should().BeTrue();
osCheckActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbTable && t.Value == InMemoryAttributes.DbTable).Should().BeTrue();
osCheckActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbSystem && t.Value == DbSystem.Brighter.ToDbName()).Should().BeTrue();
osCheckActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbName && t.Value == InMemoryAttributes.DbName).Should().BeTrue();

//check the tages for the delete messages span
deleteActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbOperation && t.Value == OutboxDbOperation.Delete.ToSpanName()).Should().BeTrue();
deleteActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbTable && t.Value == InMemoryAttributes.DbTable).Should().BeTrue();
deleteActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbSystem && t.Value == DbSystem.Brighter.ToDbName()).Should().BeTrue();
deleteActivity?.Tags.Any(t => t.Key == BrighterSemanticConventions.DbName && t.Value == InMemoryAttributes.DbName).Should().BeTrue();

}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ New issue: Complex Method
When_archiving_from_the_outbox has a cyclomatic complexity of 10, threshold = 9

Suppress

Comment thread src/Paramore.Brighter/ExternalBusService.cs
Comment on lines +78 to +79
public OutboxProducerMediator(
IAmAProducerRegistry producerRegistry,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ New issue: Complex Method
OutboxProducerMediator has a cyclomatic complexity of 14, threshold = 9

Comment thread src/Paramore.Brighter/ExternalBusService.cs
Comment on lines +78 to +79
public OutboxProducerMediator(
IAmAProducerRegistry producerRegistry,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ New issue: Constructor Over-Injection
OutboxProducerMediator has 14 arguments, threshold = 5

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: FAILED

Change in average Code Health of affected files: -0.03 (9.17 -> 9.13)

  • Declining Code Health: 9 findings(s) 🚩
  • Improving Code Health: 3 findings(s) ✅
  • Affected Hotspots: 2 files(s) 🔥

View detailed results in CodeScene

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: FAILED

Change in average Code Health of affected files: -0.03 (9.17 -> 9.13)

  • Declining Code Health: 9 findings(s) 🚩
  • Improving Code Health: 3 findings(s) ✅
  • Affected Hotspots: 2 files(s) 🔥

View detailed results in CodeScene

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: FAILED

Change in average Code Health of affected files: -0.03 (9.17 -> 9.13)

  • Declining Code Health: 9 findings(s) 🚩
  • Improving Code Health: 4 findings(s) ✅
  • Affected Hotspots: 3 files(s) 🔥

View detailed results in CodeScene

return commandProcessor;
}

private static IAmAnExternalBusService BuildExternalBus(IServiceProvider serviceProvider,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No longer an issue: Excess Number of Function Arguments
BuildExternalBus is no longer above the threshold for number of arguments

}

private static IAmAnExternalBusService BuildExternalBus(IServiceProvider serviceProvider,
private static IAmAnOutboxProducerMediator BuildOutBoxProducerMediator(IServiceProvider serviceProvider,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ New issue: Excess Number of Function Arguments
BuildOutBoxProducerMediator has 5 arguments, threshold = 4

@codescene-delta-analysis codescene-delta-analysis Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Health Quality Gates: FAILED

Change in average Code Health of affected files: -0.03 (9.17 -> 9.13)

  • Declining Code Health: 9 findings(s) 🚩
  • Improving Code Health: 4 findings(s) ✅
  • Affected Hotspots: 3 files(s) 🔥

View detailed results in CodeScene

@iancooper iancooper merged commit 262c7d7 into master Nov 6, 2024
@iancooper iancooper deleted the arch_obser branch November 6, 2024 21:38
DevJonny pushed a commit to DevJonny/Brighter that referenced this pull request Feb 28, 2026
* feat: add an archive test without observablity, to ensure test passes first, ahead of oberservability

* chore: switch branch; commit to save wip

* chore: switch to another device

* feat: we need a create span for the archive batch, if issued through the service bus. Presumably **your** archiver would have to do the same.

* fix: missing mark dispatched in clear otel checks

* feat: add tagobjects for archive batch

* feat: opentelemetry for the archive operation

* feat: add async archive operation opentelemetry

* feat: cleanup old archiver telemetry

* feat: add archive async telemetry

* feat: move archive functionality to OutboxArchiver.cs

* feat: rename of ExternalBusService.cs to OutboxProducerMediator.cs

* fix: name the mediator correctly

* fix: issue with PackageReference not PackageVersion

* fix: failing reflection based construction of OutBoxProducerMediator due to additional parameters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3 - Done feature request .NET Pull requests that update .net code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant