Skip to content

Make minor tweaks before launch - #4111

Merged
hawflau merged 15 commits into
aws:customer-journeyfrom
Leo10Gama:minor-fixes
Aug 15, 2022
Merged

Make minor tweaks before launch#4111
hawflau merged 15 commits into
aws:customer-journeyfrom
Leo10Gama:minor-fixes

Conversation

@Leo10Gama

@Leo10Gama Leo10Gama commented Aug 10, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue(s) does this change fix?

N/A

Why is this change necessary?

There were a few things I wasn't happy with reflecting on some of my previous code, and they were too small to fit into PRs on their own. Thus, they've been delegated to this PR, which has many tiny adjustments and tweaks, outlined below in order of significance.

How does it address the issue?

  • EventTracker.send_events now executes on a thread! Previously, this logic was only implemented when the function was called at capacity. Now, it's the default!
  • EventTracker now stores session_id locally. This fixes a bug I was only recently made aware of, where Events sent in a thread did not contain the session ID, thus preventing us from linking the data. With the new implementation, the session ID is appended to the Metric before it is sent out.
  • The decorators on sam sync are rearranged (namely, track_command and track_long_event have swapped places). This will prevent one send_events from being called, then the "end event" being added, and then one more send_events being called.
    • This reasoning has also been added to the docstring for the track_long_event decorator.
  • The attribute EventType._events has been renamed to EventType._event_values. This not only prevents ambiguity, but also prevents confusion with EventTracker._events.
  • Added a docstring to EventTracker.get_tracked_events method.
  • Added a comment beside EventType._event_values to provide explanation as to what the value represents.
  • Added comments in the track_long_event decorator to provide clarity on the process of the method.

What side effects does this change have?

Aside from some readability improvements within my code, the only notable side effect is the EventTracker.send_events executing on a separate thread always.

Mandatory Checklist

PRs will only be reviewed after checklist is complete

  • Add input/output type hints to new functions/methods
  • Write design document if needed (Do I need to write a design document?)
  • Write/update unit tests
  • Write/update integration tests
  • Write/update functional tests if needed
  • make pr passes
  • make update-reproducible-reqs if dependencies were changed
  • Write documentation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@github-actions github-actions Bot added area/sync sam sync command pr/external stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. labels Aug 10, 2022
@Leo10Gama
Leo10Gama marked this pull request as ready for review August 11, 2022 19:54
hawflau
hawflau suggested changes Aug 12, 2022
View reviewed changes
Comment thread samcli/lib/telemetry/event.py
hawflau
hawflau suggested changes Aug 12, 2022
View reviewed changes
Comment thread samcli/lib/telemetry/event.py Outdated
hawflau
hawflau approved these changes Aug 12, 2022
View reviewed changes

@hawflau hawflau left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating the PR! LGTM!

torresxb1
torresxb1 approved these changes Aug 15, 2022
View reviewed changes

@torresxb1 torresxb1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯 thanks for the improvements!

@hawflau
hawflau merged commit e46d19a into aws:customer-journey Aug 15, 2022
@Leo10Gama
Leo10Gama deleted the minor-fixes branch August 15, 2022 21:59
hawflau added a commit that referenced this pull request Aug 23, 2022
@Leo10Gama @hawflau
Merge EventTracker project into develop (#4134)
61bf353 
* Add project metadata tracking with telemetry (#3929)

* Implement project metadata metrics

* Implement metric metadata integration tests

* Tweak methods so tests don't fail on Linux machines

* Repair failing integration test

* Implement proper exception handling for subprocesses

* Format according to standard

* Refactor function names and encryption logic

* Remove unnecessary file

* Add type hints to new functions

* Fix retrieval process for directory name (#3962)

* Implement event data structure and relevant unit tests (#3955)

* Implement event data structure and relevant unit tests

* Implement requested changes

* Remove unused (useless) file

* Remove unnecessary ABC class

* Implement tracker methods (#3970)

* Implement tracker methods

* Improve readability of Event's __repr__ method

* Add method to convert Event to dictionary for JSON-compatibility

* Send Event metrics via Telemetry (#3979)

* Send Event metrics via Telemetry

* Add unit tests for event telemetry

* Simplify event telemetry unit tests

* Clean up unit tests

* Add BuildRuntime event and trackers (#3994)

* Add BuildRuntime Events

* Remove unnecessary file

* Prevent tracked Event 'spillage' from previous tests

* Fix builder logic

* Redefine BuildRuntime to focus on function runtimes

* Remove unnecessary file (again)

* Add UsedFeature event and trackers (#4003)

* Add UsedFeature command and trackers

* Remove LocalTest events
Events regarding sam local will be implemented at a later time.

* Prevent methods from failing if track_event fails

* Reformat code to match current standard

* Use correct string formatting

* Use proper LOG message

* Add thread lock to EventTracker (#4019)

* Add thread lock to EventTracker

* Place lock in appropriate methods

* Refactor Event data structure (#4027)

* Add new attributes to Event datastructure

* Fix non-JSON-serializable field in Event

* Fix repr to ensure SQL casting will work

* Add comment for timestamp method clarity

* Update tests to verify EventTracker thread lock is working (#4035)

* Add send_events method to EventTracker (#4039)

* Add framework for Event-sending method

* Perform some general refactors and cleanup

* Implement EventTracker Telemetry unit tests

* Cleanup tests and unused imports

* Update Metric name for consistency

* Repair failing unit test

* (Hopefully) repair broken integration test

* *Actually* repair broken integration test
I had the >= logic backwards lol

* Fix integration test
For sure this time

* Really fix the integration test
Windows development is not fun

* Properly fix integration test

* Reformat files

* Implement requested changes

* Add Lock logic to send_events method

* Implement requested changes

* Repair failing tests

* Upgrade encryption algorithm from SHA-1 to SHA-256 (#4061)

* Upgrade encryption algorithm from SHA1 to SHA256

* Make return value of hash JSON-serializable

* Add long event tracker (#4067)

* Send events via thread when at capacity (#4072)

* Switch send_events to thread when at capacity

* Repair failing unit test

* Add `sam sync` Events (#4076)

* Add sam sync Events

* Refactor methods for tracking sync events

* Improve SyncFlow tracking

* Add additional SyncFlow value

* Add workflow events (#4096)

* Add workflow_config events

* Refactor workflow Events

* Add tests for verifying workflow events

* Remove redundant (UsedFeature, ESBuild) event

* Repair failing unit test

* Rename workflow event to prevent ambiguity

* Fix send_events logic (#4104)

* Make minor tweaks before launch (#4111)

* Apply minor changes (some major)

* Add session_id for multithreaded send_events

* Adjust tests for new multithreading

* Fix failing integration test

* *Actually* fix failing integration test

* Properly fix integration test

* This time for sure

* Really really repair that test

* OK this one SHOULD work I swear

* This time for sure

* FIX INTEGRATION TEST

* IT'S A LIST

* Remove unused imports

* Add debug log for EventTracker session ID

* Place debug log in better position

Co-authored-by: Wing Fung Lau <4760060+hawflau@users.noreply.github.com>
torresxb1 pushed a commit to torresxb1/aws-sam-cli that referenced this pull request Sep 2, 2022
@Leo10Gama @hawflau @torresxb1
Merge EventTracker project into develop (aws#4134)
53b86ba 
* Add project metadata tracking with telemetry (aws#3929)

* Implement project metadata metrics

* Implement metric metadata integration tests

* Tweak methods so tests don't fail on Linux machines

* Repair failing integration test

* Implement proper exception handling for subprocesses

* Format according to standard

* Refactor function names and encryption logic

* Remove unnecessary file

* Add type hints to new functions

* Fix retrieval process for directory name (aws#3962)

* Implement event data structure and relevant unit tests (aws#3955)

* Implement event data structure and relevant unit tests

* Implement requested changes

* Remove unused (useless) file

* Remove unnecessary ABC class

* Implement tracker methods (aws#3970)

* Implement tracker methods

* Improve readability of Event's __repr__ method

* Add method to convert Event to dictionary for JSON-compatibility

* Send Event metrics via Telemetry (aws#3979)

* Send Event metrics via Telemetry

* Add unit tests for event telemetry

* Simplify event telemetry unit tests

* Clean up unit tests

* Add BuildRuntime event and trackers (aws#3994)

* Add BuildRuntime Events

* Remove unnecessary file

* Prevent tracked Event 'spillage' from previous tests

* Fix builder logic

* Redefine BuildRuntime to focus on function runtimes

* Remove unnecessary file (again)

* Add UsedFeature event and trackers (aws#4003)

* Add UsedFeature command and trackers

* Remove LocalTest events
Events regarding sam local will be implemented at a later time.

* Prevent methods from failing if track_event fails

* Reformat code to match current standard

* Use correct string formatting

* Use proper LOG message

* Add thread lock to EventTracker (aws#4019)

* Add thread lock to EventTracker

* Place lock in appropriate methods

* Refactor Event data structure (aws#4027)

* Add new attributes to Event datastructure

* Fix non-JSON-serializable field in Event

* Fix repr to ensure SQL casting will work

* Add comment for timestamp method clarity

* Update tests to verify EventTracker thread lock is working (aws#4035)

* Add send_events method to EventTracker (aws#4039)

* Add framework for Event-sending method

* Perform some general refactors and cleanup

* Implement EventTracker Telemetry unit tests

* Cleanup tests and unused imports

* Update Metric name for consistency

* Repair failing unit test

* (Hopefully) repair broken integration test

* *Actually* repair broken integration test
I had the >= logic backwards lol

* Fix integration test
For sure this time

* Really fix the integration test
Windows development is not fun

* Properly fix integration test

* Reformat files

* Implement requested changes

* Add Lock logic to send_events method

* Implement requested changes

* Repair failing tests

* Upgrade encryption algorithm from SHA-1 to SHA-256 (aws#4061)

* Upgrade encryption algorithm from SHA1 to SHA256

* Make return value of hash JSON-serializable

* Add long event tracker (aws#4067)

* Send events via thread when at capacity (aws#4072)

* Switch send_events to thread when at capacity

* Repair failing unit test

* Add `sam sync` Events (aws#4076)

* Add sam sync Events

* Refactor methods for tracking sync events

* Improve SyncFlow tracking

* Add additional SyncFlow value

* Add workflow events (aws#4096)

* Add workflow_config events

* Refactor workflow Events

* Add tests for verifying workflow events

* Remove redundant (UsedFeature, ESBuild) event

* Repair failing unit test

* Rename workflow event to prevent ambiguity

* Fix send_events logic (aws#4104)

* Make minor tweaks before launch (aws#4111)

* Apply minor changes (some major)

* Add session_id for multithreaded send_events

* Adjust tests for new multithreading

* Fix failing integration test

* *Actually* fix failing integration test

* Properly fix integration test

* This time for sure

* Really really repair that test

* OK this one SHOULD work I swear

* This time for sure

* FIX INTEGRATION TEST

* IT'S A LIST

* Remove unused imports

* Add debug log for EventTracker session ID

* Place debug log in better position

Co-authored-by: Wing Fung Lau <4760060+hawflau@users.noreply.github.com>
hawflau added a commit that referenced this pull request May 17, 2023
@Leo10Gama @hawflau Leonardo Gama
Thread id bug (#5178)
1119525 
* Add project metadata tracking with telemetry (#3929)

* Implement project metadata metrics

* Implement metric metadata integration tests

* Tweak methods so tests don't fail on Linux machines

* Repair failing integration test

* Implement proper exception handling for subprocesses

* Format according to standard

* Refactor function names and encryption logic

* Remove unnecessary file

* Add type hints to new functions

* Fix retrieval process for directory name (#3962)

* Implement event data structure and relevant unit tests (#3955)

* Implement event data structure and relevant unit tests

* Implement requested changes

* Remove unused (useless) file

* Remove unnecessary ABC class

* Implement tracker methods (#3970)

* Implement tracker methods

* Improve readability of Event's __repr__ method

* Add method to convert Event to dictionary for JSON-compatibility

* Send Event metrics via Telemetry (#3979)

* Send Event metrics via Telemetry

* Add unit tests for event telemetry

* Simplify event telemetry unit tests

* Clean up unit tests

* Add BuildRuntime event and trackers (#3994)

* Add BuildRuntime Events

* Remove unnecessary file

* Prevent tracked Event 'spillage' from previous tests

* Fix builder logic

* Redefine BuildRuntime to focus on function runtimes

* Remove unnecessary file (again)

* Add UsedFeature event and trackers (#4003)

* Add UsedFeature command and trackers

* Remove LocalTest events
Events regarding sam local will be implemented at a later time.

* Prevent methods from failing if track_event fails

* Reformat code to match current standard

* Use correct string formatting

* Use proper LOG message

* Add thread lock to EventTracker (#4019)

* Add thread lock to EventTracker

* Place lock in appropriate methods

* Refactor Event data structure (#4027)

* Add new attributes to Event datastructure

* Fix non-JSON-serializable field in Event

* Fix repr to ensure SQL casting will work

* Add comment for timestamp method clarity

* Update tests to verify EventTracker thread lock is working (#4035)

* Add send_events method to EventTracker (#4039)

* Add framework for Event-sending method

* Perform some general refactors and cleanup

* Implement EventTracker Telemetry unit tests

* Cleanup tests and unused imports

* Update Metric name for consistency

* Repair failing unit test

* (Hopefully) repair broken integration test

* *Actually* repair broken integration test
I had the >= logic backwards lol

* Fix integration test
For sure this time

* Really fix the integration test
Windows development is not fun

* Properly fix integration test

* Reformat files

* Implement requested changes

* Add Lock logic to send_events method

* Implement requested changes

* Repair failing tests

* Upgrade encryption algorithm from SHA-1 to SHA-256 (#4061)

* Upgrade encryption algorithm from SHA1 to SHA256

* Make return value of hash JSON-serializable

* Add long event tracker (#4067)

* Send events via thread when at capacity (#4072)

* Switch send_events to thread when at capacity

* Repair failing unit test

* Add `sam sync` Events (#4076)

* Add sam sync Events

* Refactor methods for tracking sync events

* Improve SyncFlow tracking

* Add additional SyncFlow value

* Add workflow events (#4096)

* Add workflow_config events

* Refactor workflow Events

* Add tests for verifying workflow events

* Remove redundant (UsedFeature, ESBuild) event

* Repair failing unit test

* Rename workflow event to prevent ambiguity

* Fix send_events logic (#4104)

* Make minor tweaks before launch (#4111)

* Apply minor changes (some major)

* Add session_id for multithreaded send_events

* Adjust tests for new multithreading

* Fix failing integration test

* *Actually* fix failing integration test

* Properly fix integration test

* This time for sure

* Really really repair that test

* OK this one SHOULD work I swear

* This time for sure

* FIX INTEGRATION TEST

* IT'S A LIST

* Remove unused imports

* Add debug log for EventTracker session ID

* Place debug log in better position

* Fix thread ID bug in EventTracker

* Reformat files to standard

* Clean up unnecessary files

* Add thread ID tracking to infra sync executor

* Add missing thread ID to tracked event

---------

Co-authored-by: Wing Fung Lau <4760060+hawflau@users.noreply.github.com>
Co-authored-by: Leonardo Gama <leogama@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers
@hawflau hawflau hawflau approved these changes
@torresxb1 torresxb1 torresxb1 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area/sync sam sync command pr/external stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Leo10Gama @hawflau @torresxb1