Repository navigation
[Hold for Payment 2024-09-10][$2000][Tracking] ProposalPolice™ 👮🏼 #35108
Description
Slack context: https://expensify.slack.com/archives/C01GTK53T8Q/p1706120841703079
Proposal
develop a GitHub 🤖 using GPT-4 capabilities to standardize and monitor proposals.
Strategy
The purpose of the Expensify Open-Source Community is to provide a global community of all the best React Native developers in the world streamlined access to contribute to the Expensify product. In order to attract and retain talented developers, the Expensify Open-Source Community must be inclusive, fair, and provide equitable opportunity to all community members, including newcomers.
Problem
There is a tendency among the open-source community to perform what I call speedrun proposals – a contributor will post an incomplete or otherwise low-quality proposal very quickly in order to be the first proposal, then subsequently edit it multiple times. Sometimes these edits significantly alter the RCA or solution and/or derive from later proposals without clear acknowledgement.
When the C+ or Expensify engineer go to review those proposals, they will tend to choose the first proposal of sufficient quality that solves the problem – and in the case of edited proposals it takes some extra sleuthing to understand the true timeline of proposals.
In short, this practice of speedrun proposals inhibits open collaboration amongst contributors and makes it harder for reviewers to fairly evaluate proposals against each other.
Solution
Create a open-source GitHub bot (tentatively named the proposal-police™ 👮🏼🚨) that helps to enforce the documented best practices for proposals, and aids the reviewers in maintaining a clearer timeline of proposals.
More specifically, the proposal-police™ would leverage GPT-4 to:
- Enforce the use of the proposal template by scanning new issue comments. If a proposal does not follow the template, the bot will comment, tagging the submitter and providing a link to the proposal template to encourage standardization.
- Monitor issue edits of proposals, classifying them as either minor or substantive. When a substantive change is detected that alters the initial problem statement, RCA, or proposed solution, the bot will append a suffix to the comment like:
Last edited: timestamp, making the timeline clearer to proposal reviewers.
In conclusion, the proposal-police™ will help promote higher quality proposals and a more informed and trust-based review process 🤝, improving baseline fairness and allowing community members and Expensify employees to better focus on getting shit :done:!
Activity
melvin-bot commentedon Jan 24, 2024
Current assignee @mallenexpensify is eligible for the NewFeature assigner, not assigning anyone new.
roryabraham commentedon Jan 24, 2024
Credit to @ikevin127 for the proposal. Comment for assignment 😉
marcochavezf commentedon Jan 24, 2024
Happy to assist with the review, co-assigning myself
ikevin127 commentedon Jan 24, 2024
Comment ? Cheers!
mallenexpensify commentedon Jan 24, 2024
@roryabraham @marcochavezf what are you thoughts about starting with an MVP that adds the above to a comment that's been edited? Then, we (all) can review how it's working then consider additional improvements.
roryabraham commentedon Jan 24, 2024
@mallenexpensify are you basically suggesting that for the MVP we remove this piece?
I don't want the edited by proposal-police to appear above any comment that's edited because that will cause false positives and discourage people from fixing typos or adding minor clarifications. That's where ChatGPT comes in to help enforce Rule #2 - only substantive edits need to be called out.
ikevin127 commentedon Jan 25, 2024
(POC) Bot source: https://github.com/ikevin127/proposal-police
For reference these are the current (POC) assistant instructions:
Instructions
You are a GitHub bot using GPT-4 to monitor and enforce proposal comments within a GitHub repo’s issues.
PROPOSAL TEMPLATE:
Proposal (mandatory line)
Please re-state the problem that we are trying to solve in this issue. - (mandatory line)
{user content here}
What is the root cause of that problem? - (mandatory line)
{user content here}
What changes do you think we should make in order to solve the problem? - (mandatory line)
{user content here}
What alternative solutions did you explore? (Optional) - (optional line)
{optional user content here}
IMPORTANT NOTES ON THE PROPOSAL TEMPLATE:
BOT ACTIONS:
CHANGES CLASSIFICATION:
When comparing an initial proposal (unedited) with the latest edit of a proposal comment, only consider the following ‘CHANGES’ CLASSIFICATIONS:
a. MINOR: These will be small differences like correcting typos, adding permalinks, videos, screenshots to either the first, second or third proposal template mandatory lines or adding the (Optional) alternative - all these without considerable changes to the RCA aka (### What is the root cause of that problem?) or SOLUTION aka (### What changes do you think we should make in order to solve the problem?) initial text.
b. SUBSTANTIAL: With focus on the RCA / SOLUTION sections, these will be big differences on the RCA and / or SOLUTION sections (either one of them, or both of them) - meaning if initially the proposal’s RCA / SOLUTION text was mentioning a certain root cause or suggesting a certain solution and the latest edit is mentioning a completely different RCA and / or considerable solution changes.
IMPORTANT NOTE: It is paramount to ALWAYS respond with the exact text provided in quotes (""). If neither case 1 nor 2 apply, only respond with "[NO_ACTION] - {describe the edge case here}." - it is PARAMOUNT that there's no other prose in the response.
Feel free to take apart the paragraphs where needed in order to change the AI's behaviour / response.
Currently on the AI side we return templates of either "NO_ACTION" (when proposal template was followed or edit changes are minor) or cases like "{user} Your proposal will be dismissed because you did not follow the proposal template." that include variables which we replace within the bot's code, variables we get from the github context depending on the triggered webhook (new comment, edited comment, etc).
mallenexpensify commentedon Jan 27, 2024
How does Chat GPT inclusion help keep the _edited by proposal-police _ from being above the comment? I might me missing or not understanding a step or detail.
ikevin127 commentedon Jan 28, 2024
Hey y'all 👋
I created the expensify-proposal-testing repo for testing purposes.
Sent you all collaborator invitations to the repo so you can create issues and play around.
Already created an issue here where I posted a few comments to test it out - feel free to use it or just create your own 👈
Note: make sure to add the Help Wanted label to new issues, otherwise the proposal-police won't act.
Currently we're using this format
Edited by proposal-police: This proposal was added at 2024-01-24 13:15:24 UTCfor when a proposal was edited substantially.98 remaining items
mallenexpensify commentedon Sep 3, 2024
@ikevin127 , what's status here? Anything else we're waiting on?
ikevin127 commentedon Sep 3, 2024
@mallenexpensify Hey, as mentioned in this expensify comment:
The latest discussion regarding additional work / changes in code on the proposal police is this slack comment ragarding some issues with the date on a small percentage of proposals that were had significant changes in edits and the date outputted by the proposal police bot, which I plan to work on whenever I get up to date with current PRs / issues.
^ This is not a disruptive issue in any way as, in case the wrong (same 2023 date) is posted by the bot when the proposal is edited significantly, the C+ can click on the comment's edit history and see the edit before the bot's edit and get the correct date - which I'm sure most C+ already do because of the UTC format which we chose for the bot output date.
Other than that one date-related issue which happens every now an then which I plan to address in the near future, I think we're good here for moving forward with the final checklist of requirements before we can proceed to payment.
mallenexpensify commentedon Sep 4, 2024
Thanks @ikevin127
What are the requirements we need to ✅ ? For payment, I think we discussed finishing the project first, then discussing payment. If so, what do you propose for the amount? (let's only focus on this core project and not any future work for deciding the payment amount).
ikevin127 commentedon Sep 4, 2024
@mallenexpensify By requirements I was referring to things like waiting a bit after proposal police is live to give it some time and show success before discussing payment - I think that happened and we can move forward here.
♻️ Summary
⏳ Timeline of events
true(Slack link) via 3rd pull request [NoQA] FIX / ProposalPolice™ GH Actions Workflow - Response: true #46694🟢 Present day: Overall the bot's activity good, it's doing its job as instructed and polices proposals for new comments (checking for proposal template match) and for significantly edited proposals.
🤖 Some recent examples of Proposal Police in action
CASE 1 - Proposal template match check
CASE 2 - Proposal edited w/ significant changes
💰 Payment amount
Given the amount of work involved to bring this from a 📓 P/S state to 🔴 live and considering the somewhat standard amount paid to C+ for completing projects, I think $2000 would be a fair amount as payment for this project.
[-][Tracking] ProposalPolice™ 👮🏼 [/-][+][$2000][Tracking] ProposalPolice™ 👮🏼 [/+]melvin-bot commentedon Sep 7, 2024
mallenexpensify commentedon Sep 7, 2024
Thanks for the thorough breakdown @ikevin127 , $2000 seems fair. I'm created a job, please accept
https://www.upwork.com/jobs/~021832212739812288245
I'll wait til Tuesday to pay, in case @marcochavezf or others have feedback
[-][$2000][Tracking] ProposalPolice™ 👮🏼 [/-][+][Hold for Payment 2024-09-10][$2000][Tracking] ProposalPolice™ 👮🏼 [/+]ikevin127 commentedon Sep 7, 2024
Sure, thanks Matt! 💯
ikevin127 commentedon Sep 11, 2024
@mallenexpensify Everything set for payment or we're waiting on more feedback ?
mallenexpensify commentedon Sep 11, 2024
Contributor: @ikevin127 paid $2000 via Upwork
Do we need any regression tests for this? Guessing not cuz it's only in GH.
For additional work/tweaks/improvements, what's fair? @ikevin127 maybe you work on any small stuff for free the next 30 days? Then, after that, we discuss payment, per issue, for future improvements/fixes? (and.. if any not-small stuff comes up in 30 days, we can discuss potential payment there too)
ikevin127 commentedon Sep 11, 2024
@mallenexpensify ✅ Correct, no regression tests needed here. Regarding small stuff like tweaks / improvements there's no need for payments in the next 30 days or ever. In case some major work / changes are required (doubt) then the standard issue rate works for me 👍
🗒️ To do
mallenexpensify commentedon Sep 11, 2024
Thanks, sounds good and makes sense. Closing.