Skip to content

Change Naomi TTI score to a confidence percent - #307

Merged
aaronchantrill merged 1 commit into
NaomiProject:naomi-devfrom
aaronchantrill:naomi_tti
Jan 4, 2021
Merged

aaronchantrill merged 1 commit into
NaomiProject:naomi-devfrom
aaronchantrill:naomi_tti

Conversation

@aaronchantrill

Copy link
Copy Markdown
Contributor

Description

Just a slight change to the Naomi TTI scoring. After generating
scores for all the intents, add those scores up and divide
individual scores by the total. This makes the score always fall
between 0 and 1 (currently there is no upper limit and I have
seen scores as high as between 6 and 7).

This should also make Naomi more confident in some cases. Right
now if one intent scores .01 and all the other intents are zero,
Naomi will respond with "unclear" since no intent has scored
above .05 (5%). With this new system, .01/.01=1 (100%). I don't
know if this will turn out to be a good thing or not. I think
it will be good, but if it causes problems, we might need to add
a filter before the conversion.

Related Issue

Motivation and Context

Responding with a confidence score is just more normal for an intent parser.

How Has This Been Tested?

Tested on Raspberry Pi 4 running Raspbian Buster

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Just a slight change to the Naomi TTI scoring. After generating
scores for all the intents, add those scores up and divide
individual scores by the total. This makes the score always fall
between 0 and 1 (currently there is no upper limit and I have
seen scores as high as between 6 and 7).

This should also make Naomi more confident in some cases. Right
now if one intent scores .01 and all the other intents are zero,
Naomi will respond with "unclear" since no intent has scored
above .05 (5%). With this new system, .01/.01=1 (100%). I don't
know if this will turn out to be a good thing or not. I think
it will be good, but if it causes problems, we might need to add
a filter before the conversion.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant