Repository navigation
Version 3.0 - #316
Merged
Merged
Version 3.0#316
Conversation
Updating Forked Dev Branch
Added the option to view the transcript of what naomi is hearing and saying. This is mostly for debugging purposes. Added the command-line option "--print_transcript" Added the profile.yml option "print_transcript:" The command-line option overrides the profile option. Prints things naomi says with two less-than's: "<<" Prints things naomi hears in passive mode with a single greater-than: ">" Prints things naomi hears in active mode with a double greater-than: ">>" This could be a privacy issue if someone's conversation is transcribed when they aren't expecting it to be, which is why I made it opt-in. We might want to print or say a warning that the operator is responsible for warning people that what they say near Naomi might be recorded into a text log.
Removed whitespace from an empty line to make Codacy happy.
This gives the user three possible ways to install Naomi. Unfortunately, all have their own benefits and issues, mostly due to the fact that Python itself is still in transition from version 2 to version 3. PROS This is my current recommendation since the option of downloading and compiling a special version of python just for Naomi turned out to be problematic, and because this way the python interpreter can be kept up to date through the use of the distro's package manager. Once the user has activated the virtual environment, they can just use 'pip' and 'python' to manage the environment. CONS I have had the experience of virtualenvs breaking when the distro moves from one minor version of python to another, but then I am on a rolling distro, and that is less likely to happen with more stable distros. It requires installing python and python-pip in addition to python3 and python3-pip (pip3) because if you use virtualenvwrapper with python3 right now, you have to specify the python3 interpreter with the VIRTUALENVWRAPPER_PYTHON and VIRTUALENVWRAPPER_VIRTUALENV environment variables. If the user is using Pocketsphinx and did not have python3 installed while running through the install procedure, then they may need to "workon Naomi" and re-install the pocketsphinx, sphinxbase and cmuclmtk python modules, which adds complexity. PROS This leaves the user's copy of python untouched. It creates a stable, independent environment for Naomi to work from so it is less likely that a distro update will break stuff. CONS First, we had problems right away with Ubuntu 18.10 and derivative distros. The gcc compiler included with them does not appear to be compatible with building python3, so these users really need to use the pre-built python3 binaries. Building python3 was also complex in that there are a lot of additional dev library requirements when building packages like "ssl" and "curses" that are required for Naomi. This copy of python will not receive updates when the user updates the system using the standard package manager, meaning that exploits are less likely to get fixed in a timely fashion. This is fairly complex, and I have implemented redragonx's signature and checksum checks. Any minor change to any of the setup files would require updating this script since the signature and checksum hashes are embedded. When installing or updating packages, the user and any scripts have to know to use the ~/.naomi/local/bin/pip command rather than the system "pip" command. PROS Easy to setup, easy to use, user and scripts don't have to remember to do anything when updating and installing libraries and can just use 'pip3' and 'python3' commands. CONS More and more software is written in python, so there are a lot of system dependancies you won't necessarily be aware of. Since we are installing specific versions of pip modules and not using they system package manager, there is a very real possibility that this might break a dependancy for another program on the user's system.
Codacy doesn't like the use of backticks when checking the return value of a command, so switched to $(...) in one place.
Since I am not automatically appending the "source virtualenvwrapper.sh" to the end of the line, print out some instructions asking the user to do so.
There are still things that users need to know and be aware of after running this installer. If they choose the first option, then they will need to add the virtualenv initialization script to their shell startup script. If they choose the second option, they need to be aware that they need to use Naomi's special copy of python when installing modules or running tests.
Responding to CVE-2017-18342, versions of YaML before 4.1 could be made to execute arbitrary code. This is not much of a concern for us at the moment, but might as well deal with it. Modified the line in python_requirements.txt
Add Audio Transcript
Naomi.py - Complains and exits if called with python 2. naomi-setup.py - Added notices when using sudo with an out of the user pressing 'q' - Added option 4 to quit program without installing - Added ability to check if header file and/or executable is installed so we can check if the environment is missing anything and warn users who don't have apt-get. - Added some color and definitions for colors so others can add additional colors. - Added second fallback keyserver for downloading gpg key when verifying the signature on python.
Removed the unnecessary definition for SystemError from Naomi.py.
Changed --primary to --system. Changed --local to --local-compile
Additional options for naomi-setup.sh
In python 3, the output from pico2wave -l (list languages) needs to be converted from bytes to string before we can check to make sure the currently selected language is a valid choice.
Python3 warning for dev branch Readme
Variable type mismatch in Pico
"Uhr" is more like "o'clock" but not "time".
Correction of German translation for clock plugin
On version 0.4.1 of deepspeech, the line "from deepspeech.model import Model" has changed to just "from deepspeech import Model" This just changes the import to work in both cases.
Fix MaryTTS plugin
Replaced imp module with importlib module due to deprecation warning. Simplified import of configparser so it is no longer trying to handle Python2 imports. Re-wrote the parse_plugin_class function to use importlib rather than imp. Changed plugin_classes initialization to fix a pep8 complaint: W504 line break after binary operator
naomi/application.py -------------------- Attached input device parameters (input_samplerate, input_samplewidth, input_channels, input_chunksize) to the input_device object so they are all available to the mic and vad objects as the input_device object is passed around. For consistency, also moved the output_chunksize and output_padding parameters to the output_device object. Added initialization of Voice Activity Detection object and passed it to the initialization of the mic object. naomi/mic.py ------------ Removed the main logic around listen and active listen so I could move it into the VADPlugin class. This should make it easier to implement the "Passive Listen for Commands" project, because once the passive listener identifies a keyword in the audio returned, we can just pass the same block of audio to the active listener for transcription. Simplified a lot of stuff. The original authors were running two threads constantly scanning the audio input for keywords, and it appears that the only reason was to speed up keyword detection. The new VAD method works much differently, but I'm interested in hearing whether anyone notices a difference. naomi/plugin.py --------------- Added skeleton for VADPlugin class naomi/pluginstore.py -------------------- Added the VADPlugin as a new plugin class plugins/vad ----------- Added two new plugins, snr_vad and webrtc_vad. plugins/vad/snr_vad ------------------- This is based on the way voice activity currently works with naomi, which is basically just waiting for audio levels to go above a certain threshold and below a certain threshold. I have always had trouble with this method, as different sound cards and different microphones register sound quite differently, so choosing a proper threshold level is often problematic. I am now saying that anything over the mean plus one and a half times the standard deviation should be considered audio to pay attention to. I also reset every 100 samples by cutting all the counts in half, thus ensuring that we aren't counting to rediculous numbers over time and allowing noise levels to adjust fairly quickly to changes in the environment. plugins/webrtc_vad ------------------ Uses the webrtcvad module, which can be installed via pip. This module requires that all chunks be 10, 20, or 30 ms. The default chunk size for Naomi is 64 ms, so you have to adjust the value of audio: input_chunksize: to either 160 (10ms), 320 (20ms), or 480 (30ms) in profile.yml.
Modified the two VAD plugins so they can receive configuration values passed through from the profile.yml For SNR_VAD, the following options can be set: snr_vad: timeout: 1 minimum_capture: 0.25 threshold: 20 For webrtc_vad, the following options can be set: webrtc_vad: timeout: 1 minimum_capture: 0.25 aggressiveness: 1
Oops, I changed the name of the SNR VAD plugin from snr to snr_vad. Left the default at snr, though, so if you have no vad selected in profile, then you get an error that plugin "vad" does not exist. Fixed, set default to "snr_vad" matching documentation and reality.
Implementing Contribution Changes
Python appears to have changed some of the ways it works with
strings since I worked on this last. The \b boundary regexp
markers no longer match outside curly brackets, so I had to
use re.finditer("{keyword}", text) instead of the original
re.finditer("\b{keyword}\b", text).
Also, I needed to add an additional "<END>" term to the template
and text so they could match the end of the string.
There's still a little cleanup to do, but it at least works again
now.
I imported the pprint module for testing. No longer needed because all the test lines are commented out.
Change the templates to upper case as they are read in.
Since Austin has added a couple of additional joke plugins (Dad Jokes and Chuck Norris Jokes) there needs to be a way to activate the default joke plugin. Currently if you have all three of these plugins installed and request "Naomi, tell me a joke" Naomi defaults to "Dad Joke". Since the default "joke" plugin is specifically designed to tell knock knock jokes, I have added knock knock to three of the intents. In a perfect world, since the Dad jokes and Chuck Norris jokes have a specific phrase associated with them other than "joke" leaving out any specific request should default to the plugin with the least requirements. I'll look into that later and see if I can get "Naomi, tell me a joke" to default to the joke plugin.
Looks good! Add "knock knock" to default joke plugin templates
Naomi tti fix
fixing nightly builds
push vs hitting server every night with no change
I have noticed that Naomi seems to grow hard of hearing as time goes on when using the SNR_VAD Voice Activity Detector. This appears to be because I am updating entries by adding 1 and dividing by 2. If the count is at 1, then the count will be updated to 1, and that level will never be removed from the list. over time, bumps and crashes will build up and affect the standard deviation calculation. This update only updates keys with a count greater than 1, so allows older keys to time out, allowing the system to generate a more accurate standard deviation. This should allow Naomi to recover from hearing a loud background noise or adjust to a new noise in the environment in only a few seconds to a minute, and make it listen harder when things get quiet.
* Separate NPE into its own module This just removes the Naomi Plugin Exchange functions from application.py and moves them into their own module (npe.py). This makes application.py smaller and easier to work with and should make it easier to troubleshoot NPE issues. * Codacy complaint Codacy is compaining that the global _ variable is not defined at the module level.
* Update Mozilla DeepSpeech
Update the implementation of Mozilla's DeepSpeech STT engine.
This version tries to set up deepspeech as much as possible. It
downloads the source code to ~/.config/naomi/sources/STTv{version}
and the acoustic model to ~/.config/naomi/deepspeech/model_{version}.pbmm
The big improvement here is that it uses the KenLM engine to build
language models based on phrases that Naomi is likely to hear based
on the templates provided in the speechhandler plugins. KenLM pays
attention to words and word order, so it will be interesting to see
how word error rate compares to pocketsphinx which just listens for
individual words from a dictionary, or how word error rates improve
if we can get pocketsphinx to use kenlm also.
I had to modify all the text to intent parsers to return a list of
full phrases from the intents instead of a list of words. This seems
to work fine with pocketsphinx as it is, but allows KenLM to
analyze word order.
I had to make one small change to sphinxvocab, where I do not
include keywords surrounded by curly brackets when passing the list
of words to Phonetisaurus.
* Minor fixes for Deepspeech
This adds the requirement for the deepspeech python module plus
the apt requirements for Naomi to set up the tools needed to
generate language models. It also now correctly downloads the
tensorflow-lite version of the language model when running on
the Raspberry Pi.
* Provide additional details
Provide some additional details about where the DeepSpeech source
code is being downloaded to.
* Update sphinxvocab.py
Removed an errant print
* Fixed deepspeech model download There were a couple of issues here. One was that if Naomi needed to download the model, it reconstructed the URL without paying attention to the architecture, so on the raspberry pi it still downloaded the pbmm model instead of the tflite model. I also seemed to be having some trouble downloading the binary correctly and kept getting an error message about the file header being malformed. I "borrowed" some code from stack overflow which seems to work pretty well. I put this code into app_utils as a start to working on Issue #303 - Standard functions with progress bars for downloads. At least we don't have to reinvent the wheel every time we need to download a binary file. * Cleaning up some debugging text
Just a slight change to the Naomi TTI scoring. After generating scores for all the intents, add those scores up and divide individual scores by the total. This makes the score always fall between 0 and 1 (currently there is no upper limit and I have seen scores as high as between 6 and 7). This should also make Naomi more confident in some cases. Right now if one intent scores .01 and all the other intents are zero, Naomi will respond with "unclear" since no intent has scored above .05 (5%). With this new system, .01/.01=1 (100%). I don't know if this will turn out to be a good thing or not. I think it will be good, but if it causes problems, we might need to add a filter before the conversion.
* Change Naomi TTI score to a confidence percent Just a slight change to the Naomi TTI scoring. After generating scores for all the intents, add those scores up and divide individual scores by the total. This makes the score always fall between 0 and 1 (currently there is no upper limit and I have seen scores as high as between 6 and 7). This should also make Naomi more confident in some cases. Right now if one intent scores .01 and all the other intents are zero, Naomi will respond with "unclear" since no intent has scored above .05 (5%). With this new system, .01/.01=1 (100%). I don't know if this will turn out to be a good thing or not. I think it will be good, but if it causes problems, we might need to add a filter before the conversion. * committing change so I can pull upstream * Fix buffer underrun error After the latest Raspbian Buster update on my Raspberry Pi 4, I was getting the following error message: ALSA lib pcm.c:8424:(snd_pcm_recover) underrun occurred This was causing Naomi to make raspberry noises instead of talking when trying to play audio generated from flite. I tried increasing the chunk size in the profile.yml file, but it did not seem to affect the playback of a file in play_fp. It looks like the chunksize and output padding both get written to attributes of the audioengine plugin but I didn't see anywhere that they were actually passed to the play_file or play_fp audiodevice methods. Since we can read the profile directly from any method just by including the profile module, I'm just reading them directly from the play_fp method. Increasing the chunk size worked, but the audio kept getting clipped at the end. I tried enabling the padding option in profile.yml, but that didn't help. I ended up adding a new option under audio -> output_pause to have the computer pause after the last chunk is written to the device and before the stream is closed, and that finally worked. I set it to 0.5 seconds on my system which seemed to work fine. 0.1 was not enough to prevent clipping.
In pull request #309, I removed the chunksize and add_padding optional parameters from play_fp, and pulled those values directly from the profile. This worked fine in normal operation, but introduced issues with the configuration routines in application.py which are passing parameters to play_file which are being passed along to play_fp. This update restores legacy compatibility by using *args and **kwargs to allow values to be passed in optionally, which makes it compatible with the call from play_file. If chunksize and add_padding are passed in as keyword arguments for some reason, those values are used instead of the profile values. In general, developers should not pass values directly to play_file or play_fp. I also removed the parameters from the play_file calls in application.py.
Prep for version 3.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Related Issue
Motivation and Context
How Has This Been Tested?
Screenshots (if appropriate):
Types of changes
Checklist: