Model test results - model 20240117

[jwebmeister]

Post test results + useful remarks here, ideally of both:

• the new model (20240117), and
• the base model (kaldi_model_daanzu_20211030-mediumlm)

, using the same test data, and using the default Ready or Not grammar module.

Useful remarks include:

• specific words or phrases that were consistently misrecognised
• rate of false positives / false negatives
• subjective opinion, which model works better or worse, and in which areas

Important instructions:

• It's important to manually review, clean and update retain.tsv with the correct rules + text, see example workflow near the end of these instructions
• See this YouTube video
• Please only include "normal commands" in the test data, please exclude "Freeze", etc.
  • There's a script ./scripts/copy_retain_item_cmds_only.ps1 that can be used in PowerShell to copy only "normal commands" out of ./retain/ and into ./cleanaudio_cmds/
• Please, if possible, only use the default _readyornot.py grammar module, or very minor modifications, i.e. no new words.
• Example command to run test ./tacspeak.exe --test_model './cleanaudio_cmds/retain.tsv' './kaldi_model/' './kaldi_model/lexicon.txt' 4
• There are a number of useful PowerShell scripts in the ./scripts/ folder related to cleaning up the retain.tsv and related .wav files.
• A workflow I use for cleaning up the data after a play session:
  • Open retain.tsv and go through each line, reviewing the rule and text
  • At the same time, load into a playlist every .wav file in the ./retain/ folder in VLC media player on single file loop, pressing 'N' to move to next .wav as I read through each line of retain.tsv
  • When there's a mismatch between the text vs the audio, but the rule is correct, I correct the text in retain.tsv to align with the audio.
  • When there's a mismatch between the recognised rule (and/or option) vs the audio, I either A) update both the rule + text manually, or B) delete the line in retain.tsv, then when I'm done reviewing I run the list_wav_missing_from_retain_tsv.ps1 first to make sure I'm deleting the right files, then run delete_wav_missing_from_retain_tsv.ps1 script (option A is preferred, but hey we're all busy and life is too short to spend cleaning all the data).
  • If the audio is so stupidly vague or garbled that I can't understand with my own ears and brain what I'm saying, I delete the line in retain.tsv, then when I'm done reviewing I run the list_wav_missing_from_retain_tsv.ps1 first to make sure I'm deleting the right files, then run delete_wav_missing_from_retain_tsv.ps1 script.

Example report:

• 0 incorrect commands out of 4 cmds (1 missions played), same result both models
• 5% WER, same result both models
• new model more often picks up baby crying as "freeze", using "listen_key_toggle":-1, using USE_NOISE_SINK = True; also picked up in base model but not as often.
• New model tended to pick up "red" as "gold" when wife was speaking
• using default _readyornot.py without any modifications
• './kaldi_model/' is new model
• './kaldi_model_base/' is base model

('./kaldi_model/', './retain/retain.tsv', 'Command', 'WER', 'Overall -> 5.00 %+/- 9.55 %N=20 C=19 S=1 D=0 I=0')
('./kaldi_model/', './retain/retain.tsv', 'Command', 'CMDERR', {'cmd_not_correct_output': 0, 'cmd_not_correct_rule': 0, 'cmd_not_correct_options': 0, 'cmd_not_recog_output': 0, 'cmd_not_recog_input': 0, 'cmds': 4})
('./kaldi_model_base/', './retain/retain.tsv', 'Command', 'WER', 'Overall -> 5.00 %+/- 9.55 %N=20 C=19 S=0 D=1 I=0')
('./kaldi_model_base/', './retain/retain.tsv', 'Command', 'CMDERR', {'cmd_not_correct_output': 0, 'cmd_not_correct_rule': 0, 'cmd_not_correct_options': 0, 'cmd_not_recog_output': 0, 'cmd_not_recog_input': 0, 'cmds': 4})

[jwebmeister]

Edited with instructions / guidance on an example workflow on how to clean up the data after a play session, and point to the helpful ./scripts/ folder, to make data cleaning a slightly easier slj.

[madmaximus101]

Currently testing the Experimental version. When speaking commands for fall in & arrest em/them/him/her blue team will be told to arrest often, sometimes red, but mostly blue. If i speak extremely clearly and with proper wording this doesnt seem to happen. But i do have to be "like a tv presenter" announciating words very properly. The Restrain command has less of this wierdness but it is still present. Again like before Blue team will be often unprompted to be given the command, with red sometimes being told to do it. The restrain command is less prone to this though which is interesting. I have noticed saying c4 in this model doesn't really work anymore. If i say c2 it pretty much works as expected. In general i noticed a habit of blue being told to do commands when i just said the command without red or blue at start of speaking. I am attempting to compile a test run with Powershell but i am getting an error: argument --testmodel: expected 4 arguments. edit: nvm...forgot the "4" lol

[jwebmeister]

Awesome! Thanks for the feedback @madmaximus101 . A few questions and comments below.

Currently testing the Experimental version. When speaking commands for fall in & arrest em/them/him/her blue team will be told to arrest often, sometimes red, but mostly blue. If i speak extremely clearly and with proper wording this doesnt seem to happen. But i do have to be "like a tv presenter" announciating words very properly. The Restrain command has less of this wierdness but it is still present. Again like before Blue team will be often unprompted to be given the command, with red sometimes being told to do it. The restrain command is less prone to this though which is interesting.

Do you have listen_key set to a loud or clicky key? I have a theory that it’s picking up some noise as “Blue” before you start speaking. It might also be retaining some audio before you press the listen_key, which would be a problem I’d need to fix in code if it’s the case (though I thought I already fixed it!). It could also be the model but I want to narrow down possibilities.
Can you try testing with listen_key_toggle=2, and try to speak the same commands that were having issues, speaking in the same manner, see if the problem persists?

You shouldn’t need to speak as a TV presenter for it to work accurately, if you do there’s something wrong.

I have noticed saying c4 in this model doesn't really work anymore. If i say c2 it pretty much works as expected.

Is C4 active in your grammar module? It isn’t by default. If it is a valid command in your grammar module, but it’s not being recognised, please confirm / let me know.

[madmaximus101]

My listen on/off key is set to my mouse thumb button it is not really noisy. i have noticed however if i breath or sigh, or if I'm typing away it will recognise noises and attempt to decode them. if i don't want any listen padding at start and end or any automatic voice on/off feature which setting do i change?

edit: will try listen key toggle 2.

edit: i am using the experimental model as provided.

[jwebmeister]

if i don't want any listen padding at start and end or any automatic voice on/off feature which setting do i change?

@madmaximus101 listen_key_padding_end_ms_max and _min are options you can change to set the amount of audio captured after releasing the listen_key.
There shouldn’t be any audio prepended before you press the listen_key (for listen_key_toggle 0 and -1), but if you’re sure it is prepending audio, let me know.

If listen_key_toggle is set to -1, it will always be listening for either YellFreeze or NoiseSink, so it’s fine for it to decode noises, as long as it doesn’t Yell without you saying “freeze” (or similar)… unfortunately it will likely yell at noises at least sometimes unless you have a very quiet environment and good mic. Just let me know if it’s truly unplayable and if it’s worse than the base model.

[madmaximus101]

Listen key toggle 2 seems to be better, hot mic always on seems to be a much better experience overall, no random ghost or added-on commands, much higher success rate in general, i did have some misheard commands, either due to not being clear enough or i assume to quick speaking. With some of the retained audio i've noticed i seem to have a tendancy to breath in or make an innitial "opening mouth sound" as i click the hot mic button or just after i did. May have to learn to not do that lol. I also noticed my mic volume was way up. That might be a contributing factor also - potential for minor distortion of sound to ruin things etc. Will lower mic volume lol.

Edit: for reference my headset is the sennhieser gsp 670. I'd say it's better than average quality for sure.

Here is a link to a gameplay session using the Listen key toggle 2 hot mic always on. Experimental Kaldi model as provided.
Edit: as of right now HD quality is still being uploaded. Text on screen will be hella blurry until it finishes - 30-45mins.
https://www.youtube.com/watch?v=Mxzgd5aaR4Y

I am attempting to run the test thing in Powershell. I think i'm running the command correctly & nothing is happening? the command runs, but i get no results, output or files generated?

[jwebmeister]

I am attempting to run the test thing in Powershell. I think i'm running the command correctly & nothing is happening? the command runs, but i get no results, output or files generated?

@madmaximus101 check in the retain.tsv, are the referenced file paths to the .wav files correct? e.g. ./cleanaudio_cmds/retain-123.wav

[madmaximus101]

i downloaded the tacspeak app & kaldi model as is. Haven't changed anything. If I forgot an instruction in regards to these needing filepaths modified i apologise.

Pic of my retain.tsv

I noticed in your powershell window. The '. after test_model were grey. In my powershell window the '. after test_model is blue. Thought i'd point that out just incase that means anything.

pic of my user settings.

[jwebmeister]

@madmaximus101 You’re running the command with ./cleanaudio_cmds/retain.tsv whereas it should probably be ./retain/retain.tsv

It doesn’t matter what the path is as long as:

• ./somedir/retain.tsv is a valid file and path
• the .wav file paths in the retain.tsv are valid files and paths.

[madmaximus101]

Ahh, a simple filepath error. I copy pasted the example command given without a second thought 😅. Appreciate the patience & help mate.

…

On Tue, 23 Jan 2024, 10:05 pm Joshua Webb, ***@***.***> wrote: @madmaximus101 <https://github.com/madmaximus101> You’re running the command with ./cleanaudio_cmds/retain.tsv whereas it should probably be ./retain/retain.tsv It doesn’t matter what the path is as long as: - ./somedir/retain.tsv is a valid file and path - the .wav file paths in the retain.tsv are valid files and paths. — Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BEXH6Y7H7AZW2D2WDPLFT6LYP6VBZAVCNFSM6AAAAABCDGEX5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBVHE3DSMZSHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[madmaximus101]

ok, i am at a point where i feel i am ready to start doing the initial collecting of data in somewhat of an organised manner, will start collating data and using the scripts etc. Is there a map in particular you would like me to play to reference for your own comparisons to make things easier? As less variables as possible etc. Any particular commands or ways of speaking you would like me to try?

[madmaximus101]

@jwebmeister

Some initial notes & basic observations so far with some newly found quirks, yay! 😛

Lowering mic gain (I had my mic gain set to silly levels (in-game) for some reason, i don't remember doing it or why i would lol? it is now at 100%) seems to have helped a lot with accuracy of words & randomly detected noise attempting to be decoded. I no longer get the tapping of my keyboard or loud sighs, or mouth noises being picked up & tacspeak trying to decode it.
I apologise for missing the stupid high mic gain levels with my earlier testing & wasting your time with that - my bad.

I am the only person in my house with 2 cats. I was using Listen key toggle -1 when it was their dinner time, they were meowing right underneath my chair & the Noise sink feature didn't detect it at all & there were no noises attempted to be decoded or false commands given.

Something of note a weird quirk i'm assuming with how Ready or Not is designed with any Kaldi model I've used. Some of the speech commands were completely different from what i said. I should of realised this instantly as I've pointed this exact issue out before lol. Took me a bit to figure out what the F**K was happening. I was seriously HUH!??
I figured out these random commands being executed different to what was spoken were happening when I was accidentally looking through multiple doorways. The commands given I guess was tacspeaks interpretation of what command I was attempting to give with what was available to be given. This quirk also happened when interacting with an ajar door. I first noticed this particular quirk when I was attempting to give a command to mirror the door when i didn't realise it was ajar. I then tested further & found any door command requiring physical interaction or placement of a device (wedge, c2, mirror) when a door was ajar would result in blue or red being given a command to stack up, or another basic command such as fall in or cover me sometimes breach & clear would be executed.

Is there a way to make a command so when a command for wedge, mirror, c2 is given on an ajar door. The operator closes the door & if possible continues the intended command(s) given?

I have also noticed another door quirk again i think due to Ready or Not's design.
If I command blue team or red team to stack up on a door & then command the other team to stack up on the same door. It defaults the command to the team already stacked up on the door, so nothing happens. It also defaults any given door related command whilst looking at the door to the team that is currently stacked up to said door. Example: Blue team stack up, I then would speak Red team breach and clear. Blue team would breach and clear.

Edit: in hindsight i realise i should of tested this with other commands such as fall in, on me, cover me, commanding one team to do these commands whilst looking at the door the second team was stacked up on. Will update comment with this result if you would like that.

Testing the listen key toggle 2 setting with the experimental kaldi model I've noticed if I stop briefly then continue it will give an unintentional command mid-sentence (blue team "slight pause" breach and clear) I am attributing this to my own cautious bias & my own learnt speech habits interacting with Tacspeak. When I speak in one fluid continuous sentence it does seem to work, although listen key toggle 2 setting does sometimes pick up my random mouth noises when i sigh louder than normal, or if I make a "tutting noise" haha. This issue is very, very much reduced with mic gain now at 100%, basically almost a non-issue at this point.
If i mucked up speaking a command (brain fart) with Listen key toggle 2 & a command was given i didn't intend I would speak "fall in". Depending on my level of panic or quick speaking this sometimes worked & sometimes didn't. Listen key toggle 2 I found requires you keep your speech in check 😅

The Listen key toggle -1 setting with the experimental Kaldi model also seems to have less unintentional noises detected with my mic gain now at 100%. If I mucked up a command (brain fart) i would let go of mouse thumb button which would lessen the impact of the error. I would then press thumb button again & issue the fall in command to stop the command currently happening which would come out correctly. I know there is a halt, cancel, stop command but my brain just thinks of fall in in the moment lol.

[madmaximus101]

@jwebmeister Noisesink seems to work as intended with high accuracy in the few times it's activated since i've corrected my mic gain to 100%. If I do make a noise detected by Noisesink such as a burp, cough, or i hit the desk it activates.

I will test Noisesink with words & phrases a person might say in surprise, fright, dissapointment or anger.

[madmaximus101]

@jwebmeister newly found quirks aside i am of the opinion the Listen key toggle -1 setting is pretty good and pretty much working as intended...now that my mic gain is at appropriate levels - again i apologise.

I will test this setting further whilst watching out for where i'm looking when giving said commands.

[madmaximus101]

The "F word & F you" are often picked up as

on_recognition (INFO): KaldiRule(16, ReadyOrNot_priority::YellFreeze) | drop
Freeze!