Voice Recognition on the Orange Pi Zero (DietPi Armbian)
November 26, 2017
Updated: April 11, 2018
Google Home Mini avec Domoticz par l'entremise de IFTTT

The installation of two Python libraries for voice recognition is taken up below. First I will go back to the hotword recognition engine snowboy from KITT.AI. Rather than relying on the work of a third party as in a previous post, it is installed from its repository on GitHub.

Secondly, the SpeechRecognition library by Anthony Zhang (Uberi) will be installed. This is a well made library that provides a uniform Python interface with many speech recognition engines. I used it to access two online services: Google Speech Recognition and Microsoft Bing Voice Recognition as well as with the off line engine Pocket Sphynx from Carnegie Mellon University. Mr Zhang has also considerably simplified adding three languages (French, Italian and Mandarin) to that engine.

Table of Contents

  1. Starting Point
  2. Installing Python 3
  3. Installing Development Prerequisites
  4. Installing Audio Prerequisites
  5. Installing snowboy
  6. Installing VoiceRecognition
  7. More languages in VoiceRecognition

  1. Starting Point
  2. I started from a clean installation of DietPi available at dietpi.com/. Two sections describe in a previous post contain details: 2. Installing DietPi and 4. Testing the OPiZ Audio.

    A word of caution. If you followed along with my previous posts and Google Assistant is running as a service on the OPiZ, it must be disabled since hot word detection will now be performed by dietpi@domopiz:~$ sudo systemctl stop google-assistant-demo.service dietpi@domopiz:~$ sudo systemctl disable google-assistant-demo.service Removed symlink /etc/systemd/system/google-assistant.service. Removed symlink /etc/systemd/system/multi-user.target.wants/google-assistant-demo.service.

  3. Installing Python 3
  4. The Armbian image from DietPi does not contain Python. So version 3 will be installed along with the virtual environment module, venv. I used the default Debian package manager to install everything.

    dietpi@domopiz:~$ sudo apt-get update && sudo apt-get upgrade ... dietpi@domopiz:~$ sudo apt-get install python3 python3-dev python3-venv ... After this operation, 84.2 MB of additional disk space will be used. Do you want to continue? [Y/n] y ... dietpi@domopiz:~$ python3 --version Python 3.4.2 dietpi@domopiz:~$ which python3 /usr/bin/python3

    There is a newer version of Python 3, (version 3.6), but hopefully version 3.4 will be recent enough for my purposes.

    I also installed the tools for creating and updating virtual environments described in Python 3 virtual environments. All Python code will be installed in virtual environments.

  5. Installing Development Prerequisites
  6. The missing instructions on how to add the Jessie backports repository have been added in this section.
    April 11, 2018

    A number of packages are needed including the GNU compilers and make utility.

    dietpi@domopiz:~$ sudo apt-get install build-essential Reading package lists... Done ... After this operation, 82.3 MB of additional disk space will be used. Do you want to continue? [Y/n] y ... dietpi@domopiz:~$ g++ --version g++ (Debian 4.9.2-10) 4.9.2

    The Simplified Wrapper and Interface Generator (SWIG) is needed to create some Python wrappers of C/C++ libraries. It turns out that snowboy needs version 3.0.10 which is not contained in the Jessie repository.

    dietpi@domopiz:~$ apt-cache policy swig swig: Installed: (none) Candidate: 2.0.12-1 Version table: 2.0.12-1 0 500 http://ftp.debian.org/debian/ jessie/main armhf Packages

    Luckily, a newer version is available in the backports repository. Details about adding the repository can be found here. Once the package signing keys and the repository listingls have been added, the list of packages has to be updated. Then a check can be made that the available version of SWIG is recent enough before finally installing it.

    dietpi@domopiz:~$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7638D0442B90D010 ... dietpi@domopiz:~$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 8B48AD6246925553 ... dietpi@domopiz:~$ echo 'deb http://httpredir.debian.org/debian jessie-backports main contrib non-free' | sudo tee -a /etc/apt/sources.list.d/jessie-backports.list dietpi@domopiz:~$ sudo apt udpate ... dietpi@domopiz:~$ apt-cache policy swig swig: Installed: (none) Candidate: 2.0.12-1 Version table: 3.0.10-1.1~bpo8+1 0 100 http://ftp.debian.org/debian/ jessie-backports/main armhf Packages 2.0.12-1 0 500 http://ftp.debian.org/debian/ jessie/main armhf Packages dietpi@domopiz:~$ sudo apt-get -t jessie-backports install swig ... After this operation, 4 851 kB of additional disk space will be used. Do you want to continue? [O/n] o Oui/Non instead of Yes/No! ... Unpacking swig3.0 (3.0.10-1.1~bpo8+1) ... Setting up swig3.0 (3.0.10-1.1~bpo8+1) ... dietpi@domopiz:~$ ls -l /usr/bin/swi* lrwxrwxrwx 1 root root 7 mai 18 2017 /usr/bin/swig -> swig3.0 -rwxr-xr-x 1 root root 1182256 mai 18 2017 /usr/bin/swig3.0

    I think the symbolic link swig to swig3.0 is not created if apt-get is asked to install swig3.0. I had to create that link in a previous run at this.

    While is possible to get the latest version of a project on github with wget or by other means, I find it useful to install the version control system git.

    dietpi@domopiz:~$ sudo apt-get install git Reading package lists... After this operation, 16.6 MB of additional disk space will be used. Do you want to continue? [Y/n] y

    If installing Pocket Sphinx then a recent version of Pulse development library is needed. Again this package must be obtained from the backports repository.

    dietpi@domopiz:~$ sudo apt-get -t jessie-backports install libpulse-dev ... After this operation, 42,7 MB of additional disk space will be used. Do you want to continue? [Y/n] y ... Setting up libjson-c2:armhf (0.11-4) ... Setting up libpython2.7-stdlib:armhf (2.7.9-2+deb8u1) ... Setting up python2.7 (2.7.9-2+deb8u1) ... Setting up libpython-stdlib:armhf (2.7.9-1) ... Setting up python (2.7.9-1) ... Setting up libasyncns0:armhf (0.8-5) ... Setting up libelf1:armhf (0.159-4.2) ... Setting up libogg0:armhf (1.3.2-1) ... Setting up libflac8:armhf (1.3.0-3) ... Setting up libglib2.0-0:armhf (2.48.0-1~bpo8+1) ... Setting up x11-common (1:7.7+7) ... Setting up libice6:armhf (2:1.0.9-1+b1) ... Setting up libpcrecpp0:armhf (2:8.35-3.3+deb8u4) ... Setting up libsm6:armhf (2:1.2.2-1+b1) ... Setting up libvorbis0a:armhf (1.3.4-2) ... Setting up libvorbisenc2:armhf (1.3.4-2) ... Setting up libsndfile1:armhf (1.0.25-9.1+deb8u1) ... Setting up libxau6:armhf (1:1.0.8-1) ... Setting up libxdmcp6:armhf (1:1.1.1-1+b1) ... Setting up libxcb1:armhf (1.10-3+b1) ... Setting up libx11-data (2:1.6.2-3+deb8u1) ... Setting up libx11-6:armhf (2:1.6.2-3+deb8u1) ... Setting up libx11-xcb1:armhf (2:1.6.2-3+deb8u1) ... Setting up libxext6:armhf (2:1.3.3-1) ... Setting up libxi6:armhf (2:1.7.4-1+deb8u1) ... Setting up libxtst6:armhf (2:1.2.2-1+deb8u1) ... Setting up libpulse0:armhf (7.1-2~bpo8+1) ... Setting up libpulse-mainloop-glib0:armhf (7.1-2~bpo8+1) ... Setting up libglib2.0-data (2.48.0-1~bpo8+1) ... Setting up libglib2.0-bin (2.48.0-1~bpo8+1) ... Setting up libpcre3-dev:armhf (2:8.35-3.3+deb8u4) ... Setting up zlib1g-dev:armhf (1:1.2.8.dfsg-2+b1) ... Setting up libglib2.0-dev (2.48.0-1~bpo8+1) ... Setting up libpulse-dev:armhf (7.1-2~bpo8+1) ...

    As can be seen that pulled in many packages including Python 2.7.9 which I did not want... oh well.

    If compiling the snowboy python wrapper from source, then ATLAS (Automatically Tuned Linear Algebra Software) must be installed. It automatically generates an optimized Basic Linear Algebra Subroutines (BLAS) library in C (and also Fortran, but I doubt that would be used here). It also provides a subset of the linear algebra routines from the Linear Algebra Package (LAPACK) library. The latter provides routines for solving systems of linear equations with classic matrix factorization methods such as LU, QR and Cholesky decompositions. It amazes me that monitoring a continuous stream of sound to pick out a key word, which involves use of BLAS/LAPACK, will occupy one of the four cores of the tiny Orange Pi Zero only a small percentage of its time. I always thought of that package as of something big and complex.

    dietpi@domopiz:~$ sudo apt-get install libatlas-base-dev ... After this operation, 26.2 MB of additional disk space will be used. Do you want to continue? [Y/n] y

  7. Installing Audio Prerequisites
  8. The module PyAudio, used by VoiceRecognition and snowboy, is a Python wrapper for the cross-platform audio I/O library PortAudio which must be installed.

    dietpi@domopiz:~$ sudo apt-get install portaudio19-dev Reading package lists... Done ... After this operation, 3,621 kB of additional disk space will be used. Do you want to continue? [Y/n] y

    VoiceRecognition also requires FLAC (Free Lossless Audio Codec). It is an open source lossless alternative to MP3.

    dietpi@domopiz:~$ sudo apt-get install flac Reading package lists... Done ... Do you want to continue? [Y/n] y ... Setting up flac (1.3.0-3) ... ...

  9. Installing snowboy
  10. At last, snowboy can be installed. First we get the source from the GitHub repository. Then a virtual environment will be created in that project. It is a good idea to exclude that environment from the version control system.

    dietpi@domopiz:~$ git clone https://github.com/Kitt-AI/snowboy.git dietpi@domopiz:~$ cd snowboy dietpi@domopiz:~/snowboy$ mkvenv pvenv creating virtual environment /home/dietpi/snowboy/pvenv updating virtual environment /home/dietpi/snowboy/pvenv dietpi@domopiz:~/snowboy$ nano .gitignore
    add at the end # exclude virtual environment /pvenv

    Activate the virtual environment and install the Python PyAudio module.

    dietpi@domopiz:~/snowboy$ ve pvenv (pvenv) dietpi@domopiz:~/snowboy$ pip install PyAudio Downloading/unpacking PyAudio Downloading PyAudio-0.2.11.tar.gz Running setup.py (path:/tmp/pip-build-_5uc4fsj/PyAudio/setup.py) egg_info for package PyAudio ... Successfully installed PyAudio Cleaning up...

    A look at the Python 3 examples shows that the three demo programs import snowboydecoder.py which in turn imports snowboydetect.py. The latter is the Python wrapper for the shared library _snowboydetect.so. These two files, which should be in the swig/Python3 directory, are missing.

    (pvenv) dietpi@domopiz:~/snowboy$ ls -l examples/Python3 -rw-r--r-- 1 dietpi dietpi 757 déc 16 16:20 demo.py ... -rw-r--r-- 1 dietpi dietpi 6909 déc 16 16:20 snowboydecoder.py lrwxrwxrwx 1 dietpi dietpi 35 déc 16 16:20 snowboydetect.py -> ../../swig/Python3/snowboydetect.py lrwxrwxrwx 1 dietpi dietpi 36 déc 16 16:20 _snowboydetect.so -> ../../swig/Python3/_snowboydetect.so (pvenv) dietpi@domopiz:~/snowboy$ ls -l swig/Python3 total 8 -rw-r--r-- 1 dietpi dietpi 2912 déc 16 22:09 Makefile -rw-r--r-- 1 dietpi dietpi 525 déc 16 22:09 snowboy-detect-swig.i (pvenv) dietpi@domopiz:~/snowboy$

    Time to create the missing shared library and Python wrapper.

    (pvenv) dietpi@domopiz:~/snowboy$ cd swig/Python3 (pvenv) dietpi@domopiz:~/snowboy/swig/Python3$ make ... (pvenv) dietpi@domopiz:~/snowboy/swig/Python3$ ls -l (pvenv) dietpi@domopiz:~/snowboy/swig/Python3$ ls -l total 1452 -rw-r--r-- 1 dietpi dietpi 2912 déc 16 22:09 Makefile -rw-r--r-- 1 dietpi dietpi 4857 déc 16 22:18 snowboydetect.py -rwxr-xr-x 1 dietpi dietpi 1043484 déc 16 22:18 _snowboydetect.so -rw-r--r-- 1 dietpi dietpi 157660 déc 16 22:18 snowboy-detect-swig.cc -rw-r--r-- 1 dietpi dietpi 525 déc 16 22:09 snowboy-detect-swig.i -rw-r--r-- 1 dietpi dietpi 266120 déc 16 22:18 snowboy-detect-swig.o

    When make invokes the C++ compiler, g++, the command line displayed on the console, makes it clear that it is the snowboy C/C++ library for the Raspian that is converted to a Python library by SWIG. No problem, it works quite well with Armbian on the Orange Pi Zero.

    Aside from a warning, everything seemed to go well, so I backed out of the swig directory and changed to the examples/Python3 directory, to try out the demonstration scripts.

    (pvenv) dietpi@domopiz:~/snowboy/swig/Python3$ cd ../../examples/Python3 (pvenv) dietpi@domopiz:~/snowboy/examples/Python3$ python demo.py resources/snowboy.umdl Traceback (most recent call last): File "demo.py", line 1, in import snowboydecoder File "/home/dietpi/snow2/examples/Python3/snowboydecoder.py", line 5, in from . import snowboydetect SystemError: Parent module '' not loaded, cannot perform relative import

    That was a bit of an anti-climax. However, it was relatively easy to fix the problem: just remove the relative path to snowboydetect.py in the import command of snowboydecoder.py.

    (pvenv) dietpi@domopiz:~/snowboy/examples/Python3$ nano snowboydecoder.py
    !/usr/bin/env python import collections import pyaudio #from . import snowboydetect import snowboydetect or just delete "from ." import time import wave import os import logging ... save and exit nano

    Now all three demonstration scripts work:

    (pvenv) dietpi@domopiz:~/snowboy/examples/Python3$ python demo.py resources/snowboy.umdl Listening... Press Ctrl+C to exit ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.front ... ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline say: "hello" say: "snowman" say: "snowbot" say: "snowboy" INFO:snowboy:Keyword 1 detected at time: 2017-11-24 01:18:21 it works! CtrlC ^C (pvenv) dietpi@domopiz:~/snowboy/examples/Python3$ python demo2.py resources/snowboy.umdl resources/alexa.umdl Listening... Press Ctrl+C to exit ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.front ... ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline say: "alexi" say: "alexa" INFO:snowboy:Keyword 2 detected at time: 2017-11-24 12:43:10 success! say: "snowboy" INFO:snowboy:Keyword 1 detected at time: 2017-11-24 12:43:12 success! CtrlC ^C (pvenv) dietpi@domopiz:~/snowboy/examples/Python3$ python demo3.py resources/snowboy.wav resources/snowboy.umdl Hotword Detected! as expected (pvenv) dietpi@domopiz:~/snowboy/examples/Python3$ python demo3.py resources/ding.wav resources/snowboy.umdl Hotword Not Detected! as expected.

  11. Installing SpeechRecognition
  12. dietpi@domopiz:~$ mkdir speechrecognition dietpi@domopiz:~$ cd speechrecognition dietpi@domopiz:~/speechrecognition$ mkvenv pvenv creating virtual environment /home/dietpi/speechrecognition/pvenv updating virtual environment /home/dietpi/speechrecognition/pvenv dietpi@domopiz:~/speechrecognition$ ve pvenv (pvenv) dietpi@domopiz:~/speechrecognition$ pip install -U PyAudio ... Successfully installed PyAudio Cleaning up... (pvenv) dietpi@domopiz:~/speechrecognition$ pip install -U SpeechRecognition Successfully installed SpeechRecognition Cleaning up... (pvenv) dietpi@domopiz:~/speechrecognition$

    Now we do not have the example scripts.

    (pvenv) dietpi@domopiz:~/speechrecognition$ wget https://github.com/Uberi/speech_recognition/archive/master.zip (pvenv) dietpi@domopiz:~/speechrecognition$ unzip -j master.zip "speech_recognition-master/examples/*" -d examples ... (pvenv) dietpi@domopiz:~/speechrecognition$ cd examples (pvenv) dietpi@domopiz:~/speechrecognition/examples$ python calibrate_energy_threshold.py ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.front ... Say something! I said: "Hello, how's it going" Google Speech Recognition thinks you said hello how's it going (pvenv) dietpi@domopiz:~/speechrecognition/examples$

    However

    (pvenv) dietpi@domopiz:~/speechrecognition/examples$ python microphone_recognition.py ... ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline Say something! I said: "Hello, how's it going" but the program did not respond ^CTraceback (most recent call last): File "microphone_recognition.py", line 11, in audio = r.listen(source) File "/home/dietpi/speechrecognition/pvenv/lib/python3.4/site-packages/speech_recognition/__init__.py", line 652, in listen buffer = source.stream.read(source.CHUNK) File "/home/dietpi/speechrecognition/pvenv/lib/python3.4/site-packages/speech_recognition/__init__.py", line 161, in read return self.pyaudio_stream.read(size, exception_on_overflow=False) File "/home/dietpi/speechrecognition/pvenv/lib/python3.4/site-packages/pyaudio.py", line 608, in read return pa.read_stream(self._stream, num_frames, exception_on_overflow) KeyboardInterrupt

    Looking at the two scripts, it became obvious that the difference was in a single line found in calibrate_energy_threshold.py just before recording sounds which was missing from microphone_recognition.py.

    #!/usr/bin/env python3 # NOTE: this example requires PyAudio because it uses the Microphone class import speech_recognition as sr # obtain audio from the microphone r = sr.Recognizer() with sr.Microphone() as source: r.adjust_for_ambient_noise(source) # listen for 1 second to calibrate the microphone print("Say something!") audio = r.listen(source)

    With that simple addition, microphone_recognition.py worked.

    (pvenv) dietpi@domopiz:~/speechrecognition/examples$ nano microphone_recognition.py (pvenv) dietpi@domopiz:~/speechrecognition/examples$ python microphone_recognition.py ... ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline Say something! Sphinx error; missing PocketSphinx module: ensure that PocketSphinx is set up correctly. Google Speech Recognition thinks you said hi what day are we Traceback (most recent call last): File "/home/dietpi/speechrecognition/pvenv/lib/python3.4/site-packages/speech_recognition/__init__.py", line 885, in recognize_google_cloud try: json.loads(credentials_json) File "/usr/lib/python3.4/json/__init__.py", line 318, in loads return _default_decoder.decode(s) File "/usr/lib/python3.4/json/decoder.py", line 343, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.4/json/decoder.py", line 361, in raw_decode raise ValueError(errmsg("Expecting value", s, err.value)) from None ValueError: Expecting value: line 1 column 1 (char 0)

    The error is caused by the missing keys for online voice recognition. Currently, only Google Speech Recognition is available freely without prior registration. I added my key for Microsoft Bing and commented out the other engines and renamed the script. I also changed the script to get a better idea of how long the speech recognition takes to analyze the sound. The ellipsis is the last thing printed before asking the engine to convert speech to text, "you said..." is printed when the results come back.

    (pvenv) dietpi@domopiz:~/speechrecognition/examples$ python microphone_recognition_en.py ... ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline Say something! a prize if you guess what I said Sphinx thinks ...Sphinx error; missing PocketSphinx module: ensure that PocketSphinx is set up correctly. Google Speech Recognition thinks ... you said good evening it's Saturday night Microsoft Bing Voice Recognition thinks ... you said Good evening it's Saturday night.

    There remains the task of installing PocketSphinx. This is easily done although it takes a while. The screen will be filled many times over as both sphinxbase and pocketsphinx are compiled.

    (pvenv) dietpi@domopiz:~/speechrecognition/examples$ pip install -U pocketsphinx Downloading/unpacking pocketsphinx ... Successfully installed pocketsphinx Cleaning up... (pvenv) dietpi@domopiz:~/speechrecognition/examples$

    Now the script will run without complaining.

    (pvenv) dietpi@domopiz:~/speechrecognition/examples$ python microphone_recognition_en.py ... ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline Say something! Sphinx thinks ... you said: good evening if the night Google Speech Recognition thinks ... you said: good evening it's Saturday night Microsoft Bing Voice Recognition thinks ... you said: Good evening it's Saturday night.

    It is painfully obvious that Pocket Sphinx runs too slowly on the Orange Pi Zero to be used in real time applications. And the combination of the quality of the microphone, my mumbling, and the power of the software gives less than optimal results. However, Google Speech Recognition and Microsoft Bing Voice Recognition work quite well, in regards to both speed and accuracy.

  13. More Languages in SpeechRecognition
  14. Doing speech recognition in other languages than (American) English is rather easy with Google Speech Recognition and Microsoft Bing Voice Recognition. All that needs to be done is to add a parameter to the r.recognize_xxxxx where xxxx is the engine. I copied microphone_recognition.py to microphone_recognition_en.py and to microphone_recognition_fr.py. Then I edited the latter.

    # recognize speech using Sphinx try: print("Sphinx thinks ...", end="", flush=True) print(" you said: " + r.recognize_sphinx(audio)) became # recognize speech using Sphinx try: print("Selon Sphinx ...", end="", flush=True) print(" vous avez dit : " + r.recognize_sphinx(audio, language="fr-FR"))

    And I did the same for the two online engines also.

    (pvenv) dietpi@domopiz:~/speechrecognition/examples$ python microphone_recognition_fr.py ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline ... Parlez! Selon Sphinx ...Sphinx error; missing PocketSphinx language data directory: "/home/dietpi/speechrecognition/pvenv/lib/python3.4/site-packages/speech_recognition/pocketsphinx-data/fr-FR" Selon Google Speech Recognition ... vous avez dit : bonjour C'est Dimanche matin Selon Microsoft Bing Voice Recognition ... vous avez dit : Bonjour c'est dimanche matin.

    As you would expect that did not work for CMU Sphinx. But we got some useful information in the error message: the location of the French dictionary

    The instructions for Installing other languages are not up to date, but the links to the French, Italian and Manadarin Chinese language packs are valid. So get the archive from Google Drive repository and extract the language pack to the directory kindly identified by the error message.

    (pvenv) dietpi@domopiz:~/speechrecognition/examples$ python microphone_recognition_fr.py ... ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline Parlez! Selon Sphinx ... vous avez dit : cette pour aujourd'hui Selon Google Speech Recognition ... vous avez dit : c'est tout pour aujourd'hui Selon Microsoft Bing Voice Recognition ... vous avez dit : Météo pour aujourd'hui.

    And the prize goes to Google. A long time ago, I had a boss known as "the Maltese Mumbler" among his staff. He probably came from Malta, but there was no doubt that he was very difficult to understand over a telephone. Perhaps I picked up his diction.