Python 3 virtual environments
November 26, 2017
Updated: December 14, 2017

The use of virtual Python environments is highly recommended but why? What is the link between the module venv and the virtualenv utility? And why are pip and setuptools playing hide and seek as I try to install them on? Indeed should they be installed at all?

This posting is about virtual Python environments on three different Linux systems.

Table of Contents

  1. Python Virtual Environments
  2. Python 2 and 3 Installed in Ubuntu 17.10
  3. Installing Python 3 in Raspbian (DietPi)
  4. Creating Python 3 Virtual Environments
  5. Updating Python 3 Virtual Environments
  6. Automating with a Script
  7. Using a Virtual Environment
  8. Examples
  9. Installing pip

  1. Python Virtual Environments
  2. Most Python projects are not standalone monoliths. They use standard libraries provided with Python itself. Often they also use third party libraries. But that can be tricky. Python project 1 could require the installation version 2.18.4 of requests a commonly used http library. What happens if Python project 2 decides to install the older 1.2.1 version of requests? It could clobber the newer version 2.18.4 and then Python project 1 might be broken. Virtual environments are meant to solve this problem. Each virtual environment contains all needed libraries for a Python project in a "box" that is independent of all other virtual environments.

    The "box" is just a directory that contains a few utilities, directories and copies of or links to the Python compiler. In the current versions of Python (version 3.5 and newer) it is created with the venv script, in older versions of Python (prior to version 3.3) that was done with virtualenv. In between (Python 3.3 and 3.4) the script was called pyvenv.

    At this point, reading A non-magical introduction to Pip and Virtualenv for Python beginners by Jamie Matthews would be a good idea. Just remember, that text was written in 2013 and since then venv is preferred to virtualenv in Python 3x (among other things, virtualenv copies the compiler binaries to the virtual environment while venv creates links). Also, I will not be installing pip et al globally, but only in virtual environments. More about that latter.

    The Python package management system is called pip (pip installs packages). It is to Python what APT (advanced package tools) is to Debian. Funny thing though, pip and apt-get can get all tangled up. Like apt, pip installs Python modules from an "official" repository, PyPi, nicknamed the Cheese Shop, a well-known sketch from Monty Python. Pip also installs packages found elsewhere such as github repositories.

    The package installation tool uses Setuptools. For the most part this will be transparent and we do not have to look at the details.

    The third part of the basic elements is Wheel. It is a "built-package format for Python". Essentially, a wheel is an appropriately named archive containing all the files needed to install a package setup in a specific hierarchy. The wheel project also created a tool that is included in setuptools and which is used by pip to actually perform the installation of a package supplied as a wheel.

    Python packages themselves are divided into two categories: system and site. The first refers to packages that are part of the standard Python library, while later refers to packages from third parties. These will be installed in an activated the virtual environment automatically by pip.

  3. Python 2 and 3 Installed on Ubuntu 17.10 and Raspbian 9.1
  4. Ubuntu 17.10 comes with two version of Python installed: 2.7.14 and 3.6.3. To quote the python.org wiki Python 2.x is legacy, Python 3.x is the present and future of the language. One would think that version 3 can handle any version 2 code. Generally, speaking that is true but not always. Version 3 of Python introduce new keywords. If a version 2 script used any of these words as variable names, then these will have to be changed. Also, some version 2 libraries may not have been ported to version 3. I am new to Python, so I have no investment in the older version and will be using version 3.

    michel@hp:~$ python -V Python 2.7.14 michel@hp:~$ python3 -V Python 3.6.3 michel@hp:~$ ls -l /usr/bin/python* lrwxrwxrwx 1 root root 9 sep 23 05:18 /usr/bin/python -> python2.7 lrwxrwxrwx 1 root root 9 sep 23 05:18 /usr/bin/python2 -> python2.7 -rwxr-xr-x 1 root root 3617176 sep 23 19:06 /usr/bin/python2.7 lrwxrwxrwx 1 root root 9 nov 29 15:46 /usr/bin/python3 -> python3.6 -rwxr-xr-x 1 root root 4568920 oct 3 18:45 /usr/bin/python3.6 -rwxr-xr-x 1 root root 4568920 oct 3 18:45 /usr/bin/python3.6m lrwxrwxrwx 1 root root 10 nov 29 15:46 /usr/bin/python3m -> python3.6m michel@hp:~$ cmp -l /usr/bin/python3.6 /usr/bin/python3.6m michel@hp:~$ michel@hp:~$ ls -i /usr/bin/python3.6* 48497800 /usr/bin/python3.6 48497801 /usr/bin/python3.6m

    As can be seen, python and python2 are symbolic links (aliases if you prefer) to the executable python2.7. Similarly, python3 is a symbolic link to the executable python3.6.

    The virtual environment utility virtualenv which is compatible with versions 2 and 3 of Python is not available nor is venv which is the version 3 replacement. Similarly the package managers pip and pip3 are not installed by default.

    Strangely, there is another Python executable, python3.6m (alias python3.m) which is exactly the same size as python3.6. A binary comparison of the two files reveals no difference between these distinct files (since they point to different inodes). Python Enhancement Proposal (PEP) 3149 provides a partial explanation

    Python implementations MAY include additional flags in the file name tag as appropriate. For example, on POSIX systems these flags will also contribute to the file name:
    • --with-pydebug (flag: d)
    • --with-pymalloc (flag: m)
    • --with-wide-unicode (flag: u)

    By default in Python 3.2, configure enables --with-pymalloc so shared library file names would appear as foo.cpython-32m.so. When the other two flags are also enabled, the file names would be foo.cpython-32dmu.so.

    I interpret that to mean that by default version 3.6 is compiled using pymalloc for heap memory allocation instead of the system malloc. Accordingly, shared library files should end in 36m.so. Actually this naming convention is even more complex.

    michel@hp:~$ ls /usr/lib/python3/dist-packages/*.so /usr/lib/python3/dist-packages/apt_inst.cpython-36m-x86_64-linux-gnu.so /usr/lib/python3/dist-packages/apt_pkg.cpython-36m-x86_64-linux-gnu.so ...

    Hopefully, this will not prove important in the future.

    The situation is almost the same on Raspbian. Both legacy Python and a current Python interpreter are included in the distribution although they are slightly older versions.

    pi@rpi3:~ $ python -V Python 2.7.13 pi@rpi3:~ $ python3 -V Python 3.5.3 pi@rpi3:~ $ pip -bash: pip: command not found pi@rpi3:~ $ pip3 -bash: pip3: command not found pi@rpi3:~ $ ls -l /usr/bin/python* lrwxrwxrwx 1 root root 9 Jan 24 2017 /usr/bin/python -> python2.7 lrwxrwxrwx 1 root root 9 Jan 24 2017 /usr/bin/python2 -> python2.7 -rwxr-xr-x 1 root root 3162224 Jan 19 2017 /usr/bin/python2.7 lrwxrwxrwx 1 root root 9 Jan 20 2017 /usr/bin/python3 -> python3.5 -rwxr-xr-x 2 root root 3976264 Jan 19 2017 /usr/bin/python3.5 -rwxr-xr-x 2 root root 3976264 Jan 19 2017 /usr/bin/python3.5m lrwxrwxrwx 1 root root 10 Jan 20 2017 /usr/bin/python3m -> python3.5m pi@rpi3:~ $ ls -i /usr/bin/python3.5* 7960 /usr/bin/python3.5 7960 /usr/bin/python3.5m

    Also, here it seems that python3.5m and python3.5 are hard links to the same inode.

    On both of these systems, a virtual environment cannot be created. A package, python3-venv must be installed.

    At the same time, python3-dev will be installed as it will be needed for the example to follow.

    The installation of those packages on the Raspberry Pi is shown below. The same procedure was followed on Ubuntu.

    pi@rpi3:~ $ sudo apt-get install --upgrade python3-dev python3-venv Reading package lists... Done After this operation, 51.1 MB of additional disk space will be used. Do you want to continue? [Y/n] y ... Setting up python3-venv (3.5.3-1) ... ... Setting up python3-dev (3.5.3-1) ... pi@rpi3:~ $

  5. Installing Python 3 in Armbian (DietPi)
  6. DietPi is a pared-down Linux server distribution (Armbian) available for many single board computers such as the different models of the Raspberry Pi, Orange Pi, Banana Pi meant to be used as servers. I am using it on an Orange Pi Zero. This distribution does not contain Python which must be installed with the apt-get utility.

    dietpi@domopiz:~$ sudo apt-get install python3 python3-dev python3-venv Reading package lists... Done ... 0 upgraded, 32 newly installed, 0 to remove and 0 not upgraded. Need to get 50.6 MB of archives. After this operation, 84.2 MB of additional disk space will be used. Do you want to continue? [Y/n] y

    It is simple to verify that Python 3.4.2 is installed in the /usr/bin directory and that a legacy version of Python is not present. As with Ubuntu, pip is not installed, which I guess is the norm for Debian based distributions.

    dietpi@domopiz:~$ python -V -bash: python: command not found dietpi@domopiz:~$ python2 -V -bash: python2: command not found dietpi@domopiz:~$ python3 -V Python 3.4.2 dietpi@domopiz:~$ which python3 /usr/bin/python3 dietpi@domopiz:~$ pip -bash: pip: command not found dietpi@domopiz:~$ pip3 -bash: pip3: command not found dietpi@domopiz:~$ ls -d /usr/lib/py* /usr/lib/python2.7 /usr/lib/python3 /usr/lib/python3.4 dietpi@domopiz:~$ ls -d /usr/local/lib/py* /usr/local/lib/python3.4

  7. Creating Python Virtual Environments
  8. Creating a virtual environment is quite simple. Use the venv module giving it the name of virtual environment. There will not be a confirmation, the command prompt will appear after a while.

    michel@hp:~$ python3 -m venv env michel@hp:~$ ls env bin include lib lib64 pyvenv.cfg share

    It is possible to create an environment into a pre-existing directory.

    michel@hp:~$ mkdir -p smith/bye michel@hp:~$ touch smith/hello michel@hp:~$ touch smith/bye/bye michel@hp:~$ python3 -m venv smith michel@hp:~$ ls smith bin bye hello include lib lib64 pyvenv.cfg share

    It looks like the venv module checks for the presence of the file pyvenv.cfg when asked to create a virtual environment in a pre-existing directory. My cursory test shows that if the cfg file is present, venv will do nothing but if the cfg file is not found, then the virtual environment will be created without disturbing files that may have existed in directories name bin, include, etc.

    The site packages installed in the virtual environment by Python 3.6 in Ubuntu 17.10 include pip and setuptools.

    michel@hp:~$ ls -l env/lib/python3.6/site-packages total 32 -rw-r--r-- 1 michel michel 126 déc 10 19:12 easy_install.py drwxr-xr-x 11 michel michel 4096 déc 10 19:12 pip drwxr-xr-x 2 michel michel 4096 déc 10 19:12 pip-9.0.1.dist-info drwxr-xr-x 5 michel michel 4096 déc 10 19:12 pkg_resources drwxr-xr-x 2 michel michel 4096 déc 10 19:12 pkg_resources-0.0.0.dist-info drwxr-xr-x 2 michel michel 4096 déc 10 19:12 __pycache__ drwxr-xr-x 5 michel michel 4096 déc 10 19:12 setuptools drwxr-xr-x 2 michel michel 4096 déc 10 19:12 setuptools-32.3.1.dist-info

    The virtual environment created by Python 3.5 in Raspbian is quite similar but strangely the version of setuptools seems newer.

    pi@rpi3:~ $ ls -l env/lib/python3.5/site-packages total 32 -rw-r--r-- 1 pi pi 126 Dec 12 19:25 easy_install.py drwxr-xr-x 11 pi pi 4096 Dec 12 19:25 pip drwxr-xr-x 2 pi pi 4096 Dec 12 19:25 pip-9.0.1.dist-info drwxr-xr-x 5 pi pi 4096 Dec 12 19:25 pkg_resources drwxr-xr-x 2 pi pi 4096 Dec 12 19:25 pkg_resources-0.0.0.dist-info drwxr-xr-x 2 pi pi 4096 Dec 12 19:25 __pycache__ drwxr-xr-x 5 pi pi 4096 Dec 12 19:25 setuptools drwxr-xr-x 2 pi pi 4096 Dec 12 19:25 setuptools-33.1.1.dist-info

    The virtual environment created by Python 3.4 in Armbian is slightly different and the versions of pip and setuptools are considerably older.

    dietpi@domopiz:~$ ls -l env/lib/python3.4/site-packages total 128 -rw-r--r-- 1 dietpi dietpi 126 Dec 10 19:41 easy_install.py drwxr-xr-x 3 dietpi dietpi 4096 Dec 10 19:41 _markerlib drwxr-xr-x 6 dietpi dietpi 4096 Dec 10 19:41 pip drwxr-xr-x 2 dietpi dietpi 4096 Dec 10 19:41 pip-1.5.6.dist-info -rw-r--r-- 1 dietpi dietpi 101340 Dec 10 19:41 pkg_resources.py drwxr-xr-x 2 dietpi dietpi 4096 Dec 10 19:41 __pycache__ drwxr-xr-x 5 dietpi dietpi 4096 Dec 10 19:41 setuptools drwxr-xr-x 2 dietpi dietpi 4096 Dec 10 19:41 setuptools-5.5.1.dist-info

  9. Updating Python Virtual Environments
  10. The Python Software Foundation recommends updating the pip and setuptools modules installed in a newly created virtual environment. It also suggests installing wheel. All this can be done with pip itself.

    On Ubuntu and Raspbian, which use version 9.0 of pip, only setuptools will be updated as pip is already the most recent version available. As can be seen, both wheel and the newer version of setuptools will be downloaded for installation.

    michel@hp:~$ python3 -m venv env michel@hp:~$ env/bin/pip install -U pip setuptools wheel Requirement already up-to-date: pip in ./env/lib/python3.6/site-packages Collecting setuptools Downloading setuptools-38.2.4-py2.py3-none-any.whl (489kB) 100% |████████████████████████████████| 491kB 2.1MB/s Collecting wheel Downloading wheel-0.30.0-py2.py3-none-any.whl (49kB) 100% |████████████████████████████████| 51kB 3.5MB/s Installing collected packages: setuptools, wheel Found existing installation: setuptools 32.3.1 Uninstalling setuptools-32.3.1: Successfully uninstalled setuptools-32.3.1 Successfully installed setuptools-38.2.4 wheel-0.30.0sudo

    After that first download, pip will use cached copies of the whl archives and will not need to download them.

    michel@hp:~$ python3 -m venv project michel@hp:~$ project/bin/pip install -U pip setuptools wheel Requirement already up-to-date: pip in ./project/lib/python3.6/site-packages Collecting setuptools Using cached setuptools-38.2.4-py2.py3-none-any.whl Collecting wheel Using cached wheel-0.30.0-py2.py3-none-any.whl Installing collected packages: setuptools, wheel Found existing installation: setuptools 32.3.1 Uninstalling setuptools-32.3.1: Successfully uninstalled setuptools-32.3.1 Successfully installed setuptools-38.2.4 wheel-0.30.0

    The downloaded whl archives are saved in the pip cache found in the default directory ~/.cache/pip. It may be elsewhere on a Linux system if the environment variable XDG_CACHE_HOME is defined.

    The behaviour is exactly the same in Raspbian. Unfortunately, there is no caching in Armbian probably related to the older installed versions of Python, pip and setuptools. This is easily verified by creating and updating a virtual environment and then doing it again a second time.

    dietpi@domopiz:~$ python3 -m venv env dietpi@domopiz:~$ env/bin/pip install -U pip setuptools wheel ... Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB): 1.3MB downloaded Downloading setuptools-38.2.4-py2.py3-none-any.whl (489kB): 489kB downloaded Downloading wheel-0.30.0-py2.py3-none-any.whl (49kB): 49kB downloaded ... Successfully installed pip setuptools wheel Cleaning up... dietpi@domopiz:~$ python3 -m venv project dietpi@domopiz:~$ project/bin/pip install -U pip setuptools wheel ... Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB): 1.3MB downloaded Downloading setuptools-38.2.4-py2.py3-none-any.whl (489kB): 489kB downloaded Downloading wheel-0.30.0-py2.py3-none-any.whl (49kB): 49kB downloaded ... Successfully installed pip setuptools wheel

    There is a way to avoid all those downloads. First use the updated pip to download the most recent pip, setuptools and wheel archives to a local directory.

    dietpi@domopiz:~$ mkdir -p .local/venvtools dietpi@domopiz:~$ env/bin/pip download -d .local/venvtools pip setuptools wheel Collecting pip Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB) 100% |████████████████████████████████| 1.3MB 137kB/s Saved ./.local/venvtools/pip-9.0.1-py2.py3-none-any.whl Collecting setuptools Downloading setuptools-38.2.4-py2.py3-none-any.whl (489kB) 100% |████████████████████████████████| 491kB 318kB/s Saved ./.local/venvtools/setuptools-38.2.4-py2.py3-none-any.whl Collecting wheel Downloading wheel-0.30.0-py2.py3-none-any.whl (49kB) 100% |████████████████████████████████| 51kB 1.2MB/s Saved ./.local/venvtools/wheel-0.30.0-py2.py3-none-any.whl Successfully downloaded pip setuptools wheel (env) dietpi@domopiz:~$ dietpi@domopiz:~$ ls -l .local/venvtools total 1760 -rw-r--r-- 1 dietpi dietpi 1254803 Dec 10 18:08 pip-9.0.1-py2.py3-none-any.whl -rw-r--r-- 1 dietpi dietpi 489661 Dec 10 18:08 setuptools-38.2.4-py2.py3-none-any.whl -rw-r--r-- 1 dietpi dietpi 49751 Dec 10 18:08 wheel-0.30.0-py2.py3-none-any.whl

    Now let's create a virtual environment and update it from these local sources. The computer could be disconnected from the network.

    dietpi@domopiz:~$ python3 -m venv project2 dietpi@domopiz:~$ project2/bin/pip install -U --no-index --find-links=.local/venvtools pip setuptools wheel Ignoring indexes: https://pypi.python.org/simple/ Unpacking ./.local/venvtools/pip-9.0.1-py2.py3-none-any.whl Unpacking ./.local/venvtools/setuptools-38.2.4-py2.py3-none-any.whl Downloading/unpacking wheel Installing collected packages: pip, setuptools, wheel Found existing installation: pip 1.5.6 Uninstalling pip: Successfully uninstalled pip Found existing installation: setuptools 5.5.1 Uninstalling setuptools: Successfully uninstalled setuptools Successfully installed pip setuptools wheel Cleaning up...

    There is one problem with this approach. As far as I know, there is no verification for newer versions any of the three packages in PyPi. Every now and then it may be worthwhile to either reload the locally cached versions or to update from the Cheese Shop.

  11. Automating with a Script
  12. Since it would be best to always update a newly created virtual environment, it seemed like a good idea to create a simple script to do both steps at once. Accordingly, I created the directory .local/bin in my home directory (in may case, the full path is /home/michel/.local/bin.

    hp@michel:~$ mkdir -p .local/bin
    Then I added that directory to the search path by appending the following to the .profile file in my home directory

    # set PATH so it includes user's private bin in ./local if it exists if [ -d "$HOME/.local/bin" ] ; then PATH="$HOME/.local/bin:$PATH" fi

    This will only take effect on the next login to my session. I then created the simple mkvenv script in the newly created directory.

    #!/bin/bash python3 -m venv $1 $1/bin/pip install -U pip setuptools wheel

    Then the file is made executable.

    michel@hp:~$ chmod +x .local/bin/mkvenv

    After that, a one line command will create and update a virtual environment (don't forget you have to restart the current session for the search path to be updated).

    michel@hp:~$ mkvenv env

    I repeated the same steps in Armbian except for the addition of an option in the line that performs the update in mkvenv.

    #!/bin/bash
    python3 -m venv $1
    $1/bin/pip install -U --no-index --find-links=.local/venvtools pip setuptools wheel

    Of course, that was too simple. I modified these scripts so that they can create multiple virtual environments at once. I also added a help (-h or --help option and an ignore cache and get the packages from PyPi (-g or --get) option for the Armbian version. To make things as simple as possible, the option can be used but it will be ignored in the Ubuntu version. These can be downloaded

    Following a suggestion by AndyG, I added a shell function and alias in my .bashrc file.

    ve() { source $1/bin/activate; } alias ev='deactivate'

    If you look at the topic, you will see what rookie mistake I made. With this shell function I can activate any virtual environment in my present working directory with a simple command.

    michel@hp:~$ ve newproject

    Where ve stands for (activating a) virtual environment, the reversed letters ev perform the opposite action: deactivate.

    Before I continue adding stuff like this I should look into virtualenvwrapper.

  13. Using a Virtual Environment
  14. With all that precedes, the final bit is something of an anti-climax. There really is not much to using virtual Python environments. As seen above a virtual environment named env is a directory named env created in the working directory when venv was invoked. When activated, it changes the prompt to indicate its presence.

    michel@hp:~$ source env/bin/activate (env) michel@hp:~$

    A virtual environment is deactivated with the deactivate command. The environment prefix is then removed.

    michel@hp:~$ deactivate michel@hp:~$

    The following dialog shows that activating the virtual environment means that the command python will now invoke python3. This is done with symbolic links and modification of the search path and probably more tricks. This is how it becomes it possible to have other versions of Python installed in other virtual environments and using any version without interference.

    dietpi@domopiz:~$ python --version -bash: python: command not found dietpi@domopiz:~$ echo $PATH /usr/local/bin:/usr/bin:/bin dietpi@domopiz:~$ source env/bin/activate (env) dietpi@domopiz:~$ python --version Python 3.4.2 (env) dietpi@domopiz:~$ echo $PATH /home/dietpi/audioenv/bin:/usr/local/bin:/usr/bin:/bin: (env) dietpi@domopiz:~$ deactivate dietpi@domopiz:~$ python --version -bash: python: command not found

    Removing a virtual environment is as simple as deleting the directory.

    dietpi@domopiz:~$ rm -r env

    Do not rename a virtual environment or any directory that contains it. The reason is that absolute paths are used. See Don’t Rename Your Virtualenv Projects by Justin Iso and read the discussion Renaming a virtualenv folder without breaking it at stack overflow.

    Someone at the Universty of Washington has written an An Introduction To Venv which I have found useful. I will again mention the introduction by Jamie Matthews A non-magical introduction to Pip and Virtualenv for Python beginners.

  15. Examples
  16. As an example of using a virtual environment, I will install Pocketsphinx which is a part of CMUShpinx an Open Source Speech Recognition Toolkit by a team at Carnegie Mellon University. There are many computer language bindings. The Python binding pocketsphinx-python by Dmitry Prazdnichnov (bambocher) is easily installed with pip from PyPi. The source is available from a Github repository.

    There are a few requirements that need to be installed before setting up the virtual environment: python3-dev (if not already done), swig and libpulse-dev. All this is done with apt-get.

    michel@hp:~$ sudo apt-get install python3-dev Lecture des listes de paquets... Fait ... Après cette opération, 16,4 Mo d'espace disque supplémentaires seront utilisés. Souhaitez-vous continuer ? [O/n] o ... Paramétrage de python3-dev (3.6.3-0ubuntu2) ... michel@hp:~$ sudo apt-get install swig Lecture des listes de paquets... Fait ... Paramétrage de swig (3.0.10-1.2) ... michel@hp:~/pocketsphinx$ sudo apt-get install libpulse-dev Lecture des listes de paquets... Fait ... Paramétrage de libpulse-dev:amd64 (1:10.0-2ubuntu3) ...

    Time to create the virtual environment that will contain our project. With remarkable imagination, I named it pocketsphinx.

    michel@hp:~$ python3 -m venv pocketsphinx michel@hp:~$ pocketspnix/bin/pip install -U pip setuptools wheel michel@hp:~$ source pocketsphinx/bin/activate (pocketsphinx) michel@hp:~$ or, using the above script michel@hp:~$ mkvenv pocketsphinx michel@hp:~$ ve pocketsphinx (pocketsphinx) michel@hp:~$

    Now we are ready to install the Python package pocketsphinx from the Cheese Store.

    (pocketsphinx) michel@hp:~$ cd pocketsphinx (pocketsphinx) michel@hp:~/pocketsphinx$ pip install -U pocketsphinx Collecting pocketsphinx ... Successfully installed pocketsphinx-0.1.3 (pocketsphinx) michel@hp:~/pocketsphinx$

    Two site packages are installed in the lib/python3.6/site-packages directory: sphinxbase and pocketsphinx. These include two shared libraries and two Python scripts.

    /home/michel/pocketsphinx/lib/python3.6/site-packages/sphinxbase/_sphinxbase.cpython-36m-x86_64-linux-gnu.so /home/michel/pocketsphinx/lib/python3.6/site-packages/sphinxbase/sphinxbase.py /home/michel/pocketsphinx/lib/python3.6/site-packages/pocketsphinx/_pocketsphinx.cpython-36m-x86_64-linux-gnu.so /home/michel/pocketsphinx/lib/python3.6/site-packages/pocketsphinx/pocketsphinx.py

    There is a model directory in pocketsphinx that contains the en_US language model. No other language model is installed although Pocket Sphinx is multilingual.

    Following the LiveSpeech instructions by Dmitry Prazdnichnovat, the livespeech.py script is created in ~/pocketsphinx.

    #!/usr/bin/env python from pocketsphinx import LiveSpeech print('Start talking, Ctrl-C to quit') for phrase in LiveSpeech(): print(phrase)

    I made the script executable (chmod +x pocketsphinx.py) and then I connected a USB headset to the computer, checked that it was the default source and output sound device and that it worked correctly. in Settings. Then I ran the script which printed out more or less what I was saying.

    (pocketsphinx) michel@hp:~/pocketsphinx$ ./livespeech.py Start talking, Ctrl-C to quit good morning what is the weather like the it looks cool it looks cool actually looks very cold ^C (pocketsphinx) michel@hp:~/pocketsphinx$

    Then, on a lark, I tried talking in French, saying
    -- Bonjour, quelle est la température?
    -- Bonjour
    -- Bonjour?
    and got the following

    bosch old kid in a competitive up bottle the lot

    Is that what French sounds like to a unilingual anglophone with no prior exposure to la langue de Molière?

    Presumably, I could have called the virtual environment something like env and I could have installed pocketsphinx in a separate directory, with the env activated. Right now, I am working on the hypothesis that would be a mistake. At the very least, I would have to add a text file in the two directories to remember how they are linked. Why not keep both the Python virtual environment and the Python project in the same directory.

    As a second example, I installed the speech recognition module for Python by Anthony Zhang (Uberi). It is available from PyPi so that it could be installed just like pocketsphinx above. Instead, I cloned the git repository in the ~/Development/python. Before doing that, there was a new system requirement to fulfill.

    michel@hp:~$ sudo apt-get install portaudio19-dev Lecture des listes de paquets... Fait Souhaitez-vous continuer ? [O/n] 0 Paramétrage de libjack-dev (1:0.125.0-2) ... Paramétrage de portaudio19-dev:amd64 (19.6.0-1) ... ... michel@hp:~$ cd Development/python or wherever you want to keep your Python projects michel@hp:~/Development/python$ git clone https://github.com/Uberi/speech_recognition.git Clonage dans 'speech_recognition'...

    I then installed a virtual Python environment in that same directory.

    michel@hp:~/Development/python$ mkvenv speech_recognition creating virtual environment /home/michel/Development/python/speech_recognition updating virtual environment /home/michel/Development/python/speech_recognition

    As recommended by Jamie Matthews, I edited the .gitignore file by adding the virtual environment directories to the list of items to be ignored by git.

    # virtual environment bin/ include/ lib/ lib64 share/ pyvenv.cfg

    The next step, installing PyAudio which contains the Python bindings for PortAudio, needed to be done in the virtual environment.

    michel@hp:~/Development/python$ ve speech_recognition (speech_recognition) michel@hp:~/Development/python$ pip install pyaudio Collecting pyaudio Installing collected packages: pyaudio Successfully installed pyaudio-0.2.11 (speech_recognition) michel@hp:~/Development/python$

    PocketSphinx-Python is another requirement that needs to be installed. The installation went very quickly, probably because it was already built in the previous virtual environment.

    (speech_recognition) michel@hp:~/Development/python/speech_recognition$ pip install pocketsphinx Collecting pocketsphinx Installing collected packages: pocketsphinx Successfully installed pocketsphinx-0.1.3

    Finally, it was time to install Speech Recognition itself.

    (speech_recognition) michel@hp:~/Development/python/speech_recognition$ python setup.py install running install running build ...

    Unfortunately, the scripts found in the examples directory did not work even if all the unit tests passed including the two Sphinx test and the Google Speech Recognition test.

    (speech_recognition) michel@hp:~/Development/python/speech_recognition$ python -m unittest discover --verbose test_aiff_mono_16_bit (tests.test_audio.TestAudioFile) ... ok ... test_google_chinese (tests.test_recognition.TestRecognition) ... ok test_google_english (tests.test_recognition.TestRecognition) ... ok test_google_french (tests.test_recognition.TestRecognition) ... ok ... test_sphinx_english (tests.test_recognition.TestRecognition) ... ok ... test_sphinx_keywords (tests.test_special_features.TestSpecialFeatures) ... ok

    This was surprising as I had tried installing the library from the PyPi repository and it had worked. Looking at the git log, it became obvious that there had been changes made since the upload to the Cheese Shop. Si I rolled back to older versions until I found that the library worked.

    (speech_recognition) michel@hp:~/Development/python/speech_recognition$ git reset --hard 0a7bf7cb2b3ad3af6635535670f2f5a639211fc3 (speech_recognition) michel@hp:~/Development/python/speech_recognition$ python setup.py install

    I will discuss installing these two packages in Armbian in a future post as it is not quite as straight forward.

    The instructions about installing other languages are not up to date in the sense that the "simple Bash script" does not have the correct address for the language and even if it had, I do not think it would extract a downloaded language pack to the correct directory. The links are working, so download any desired language pack. Then extract the enclosed directory with the typical local identifier such as fr-FR or it-IT to the following directory

      speech_recognition/lib/python3.6/site-packages/speech_recognition/pocketsphinx-data
    
    The new language pack should now appear along side of the default en-US language pack.

    To use the new language, add the parameter language=fr-FR in the r.recognize_xxxx(audio, ...) calls. Here are two examples from the microphone_recognition.py script:

    # recognize speech using Sphinx try: print("Sphinx thinks you said " + r.recognize_sphinx(audio, language="fr-FR")) ... # recognize speech using Google Speech Recognition try: # for testing purposes, we're just using the default API key # to use another API key, use `r.recognize_google(audio, key="GOOGLE_SPEECH_RECOGNITION_API_KEY")` # instead of `r.recognize_google(audio)` print("Google Speech Recognition thinks you said " + r.recognize_google(audio, language="it-IT"))

    By the way, adding the parameter before extracting the language pack is a simple way of finding out where the pack should go. Run the modified script and it will complain.

    ... Say something! Sphinx error; missing PocketSphinx language data directory: "/home/michel/Development/python/speech_recognition/lib/python3.6/site-packages/speech_recognition/pocketsphinx-data/fr-FR"

    I was able to check that the French language pack can be used with Sphinx, Google Speech Recognition and Bing Voice Recognition. That does not mean it does not work with the other engines, I simply have not tested.

  17. Installing pip
  18. By default pip is included with Python if the latter is installed from source. However, in Debian it is not present by default. It can be installed with the apt-get utility. The latest version available from the Ubuntu 17.10 repositories is quite recent.

    michel@hp:~$ apt-cache policy python-pip python-pip: Installé : (aucun) Candidat : 9.0.1-2 michel@hp:~$ apt-cache policy python3-pip python3-pip: Installé : (aucun) Candidat : 9.0.1-2

    I no longer have Ubuntu 14.04 running, but I distinctly recall that only a much older version was available. That was a problem because when an apt-get installed version of pip is used to update itself, it gives up saying the package belongs to the operating system.

    According to the Python Software Foundation (Ensure you can run pip from the command line) it may be possible to "bootstrap" an installation using ensurepip. That module is available in both installed versions of Python on Ubuntu 17.10 and Armbian 9.1/

    /usr/lib/python2.7/ensurepip /usr/lib/python3.6/ensurepip

    To be more accurate, ensurepip for Python 2.7 was installed by default, it was not there initially in Python 3 and was probably added along with the python3-dev.

    The commands would be something like this.

    michel@hp:~$ python -m ensurepip --default-pip for Python 2.7 michel@hp:~$ python -m pip install -U pip setuptools wheel michel@hp:~$ python3 -m ensurepip --default-pip for Python 3.6 michel@hp:~$ python3 -m pip install -U pip setuptools wheel

    Actually, I would assume that the sudo prefix will be necessary as root is the owner of /usr/bin where at least pip should live. I would install the Python 3.6 version first and change the name of pip to pip3 if necessary, to ensure that the latter does not get clobbered when pip for Python 2.7 is installed.

    I have not done this, there is no guarantee that it would work. It looks as it may be possible to add pip, setuptools and wheel to the system Python but I decided to not do it for the time being. I want to force myself to use virtual environments in a systematic fashion. As a side benefit, not having pip installed in the system means that it is impossible to install a Python module as a system package by mistake. Just think about how similar the two look when pip is found in the /usr/bin directory (or any other directory in the search path).

    (env) michel@hp:~/env$ pip install pyaudio michel@hp:~/env$ pip install pyaudio