Python Environments with Conda

By Daniel Chen

February 29, 2020

Anaconda (and these days miniconda) has been my go-to for getting Python and the scientific/data science software stack installed on my computer (even on my Arch linux machine!).

When it first came out it was the first time I was able to install and use pandas and the rest of the scipy stack. I’ve stuck with it ever since.

It was also the first time I used (and understood) virtual environments.

In this post I’m hoping to show you a quick guide to using virtual environments with conda. You can find more details on how to manage your conda environment here:

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Problem

Here are some situations virtual environments solve:

You have code that needs to run in a different version of python than the version you have on your computer (e.g., I have Python (3) and need to run a script in Legacy Python (2))
The code I want to run uses a different version of package(s) than what I have installed (e.g., you need an older or newer version of pandas)
You can’t install and/or import a package because of package dependency conflicts
All the code and packages work on your computer, and you want a way to give someone else the same set of packages so they can run your code
You want to install an experimental package and do not want to mess up your current environment.

What are virtual environments

A virtual environment allows you to create an isolated (i.e., virtual) “environment” for your software, in this case, Python packages. This allows you to “activate” different python environments on your computer without having to uninstall everything just to run code from different projects.

Creating conda environments

We’ll be talking about conda environments in this post. Depending on how you installed anaconda/miniconda/conda make sure you’re in an environment that can run conda commands

Getting in the right place

You should see (base) in the beginning of a prompt. This is what it looks like on my computer, the (base) tells me I am in the “base” conda environment.

(base) $

You can see here the default python is the one installed in my miniconda directory.

(base) $ which python
/home/dchen/miniconda3/bin/python

We can run conda info --envs to get a list of all the environments we have

(base) $ conda info --envs
# conda environments:
#
base                  *  /home/dchen/miniconda3

Creating an environment

I’m currently using Python 3.

(base) $ python --version
Python 3.7.6

Let’s create another environment named py27 that’s Legacy Python (i.e., Python 2). We do this with the conda create command. We pass in the --name or -n flag to give a name to the new environment. We can also pass in python=2.7 to put Python 2.7 in this environment. Here you can specify any version of Python you want. For example, it can be another Python 3.7 environment for another Python 3 project.

(base) $ conda create -n py27 python=2.7

It’ll will build the plan for you, all you need to do is hit enter (it defaults to y)

Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/dchen/miniconda3/envs/py27

  added / updated specs:
    - python=2.7


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    certifi-2019.11.28         |           py27_0         149 KB  conda-forge
    pip-20.0.2                 |             py_2         1.0 MB  conda-forge
    python-2.7.15              |    h5a48372_1009        12.7 MB  conda-forge
    setuptools-44.0.0          |           py27_0         663 KB  conda-forge
    wheel-0.34.2               |             py_1          24 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        14.6 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-0_gnu
  ca-certificates    conda-forge/linux-64::ca-certificates-2019.11.28-hecc5488_0
  certifi            conda-forge/linux-64::certifi-2019.11.28-py27_0
  libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1006
  libgcc-ng          conda-forge/linux-64::libgcc-ng-9.2.0-h24d8f2e_2
  libgomp            conda-forge/linux-64::libgomp-9.2.0-h24d8f2e_2
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-9.2.0-hdf63c60_2
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  openssl            conda-forge/linux-64::openssl-1.1.1d-h516909a_0
  pip                conda-forge/noarch::pip-20.0.2-py_2
  python             conda-forge/linux-64::python-2.7.15-h5a48372_1009
  readline           conda-forge/linux-64::readline-8.0-hf8c457e_0
  setuptools         conda-forge/linux-64::setuptools-44.0.0-py27_0
  sqlite             conda-forge/linux-64::sqlite-3.30.1-hcee41ef_0
  tk                 conda-forge/linux-64::tk-8.6.10-hed695b0_0
  wheel              conda-forge/noarch::wheel-0.34.2-py_1
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1006


Proceed ([y]/n)?

When it’s done you’ll actually see a message on how to “activate” your new environment.

Downloading and Extracting Packages
setuptools-44.0.0    | 663 KB    | ######################################################################## | 100%
wheel-0.34.2         | 24 KB     | ######################################################################## | 100%
pip-20.0.2           | 1.0 MB    | ######################################################################## | 100%
python-2.7.15        | 12.7 MB   | ######################################################################## | 100%
certifi-2019.11.28   | 149 KB    | ######################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate py27
#
# To deactivate an active environment, use
#
#     $ conda deactivate

We can run conda activate py27 to “activate” (i.e., enable) our new environment. Do “deactivate” (i.e., exit) the environment we can use conda deactivate.

Activate (i.e., “enter”) the new environment

We can see our new environment when we list our envs

(base) $ conda info --envs
# conda environments:
#
base                  *  /home/dchen/miniconda3
py27                     /home/dchen/miniconda3/envs/py27

To activate the py27 environment we just created we run conda activate py27.

(base) $ conda activate py27
(py27) $

You can see when we activate the environment, we get a new prompt. The (base) now changed to let us know which environment we’re in, (py27).

Now when we run Python commands, it’s actually an entirely different Python from what we were using before.

Here’s the location of the Python binary

(py27) $ which python
/home/dchen/miniconda3/envs/py27/bin/python

The version is different from before (because we used python=2.7 when we created this py27 environment).

(py27) $ python --version
Python 2.7.15

Installing packages into environment (conda)

I’ll write a separate post about setting up conda-forge for installing packages (the commands to run are on the website).

In general, I try to conda install as many packages as I can. If the package I’m trying to install does not exist on conda, then I pip install the package.

This is a brand new Legacy Python environment, without any packages. In this example we’ll try to install the networkx package. You can see we currently do not have this package in this environment.

(py27) $ python
Python 2.7.15 | packaged by conda-forge | (default, Jul  2 2019, 00:39:44)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import networkx as nx
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named networkx
>>> quit()

We can install networkx with conda install networkx. If you need a specific version of a package, this is where you will install it.

(py27) $ conda install networkx
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/dchen/miniconda3/envs/py27

  added / updated specs:
    - networkx


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    networkx-2.1               |           py27_0         1.8 MB  conda-forge
    ------------------------------------------------------------
                                           Total:         1.8 MB

The following NEW packages will be INSTALLED:

  decorator          conda-forge/noarch::decorator-4.4.1-py_0
  networkx           conda-forge/linux-64::networkx-2.1-py27_0


Proceed ([y]/n)?


Downloading and Extracting Packages
networkx-2.1         | 1.8 MB    | ######################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Now we have the package in our environment!

(py27) $ python
Python 2.7.15 | packaged by conda-forge | (default, Jul  2 2019, 00:39:44)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import networkx as nx
>>> G = nx.Graph()
>>> quit()

Just to show that the package is in an entirely different environment, we can deativate our current environment to go back into (base), and can see we don’t have networkx installed in our base Python 3 environment.

(py27) $ conda deactivate
(base) $ python
Python 3.7.6 | packaged by conda-forge | (default, Jan  7 2020, 22:33:48)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import networkx as nx
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'networkx'
>>> quit()
(base) $

Reproducing and sharing environments

Now that you have all the packages you need in an environment and all your code is working, next is to document your work. Even if you’re working on your own, you may be working on your laptop and on a remote server, and need to replicate your environment in both locations. Another scenario is if you need to share your code with another person and want to make sure your code is executed in the same environment on their machine.

We can use the conda env export command to generate all the packages in the current environment. So if we’re in the py27 environment we created, you can get a list of all the packages in it.

(py27) $ conda env export
name: py27
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=0_gnu
  - ca-certificates=2019.11.28=hecc5488_0
  - certifi=2019.11.28=py27_0
  - decorator=4.4.1=py_0
  - libffi=3.2.1=he1b5a44_1006
  - libgcc-ng=9.2.0=h24d8f2e_2
  - libgomp=9.2.0=h24d8f2e_2
  - libstdcxx-ng=9.2.0=hdf63c60_2
  - ncurses=6.1=hf484d3e_1002
  - networkx=2.1=py27_0
  - openssl=1.1.1d=h516909a_0
  - pip=20.0.2=py_2
  - python=2.7.15=h5a48372_1009
  - readline=8.0=hf8c457e_0
  - setuptools=44.0.0=py27_0
  - sqlite=3.30.1=hcee41ef_0
  - tk=8.6.10=hed695b0_0
  - wheel=0.34.2=py_1
  - zlib=1.2.11=h516909a_1006
prefix: /home/dchen/miniconda3/envs/py27

We can export this to an environment.yml file so all the packages are saved and documented. You can do this by copying and pasting the results in to a file called environment.yml manually, or you can pipe the output into the file conda env export > environment.yml. If your python code is in a project structure you would run this command to put the environment.yml file in the root project directory.

Now you can send your Python project to another person or to another computer. To reproduce the environment, you can run conda env create -f environment.yml to use the environment.yml file to automatically recreate the environment. The name field in the first line will be used as the environment name.

In this example let’s edit the name to py27copy since we already have a py27 environment.

Now if we run conda env create -f environment.yml, it will “reinstall” all the packages needed as specified in the environment.yml file, and we can activate/deactivate just like before

(py27) $ conda env create -f ~/Desktop/environment.yml
Collecting package metadata (repodata.json): done
Solving environment: done
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate py27dup
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Installing packages into environment (pip)

Not every package will be conda installable, for those packages we can pip install them just like normal. Here we’ll install the flask package for web development into our py27 environment.

(py27) $ pip install flask
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting flask
  Using cached Flask-1.1.1-py2.py3-none-any.whl (94 kB)
Collecting Werkzeug>=0.15
  Downloading Werkzeug-1.0.0-py2.py3-none-any.whl (298 kB)
     |████████████████████████████████| 298 kB 1.1 MB/s
Collecting itsdangerous>=0.24
  Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting click>=5.1
  Downloading Click-7.0-py2.py3-none-any.whl (81 kB)
     |████████████████████████████████| 81 kB 2.0 MB/s
Collecting Jinja2>=2.10.1
  Downloading Jinja2-2.11.1-py2.py3-none-any.whl (126 kB)
     |████████████████████████████████| 126 kB 306 kB/s
Collecting MarkupSafe>=0.23
  Downloading MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl (24 kB)
Installing collected packages: Werkzeug, itsdangerous, click, MarkupSafe, Jinja2, flask
Successfully installed Jinja2-2.11.1 MarkupSafe-1.1.1 Werkzeug-1.0.0 click-7.0 flask-1.1.1 itsdangerous-1.1.0

Now if you want to export the enviornment to the environment.yml you’ll notice that pip will be part of the installation, and also a new pip section that describes the packages that will be installed via pip instead of conda.

(py27) $ conda env export
name: py27
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=0_gnu
  - ca-certificates=2019.11.28=hecc5488_0
  - certifi=2019.11.28=py27_0
  - decorator=4.4.1=py_0
  - libffi=3.2.1=he1b5a44_1006
  - libgcc-ng=9.2.0=h24d8f2e_2
  - libgomp=9.2.0=h24d8f2e_2
  - libstdcxx-ng=9.2.0=hdf63c60_2
  - ncurses=6.1=hf484d3e_1002
  - networkx=2.1=py27_0
  - openssl=1.1.1d=h516909a_0
  - pip=20.0.2=py_2
  - python=2.7.15=h5a48372_1009
  - readline=8.0=hf8c457e_0
  - setuptools=44.0.0=py27_0
  - sqlite=3.30.1=hcee41ef_0
  - tk=8.6.10=hed695b0_0
  - wheel=0.34.2=py_1
  - zlib=1.2.11=h516909a_1006
  - pip:
    - click==7.0
    - flask==1.1.1
    - itsdangerous==1.1.0
    - jinja2==2.11.1
    - markupsafe==1.1.1
    - werkzeug==1.0.0
prefix: /home/dchen/miniconda3/envs/py27

Summary

Creating environments with the conda create -n <name> python=<version> is an easy way to create new environments with a specified version of Python. You can then activate the environment with conda activate <name> and proceed to install packages with conda install <package> or pip install <package. When you’re ready to document and share your environment with another person or on another machine, you can run conda env export > environment.yml to save the current environment to a environment.yml. This file can be used to create a new environment with conda env create -f environment.yml and now you can replicate your environment across multiple machines!

You don’t need to use conda to create your Python environments. If you’re not using conda for your environments, you can create a requirements.txt file instead of an environment.yml with pip freeze > requirements.txt and install everything into the current environment with pip install -r requirements.txt. Just know you need to create the environment first as pip will not create the environment for you.

Hope this helps people get started with environments and getting their code working on multiple computers. You can checkout the main documentation page on conda environment management here:

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Posted on:: February 29, 2020

Length:: 10 minute read, 2124 words

Tags:: python tutorial

See Also:: R or Python, which One to Learn (First)?; Getting Started with Data Science and Analysis; Open Source A Python Project I