# Python Environments with Conda

Anaconda (and these days miniconda) has been my go-to for getting Python and the scientific/data science software stack installed on my computer (even on my Arch linux machine!).

When it first came out it was the first time I was able to install and use pandas and the rest of the scipy stack. I’ve stuck with it ever since.

It was also the first time I used (and understood) virtual environments.

In this post I’m hoping to show you a quick guide to using virtual environments with conda. You can find more details on how to manage your conda environment here:

## Problem

Here are some situations virtual environments solve:

1. You have code that needs to run in a different version of python than the version you have on your computer (e.g., I have Python (3) and need to run a script in Legacy Python (2))
2. The code I want to run uses a different version of package(s) than what I have installed (e.g., you need an older or newer version of pandas)
3. You can’t install and/or import a package because of package dependency conflicts
4. All the code and packages work on your computer, and you want a way to give someone else the same set of packages so they can run your code
5. You want to install an experimental package and do not want to mess up your current environment.

## What are virtual environments

A virtual environment allows you to create an isolated (i.e., virtual) “environment” for your software, in this case, Python packages. This allows you to “activate” different python environments on your computer without having to uninstall everything just to run code from different projects.

## Creating conda environments

We’ll be talking about conda environments in this post. Depending on how you installed anaconda/miniconda/conda make sure you’re in an environment that can run conda commands

### Getting in the right place

You should see (base) in the beginning of a prompt. This is what it looks like on my computer, the (base) tells me I am in the “base” conda environment.

(base) $ You can see here the default python is the one installed in my miniconda directory. (base)$ which python
/home/dchen/miniconda3/bin/python

We can run conda info --envs to get a list of all the environments we have

(base) $conda info --envs # conda environments: # base * /home/dchen/miniconda3 ### Creating an environment I’m currently using Python 3. (base)$ python --version
Python 3.7.6

Let’s create another environment named py27 that’s Legacy Python (i.e., Python 2). We do this with the conda create command. We pass in the --name or -n flag to give a name to the new environment. We can also pass in python=2.7 to put Python 2.7 in this environment. Here you can specify any version of Python you want. For example, it can be another Python 3.7 environment for another Python 3 project.

(base) $conda create -n py27 python=2.7 It’ll will build the plan for you, all you need to do is hit enter (it defaults to y) Collecting package metadata (current_repodata.json): done Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): done Solving environment: done ## Package Plan ## environment location: /home/dchen/miniconda3/envs/py27 added / updated specs: - python=2.7 The following packages will be downloaded: package | build ---------------------------|----------------- certifi-2019.11.28 | py27_0 149 KB conda-forge pip-20.0.2 | py_2 1.0 MB conda-forge python-2.7.15 | h5a48372_1009 12.7 MB conda-forge setuptools-44.0.0 | py27_0 663 KB conda-forge wheel-0.34.2 | py_1 24 KB conda-forge ------------------------------------------------------------ Total: 14.6 MB The following NEW packages will be INSTALLED: _libgcc_mutex conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-0_gnu ca-certificates conda-forge/linux-64::ca-certificates-2019.11.28-hecc5488_0 certifi conda-forge/linux-64::certifi-2019.11.28-py27_0 libffi conda-forge/linux-64::libffi-3.2.1-he1b5a44_1006 libgcc-ng conda-forge/linux-64::libgcc-ng-9.2.0-h24d8f2e_2 libgomp conda-forge/linux-64::libgomp-9.2.0-h24d8f2e_2 libstdcxx-ng conda-forge/linux-64::libstdcxx-ng-9.2.0-hdf63c60_2 ncurses conda-forge/linux-64::ncurses-6.1-hf484d3e_1002 openssl conda-forge/linux-64::openssl-1.1.1d-h516909a_0 pip conda-forge/noarch::pip-20.0.2-py_2 python conda-forge/linux-64::python-2.7.15-h5a48372_1009 readline conda-forge/linux-64::readline-8.0-hf8c457e_0 setuptools conda-forge/linux-64::setuptools-44.0.0-py27_0 sqlite conda-forge/linux-64::sqlite-3.30.1-hcee41ef_0 tk conda-forge/linux-64::tk-8.6.10-hed695b0_0 wheel conda-forge/noarch::wheel-0.34.2-py_1 zlib conda-forge/linux-64::zlib-1.2.11-h516909a_1006 Proceed ([y]/n)? When it’s done you’ll actually see a message on how to “activate” your new environment. Downloading and Extracting Packages setuptools-44.0.0 | 663 KB | ######################################################################## | 100% wheel-0.34.2 | 24 KB | ######################################################################## | 100% pip-20.0.2 | 1.0 MB | ######################################################################## | 100% python-2.7.15 | 12.7 MB | ######################################################################## | 100% certifi-2019.11.28 | 149 KB | ######################################################################## | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # #$ conda activate py27
#
# To deactivate an active environment, use
#
#     $conda deactivate We can run conda activate py27 to “activate” (i.e., enable) our new environment. Do “deactivate” (i.e., exit) the environment we can use conda deactivate. ## Activate (i.e., “enter”) the new environment We can see our new environment when we list our envs (base)$ conda info --envs
# conda environments:
#
base                  *  /home/dchen/miniconda3
py27                     /home/dchen/miniconda3/envs/py27

To activate the py27 environment we just created we run conda activate py27.

(base) $conda activate py27 (py27)$

You can see when we activate the environment, we get a new prompt. The (base) now changed to let us know which environment we’re in, (py27).

Now when we run Python commands, it’s actually an entirely different Python from what we were using before.

Here’s the location of the Python binary

(py27) $which python /home/dchen/miniconda3/envs/py27/bin/python The version is different from before (because we used python=2.7 when we created this py27 environment). (py27)$ python --version
Python 2.7.15

## Installing packages into environment (conda)

I’ll write a separate post about setting up conda-forge for installing packages (the commands to run are on the website).

In general, I try to conda install as many packages as I can. If the package I’m trying to install does not exist on conda, then I pip install the package.

This is a brand new Legacy Python environment, without any packages. In this example we’ll try to install the networkx package. You can see we currently do not have this package in this environment.

(py27) $python Python 2.7.15 | packaged by conda-forge | (default, Jul 2 2019, 00:39:44) [GCC 7.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import networkx as nx Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named networkx >>> quit() We can install networkx with conda install networkx. If you need a specific version of a package, this is where you will install it. (py27)$ conda install networkx
Solving environment: done

## Package Plan ##

environment location: /home/dchen/miniconda3/envs/py27

- networkx

package                    |            build
---------------------------|-----------------
networkx-2.1               |           py27_0         1.8 MB  conda-forge
------------------------------------------------------------
Total:         1.8 MB

The following NEW packages will be INSTALLED:

decorator          conda-forge/noarch::decorator-4.4.1-py_0
networkx           conda-forge/linux-64::networkx-2.1-py27_0

Proceed ([y]/n)?

networkx-2.1         | 1.8 MB    | ######################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Now we have the package in our environment!

(py27) $python Python 2.7.15 | packaged by conda-forge | (default, Jul 2 2019, 00:39:44) [GCC 7.3.0] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import networkx as nx >>> G = nx.Graph() >>> quit() Just to show that the package is in an entirely different environment, we can deativate our current environment to go back into (base), and can see we don’t have networkx installed in our base Python 3 environment. (py27)$ conda deactivate
(base) $python Python 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48) [GCC 7.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import networkx as nx Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'networkx' >>> quit() (base)$

## Reproducing and sharing environments

Now that you have all the packages you need in an environment and all your code is working, next is to document your work. Even if you’re working on your own, you may be working on your laptop and on a remote server, and need to replicate your environment in both locations. Another scenario is if you need to share your code with another person and want to make sure your code is executed in the same environment on their machine.

We can use the conda env export command to generate all the packages in the current environment. So if we’re in the py27 environment we created, you can get a list of all the packages in it.

(py27) $conda env export name: py27 channels: - conda-forge - defaults dependencies: - _libgcc_mutex=0.1=conda_forge - _openmp_mutex=4.5=0_gnu - ca-certificates=2019.11.28=hecc5488_0 - certifi=2019.11.28=py27_0 - decorator=4.4.1=py_0 - libffi=3.2.1=he1b5a44_1006 - libgcc-ng=9.2.0=h24d8f2e_2 - libgomp=9.2.0=h24d8f2e_2 - libstdcxx-ng=9.2.0=hdf63c60_2 - ncurses=6.1=hf484d3e_1002 - networkx=2.1=py27_0 - openssl=1.1.1d=h516909a_0 - pip=20.0.2=py_2 - python=2.7.15=h5a48372_1009 - readline=8.0=hf8c457e_0 - setuptools=44.0.0=py27_0 - sqlite=3.30.1=hcee41ef_0 - tk=8.6.10=hed695b0_0 - wheel=0.34.2=py_1 - zlib=1.2.11=h516909a_1006 prefix: /home/dchen/miniconda3/envs/py27 We can export this to an environment.yml file so all the packages are saved and documented. You can do this by copying and pasting the results in to a file called environment.yml manually, or you can pipe the output into the file conda env export > environment.yml. If your python code is in a project structure you would run this command to put the environment.yml file in the root project directory. Now you can send your Python project to another person or to another computer. To reproduce the environment, you can run conda env create -f environment.yml to use the environment.yml file to automatically recreate the environment. The name field in the first line will be used as the environment name. In this example let’s edit the name to py27copy since we already have a py27 environment. Now if we run conda env create -f environment.yml, it will “reinstall” all the packages needed as specified in the environment.yml file, and we can activate/deactivate just like before (py27)$ conda env create -f ~/Desktop/environment.yml
Solving environment: done
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $conda activate py27dup # # To deactivate an active environment, use # #$ conda deactivate


## Installing packages into environment (pip)

Not every package will be conda installable, for those packages we can pip install them just like normal. Here we’ll install the flask package for web development into our py27 environment.

(py27) $pip install flask DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support Collecting flask Using cached Flask-1.1.1-py2.py3-none-any.whl (94 kB) Collecting Werkzeug>=0.15 Downloading Werkzeug-1.0.0-py2.py3-none-any.whl (298 kB) |████████████████████████████████| 298 kB 1.1 MB/s Collecting itsdangerous>=0.24 Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB) Collecting click>=5.1 Downloading Click-7.0-py2.py3-none-any.whl (81 kB) |████████████████████████████████| 81 kB 2.0 MB/s Collecting Jinja2>=2.10.1 Downloading Jinja2-2.11.1-py2.py3-none-any.whl (126 kB) |████████████████████████████████| 126 kB 306 kB/s Collecting MarkupSafe>=0.23 Downloading MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl (24 kB) Installing collected packages: Werkzeug, itsdangerous, click, MarkupSafe, Jinja2, flask Successfully installed Jinja2-2.11.1 MarkupSafe-1.1.1 Werkzeug-1.0.0 click-7.0 flask-1.1.1 itsdangerous-1.1.0 Now if you want to export the enviornment to the environment.yml you’ll notice that pip will be part of the installation, and also a new pip section that describes the packages that will be installed via pip instead of conda. (py27)$ conda env export
name: py27
channels:
- conda-forge
- defaults
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=0_gnu
- ca-certificates=2019.11.28=hecc5488_0
- certifi=2019.11.28=py27_0
- decorator=4.4.1=py_0
- libffi=3.2.1=he1b5a44_1006
- libgcc-ng=9.2.0=h24d8f2e_2
- libgomp=9.2.0=h24d8f2e_2
- libstdcxx-ng=9.2.0=hdf63c60_2
- ncurses=6.1=hf484d3e_1002
- networkx=2.1=py27_0
- openssl=1.1.1d=h516909a_0
- pip=20.0.2=py_2
- python=2.7.15=h5a48372_1009
- setuptools=44.0.0=py27_0
- sqlite=3.30.1=hcee41ef_0
- tk=8.6.10=hed695b0_0
- wheel=0.34.2=py_1
- zlib=1.2.11=h516909a_1006
- pip:
- click==7.0
- itsdangerous==1.1.0
- jinja2==2.11.1
- markupsafe==1.1.1
- werkzeug==1.0.0
prefix: /home/dchen/miniconda3/envs/py27

## Summary

Creating environments with the conda create -n <name> python=<version> is an easy way to create new environments with a specified version of Python. You can then activate the environment with conda activate <name> and proceed to install packages with conda install <package> or pip install <package. When you’re ready to document and share your environment with another person or on another machine, you can run conda env export > environment.yml to save the current environment to a environment.yml. This file can be used to create a new environment with conda env create -f environment.yml and now you can replicate your environment across multiple machines!

You don’t need to use conda to create your Python environments. If you’re not using conda for your environments, you can create a requirements.txt file instead of an environment.yml with pip freeze > requirements.txt and install everything into the current environment with pip install -r requirements.txt. Just know you need to create the environment first as pip will not create the environment for you.

Hope this helps people get started with environments and getting their code working on multiple computers. You can checkout the main documentation page on conda environment management here: