Difference between revisions of "Python/pip"

From HPC Wiki
Jump to navigation Jump to search
m
m
 
(13 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
[[Category:Basics]][[Category:HPC-User]]
 
== General ==
 
== General ==
 
Python is an Open Source, interpreted high-level programming language which is popular for prototyping high end simulations and data science applications. In addition to a large standard library there are many thirdparty libraries available (e.g. Numpy, Scipy). However, it is a single threaded environment and therefore may not be most suitable for very large simulations.  
 
Python is an Open Source, interpreted high-level programming language which is popular for prototyping high end simulations and data science applications. In addition to a large standard library there are many thirdparty libraries available (e.g. Numpy, Scipy). However, it is a single threaded environment and therefore may not be most suitable for very large simulations.  
Line 5: Line 6:
  
 
== Usage ==
 
== Usage ==
In Linux, there is a version of Python boxed in the distribution, delivered as RPM. This Python and some Py-modules (NumPy, SciPy, Matplotlib, ...) are available by default. The RPMs installed in exactly that versions which are available by OS, also typically older versions. The RPMs are updated accordingly to the OS distribution update rules. We do not offer any other versions/updates/fixes besides the distribution RPMs. Users cannot modify this installation. However, you are welcome to add needed Py-modules in your $HOME directory, e.g. by 'pip' or 'easy_install'.
+
Python and its most commonly used libraries (NumPy, SciPy, Matplotlib) are usually available as a module. Please notice that an older version may be set as default and there might be modules for different versions. If the version you need is not available, you can install it in your directory by using 'pip' or 'easy_install'.
Python software often rely on GCC compilers and do not work with Intel compilers; thus type 'module switch intel gcc' before adding any Py-Modules.
 
  
Examples:
+
You can check the used Version with:
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
$ module switch intel gcc
+
$ python3 --version
$ export MYPY=$HOME/SomeDirForPythonInstall
 
$ export PYTHONPATH=$PYTHONPATH:$MYPY/lib/python2.7/site-packages
 
$ mkdir -p $MYPY/lib/python2.7/site-packages
 
$ easy_install --prefix $MYPY  theano
 
 
</syntaxhighlight>
 
</syntaxhighlight>
Note: you need to set PYTHONPATH anytime you wish to use 'teano' Py-module!
 
  
 +
and the corresponding install directory with
 +
<syntaxhighlight lang="bash">
 +
$ which python3
 +
</syntaxhighlight>
 +
 +
and then use
 +
<syntaxhighlight lang="bash">
 +
$ python3 my_program.py
 +
</syntaxhighlight>
 +
to execute your code.
 +
 +
Often on a single system there are two environments available simultaneously: Python 2.x.x (callable with <code>python</code>) and Python 3.x.x (callable as shown above). If not otherwise necessary, it is advisable to use and program in 3.x.x
 +
 +
 +
== Pip ==
 +
Pip is a package management system for Python and is automatically included since Python 2.7.9/3.4. It enables managing lists of packages and their versions. If you aim to install packages for Python3, use <code>pip3</code> instead of <code>pip</code> with the commands below.
 +
 +
Example:
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
$ module switch intel gcc
 
 
$ pip install --user theano
 
$ pip install --user theano
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Note: the installation is in $HOME/.local/[bin|lib] directories. Beware of clash if using multiple Py-modules!
+
Getting the list of the installed packages:
 +
<syntaxhighlight lang="bash">
 +
$ pip list
 +
</syntaxhighlight>
  
== Other Python versions ==
+
Pip can also list the packages with outdated versions or available prereleases:
If you need yet another version of Python as the one available by default, take a look at the output of 'module avail python' command. It lists a number of Python installations available in the HPC Cluster. E.g. type 'module load python/3.6.0'  for loading of the the appropriate version of Python3.
+
<syntaxhighlight lang="bash">
 +
$ pip list --outdated
 +
$ pip list --pre
 +
</syntaxhighlight>
  
Note: the name of the Python binary version 3 is 'python2' and not just 'python'!
+
Packages can easily be installed through the pip install command (specify <code>--user</code> to install locally):
  
You can add Py-Modules in your $HOME in same ways as described above. In addition,
+
<syntaxhighlight lang="bash">
some Py-Modules (NumPy, SciPy, ...) are bundled in these installations, and made available by PYTHONPATH environment variable (set by loading the python environment module). Whenever modifying this environment variable, you will likely wish to enhance the value instead to overwrite it, e.g. 'export PYTHONPATH=$HOME/my/newpatho/added:$PYTHONPATH'
+
$ pip install [--user] my-package
By unsetting the PYTHONPATH envvar you have the ultimate power to install any wished versions of these bundled Py-modules in your $HOME.
+
</syntaxhighlight>
  
Intel also offers Python releases, available via the environment modules. List available versions with  'module avail pythoni' and load one by e.g. 'module load pythoni/3.5' for Python3 version.
+
The same goes for uninstalling packages:
All notes above about installing software in $HOME and PYTHONPATH apply. Note that Intel also call Python-3 binary 'python' in addition to of 'python3', which differs the Intel release of python from other ones.
 
  
You're also free to install your own version of Python in your $HOME. All notes above about installing software in $HOME and PYTHONPATH apply.
+
<syntaxhighlight lang="bash">
 +
$ pip uninstall my-package
 +
</syntaxhighlight>
 +
 
 +
Upgrade the packages specified to the latest version:
 +
 
 +
<syntaxhighlight lang="bash">
 +
$ pip install --upgrade package1 [package2 ...]
 +
</syntaxhighlight>
 +
 
 +
The present packages can be stored as a 'requirements' file. If properly formatted, this file can then be used to recreate the given environment on another system with exactly the same packages and versions:
 +
 
 +
<syntaxhighlight lang="bash">
 +
$ pip install -r requirements.txt
 +
</syntaxhighlight>
 +
 
 +
Note: <code>pip3 [...]</code> can be used interchangeably with:
 +
 
 +
<syntaxhighlight lang="bash">
 +
$ python3 -m pip [...]
 +
</syntaxhighlight>
  
 
== Tutorials ==
 
== Tutorials ==
Line 50: Line 87:
 
== References ==
 
== References ==
 
https://www.python.org/about/
 
https://www.python.org/about/
 +
https://pypi.org/project/pip/

Latest revision as of 15:52, 3 September 2019

General

Python is an Open Source, interpreted high-level programming language which is popular for prototyping high end simulations and data science applications. In addition to a large standard library there are many thirdparty libraries available (e.g. Numpy, Scipy). However, it is a single threaded environment and therefore may not be most suitable for very large simulations.

Usage

Python and its most commonly used libraries (NumPy, SciPy, Matplotlib) are usually available as a module. Please notice that an older version may be set as default and there might be modules for different versions. If the version you need is not available, you can install it in your directory by using 'pip' or 'easy_install'.

You can check the used Version with:

$ python3 --version

and the corresponding install directory with

$ which python3

and then use

$ python3 my_program.py

to execute your code.

Often on a single system there are two environments available simultaneously: Python 2.x.x (callable with python) and Python 3.x.x (callable as shown above). If not otherwise necessary, it is advisable to use and program in 3.x.x


Pip

Pip is a package management system for Python and is automatically included since Python 2.7.9/3.4. It enables managing lists of packages and their versions. If you aim to install packages for Python3, use pip3 instead of pip with the commands below.

Example:

$ pip install --user theano

Getting the list of the installed packages:

$ pip list

Pip can also list the packages with outdated versions or available prereleases:

$ pip list --outdated
$ pip list --pre

Packages can easily be installed through the pip install command (specify --user to install locally):

$ pip install [--user] my-package

The same goes for uninstalling packages:

$ pip uninstall my-package

Upgrade the packages specified to the latest version:

$ pip install --upgrade package1 [package2 ...]

The present packages can be stored as a 'requirements' file. If properly formatted, this file can then be used to recreate the given environment on another system with exactly the same packages and versions:

$ pip install -r requirements.txt

Note: pip3 [...] can be used interchangeably with:

$ python3 -m pip [...]

Tutorials

As there are plenty of python tutorials available online we decided not to create our very own but instead simply list a few we consider to be most helpful.

https://docs.python.org/3/tutorial/

https://www.w3schools.com/python/

https://www.tutorialspoint.com/python/

References

https://www.python.org/about/ https://pypi.org/project/pip/