Installing Python 3.6+ on Nodes on CentOS 6.5 (also install PIKA)

We recently upgrade our Vertica cluster to 9.x, however our OS is still CentOS 6.5. OS upgrade to 7.x will be scheduled in near future.

CentOS 6.5 still comes with base Python 2.6.6 and Vertica processes are DEPENDENT on core Python. Upgrading Python Core on Vertica cluster/nodes presents its reiks and challenges. To be safe you should install more recent versions of Python as local/parallel installations. This document walks through the steps to install Python 3.6.x on a Vertica node. Let us first check the environment, and then I’ll install Python 3.6.1 from source as local install. We see our core Python version is 2.6.6. We will NOT step on this but install newer version of Python under /usr/local/bin

sbalas002c@PAWAYL-31V7HC2 MINGW64 ~
$ ssh sbalas002c@VERTICA-PROD01
sbalas002c@VERTICA-PROD01's password:
Last login: Fri Apr 13 11:58:09 2018 from 10.45.143.151
[sbalas002c@VERTICA-PROD01~]$
[sbalas002c@VERTICA-PROD01~]$ sudo su -
[sudo] password for sbalas002c:
sbalas002c is not in the sudoers file.  This incident will be reported.
[sbalas002c@VERTICA-PROD01~]$ su - dbadmin
Password:
[dbadmin@VERTICA-PROD01~]$ sudo su -
[root@VERTICA-PROD01~]# which python
/usr/bin/python
[root@VERTICA-PROD01~]# python -V
Python 2.6.6
[root@VERTICA-PROD01~]#

We are behind the corporate firewalls, so we need to use proxy servers to get to the out-side world. For convenience add the proxy setting to .bash_profile.

# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

[root@VERTICA-PROD01~]# env | grep proxy
http_proxy=http://VERTICA-PROD01:8080/
ftp_proxy=http://VERTICA-PROD01:8080/
https_proxy=https://VERTICA-PROD01:8080/
[root@VERTICA-PROD01~]#

Let us check the development environment is up-to-date. We need to install latest development environment.

[root@VERTICA-PROD01~]# yum list | grep gcc-c++
avr-gcc-c++.x86_64                   4.5.0-2.el6            tsg_epel_6
gcc-c++.x86_64                       4.4.7-18.el6           tsg_centos_6_base
mingw32-gcc-c++.x86_64               4.9.2-1.el6            tsg_epel_6
mingw64-gcc-c++.x86_64               4.9.2-1.el6            tsg_epel_6
[root@VERTICA-PROD01~]# yum list readline-devel
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
Available Packages
readline-devel.i686                                                              6.0-4.el6                                                             tsg_centos_6_base
readline-devel.x86_64                                                            6.0-4.el6                                                             tsg_centos_6_base
[root@VERTICA-PROD01~]#

Now that the DEV packages are available, let us install them on our node. This step will install all the required development tools on the node.

[root@VERTICA-PROD01~]# yum groupinstall 'Development tools'
Loaded plugins: fastestmirror, security
Setting up Group Process
Loading mirror speeds from cached hostfile
Package gcc-4.4.7-18.el6.x86_64 already installed and latest version
Package 1:make-3.81-23.el6.x86_64 already installed and latest version
Package 1:pkgconfig-0.23-9.1.el6.x86_64 already installed and latest version
Dependencies Resolved
Total                                                                                                                                   7.8 MB/s |  53 MB     00:06
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : patch-2.6-6.el6.x86_64                                                                                                                              1/39
  
Complete!
[root@VERTICA-PROD01~]#

Next step will install several supporting libraries that are required for Python compilation.

[root@VERTICA-PROD01~]# yum install -y zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel expat-devel
Loaded plugins: fastestmirror, security
Setting up Install Process
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
Install      30 Package(s)
Total download size: 17 M
Installed size: 52 M
Downloading Packages:
Total                                                                                                                                   5.8 MB/s |  17 MB     00:03
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : xorg-x11-proto-devel-7.7-14.el6.noarch                                                                                                              
Complete!
[root@VERTICA-PROD01~]#

NOW that our development environment is installed and configured, we can down load Python source, compile and install them. I’ll create a local python directory for download.

[root@VERTICA-PROD01~]# ll
total 68
-rw-------. 1 root root 4001 May 20 2015 anaconda-ks.cfg
-rw-r--r--. 1 root root 21457 May 20 2015 install.log
-rw-r--r--. 1 root root 5820 May 20 2015 install.log.syslog
-rw-r--r--. 1 root root 13 May 20 2015 jal_test.out
-rw-------. 1 root root 15783 May 20 2015 post-install.log
-rw-r--r-- 1 root root 1530 Dec 7 16:10 repos_bak_20171207_161001.tar.gz
drwxr-xr-x 2 root root 4096 Nov 10 13:15 vertica-packages
-rw-r--r-- 1 root root 494 Jul 28 2017 yum.conf
[root@VERTICA-PROD01~]# mkdir python36
[root@VERTICA-PROD01~]# cd python36/
[root@VERTICA-PROD01python36]# ll
total 0
[root@VERTICA-PROD01 python36]# wget http://python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz
--2018-04-13 16:57:37-- http://python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz
Resolving spotlightmonprx.resource.comcast.net... 96.115.77.99
Connecting to spotlightmonprx.resource.comcast.net|96.115.77.99|:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz [following]
--2018-04-13 16:57:37-- https://python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz
Connecting to spotlightmonprx.resource.comcast.net|96.115.77.99|:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz [following]
--2018-04-13 16:57:37-- https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tar.xz
Connecting to spotlightmonprx.resource.comcast.net|96.115.77.99|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 16872064 (16M) [application/octet-stream]
Saving to: “Python-3.6.1.tar.xz”
100%[==============================================================================================================================>] 16,872,064 16.4M/s in 1.0s
2018-04-13 16:57:38 (16.4 MB/s) - “Python-3.6.1.tar.xz” saved [16872064/16872064]
[root@VERTICA-PROD01 python36]# ll
total 16504
-rw-r--r-- 1 root root 16872064 Mar 21 2017 Python-3.6.1.tar.xz
[root@VERTICA-PROD01 python36]# tar xf Python-3.6.1.tar.xz
[root@VERTICA-PROD01 python36]# ll
total 16508
drwxr-xr-x 17 mradmin mradmin 4096 Mar 21 2017 Python-3.6.1
-rw-r--r-- 1 root root 16872064 Mar 21 2017 Python-3.6.1.tar.xz
[root@VERTICA-PROD01 python36]#
PLEASE note the –prefix FLAG in the next command. THIS is REQUIRED so that Core does not get overlayed.

Now that we have the development environment, and downloaded the required version of Python 3.6.1 source, let us configure the local compilation settings, compile and install Python 3.3.6.

[root@VERTICA-PROD01 python36]# cd Python-3.6.1
[root@VERTICA-PROD01 Python-3.6.1]# ./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for python3.6... no
configure: creating ./config.status
config.status: creating Makefile.pre
config.status: creating Modules/Setup.config
config.status: creating Misc/python.pc
config.status: creating Misc/python-config.sh
config.status: creating Modules/ld_so_aix
config.status: creating pyconfig.h
creating Modules/Setup
creating Modules/Setup.local
creating Makefile

If you want a release build with all optimizations active (LTO, PGO, etc),
please run ./configure --enable-optimizations

[root@VERTICA-PROD01 Python-3.6.1]#

Compile the Python source… PLEASE NOTE altinstall.

[root@VERTICA-PROD01 python36]# cd Python-3.6.1
[root@VERTICA-PROD01 Python-3.6.1]# ./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for python3.6... no
checking for python3... no
checking for python... python
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... linux
checking for --without-gcc... no
checking for --with-icc... no
checking for gcc... gcc
checking whether the C compiler works... yes


. . . . SNIP SNIP SNIP  . . . .
 
changing mode of /usr/local/lib/python3.6/lib-dynload/_multiprocessing.cpython-36m-x86_64-linux-gnu.so to 755
changing mode of /usr/local/lib/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so to 755
changing mode of /usr/local/lib/python3.6/lib-dynload/math.cpython-36m-x86_64-linux-gnu.so to 755
changing mode of /usr/local/lib/python3.6/lib-dynload/cmath.cpython-36m-x86_64-linux-gnu.so to 755
changing mode of /usr/local/lib/python3.6/lib-dynload/_testmultiphase.cpython-36m-x86_64-linux-gnu.so to 755
changing mode of /usr/local/lib/python3.6/lib-dynload/spwd.cpython-36m-x86_64-linux-gnu.so to 755
changing mode of /usr/local/lib/python3.6/lib-dynload/ to 755
changing mode of /usr/local/lib/python3.6/lib-dynload/__pycache__ to 755
running install_scripts
copying build/scripts-3.6/pyvenv-3.6 -> /usr/local/bin
copying build/scripts-3.6/pydoc3.6 -> /usr/local/bin
copying build/scripts-3.6/2to3-3.6 -> /usr/local/bin
copying build/scripts-3.6/idle3.6 -> /usr/local/bin
changing mode of /usr/local/bin/pyvenv-3.6 to 755
changing mode of /usr/local/bin/pydoc3.6 to 755
changing mode of /usr/local/bin/2to3-3.6 to 755
changing mode of /usr/local/bin/idle3.6 to 755
rm /usr/local/lib/python3.6/lib-dynload/_sysconfigdata_m_linux_x86_64-linux-gnu.py
rm -r /usr/local/lib/python3.6/lib-dynload/__pycache__
/usr/bin/install -c -m 644 ./Misc/python.man \
                /usr/local/share/man/man1/python3.6.1
if test "xupgrade" != "xno"  ; then \
                case upgrade in \
                        upgrade) ensurepip="--altinstall --upgrade" ;; \
                        install|*) ensurepip="--altinstall" ;; \
                esac; \
                LD_LIBRARY_PATH=/root/python36/Python-3.6.1 ./python -E -m ensurepip \
                        $ensurepip --root=/ ; \
        fi
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-9.0.1 setuptools-28.8.0
[root@VERTICA-PROD01 Python-3.6.1]#

Now that Python 3.6.x is installed, let us verify

[root@VERTICA-PROD01 Python-3.6.1]# python3.6 -V
Python 3.6.1
[root@VERTICA-PROD01 Python-3.6.1]# which python3.6
/usr/local/bin/python3.6
[root@VERTICA-PROD01 Python-3.6.1]#

Installing some additional IMPORTANT PY packages…

[root@VERTICA-PROD01 Python-3.6.1]# wget https://bootstrap.pypa.io/get-pip.py
--2018-04-13 17:38:28--  https://bootstrap.pypa.io/get-pip.py
Resolving spotlightmonprx.resource.comcast.net... 96.115.77.99
Connecting to spotlightmonprx.resource.comcast.net|96.115.77.99|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1780465 (1.7M) [text/x-python]
Saving to: “get-pip.py”
100%[==============================================================================================================================>] 1,780,465   6.77M/s   in 0.3s
2018-04-13 17:38:29 (6.77 MB/s) - “get-pip.py” saved [1780465/1780465]
[root@VERTICA-PROD01 Python-3.6.1]# python3.6 get-pip.py
Collecting pip
  Downloading pip-9.0.3-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 739kB/s
Collecting wheel
  Downloading wheel-0.31.0-py2.py3-none-any.whl (41kB)
    100% |████████████████████████████████| 51kB 7.9MB/s
Installing collected packages: pip, wheel
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-9.0.3 wheel-0.31.0
[root@VERTICA-PROD01 Python-3.6.1]#

NOW Python 3.6 is INSTALLED.

However to install Python 3.6 dependent modules, we would PREFER to initiate a virtual environment. We’ll do that in next post! Here we’ll continue with installing PICA, which is an open source messaging tool for Windows.

[root@VERTICA-PROD01 pica]# wget https://pypi.python.org/pypi/pika
--2018-04-13 18:06:40--  https://pypi.python.org/pypi/pika
Resolving spotlightmonprx.resource.comcast.net... 96.115.77.99
Connecting to spotlightmonprx.resource.comcast.net|96.115.77.99|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 24064 (24K) [text/html]
Saving to: “pika”
100%[==============================================================================================================================>] 24,064      --.-K/s   in 0.02s
2018-04-13 18:06:41 (1.34 MB/s) - “pika” saved [24064/24064]
[root@VERTICA-PROD01 pica]# ll
total 24
-rw-r--r-- 1 root root 24064 Apr 13 18:06 pika
[root@VERTICA-PROD01 pica]#

PIKA is installed in production Vertica Cluster:

[root@VERTICA-PROD01 pica]# wget https://pypi.python.org/pypi/pika
--2018-04-13 18:06:40--  https://pypi.python.org/pypi/pika
Resolving spotlightmonprx.resource.comcast.net... 96.115.77.99
Connecting to spotlightmonprx.resource.comcast.net|96.115.77.99|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 24064 (24K) [text/html]
Saving to: “pika”
100%[==============================================================================================================================>] 24,064      --.-K/s   in 0.02s
2018-04-13 18:06:41 (1.34 MB/s) - “pika” saved [24064/24064]
[root@VERTICA-PROD01 pica]# ll
total 24
-rw-r--r-- 1 root root 24064 Apr 13 18:06 pika
[root@VERTICA-PROD01 pica]#
[root@VERTICA-PROD01 pica]# cd ..
[root@VERTICA-PROD01 python36]# which pip
/usr/local/bin/pip
[root@VERTICA-PROD01 python36]# pip install pika
Collecting pika
  Downloading pika-0.11.2-py2.py3-none-any.whl (107kB)
    100% |████████████████████████████████| 112kB 3.5MB/s
Installing collected packages: pika
Successfully installed pika-0.11.2
[root@VERTICA-PROD01 python36]# python3.6
Python 3.6.1 (default, Apr 13 2018, 17:20:29)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pika
>>> quit()
[root@VERTICA-PROD01 python36]#

Enjoy experimenting with PIKA!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s