Create 1N Vertica 9.2 cluster in AWS EC2 RHEL 8 (Part 2)

In my previous post, I prepared a single node RHEL 8 EC2 instance in AWS for Vertica 9.x installation. In this post I’ll continue with following installation steps.

The installation script runs on a single node, using a Bash shell, and it copies the Vertica package to all other hosts (identified by the --hosts argument) in your planned cluster. Complete install_vertica option are documented here.

[root@ip-172-31-1-75 ~]# /opt/vertica/sbin/install_vertica --hosts node0001  --rpm /tmp/vertica-9.2.1-0.x86_64.RHEL6.rpm   --dba-user dbadmin
Vertica Analytic Database 9.2.1-0 Installation Tool

AWS Detected. Using AWS defaults.
    AWS Default: --data-dir was not specified, using default: /vertica/data
    AWS Default: --dba-user-password-disabled was not specified,  disabling dba password by default while on AWS
    AWS Default: --point-to-point was not specified,  enabling point-to-point spread communication by default while on AWS

>> Validating options...


Mapping hostnames in --hosts (-s) to addresses...
        Unable to resolve to IPv4: node0001
        Error: Unable to resolve 'node0001'
Installation FAILED with errors.

Installation stopped before any changes were made.
[root@ip-172-31-1-75 ~]#

I’ll specify data directory at the correct mount point and specify the dbadmin password. Next attempt failed with three errors.

[root@ip-172-31-1-75 ~]# /opt/vertica/sbin/install_vertica --hosts 172.31.1.75  --rpm /tmp/vertica-9.2.1-0.x86_64.RHEL6.rpm --data-dir /vdata  --dba-user dbadmin --dba-user-password KasaMusa --license CE --accept-eula
Vertica Analytic Database 9.2.1-0 Installation Tool

AWS Detected. Using AWS defaults.
    AWS Default: --dba-user-password-disabled was not specified,  disabling dba password by default while on AWS
    AWS Default: --point-to-point was not specified,  enabling point-to-point spread communication by default while on AWS

>> Validating options...


Mapping hostnames in --hosts (-s) to addresses...

>> Starting installation tasks.
>> Getting system information for cluster (this may take a while)...

Default shell on nodes:
172.31.1.75 /bin/bash

>> Validating software versions (rpm or deb)...


>> Beginning new cluster creation...

successfully backed up admintools.conf on 172.31.1.75

>> Creating or validating DB Admin user/group...

Successful on hosts (1): 172.31.1.75
    Provided DB Admin account details: user = dbadmin, group = verticadba, home = /home/dbadmin
    Creating group... Adding group
    Validating group... Okay
    Creating user... Adding user, Setting credentials
    Validating user... Okay


>> Validating node and cluster prerequisites...

Prerequisites not fully met during local (OS) configuration for
verify-172.31.1.75.xml:
    HINT (S0305): https://www.vertica.com/docs/9.2.x/HTML/index.htm#cshid=S0305
        TZ is unset for dbadmin. Consider updating .profile or .bashrc
    FAIL (S0150): https://www.vertica.com/docs/9.2.x/HTML/index.htm#cshid=S0150
        These disks do not have 'deadline' or 'noop' IO scheduling: '/dev/xvdb'
        ('xvdb') = 'mq-deadline'
    FAIL (S0310): https://www.vertica.com/docs/9.2.x/HTML/index.htm#cshid=S0310
        Transparent hugepages is set to 'always'. Must be 'never' or 'madvise'.

System prerequisites failed.  Threshold = WARN
        Hint: Fix above failures or use --failure-threshold

Installation FAILED with errors.

****
AdminTools and your existing Vertica databases may be unavailable.
Investigate the above warnings/errors and re-run installation.
****
[root@ip-172-31-1-75 ~]#

As noted in Part 1, both IO Scheduling and Transparent hugepages setting errored out during installation. The installer is expecting RHEL 7.x values, which have been enhanced in RHEL 8.0. I’ll address, the two failures and one hint below, by forcing 7.x values to see what happens.

[root@ip-172-31-1-75 ~]# echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
[root@ip-172-31-1-75 ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
[root@ip-172-31-1-75 ~]# vi /etc/rc.local
[root@ip-172-31-1-75 ~]# vi /etc/rc.local
[root@ip-172-31-1-75 ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
[root@ip-172-31-1-75 ~]#

Now on retry, the installation fails with IO Scheduler error only.

[root@ip-172-31-1-75 ~]# /opt/vertica/sbin/install_vertica --hosts 172.31.1.75  --rpm /tmp/vertica-9.2.1-0.x86_64.RHEL6.rpm --data-dir /vdata  --dba-user dbadmin --dba-user-password KasaMusa --license CE --accept-eula
Vertica Analytic Database 9.2.1-0 Installation Tool

AWS Detected. Using AWS defaults.
    AWS Default: --dba-user-password-disabled was not specified,  disabling dba password by default while on AWS
    AWS Default: --point-to-point was not specified,  enabling point-to-point spread communication by default while on AWS

>> Validating options...


Mapping hostnames in --hosts (-s) to addresses...

>> Starting installation tasks.
>> Getting system information for cluster (this may take a while)...

Default shell on nodes:
172.31.1.75 /bin/bash

>> Validating software versions (rpm or deb)...


>> Beginning new cluster creation...

successfully backed up admintools.conf on 172.31.1.75

>> Creating or validating DB Admin user/group...

Successful on hosts (1): 172.31.1.75
    Provided DB Admin account details: user = dbadmin, group = verticadba, home = /home/dbadmin
    Creating group... Group already exists
    Validating group... Okay
    Creating user... User already exists
    Validating user... Okay


>> Validating node and cluster prerequisites...

Prerequisites not fully met during local (OS) configuration for
verify-172.31.1.75.xml:
    HINT (S0305): https://www.vertica.com/docs/9.2.x/HTML/index.htm#cshid=S0305
        TZ is unset for dbadmin. Consider updating .profile or .bashrc
    FAIL (S0150): https://www.vertica.com/docs/9.2.x/HTML/index.htm#cshid=S0150
        These disks do not have 'deadline' or 'noop' IO scheduling: '/dev/xvdb'
        ('xvdb') = 'mq-deadline'

System prerequisites failed.  Threshold = WARN
        Hint: Fix above failures or use --failure-threshold

Installation FAILED with errors.

****
AdminTools and your existing Vertica databases may be unavailable.
Investigate the above warnings/errors and re-run installation.
****
[root@ip-172-31-1-75 ~]#

When I tried setting IO Scheduler value to noop, I got invalid argument error, whereas none was accepted.

[root@ip-172-31-1-75 ~]# echo noop > /sys/block/xvdb/queue/scheduler
-bash: echo: write error: Invalid argument
[root@ip-172-31-1-75 ~]# echo none > /sys/block/xvdb/queue/scheduler
[root@ip-172-31-1-75 ~]#
[root@ip-172-31-1-75 ~]# echo 'echo none > /sys/block/xvdb/queue/scheduler' >> /etc/rc.local
[root@ip-172-31-1-75 ~]#

Now the installation script was successful, with only hints (including for IO schedulers) which I’ll address later.

[root@ip-172-31-1-75 ~]# /opt/vertica/sbin/install_vertica --hosts 172.31.1.75  --rpm /tmp/vertica-9.2.1-0.x86_64.RHEL6.rpm --data-dir /vdata  --dba-user dbadmin --dba-user-password KasaMusa --license CE --accept-eula
Vertica Analytic Database 9.2.1-0 Installation Tool

AWS Detected. Using AWS defaults.
    AWS Default: --dba-user-password-disabled was not specified,  disabling dba password by default while on AWS
    AWS Default: --point-to-point was not specified,  enabling point-to-point spread communication by default while on AWS

>> Validating options...


Mapping hostnames in --hosts (-s) to addresses...

>> Starting installation tasks.
>> Getting system information for cluster (this may take a while)...

Default shell on nodes:
172.31.1.75 /bin/bash

>> Validating software versions (rpm or deb)...


>> Beginning new cluster creation...

successfully backed up admintools.conf on 172.31.1.75

>> Creating or validating DB Admin user/group...

Successful on hosts (1): 172.31.1.75
    Provided DB Admin account details: user = dbadmin, group = verticadba, home = /home/dbadmin
    Creating group... Group already exists
    Validating group... Okay
    Creating user... User already exists
    Validating user... Okay


>> Validating node and cluster prerequisites...

Prerequisites not fully met during local (OS) configuration for
verify-172.31.1.75.xml:
    HINT (S0151): https://www.vertica.com/docs/9.2.x/HTML/index.htm#cshid=S0151
        These disks do not have known IO schedulers: '/dev/xvdb' ('xvdb') =
        'none'
    HINT (S0305): https://www.vertica.com/docs/9.2.x/HTML/index.htm#cshid=S0305
        TZ is unset for dbadmin. Consider updating .profile or .bashrc

System prerequisites passed.  Threshold = WARN


>> Establishing DB Admin SSH connectivity...

Installing/Repairing SSH keys for dbadmin


>> Setting up each node and modifying cluster...

Creating Vertica Data Directory...

Updating agent...

>> Sending new cluster configuration to all nodes...

Starting agent...

>> Completing installation...

Running upgrade logic
No spread upgrade required: /opt/vertica/config/vspread.conf not found on any node
Installation complete.

Please evaluate your hardware using Vertica's validation tools:
    https://www.vertica.com/docs/9.2.x/HTML/index.htm#cshid=VALSCRIPT

To create a database:
  1. Logout and login as dbadmin. (see note below)
  2. Run /opt/vertica/bin/adminTools as dbadmin
  3. Select Create Database from the Configuration Menu

  Note: Installation may have made configuration changes to dbadmin
  that do not take effect until the next session (logout and login).

To add or remove hosts, select Cluster Management from the Advanced Menu.
[root@ip-172-31-1-75 ~]#

I’ll install Management Console and log out and login as dbadmin, and create the DB.

[root@ip-172-31-1-75 ~]# rpm -Uvh /tmp/vertica-console-9.2.1-0.x86_64.RHEL6.rpm
 Verifying…                          ################################# [100%]
 Preparing…                          ################################# [100%]
 ERROR: This version of Red Hat is not supported.
 error: %prein(vertica-console-9.2.1-0.x86_64) scriptlet failed, exit status 1
 error: vertica-console-9.2.1-0.x86_64: install failed
 [root@ip-172-31-1-75 ~]#

MC installation failed!

Let us try login as dbadmin and create a database.

Using admintools to create a database.

*** Creating database: tsptestdb ***
         Creating database tsptestdb
         Starting bootstrap node v_tsptestdb_node0001 (172.31.1.75)
         Starting nodes:
                 v_tsptestdb_node0001 (172.31.1.75)
         Starting Vertica on all nodes. Please wait, databases with a large catalog may take a while to initialize.
         Node Status: v_tsptestdb_node0001: (DOWN)
         Node Status: v_tsptestdb_node0001: (DOWN)
         Node Status: v_tsptestdb_node0001: (DOWN)
         Node Status: v_tsptestdb_node0001: (DOWN)
         Node Status: v_tsptestdb_node0001: (DOWN)
         Node Status: v_tsptestdb_node0001: (DOWN)
         Node Status: v_tsptestdb_node0001: (DOWN)
         Node Status: v_tsptestdb_node0001: (DOWN)
         Node Status: v_tsptestdb_node0001: (DOWN)
         Node Status: v_tsptestdb_node0001: (DOWN)
 ERROR:  Database did not start cleanly on initiator node!
         Stopping all nodes
 Could not connect to database (EOF received)/opt/vertica/bin/vsql: error while loading shared libraries: libnsl.so.1: cannot open shared object file: No such file or directory
 Press RETURN to continue

Database creation FAILED with ERROR: error while loading shared libraries: libnsl.so.1: cannot open shared object file: No such file or directory.

At this point I’ll terminate the EC2 instance and create an RHEL 7.x instance and install Vertica 9.2 in my next post.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s