Step-by-Step Installation of Oracle 11g Two-Node RAC Linux 7

Installation Description:

Operating System: Oracle Linux 7.5 64 bit
Oracle Database Version: 11.2.0.4
Database Type: Oracle RAC with ASM

Minimum Requirement:

• 20 GB for both grid and oracle home on each node
• At least 1024 x 768 display resolution, so that OUI displays correctly
• 1 GB of space in the /tmp directory.
• If you intend to install Oracle Databases or an Oracle RAC database on the cluster, be aware that the size of the /dev/shm mount area on each server must be greater than the system global areal (SGA) and the program global area (PGA) of the databases on the servers.
• At least 2.5 GB of RAM for Oracle Grid Infrastructure for a Cluster installations, including installations where you plan to install Oracle RAC. 
• Swap space equivalent to the multiple of the available RAM:    

- Between 2.5 GB and 32 GB => Equal to the size of RAM      

-  More than 16 GB => 16 GB of RAM

• Oracle Linux 7 and Red Hat Enterprise Linux 7:The following packages (or later versions) must be installed:

• binutils-2.23.52.0.1-12.el7.x86_64
• compat-libcap1-1.10-3.el7.x86_64
• gcc-4.8.2-3.el7.x86_64
• gcc-c++-4.8.2-3.el7.x86_64
• glibc-2.17-36.el7.i686
• glibc-2.17-36.el7.x86_64
• glibc-devel-2.17-36.el7.i686
• glibc-devel-2.17-36.el7.x86_64
• ksh
• libaio-0.3.109-9.el7.i686
• libaio-0.3.109-9.el7.x86_64
• libaio-devel-0.3.109-9.el7.i686
• libaio-devel-0.3.109-9.el7.x86_64
• libgcc-4.8.2-3.el7.i686
• libgcc-4.8.2-3.el7.x86_64
• libstdc++-4.8.2-3.el7.i686
• libstdc++-4.8.2-3.el7.x86_64
• libstdc++-devel-4.8.2-3.el7.i686
• libstdc++-devel-4.8.2-3.el7.x86_64
• libXi-1.7.2-1.el7.i686
• libXi-1.7.2-1.el7.x86_64
• libXtst-1.2.2-1.el7.i686
• libXtst-1.2.2-1.el7.x86_64
• make-3.82-19.el7.x86_64
• sysstat-10.1.5-1.el7.x86_64

Step 1: First you have to download virtual box. Go to google and search download virtual box 5.2.4 64 bit for Windows OS.

Step 2: Install Virtual Box by clicking on virtual box setup.exe file. Use exactly same steps to install Virtual box.

Step 3: RAC Configuration Setting on Virtual Box
Click on VirtualBox icon from your desktop machine. You will see below screen. Click on new to create new virtual server as rac1. 

Name: This should be anything
Type: This should be Linux since we are using Linux OS.
Version: This should be Oracle(64 bit). Sometime you may be able to see Oracle(32 bit) for which you have to call windows admin and make changes at Windows BIOS level to change from 32-bit to 64 bit.

Here you have to select memory for 1st node of RAC.

Click on folder icon and select directory on your machine where rac1 machine will be saved. Also, select the size of machine.(Here 30 GB). Finally click on create.

Your rac1 machine will be created like this:

Now its time to change Virtual Box settings for RAC environment:

Go to Audio and disable everything.

Go to Network:
For Adapter1,select "Internal Network" and give name as pubnet.(This will be your public network)
For Adapter2, select "Internal Network" give name as privnet.(This will be your private network)
For Adapter3, select "Internal Network" give name as privnet.(This is not required, but you can use this for internet access). I have not used in my environment.

 

Go to Storage, click on empty, you will see below screen. Select image file of Oracle Linux Operating system setup from your desktop machine.

Once you selected OS setup then you will see below screen. If all changes are done then click ok to save. Now start the machine.

You will see below screen for Linux installation. Click ENTER. Wait for few seconds.

Click continue by selecting default language.

In below screen, you will see different settings. Click on "Installation Destination" of SYSTEM.
In "Other Storage Option", you can create partitions manually. I have selected 1st option which will create partition automatically. Click on Done.

Now click on "Network & Hostname" of SYSTEM. Select Ethernet(enpOs3) and make below changes and click save.
General: 

• Automatically connect
• All users may connect

IPV4 Settings:
IP Address: 192.168.43.91 => This is your public/physical RAC1 machine IP
Netmask: 255.255.255.0
Gateway: 0.0

IPV6 Settings:
Select method as ignore.

Once changes are done then you will see below screen. Click ON.

You have to do same settings for Ethernet(enpOs8) of RAC1 machine, just change the IP address as 192.168.100.1 as private IP.

Select "Software Selection" and select last option "Server with GUI". On Right side, select options highlighted in screen. You can untick options which are not suitable.

Once all settings are done then click on Begin Installation.

Here you have to set password for root user. Note down this password and click on Reboot.

In Licensing, accept the agreement and click on Done and then Finish Configuration.

Here, you have to select No option.

Again select option NO.

Click on Forward then Next,Next, and create one account(Here i have created oracle).

Select India as country and click START.

You have successfully installed Oracle Linux 7.5 on Virtual Box. Now its time to do some settings post installation.

Step 4: Server Settings

For enabling mouse-pointer/keyboard without pressing any alt+ctl button from your desktop machine to Linux Virtual Machine, you have to follow below steps:

• By oracle user, go to Devices
• Click on "Install Guest Addition CD Image".
• You will get below pop-up box. Click Run.
• It will ask authentication. Enter password for oracle user.
• You will get one terminal. Once all operations on terminal gets completed then it will ask you to Close terminal. Close that terminal.
• Go to Devices and then "Shared Clipboard" &Drag and Drop" and select "Bidirectional".

Now you can easily enter into Virtual machine from your desktop and vice-versa. Also, you can drag and drop any file from your desktop machine to virtual machine and vice-versa.

If this will not work then ,reboot virtual machine and  check. 

Now you have to add rac IP entries in /etc/hosts file of both nodes. Note that below two entries are mandatory and don't try to delete these entries from this files otherwise you will face network related errors.

For 2-node RAC, you will require below IP configuration:

• 2 public IPs - These will be physical IPs
• 2 Private IPs - These will be physical IPs - Private IP segment should be different than other IP segment
• 2 VIPs - These will be virtual IPs in segment of public IP
• 1 or 3 SCAN IP - These will be virtual IPs in segment of public IP
• GNS IP is not mandatory which is optional and features of 11gR2.
• DNS IP - We are not using DNS.(If you are using 3 scan IPs then these should be added in DNS server).

Note none of the IPs should be ping prior installation, except public and private IPs.

 

Also, public IPs ethernet card name should be same on all nodes.(For example, if eth0 is for public IP on 1st node then eth0 should be on 2nd node for public IP).

Private IPs ethernet card name should be same on all nodes.(For example, if eth1 is for private IP on 1st node then eth1 should be on 2nd node for private IP).

Firewall should be stopped on all nodes.

SELINUX should be disabled on all nodes.

Step 5: Create 2nd node of rac by cloning 1st node. Follow below steps to create colne of 1st node and give names as rac2.

Since this is clone of 1st node, everything will be same as first node including network settings,IP addresses,host name,mac addresses which will not work for 2nd node. You have to change below settings.

• Host name:
• IP Addresses - Both public and private (Edit Network settings and change IPs).
• Mac addresses - Stop rac2 virtual machine and change mac address of rac2 machine.Just change last character of address. (Mac address of both rac1 and rac2 machine should not be same).

You can verify ping status of both rac1 and rac2 machines.

[oracle@rac2 ~]$ ping rac2

PING rac2.localdomain (192.168.56.92) 56(84) bytes of data.

64 bytes from rac2.localdomain (192.168.56.92): icmp_seq=1 ttl=64 time=0.026 ms

64 bytes from rac2.localdomain (192.168.56.92): icmp_seq=2 ttl=64 time=0.022 ms

^C

--- rac2.localdomain ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 999ms

rtt min/avg/max/mdev = 0.022/0.024/0.026/0.002 ms

[oracle@rac2 ~]$

[oracle@rac2 ~]$

[oracle@rac2 ~]$

[oracle@rac2 ~]$

[oracle@rac2 ~]$ cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# Public

192.168.56.91    rac1.localdomain        rac1

192.168.56.92    rac2.localdomain        rac2

# Private

192.168.100.1    priv-rac1.localdomain   priv-rac1

192.168.100.2    priv-rac2.localdomain   priv-rac2

# Virtual

192.168.56.81    vip-rac1.localdomain    vip-rac1

192.168.56.82    vip-rac2.localdomain    vip-rac2

# SCAN

192.168.56.11    scan-rac.localdomain    scan-rac

192.168.56.12    scan-rac.localdomain    scan-rac

192.168.56.13    scan-rac.localdomain    scan-rac

[oracle@rac2 ~]$ ping priv-rac2

PING priv-rac2.localdomain (192.168.100.2) 56(84) bytes of data.

64 bytes from priv-rac2.localdomain (192.168.100.2): icmp_seq=1 ttl=64 time=0.025 ms

64 bytes from priv-rac2.localdomain (192.168.100.2): icmp_seq=2 ttl=64 time=0.023 ms

^C

Step 6: Create shared storage.

• Right click on rac1 machine and create storage as per below steps.
• Go to File =>Virtual Media Manager and mark above storage disk to shared.
• You can view both rac1 and rac2 machines and you will see above created disk is shareable now.
• Go to rac2 machine and go to storage and select "Choose existing disk option" and select above disk to add to rac2 machine.
• You can create multiple shared disk like this.

ASM RAW Disk Creation:

You have created shared storage and now raw disks are created on server as /dev/sdb, /dev/sdc, and /dev/sdd.

You have to create logical partitions based on these raw devices by using below steps. You can mark all partitions as Primary so that they can not be extended later. 

You have to login as root user and execute below commands:

• fdisk  /dev/sdb
• n
• p
• 1
• Enter
• Enter
• w

[root@rac1 ~]# fdisk -l

Disk /dev/sda: 32.2 GB, 32212254720 bytes, 62914560 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x00094990

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *        2048     1026047      512000   83  Linux

/dev/sda2         1026048    62914559    30944256   8e  Linux LVM

Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x79b549fc

   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1            2048    10485759     5241856   83  Linux

Disk /dev/sdc: 5368 MB, 5368709120 bytes, 10485760 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdd: 5368 MB, 5368709120 bytes, 10485760 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/ol-root: 28.4 GB, 28420603904 bytes, 55508992 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/ol-swap: 3221 MB, 3221225472 bytes, 6291456 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

[root@rac1 ~]# ls -ltr /dev/sd*

brw-rw----. 1 root disk 8,  0 Jan  4 20:52 /dev/sda

brw-rw----. 1 root disk 8,  2 Jan  4 20:52 /dev/sda2

brw-rw----. 1 root disk 8, 48 Jan  4 20:52 /dev/sdd

brw-rw----. 1 root disk 8, 16 Jan  4 20:52 /dev/sdb

brw-rw----. 1 root disk 8, 32 Jan  4 20:52 /dev/sdc

brw-rw----. 1 root disk 8,  1 Jan  4 20:52 /dev/sda1

brw-rw----. 1 root disk 8, 17 Jan  4 20:52 /dev/sdb1

[root@rac1 ~]#

[root@rac1 ~]# fdisk /dev/sdb

Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Command (m for help): p

Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x79b549fc

   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1            2048    10485759     5241856   83  Linux

Command (m for help): q

[root@rac1 ~]# fdisk /dev/sdc

Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Device does not contain a recognized partition table

Building a new DOS disklabel with disk identifier 0x254ec796.

Command (m for help): p

Disk /dev/sdc: 5368 MB, 5368709120 bytes, 10485760 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x254ec796

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n

Partition type:

   p   primary (0 primary, 0 extended, 4 free)

   e   extended

Select (default p): p

Partition number (1-4, default 1): 1

First sector (2048-10485759, default 2048):

Using default value 2048

Last sector, +sectors or +size{K,M,G} (2048-10485759, default 10485759):

Using default value 10485759

Partition 1 of type Linux and of size 5 GiB is set

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

[root@rac1 ~]#

[root@rac1 ~]# fdisk /dev/sdd

Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.

Be careful before using the write command.

Device does not contain a recognized partition table

Building a new DOS disklabel with disk identifier 0x66d6c3d8.

Command (m for help): p

Disk /dev/sdd: 5368 MB, 5368709120 bytes, 10485760 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x66d6c3d8

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n

Partition type:

   p   primary (0 primary, 0 extended, 4 free)

   e   extended

Select (default p): p

Partition number (1-4, default 1): 1

First sector (2048-10485759, default 2048):

Using default value 2048

Last sector, +sectors or +size{K,M,G} (2048-10485759, default 10485759):

Using default value 10485759

Partition 1 of type Linux and of size 5 GiB is set

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

[root@rac1 ~]#

[root@rac1 ~]#

[root@rac1 ~]# ls -ltr /dev/sd*

brw-rw----. 1 root disk 8,  0 Jan  4 20:52 /dev/sda

brw-rw----. 1 root disk 8,  2 Jan  4 20:52 /dev/sda2

brw-rw----. 1 root disk 8,  1 Jan  4 20:52 /dev/sda1

brw-rw----. 1 root disk 8, 17 Jan  4 20:52 /dev/sdb1

brw-rw----. 1 root disk 8, 16 Jan  4 20:55 /dev/sdb

brw-rw----. 1 root disk 8, 33 Jan  4 20:55 /dev/sdc1

brw-rw----. 1 root disk 8, 32 Jan  4 20:55 /dev/sdc

brw-rw----. 1 root disk 8, 49 Jan  4 20:55 /dev/sdd1

brw-rw----. 1 root disk 8, 48 Jan  4 20:55 /dev/sdd

[root@rac1 ~]#

[root@rac1 ~]#

[root@rac1 ~]# oracleasm createdisk DATA /dev/sdb1

Writing disk header: done

Instantiating disk: failed

Clearing disk header: done

[root@rac1 ~]#

[root@rac1 ~]#

[root@rac1 ~]# /usr/sbin/oracleasm init

Creating /dev/oracleasm mount point: /dev/oracleasm

Loading module "oracleasm": oracleasm

Configuring "oracleasm" to use device physical block size

Mounting ASMlib driver filesystem: /dev/oracleasm

[root@rac1 ~]#

[root@rac1 ~]#

[root@rac1 ~]# oracleasm createdisk DISK1 /dev/sdb1

Writing disk header: done

Instantiating disk: done

[root@rac1 ~]#

[root@rac1 ~]# oracleasm createdisk DISK2 /dev/sdc1

Writing disk header: done

Instantiating disk: done

[root@rac1 ~]#

[root@rac1 ~]# oracleasm createdisk DISK3 /dev/sdd1

Writing disk header: done

Instantiating disk: done

[root@rac1 ~]#

[root@rac1 ~]#

[root@rac1 ~]# oracleasm listdisks

DISK1

DISK2

DISK3

[root@rac1 ~]#

[root@rac1 ~]# oracleasm configure -i

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver.  The following questions will determine whether the driver is

loaded on boot and what permissions it will have.  The current values

will be shown in brackets ('[]').  Hitting <ENTER> without typing an

answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid

Default group to own the driver interface []: oinstall

Start Oracle ASM library driver on boot (y/n) [n]: y

Scan for Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done

[root@rac1 ~]#

[root@rac2 ~]# oracleasm configure -i

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver.  The following questions will determine whether the driver is

loaded on boot and what permissions it will have.  The current values

will be shown in brackets ('[]').  Hitting <ENTER> without typing an

answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid

Default group to own the driver interface []: oinstall

Start Oracle ASM library driver on boot (y/n) [n]: y

Scan for Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done

[root@rac2 ~]#

[root@rac2 ~]# oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

Instantiating disk "DISK1"

Instantiating disk "DISK2"

Instantiating disk "DISK3"

[root@rac2 ~]#

[root@rac2 ~]# oracleasm listdisks

DISK1

DISK2

DISK3

[root@rac2 ~]#

You can make your desktop machine folder for sharing software and other files by making below changes so that you can easily access this shared folder on rac1 and rac2 Linux servers.

You can verify whether folder is shared or not.

[root@rac1 sf_RAC_Setup]# cd RAC\ 11g\ Installation/

[root@rac1 RAC 11g Installation]# pwd

/media/sf_RAC_Setup/RAC 11g Installation

[root@rac1 RAC 11g Installation]# ll

total 6327958

-rwxrwx---. 1 root vboxsf       1640 Sep 12 10:21 Create groups and users for asm.txt

-rwxrwx---. 1 root vboxsf        192 Jan 12  2017 Errors.txt

-rwxrwx---. 1 root vboxsf       1013 Nov 21 20:56 Error while installing Oracle RAC 11g R2 on OEL 7.1.txt

-rwxrwx---. 1 root vboxsf     670944 Jan 13  2017 ksh-20100621-2.el6.x86_64.rpm

-rwxrwx---. 1 root vboxsf      10256 Nov 21 20:05 libaio-devel-0.3.105-2.x86_64.rpm

drwxrwx---. 1 root vboxsf       4096 Dec 22  2016 Linux 6.3

-rwxrwx---. 1 root vboxsf        261 Aug 24  2016 List of rpms for asm.txt

-rwxrwx---. 1 root vboxsf      17168 Dec 24  2016 oracleasmlib-2.0.12-1.el6.x86_64.rpm

-rwxrwx---. 1 root vboxsf      13336 Dec 24  2016 oracleasmlib-2.0.4-1.el6.i686.rpm

-rwxrwx---. 1 root vboxsf      85303 Aug 25  2016 oracleasm-support-2.1.8-1.el5.i386.rpm

-rwxrwx---. 1 root vboxsf      90225 Aug  5  2016 oracleasm-support-2.1.8-1.el5.x86_64.rpm

Step 7: Perform OS pre-requisites like:

• RPM installation
• Limits/Kernel parameter settings
• Oracle/grid Base/Home directory creation
• User and group creation

[root@rac1 RAC 11g Installation]#

[root@rac1 RAC 11g Installation]# rpm -ivh oracleasmlib-2.0.12-1.el6.x86_64.rpm

warning: oracleasmlib-2.0.12-1.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:oracleasmlib-2.0.12-1.el6        ################################# [100%]

[root@rac1 RAC 11g Installation]#

[root@rac1 RAC 11g Installation]# rpm -ivh oracleasm-support-2.1.8-1.el5.x86_64.rpm

warning: oracleasm-support-2.1.8-1.el5.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 1e5e0159: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:oracleasm-support-2.1.8-1.el5    ################################# [100%]

[root@rac1 RAC 11g Installation]#

[root@rac1 RAC 11g Installation]#

[root@rac1 RAC 11g Installation]#

[root@rac1 RAC 11g Installation]# oracleasm listdisks

[root@rac1 RAC 11g Installation]#

[root@rac1 RAC 11g Installation]#

[root@rac1 RAC 11g Installation]#

[root@rac1 RAC 11g Installation]# oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

[root@rac1 RAC 11g Installation]#

Add below parameters in below locations on both nodes(rac1 and rac2):

[root@rac1 ~]# cat /etc/security/limits.conf | tail -10

#*               soft    core            0

#*               hard    rss             10000

#@student        hard    nproc           20

#@faculty        soft    nproc           20

#@faculty        hard    nproc           50

#ftp             hard    nproc           0

#@student        -       maxlogins       4

# End of file

[root@rac1 ~]# vi /etc/security/limits.conf

[root@rac1 ~]# cat /etc/security/limits.conf | tail -10

# End of file

grid soft nproc 2047

grid hard nproc 16384

grid soft nofile 1024

grid hard nofile 65536

oracle soft nproc 2047

oracle hard nproc 16384

oracle soft nofile 1024

oracle hard nofile 65536

[root@rac1 ~]#

[root@rac1 ~]# vi /etc/sysctl.conf

[root@rac1 ~]#

[root@rac1 ~]#

[root@rac1 ~]# cat /etc/sysctl.conf

# System default settings live in /usr/lib/sysctl.d/00-system.conf.

# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file

#

# For more information, see sysctl.conf(5) and sysctl.d(5).

fs.aio-max-nr = 1048576

fs.file-max = 6815744

kernel.shmall = 2097152

kernel.shmmax = 2009298944

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048586

[root@rac1 ~]#

[root@rac1 ~]# sysctl -p

fs.aio-max-nr = 1048576

fs.file-max = 6815744

kernel.shmall = 2097152

kernel.shmmax = 2009298944

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048586

[root@rac1 ~]# groupadd oinstall

[root@rac1 ~]# groupadd dba

[root@rac1 ~]# groupadd asmadmin

[root@rac1 ~]# groupadd oper

[root@rac1 ~]# groupadd asmdba

[root@rac1 ~]# groupadd asmoper

[root@rac1 ~]# usermod -g oinstall -G dba oracle

[root@rac1 ~]# useradd -g oinstall -G asmadmin grid

[root@rac1 ~]# usermod -g dba grid

[root@rac1 ~]# usermod -g oinstall grid

[root@rac1 ~]# usermod -g oper grid

[root@rac1 ~]# usermod -g asmdba grid

[root@rac1 ~]# usermod -g asmoper grid

[root@rac1 ~]#

Create Directory structure on both nodes:

[root@rac1 ~]# cd /

[root@rac1 /]# mkdir /u01

[root@rac1 /]# df -h /u01

Filesystem           Size  Used Avail Use% Mounted on

/dev/mapper/ol-root   27G  5.1G   22G  20% /

[root@rac1 /]# chown -R grid:oinstall /u01/

[root@rac1 /]# chmod -R 775 /u01/

[root@rac1 /]#

[root@rac1 /]# ls -ld /u01

drwxrwxr-x. 2 grid oinstall 6 Jan  3 20:26 /u01

[root@rac1 /]# passwd grid

Changing password for user grid.

New password:

BAD PASSWORD: The password is shorter than 8 characters

Retype new password:

passwd: all authentication tokens updated successfully.

[root@rac1 /]#

[grid@rac1 ~]$ cd /u01

[grid@rac1 u01]$ ls -ltr

total 0

[grid@rac1 u01]$ mkdir setup

[root@rac1 RAC 11g Installation]# cp p13390677_112040_Linux-x86-64_3of7.zip /u01/setup/

[root@rac1 RAC 11g Installation]#

[root@rac1 setup]# chmod 777 p13390677_112040_Linux-x86-64_3of7.zip *

[root@rac1 setup]# ls -tlr

total 1177004

-rwxrwxrwx. 1 root root 1205251894 Jan  3 20:33 p13390677_112040_Linux-x86-64_3of7.zip

[grid@rac1 setup]$ ls -ltr

total 1177004

-rwxrwxrwx. 1 root root 1205251894 Jan  3 20:33 p13390677_112040_Linux-x86-64_3of7.zip

Step 8: Grid/Oracle Software Installation and Configuration:

For Grid installation and Configuration,you have to unzip grid setup.

[grid@rac1 setup]$ unzip p13390677_112040_Linux-x86-64_3of7.zip

Archive:  p13390677_112040_Linux-x86-64_3of7.zip

   creating: grid/

  inflating: grid/runInstaller      

   creating: grid/response/

  inflating: grid/response/grid_install.rsp 

   creating: grid/sshsetup/

  inflating: grid/sshsetup/sshUserSetup.sh 

  inflating: grid/readme.html       

   creating: grid/install/

   creating: grid/install/images/

  inflating: grid/install/images/billboards.gif 

  inflating: grid/install/runInstaller.sh 

  inflating: grid/install/unzip     

  inflating: grid/install/attachHome.sh 

 extracting: grid/install/addLangs.sh 

  …

…

…

  inflating: grid/stage/invDetails.properties 

   creating: grid/stage/globalvariables/

  inflating: grid/stage/globalvariables/variable.properties 

  inflating: grid/stage/globalvariables/globalvar.xml 

   creating: grid/rpm/

  inflating: grid/rpm/cvuqdisk-1.0.9-1.rpm 

  inflating: grid/runcluvfy.sh      

  inflating: grid/welcome.html

[grid@rac1 setup]$ ls -ltr

total 1177008

drwxr-xr-x. 7 grid oinstall       4096 Aug 27  2013 grid

-rwxrwxrwx. 1 grid oinstall 1205251894 Jan  3 20:33 p13390677_112040_Linux-x86-64_3of7.zip

[grid@rac1 setup]$ cd grid

[grid@rac1 grid]$ ls -ltr

total 56

drwxr-xr-x.  2 grid oinstall    28 Aug 26  2013 sshsetup

-rwxr-xr-x.  1 grid oinstall  3268 Aug 26  2013 runInstaller

-rwxr-xr-x.  1 grid oinstall  4878 Aug 26  2013 runcluvfy.sh

drwxr-xr-x.  2 grid oinstall    33 Aug 26  2013 rpm

drwxr-xr-x.  2 grid oinstall    29 Aug 26  2013 response

drwxr-xr-x.  4 grid oinstall  4096 Aug 26  2013 install

drwxr-xr-x. 14 grid oinstall  4096 Aug 26  2013 stage

-rw-r--r--.  1 grid oinstall 30016 Aug 27  2013 readme.html

-rw-r--r--.  1 grid oinstall   500 Aug 27  2013 welcome.html

[grid@rac1 grid]$

[grid@rac1 grid]$ cd rpm

[grid@rac1 rpm]$ ls -tlr

total 12

-rw-r--r--. 1 grid oinstall 8288 Aug 26  2013 cvuqdisk-1.0.9-1.rpm

[grid@rac1 rpm]$ pwd

/u01/setup/grid/rpm

[grid@rac1 rpm]$ su -

Password:

Last login: Wed Jan  3 20:31:52 IST 2018 on pts/0

[root@rac1 ~]# cd /u01/setup/grid/rpm

[root@rac1 rpm]# ls -ltr

total 12

-rw-r--r--. 1 grid oinstall 8288 Aug 26  2013 cvuqdisk-1.0.9-1.rpm

[root@rac1 rpm]# rpm -ivh cvuqdisk-1.0.9-1.rpm

Preparing...                          ################################# [100%]

Using default group oinstall to install package

Updating / installing...

   1:cvuqdisk-1.0.9-1                 ################################# [100%]

[root@rac1 rpm]#  scp cvuqdisk-1.0.9-1.rpm root@rac2:/home/grid/

The authenticity of host 'rac2 (192.168.56.92)' can't be established.

ECDSA key fingerprint is 09:79:8a:4b:c4:fd:86:7b:6a:e6:1b:1f:33:c8:db:f9.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'rac2,192.168.56.92' (ECDSA) to the list of known hosts.

root@rac2's password:

cvuqdisk-1.0.9-1.rpm                                                                                                        100% 8288     8.1KB/s   00:00   

[root@rac2 u01]# cd /home/grid

[root@rac2 grid]# ls -tlr

total 12

-rw-r--r--. 1 root root 8288 Jan  3 20:37 cvuqdisk-1.0.9-1.rpm

[root@rac2 grid]# rpm -ivh cvuqdisk-1.0.9-1.rpm

Preparing...                          ################################# [100%]

Using default group oinstall to install package

Updating / installing...

   1:cvuqdisk-1.0.9-1                 ################################# [100%]

[root@rac2 grid]# hostname

rac2.localdomain

From 1st rac1 node ,go to Software location directory where you unzipped the software and by grid user, execute ./ruInstaller command.

Click Skip option.

Select "Install and Configure Oracle Grid Infrastructure for a Cluster" option.

1st option: This is used when you install and configure cluster (RAC).

2nd option: This is used when you install and configure standalone cluster without RAC.

3rd option: This is used when you upgrade existing cluster to 11g cluster.

4th option: This is just software software installation.

Select "Advanced Installation" option.

Select default language as English.

You can give cluster as anything.(Here it is rac-cluster). Scan name should be your scan-name set in /etc/hosts file and port can be anything which is opened on server. GNS (Grid naming Service) is optional feature of Oracle 11gR2 cluster. Here i am not using GNS.

Here you will see only one entry is present on screen. You have to add second entry of rac2 machine by clicking on add button. Check you /etc/hosts file for public Hostname and Virtual Hostname.

You can verifying after adding entries.

Here i have added wrong entry for virtual name of 2nd node and hence below error occurred. You need to correct the virtual host name of rac2 server.

Here you have to select Public option for Public ethernet and Private for Private ethernet and also, verify subnet mask for both public and private cards.

Select 1st option. If you are using ACFS file system then select 2nd option.

Now you will not see any raw disks here, you have to change disk location as /dev/oracleasm/disks

As per normal redundancy, minimum 2 disks are required, but during installation it gives error if you select 2 disks in Normal Redundancy. Oracle cluster requires odd number of voting disks and hence this is the reason it does not accept 2 disks. You have to select 3 disks instead of 2. 

Enter password for both SYS and ASMSNMP users and note the same.

Do not select 1st option. Select 2nd option.

Make sure oinstall is primary group for both oracle and grid users.

Select oinstall/dba group as OSASM group

Select oinstall/dba group as OSDBA group

Select asmoper/dba  group as OSOPER group.

Click YES to continue.

Select your GRID_BASE and GRID_HOME locations.

GRID_HOME can not be inside your GRID_BASE., both are different since GRID_HOME contains few files owned by root user.

Select inventory location.

During prerequisites check screen, i faced below errors/warning messages. Please find below solutions to resolve this issue:

Error1 and Error3 : Package: elfutils-libelf-devel-0.97 and Package: pdksh-5.2.14

Solution: As per Oracle Doc ID 1454982.1,  we can ignore tese and click on ignore all option and go ahead.

For NTP error:

I am stopping NTP protocol on both nodes since CTSS(Cluster Time Synchronisation Service) is used instead NTP.

[root@rac1 disks]# service ntpd stop

Redirecting to /bin/systemctl stop  ntpd.service

[root@rac1 disks]#

[root@rac2 ~]#  service ntpd stop

Redirecting to /bin/systemctl stop  ntpd.service

[root@rac2 ~]#

[root@rac1 RAC 11g Installation]# cd /etc/sysconfig/

[root@rac1 sysconfig]#

[root@rac1 sysconfig]#

[root@rac1 sysconfig]# ls -ltr ntpd

-rw-r--r--. 1 root root 45 Dec 20  2014 ntpd

[root@rac1 sysconfig]#

[root@rac1 sysconfig]#

[root@rac1 sysconfig]#

[root@rac1 sysconfig]# cat ntpd

# Command line options for ntpd

OPTIONS="-g"

[root@rac1 sysconfig]#

[root@rac1 sysconfig]#

[root@rac1 sysconfig]# vi ntpd

[root@rac1 sysconfig]#

[root@rac1 sysconfig]# cat ntpd

# Command line options for ntpd

#OPTIONS="-g"

OPTIONS="-x -g -p /var/run/ntpd.pid"

[root@rac2 sysconfig]# cat ntpd

# Command line options for ntpd

OPTIONS="-g"

[root@rac2 sysconfig]#

[root@rac2 sysconfig]#

[root@rac2 sysconfig]# vi ntpd

[root@rac2 sysconfig]#

[root@rac2 sysconfig]#

[root@rac2 sysconfig]# cat ntpd

# Command line options for ntpd

#OPTIONS="-g"

OPTIONS="-x -g -p /var/run/ntpd.pid"

[root@rac2 sysconfig]#

PRVF-5507 : NTP daemon or service is not running on any node but NTP configuration file exists on the following node(s):rac1,rac2

[root@rac1 etc]# pwd

/etc

[root@rac1 etc]# ls -ltr ntp.conf

-rw-r--r--. 1 root root 1992 Dec 20  2014 ntp.conf

[root@rac1 etc]#

[root@rac1 etc]# mv ntp.conf bkp_ntp.conf

[root@rac1 etc]#

[root@rac2 sysconfig]# cd /etc/

[root@rac2 etc]# ls -ltr ntp.conf

-rw-r--r--. 1 root root 1992 Dec 20  2014 ntp.conf

[root@rac2 etc]#

[root@rac2 etc]# mv ntp.conf bkp_ntp.conf

[root@rac2 etc]#

For libaio-devel-0.3.105-2.x86_64.rpm:

Install this rpm which i missed to install eralier.

[root@rac1 RAC 11g Installation]# rpm -ivh libaio-devel-0.3.105-2.x86_64.rpm

warning: libaio-devel-0.3.105-2.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 73307de6: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:libaio-devel-0.3.105-2           ################################# [100%]

[root@rac1 RAC 11g Installation]#

[root@rac2 RAC 11g Installation]# rpm -ivh libaio-devel-0.3.105-2.x86_64.rpm

warning: libaio-devel-0.3.105-2.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 73307de6: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:libaio-devel-0.3.105-2           ################################# [100%]

[root@rac2 RAC 11g Installation]#

For Device Checks for ASM: you can ignore this option.

For Task resolv.conf integrity, you can ignore this since we are not using DNS.

Once above installation task is done then you will get below pop-up box. You have to open new terminal windows as root user and have to execute both scripts one by one as per steps given in pop-up box. Output looks like below:

[root@rac1 etc]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@rac1 etc]#

[root@rac2 etc]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@rac2 etc]#

On RAC1=>

[root@rac1 ~]# /u01/app/11.2.0/grid/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

    ORACLE_OWNER= grid

    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

Installing Trace File Analyzer

OLR initialization - successful

  root wallet

  root wallet cert

  root cert export

  peer wallet

  profile reader wallet

  pa wallet

  peer wallet keys

  pa wallet keys

  peer cert request

  pa cert request

  peer cert

  pa cert

  peer root cert TP

  profile reader root cert TP

  pa root cert TP

  peer pa cert TP

  pa peer cert TP

  profile reader pa cert TP

  profile reader peer cert TP

  peer user cert

  pa user cert

Adding Clusterware entries to inittab

CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'

CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'

CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'

CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'

CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'rac1'

CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'

CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded

CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

clscfg: -install mode specified

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

CRS-4256: Updating the profile

Successful addition of voting disk 4fd602bf28614f16bf3012800492d6ec.

Successful addition of voting disk 63db0c1e6f964f98bfc0b722bc6c20c6.

Successful addition of voting disk 6cd2f7fdf7404f38bf8db42a48affc62.

Successfully replaced voting disk group with +DATA.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

 1. ONLINE   4fd602bf28614f16bf3012800492d6ec (/dev/oracleasm/disks/DISK1) [DATA]

 2. ONLINE   63db0c1e6f964f98bfc0b722bc6c20c6 (/dev/oracleasm/disks/DISK2) [DATA]

 3. ONLINE   6cd2f7fdf7404f38bf8db42a48affc62 (/dev/oracleasm/disks/DISK3) [DATA]

Located 3 voting disk(s).

CRS-2672: Attempting to start 'ora.asm' on 'rac1'

CRS-2676: Start of 'ora.asm' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'

CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[root@rac1 ~]#

On RAC2=>

[root@rac2 ~]# /u01/app/11.2.0/grid/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

    ORACLE_OWNER= grid

    ORACLE_HOME=  /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

Installing Trace File Analyzer

OLR initialization - successful

Adding Clusterware entries to inittab

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[root@rac2 ~]#

Once above scripts are done then click OK in pop-up box and proceed further.

You can ignore below error message since all tasks are completed successfully.

You have successfully installed and configured GRID Infrastructure. Refer below steps to install and configure RDBMS 

Step 9: Install and configure RDBMS RAC Database.  On 1st node, login as oracle user and go to database software location and execute ./runInstaller.

[oracle@rac1 oracle]$ unzip p13390677_112040_Linux-x86-64_1of7.zip
Archive:  p13390677_112040_Linux-x86-64_1of7.zip
   creating: database/
  inflating: database/readme.html   
   creating: database/sshsetup/
  inflating: database/sshsetup/sshUserSetup.sh 
   creating: database/rpm/
...........
..............
[oracle@rac1 oracle]$
[oracle@rac1 oracle]$ unzip p13390677_112040_Linux-x86-64_2of7.zip
Archive:  p13390677_112040_Linux-x86-64_2of7.zip
   creating: database/stage/Components/oracle.ctx/
   creating: database/stage/Components/oracle.ctx/11.2.0.4.0/
   creating: database/stage/Components/oracle.ctx/11.2.0.4.0/1/
.....................
..................
[oracle@rac1 oracle]$
[oracle@rac1 oracle]$ pwd
/u01/setup/oracle
[oracle@rac1 oracle]$ ls -ltr
drwxr-xr-x. 7 oracle oinstall       4096 Aug 27  2013 database

Click on "Skip software update".

Click on "Create and configure a database" option.

Select "Server Class" option.

Select "Oracle Real Application Clusters database installation" option and select 2nd node and you can test ssh connectivity as well.

Select "Advanced install" option.

Select "Enterprise Edition" option which will give you below features:

• Oracle Partitioning
• Oracle OLAP
• Oracle Label Security
• Oracle Data Mining RDBMS Files
• Oracle Database Vault Options
• Oracle Real Application Testing

Select Oracle Base location which will create Oracle Home.

Here, you have to give SID and Global database name.

Below option will configure and create memory parameters, characters, security, and sample schemas.

Select default characteristics. You can select different characteristics as per your requirement.

Select below option which will create and configure OEM console.

Select ASM (Automatic Storage Management) option as storage.

Here, you have to specify password for ASMSNMP user.

You can select 1st option to disable automated backups.

Select Diskgroup where you want to store database (Here it is DATA).

Configure password for mentioned user separately or you can keep common password for all.

Click YES to proceed.

Select operating system group.

You will get below requirement warnings. Ignore below options and proceed further.

Click YES to proceed.

You will face below error message while running installation task. Apply below workaround on both nodes to fix the issue. Click on Retry to continue.

Error:

-------

error in invoking target 'agent nmhs' of make file ins_emagent.mk

Solution:

------------

Edit $ORACLE_HOME/sysman/lib/ins_emagent.mk, search for the line $(MK_EMAGENT_NMECTL) and replace the line with $(MK_EMAGENT_NMECTL) -lnnz11

Database Creation screen will appear.

Click OK to proceed further.

On node1:

[root@rac1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

On node2:

[root@rac2 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

On node1:

[root@rac1 ~]# /u01/app/oracle/product/11.2.0/dbhome_1/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0/dbhome_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@rac1 ~]#

On node2:
[root@rac2 ~]# id
uid=0(root) gid=0(root) groups=0(root)
[root@rac2 ~]#
[root@rac2 ~]# hostname
rac2.localdomain
[root@rac2 ~]#
[root@rac2 ~]# /u01/app/oracle/product/11.2.0/dbhome_1/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.0/dbhome_1
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

Click on Skip and continue.

Click on YES.

 

Thanks for reading this post ! Please comment if you like this post !