Oracle集群搭建步骤.docx

一、版本说明发现网上相关文章很多但是不是很专，因此细化版本为了使文章更加实用。这次搭建的是Oracle 11g R2 11.2.0.4的RAC环境，使用的操作系统版本为 Oracle Linux 6.4点击（此处）折叠或打开1. rootnode1 # lsb_release -a2. LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0- noarch3. Distributor ID: OracleServer4. Description: Oracle Linux Server release 6.45. Release: 6.46. Codename: n/a内核版本：点击（此处）折叠或打开1. rootnode1 # uname -r2. 2.6.39-400.17.1.el6uek.x86_64UEK 代表 Unbreakable Enterprise Kernel， 是 Oracle 专门为 Oracle Linux 定 制的内核，可以提供在线支持和对数据库的支持，算是OracleLinux的卖点之 一。（本文中一些配置过程会因实用UEK版本而得到简化）二、Oracle 11g R2 RAC安装整体步骤1. 服务器准备(system, storage, and network administration):安装操作系统并升级到满足Oracle安装要求的版本。创建安装所需要的组、用户以及软件的家目录。 设置GNS域名如果您打算部署GNS，并且完成网络地址在DNS和服务器上的配置。设置所要求的存储。将所有安装文件拷贝到一个节点上。2. 安装 Oracle Grid 集群基础架构，包括 Oracle Clusterware 和 Oracle ASM (system and storage administration):为集群安装Oracle Grid基础架构软件。在安装过程中，Fixup脚本进行操作系统参数、 SSH和用户环境变量等参数的附加调整。 升级 Oracle Clusterware 和 Oracle ASM 到最新补丁。3. 安装 Oracle RAC (database administration): 安装 Oracle RAC 打补丁到最新版本 完成安装后的调试三、详细安装过程及说明(参考官方文档)1. 通过SecureCRT或TerminalX建立命令行连接。2. 在每一个节点上添加安装Oracle Grid的用户、组和家目录，并设置 权限。点击(此处)折叠或打开1. # /usr/sbin/groupadd -g 1000 oinstall2. # /usr/sbin/groupadd -g 1020 asmadmin3. # /usr/sbin/groupadd -g 1021 asmdba4. # /usr/sbin/groupadd -g 1022 asmoper5. # /usr/sbin/groupadd -g 1031 dba6. # /usr/sbin/groupadd -g 1032 oper7. # useradd -u 1100 -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid8. # useradd -u 1101 -g oinstall -G dba,asmdba,oper oracle9. #mkdir-p/u01/app/11.2.0/grid10. #mkdir-p/u01/app/grid11. #chown-Rgrid:oinstall/u0112. #mkdir/u01/app/oracle13. # chown oracle:oinstall /u01/app/oracle14. # chmod -R 775 /u01/参照官方文档，采用GI与DB分开安装和权限的策略，对于多实例管理有利。3.检查服务器各个节点的配置是否符合安装要求I.服务器硬盘空间要求/tmp目录大小至少：1GB安装 Grid Infrastracture 所需空间：6.6GB安装Oracle Database所需空间：4GB此外安装过程中分析、收集、跟踪文件所需空间：10GB建议总共至少30GB，放心！（此处不包含ASM或NFS的空间需求）II.服务器内存要求T曲小Swap Space Roqwretf Mr LinuxAvailable RAMSwap Space RequiredEqual Id the of RAM16 GBBetw-een 2.5 GB and 16 G0More than 16 GB内存大小：至少2.5GBSwap大小：当内存为2.5GB-16GB时，Swap需要大于等于系统内存。当内存大于16GB时，Swap等于16GB即可。III.检查和调试代码 查看内存及Swap大小的命令如下。点击（此处）折叠或打开1. # grep MemTotal /proc/meminfo2. # grep SwapTotal /proc/meminfo查看/tmp目录以及配置单独lv的命令如下。点击（此处）折叠或打开1. # df -h /tmp2. # lvcreate -L 2G -n lv_tmp vg_temp3. # mount /dev/vg_temp/lv_tmp /tmp4. # df -h /tmp4. 设置操作系统相关参数这一项在非Oracle Linux的操作系统中算是一项比较繁琐的工作，然而我们这次安装的 版本是 Oracle Linux 6.4 with Unbreakable Enterprise Kernel，因此我们有了更简单的办法，就是通过安装Oracle Preinstallation RPM来实现相关操作系统参数的调整和软件包 的安装。这个安装包主要完成以下工作： Automatically downloads and installs any additional RPM packages needed for installing Oracle Grid Infrastructure and Oracle Database, and resolves any dependencies Creates an oracle user, and creates the oraInventory (oinstaii) and OSDBA (dba) groups for that user As needed, sets syscti.conf settings, system startup parameters, and driver parameters to values based on recommendations from the Oracle Preinstallation RPM program Sets hard and soft resource limits Sets other recommended parameters, depending on your kernel version此安装包位于操作系统安装盘的Packages目录下。点击(此处)折叠或打开1. # cd /mnt/install_DVD2. # cd Packages3. # ll | grep preinstall4. -rw-r-r- 1 root root 15524 Jan 16 2013 oracle-rdbms-server-11gR2-preinstall-1.0-7. el6.x86_64.rpm5. # rpm -ivh oracle-rdbms-server-11gR2-preinstall-1.0-7.el6.x86_64.rpm安装包修改/etc/sysctl.conf的内容如下：其中标有oracle-rdbms-server-11gR2-preinstall字样注释的便是安装包所添加的参数。点击(此处)折叠或打开1. # cat /etc/sysctl.conf2. # Kernel sysctl configuration file for Red Hat Linux3. #4. # For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and5. # sysctl.conf(5) for more details.6.6. # Controls IP packet forwarding7. net.ipv4.ip_forward = 08. -9. # Controls source route verification10. net.ipv4.conf.default.rp_filter = 112.11. # Do not accept source routing12. net.ipv4.conf.default.accept_source_route = 015.13. # Controls the System Request debugging functionality of the kernel14. kernel.sysrq = 018.15. # Controls whether core dumps will append the PID to the core filename.16. # Useful for debugging multi-threaded applications.17. kernel.core_uses_pid = 122.18. # Controls the use of TCP syncookies19. net.ipv4.tcp_syncookies = 125.20. # Disable netfilter on bridges.21. net.bridge.bridge-nf-call-ip6tables = 022. net.bridge.bridge-nf-call-iptables = 023. net.bridge.bridge-nf-call-arptables = 030.24. # Controls the default maxmimum size of a mesage queue25. kernel.msgmnb = 6553633.26. # Controls the maximum size of a message, in bytes27. kernel.msgmax = 6553636.28. # Controls the maximum shared segment size, in bytes38.29. # Controls the maximum number of shared memory segments, in pages40.30. # oracle-rdbms-server-11gR2-preinstall setting for fs.file-max is 681574431. fs.file-max = 681574443.32. # oracle-rdbms-server-11gR2-preinstall setting for kernel.sem is '250 32000 100 128'33. kernel.sem = 250 32000 100 12846.34. # oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmni is 409635. kernel.shmmni = 409649.36. # oracle-rdbms-server-11gR2-preinstall setting for kernel.shmall is 1073741824 on x86_6437. # oracle-rdbms-server-11gR2-preinstall setting for kernel.shmall is 2097152 on i38638. kernel.shmall = 107374182453.39. # oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmax is 4398046511104 on x86_6440. # oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmax is 4294967295 on i38641. kernel.shmmax = 439804651110457.42. # oracle-rdbms-server-11gR2-preinstall setting for net.core.rmem_default is 26214443. net.core.rmem_default = 26214460.44. # oracle-rdbms-server-11gR2-preinstall setting for net.core.rmem_max is 419430445. net.core.rmem_max = 419430463.46. # oracle-rdbms-server-11gR2-preinstall setting for net.core.wmem_default is 26214447. net.core.wmem_default = 26214466.48. # oracle-rdbms-server-11gR2-preinstall setting for net.core.wmem_max is 104857649. net.core.wmem_max = 104857669.50. # oracle-rdbms-server-11gR2-preinstall setting for fs.aio-max-nr is 104857651. fs.aio-max-nr = 104857672.52. # oracle-rdbms-server-11gR2-preinstall setting for net.ipv4.ip_local_port_range is 9000 6550053. net.ipv4.ip_local_port_range = 9000 655005.配置Oracle网络I.硬件配置要求每个服务器节点至少需要2块网卡，一块对外网络接口，一块私有网络接口（心跳）。 如果你通过OUI安装Oracle集群软件，需要保证每个节点用于外网或私网接 口（网卡名）保证一致。比如，nodel使用eth0作为对外接口，node2就不能 使用ethl作为对外接口。II.IP配置要求存在为集群服务的DHCP DHCP能为每个节点提供足够的IP，一个虚拟IP、3个SCAN IPIII.非 GNS下手动配置IP实例Table 2-4 Manual Network Configuration Example| Identity-Home NodeHost NodeGiven NameTypeAddressAddress Assigned ByResoKByNode 1PublicNode 1ncidelnodel1Public192.0.2.101ixed3NSNode 1VIPNode 1Seleclcd by Oracle Clustenuarenodsl-vdpVirtual192.0.2.104Fixed3NS an"iQSli TiNode 1PrivateNode 1nodelnodel-privPrivate192J68 0 1Fixeddns an hosts fil or noneNode 2 =WlicNode 2姑如豚硕Public192.0.2.102FixedNSNode 2VIPNode 2seiecied 叫 oracle ClusterrtrarenadeS-vipvirtual1192 0 2/105Fixeddns an hosts filNode 2PrivateNode 2node 2node2-prlvPrivate192.168.0.2n>ed3NS an nosts. fi r noneSCANVIP 1noneseieciea 叫 oracle Clusterrtraremyciu sier-scanfinial192 0 2.201FinedMSSCANVIP 2noneSeleclcd t>y OracleClusterrtraremyclust&r-scanvirtual192.0.2.202Fixed3NSSCANVIP 3noneSelect配 by OracleClusterwaremycluEter-Ecanvirtual192.0.2.203FixedDNS6.检查操作系统软件包首先，根据操作系统版本到官方文档中找到安装包需求列表，如下图Table 2-9 Linux xSS-64 Oracle Grid infrastructure and Oracle RAC Package RequirementsItmRequirementsOracle Linux 6 (2 .3-100.2B.5.el6.x86_Ei4 or lEiter) Subscribe Io lhe Oracle Linux 6 channel on the Unbreakable Ln lx Network, eji then install the oracle Preinstall ail on rpm ms instai s ine oracle un&reakan Enterprise Kernel for Linux, and all required kernel packages for Oracle Grid mirastructure and Oracle Database installationsI Oracle Linux 6 and Red Hat Eilerprise Linux 6The following pacKages (or laler versions) must be in 吕由 lied.Li:i_iL.il5-2.2O.51.O.2-C . .tie：c omp az - libcapl-l. 1C -1 仪驻 _E) nouKpat-l' bstdc+->. 2 r >- f-u - p i( DDinipac-lib5tdc+-35-3-2.3-5 -elE .H6S-6 sc-4a 4a 4-13.516 (xS_4) QTCC-C+-F-4.马 4-13 - S16 US-6_-=> g 1 i.bc-2 12-1.7 eie li.bc-2 a 12-1.7 e!6)QTLLbi3-d.5V51-2 12-1 7 已 16 (XSfi_-6) glibc-dcvcl-2.12-1.7. =16 . rah1 LDorcc-9 . q .马一3. el (16S6J LibgcD-4.4. 4-13.el-6 libsEdc+-k-4.4.4-13 .el£ (kS£_jS4 LLDSGlC-Hb-'i . 4 . 4-13 . £16.1-6S«1 ib a u dc+ + -de vel-4 .-9-13.eie (盎日日日 4 ) libBE：dc+-k-davel-4. .4-13.el .16SS1 LDai O- . 3.107-10 已 1$ (its 电页壬) 1ibai-D.3.107-10 el6-±65后1iba1o-deve1-0 3-107-10 e!6 (h8_64) 1 Lbaio-d.evel-0 3-107-10 .elfimake-3.Bl-19.el63 73scat9-0 4-11-el6可以通过mount安装盘或直接通过yum源检查和安装所需要的包：点击（此处）折叠或打开1. # rpm -qa packages_name2. # rpm -ivh packages_name3. -4. # yum list packages_name5. # yum install packages_name这里还要提一下：CVUdisk这个包也可以在这个时候一起安装好，以备之后使 用CVU时会用到。点击（此处）折叠或打开1. #检查是否巳经安装2. # rpm -qi cvuqdisk3. #如果巳安装需要先卸载之前的版本4. # rpm -e cvuqdisk5. #安装新版本6. # CVUQDISK_GRP=oinstall; export CVUQDISK_GRP7. # rpm -ivh cvuqdisk-1.0.9-1.rpm7.关闭NTP及端口范围参数修改点击（此处）折叠或打开1. # Oracle 建议使用 Oracle Cluster Time Synchronization Service，因此关闭删除 NTP2. # /sbin/service ntpd stop3. # chkconfig ntpd off4. # rm /etc/ntp.conf （mv /etc/ntp.conf /etc/ntp.conf.old）5. # rm /var/run/ntpd.pid6.6. 检查TCP/UDP端口范围7. # cat /proc/sys/net/ipv4/ip_local_port_range8. 如果巳经显示9000 65500,就不用进行下面的步骤了9. # echo 9000 65500 > /proc/sys/net/ipv4/ip_local_port_range10. # vim /etc/sysctl.conf11. #添加此行：12. # TCP/UDP port range13. net.ipv4.ip_local_port_range = 9000 6550014. #重启网络15. # /etc/rc.d/init.d/network restart8调整.bash_profile内容并设置显示地址点击（此处）折叠或打开1. 设置参数：2. $ su - root3. # mkdir /mount_point/tmp4. # chmod 775 /mount_point/tmp5. # exit6.6. $ vi .bash_profile7. 加入行：8. TEMP=/mount_point/tmp9. TMPDIR=/mount_point/tmp10. export TEMP TMPDIR11. 加入行：umask 02212. 加入行：DISPLAY=local_IP:0.0 ; export DISPLAY13. local_IP为你要将安装视图所显示的桌面地址15.14. 检查设置参数15. $ umask16. $ env | more17. $ echo $DISPLAY18. $ echo $TEMP19. $ echo $TMPDIR9配置SSH互信这是很关键的一步，虽然官方文档中声称安装GI和RAC的时候OUI会自动配 置SSH，但为了在安装之前使用CVU检查各项配置，还是手动配置互信更优。点击（此处）折叠或打开1. 配置过程如下：2. 各节点生成Keys:3. rootrac1 # su - oracle4. oraclerac1 $ mkdir /.ssh5. oraclerac1 $ chmod 700 /.ssh6. oraclerac1 $ ssh-keygen -t rsa7. oraclerac1 $ ssh-keygen -t dsa8. rootrac2 # su - oracle9. oraclerac2 $ mkdir /.ssh10. oraclerac2 $ chmod 700 /.ssh11. oraclerac2 $ ssh-keygen -t rsa12. oraclerac2 $ ssh-keygen -t dsa13.13. 在节点1上进行互信配置：14. oraclerac1 $ touch /.ssh/authorized_keys15. oraclerac1 $ cd /.ssh16. oraclerac1.ssh$sshrac1cat/.ssh/id_rsa.pub>>authorized_keys17. oraclerac1.ssh$sshrac2cat/.ssh/id_rsa.pub>>authorized_keys18. oraclerac1.ssh$sshrac1cat/.ssh/id_dsa.pub>>authorized_keys19. oraclerac1.ssh$sshrac2cat/.ssh/id_dsa.pub>>authorized_keys21.20. 在rac1把存储公钥信息的验证文件传送到rac2上21. oraclerac1 .ssh$ pwd22. /home/oracle/.ssh23. oraclerac1 .ssh$ scp authorized_keys rac2:'pwd'24. oraclerac2's password:25. authorized_keys 100% 1644 1.6KB/s 00:0028.26. 设置验证文件的权限27. 在每一个节点执行：28. $ chmod 600 /.ssh/authorized_keys32.29. 启用用户一致性30. 在你要运行OUI的节点以oracle用户运行（这里选择rac1）:31. oraclerac1 .ssh$ exec /usr/bin/ssh-agent $SHELL32. oraclerac1 .ssh$ ssh-add33. Identity added: /home/oracle/.ssh/id_rsa （/home/oracle/.ssh/id_rsa）34. Identity added: /home/oracle/.ssh/id_dsa （/home/oracle/.ssh/id_dsa）39.35. 验证ssh配置是否正确36. 以oracle用户在所有节点分别执行：37. ssh rac1 date38. ssh rac2 date39. ssh rac1-priv date40. ssh rac2-priv date46.41. 如果不需要输入密码就可以输出时间，说明ssh验证配置成功。必须把以上命令在两个节点都运行，每 一个命令在第一次执行的时候需要输入yes。42. 如果不运行这些命令，即使ssh验证巳经配好，安装clusterware的时候也会出现错误：43. The specified nodes are not clusterable44. 因为，配好ssh后，还需要在第一次访问时输入yes，才算是真正的无障碍访问其他服务器。请谨记，SSH互信需要实现的就是各个节点之间可以无密码进行SSH访问。10.为Grid集群基础软件配置存储（使用ASM）Oracle所允许的存储主要有ASM、NFS和ACFS等，这里仅介绍ASM的配置过程。（1）安装ASM配置软件点击（此处）折叠或打开1. 依然是可以通过yum或者系统DVD光盘安装2. #cd /mnt/install_DVD/Packages3. #rpm-qioracleasm4. #rpm-qioracleasm-support5. #rpm-qioracleasm-support-2.1.8-1.el6.x86_64.rpm6. /我们是Oracle Linux系统，此版本巳经集成了 oracleasm和oracleasmlib，所以就不用再安装 了（2）规划ASM磁盘组官方文档中规定了不同冗余策略下OCR、Voting Disk、Database和Recovery 所需求的大小。e加亭 3-5副 oracle Clustenvare storage spaM Requtreti by Recfufttfancy TypeRedundancy LevelMinimum Number of disksoracle Cluster Registry (QCR| Filesvoting disk FilesBoth FileExternal1300 MB300 ME600 hNormal3&0C MB90DMB1.5GBfDHigh590D M0.1 5 GB?'.4GTable 3-6 Total Oracle Database Storage Space RetfUlred by Redundancy TypeR&duindanicyLevelMmimum Number of DisksDatabase FilesRecovery FilwsBnFili找到存储管理员，把规划好的磁盘都让他给你配置好（3）配置ASM点击(此处)折叠或打开1. 格式化每个磁盘2. rootrac1 # fdisk /dev/sdb3.3. 完成后我们通过fdisk -l查看一下4. rootrac1 # fdisk -l6.5. rootrac1 # /usr/sbin/oracleasm configure -i6. Default user to own the driver interface : grid7. Default group to own the driver interface : asmadmin8. Start Oracle ASM library driver on boot (y/n) n: y9. Scan for Oracle ASM disks on boot (y/n) y: y12.10. rootrac1 # /etc/init.d/oracleasm createdisk VOL1 /dev/sdb111. rootrac1 # /etc/init.d/oracleasm createdisk VOL2 /dev/sdc112. rootrac1 # /etc/init.d/oracleasm createdisk VOL3 /dev/sddl16.13. 以下操作在另一个节点上进行，RAC2上操作14. rootrac2 # oracleasm scandisks15. 这里用到的是asmlib来做磁盘设备管理的，这里大家也可以使用UDEV来管理，在RHEL6中asmlib 巳经被废弃了，以后就是UDEV的天下11. 使用CVU检查各个节点的配置工作千辛万苦终于到了安装前的最后一步点击（此处）折叠或打开1. rootnode1 # su - grid2. gridnode1 $ cd grid_sw3. gridnode1 grid_sw$ ./runcluvfy.sh stage -pre crsinst -n node1,node2 -fixup - verbose4. 其中“grid_sw”为安装包解压出来的grid安装文件。这一步的重点在于遇到报错怎样处理：我这次安装遇到了 PRVF-5636，此问题是由于没有DNS以及/etc/resolv.conf的设置引起 的，因为脚本要用nslookup命令去检测两个节点的连通性。12. 安装GI和RAC（1）安装 Oracle Grid InfrastructureDATABASEDuwnluaa Software Updates虹 Div Soft侦” UDC&te$Installaiion OptionInstallaiionTypeCluster ConfiQurationInstall LocationsPrerequisite ChecksSummaryInstall Product2 FinishCancelEBOracle Grii Iu£rastrbctwre - Setting up Grid lafrastructxire - Step 110Download Software UpdatesDownload softvare updates for this installation. Software updaier consisi of recommended u