xCAT/xcat-core
2
0
Fork 0
mirror of https://github.com/xcat2/xcat-core.git synced 2025-05-29 09:13:08 +00:00
Code Issues Releases Wiki Activity

Merge pull request #262 from hu-weihua/doc

Browse Source 
Add IB docs under advanced/network/infiniband directory
This commit is contained in:
Xiaopeng Wang 2015-10-20 11:27:10 +08:00
commit 80cb045a97
14 changed files with 717 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
docs/source/advanced/networks/infiniband/driver_and_installation.rst
View File

@ -1,2 +1,11 @@
IB Driver Preparation and Installation
======================================
xCAT provides sample postscripts to help you install the Mellanox OpenFabrics Enterprise Distribution (OFED) Infiniband Driver. These scripts are located in ``opt/xcat/share/xcat/ib/scripts/Mellanox/``. You can use these scripts directly or change them to satisfy your own environment. **xCAT 2.11 drops support of mlnxofed_ib_install and recommends using version 2 of the script: mlnxofed_ib_install.v2**.
.. toctree::
:maxdepth: 2
mlnxofed_ib_install_v2_usage.rst
mlnxofed_ib_install_v1_usage.rst

96
docs/source/advanced/networks/infiniband/firmware_updates.rst
View File

@ -1,2 +1,98 @@
Firmware Updates
================
Adapter Firmware Update
-----------------------
Please download the OFED IB adapter firmware from the Mellanox site `http://www.mellanox.com/page/firmware_table_IBM <http://www.mellanox.com/page/firmware_table_IBM>`_ .
Obtain device id: ::
lspci | grep -i mel
Check current installed fw level: ::
mstflint -d 0002:01:00.0 q | grep FW
Copy or mount firmware to host:
Burn new firmware on each ibaX: ::
mstflint -d 0002:01:00.0 -i <image location> b
Note: if this is a PureFlex MezzanineP adapater then you must select the correct image for each ibaX device. Note the difference in the firmware image at end of filename: _0.bin (iba0/iba2) & _1.bin (iba1/iba3)
Verify download successful: ::
mstflint -d 0002:01:00.0 q
Activate the new firmware: ::
reboot the image
Note: the above 0002:01:00.0 device location was used as an example only. it is not meant to imply that there is only one device location or that your device will have the same device location.
Mellanox Switch Firmware Upgrade
--------------------------------
This section provides manual procedure to help update the firmware for Mellanox Infiniband (IB) Switches. You can down load IB switch firmware like IB6131 (image-PPC_M460EX-SX_3.2.xxx.img) from the Mellanox website `http://www.mellanox.com/page/firmware_table_IBM <http://www.mellanox.com/page/firmware_table_IBM>`_ and place into your xCAT Management Node or server that can communicate to Flex IB6131 switch module. There are two ways to update the MLNX-OS switch package. This process works regardless if updating an internal PureFlex chassis Infiniband switch (IB6131 or for an external Mellanox switch.
Update via Browser
^^^^^^^^^^^^^^^^^^
This method is straight forward if your switches are on the public network or your browser is already capable to tunnel to the private address. If neither is the case then you may prefer to use option two.
After logging into the switch (id=admin, pwd=admin)
Select the "System" tab and then the "MLNX-OS Upgrade" option
Under the "Install New Image", select the "Install via scp"
URL: scp://userid@fwhost/directoryofimage/imagename
Select "Install Image"
The image will then be downloaded to the switch and the installation process will begin.
Once completed, the switch must be rebooted for the new package to be activate
Update via CLI
^^^^^^^^^^^^^^
Login to the IB switch: ::
ssh admin@<switchipaddr>
enable (get into correct CLI mode. You can use en)
configure terminal (get into correct CLI mode. You can use co t)
List current images and Remove older images to free up space: ::
show image
image delete <ibimage>
(you can paste in ibimage name from show image for image delete)
Get the new IB image using fetch with scp to a server that contains new IB image. An example of IB3161 image would be "image-PPC_M460EX-SX_3.2.0291.img" Admin can use different protocol . This image fetch scp command is about 4 minutes. ::
image fetch ?
image fetch scp://userid:password@serveripddr/<full path ibimage location>
Verify that new IB image is loaded, then install the new showIB image on IB switch. The install image process goes through 4 stages Verify image, Uncompress image, Create Filesystems, and Extract Image. This install process takes about 9 minutes. ::
show image
image install <newibimage>
(you can paste in new IB image from "show image" to execute image install)
Toggle boot partition to new IB image, verify image install is loaded , and that next boot setting is pointing to new IB image. ::
image boot next
show image
Save the changes made for new IB image: ::
configuration write
Activate the new IB image (reboot switch): ::
reload

2
docs/source/advanced/networks/infiniband/index.rst
View File

@ -1,6 +1,8 @@
Infiniband (Mellanox)
=====================
xCAT offers a certain degree support for Mellanox infiniband product, it help you to configurate Mellanox infiniband products easily.
.. toctree::
:maxdepth: 2

53
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskful.rst Normal file
View File

@ -0,0 +1,53 @@
Configuration for Diskful Installation
======================================
1. Set script ``mlnxofed_ib_install`` as postbootscript ::
chdef <node> -p postbootscripts=mlnxofed_ib_install
2. Specify dependence package **[required for RHEL and SLES]**
a) Copy the pkglist to the custom directory ::
cp /opt/xcat/share/xcat/install/<ostype>/compute.<osver>.<arch>.pkglist \
/install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist
b) Edit your /install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist and add one line::
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<osver>.<arch>.pkglist#
c) Make the related osimage use the customized pkglist ::
chdef -t osimage -o <osver>-<arch>-install-compute \
pkglist=/install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist
Take RHEL 6.4 on x86_64 for example ::
cp /opt/xcat/share/xcat/install/rh/compute.rhels6.x86_64.pkglist \
/install/custom/install/rh/compute.rhels6.x86_64.pkglist
Edit the ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` and add below line
``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#``
Then ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` looks like below ::
#Please make sure there is a space between @ and group name
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#
ntp
nfs-utils
net-snmp
rsync
yp-tools
openssh-server
util-linux-ng
Then modify related osimage ::
chdef -t osimage -o rhels6.4-x86_64-install-compute \
pkglist=/install/custom/install/rh/compute.rhels6.x86_64.pkglist
3. Install node ::
nodeset <node> osimage=<osver>-<arch>-install-compute
rsetboot <node> net
rpower <node> reset

77
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskless.rst Normal file
View File

@ -0,0 +1,77 @@
Configuration for Diskless Installation
=======================================
1. Specify dependence package **[required for RHEL and SLES]**
a) Copy a correct pkglist file **shipped by xCAT** according your environment to the ``/install/custom/netboot/<ostype>/`` directory ::
cp /opt/xcat/share/xcat/netboot/<ostype>/compute.<osver>.<arch>.pkglist \
/install/custom/netboot/<ostype>/compute.<osver>.<arch>.pkglist
b) Edit your ``/install/custom/netboot/<ostype>/<profile>.pkglist`` and add one line ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<osver>.<arch>.pkglist#``
Take RHEL 6.4 on x86_64 for example ::
cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.pkglist \
/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist
Edit the ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` and add below line
``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#``
Then ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` looks like below ::
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#
bash
nfs-utils
openssl
dhclient
.....
2. Prepare postinstall scripts
a) Specify postinstall script **shipped by xCAT** ::
mkdir -p /install/custom/netboot/<ostype>/
cp /opt/xcat/share/xcat/netboot/<ostype>/<profile>.postinstall \
/install/custom/netboot/<ostype>/
chmod +x /install/custom/netboot/<ostype>/<profile>.postinstall
Take RHEL 6.4 on x86_64 for example ::
mkdir -p /install/custom/netboot/rh/
cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.postinstall \
/install/custom/netboot/rh/
chmod +x /install/custom/netboot/rh/compute.rhels6.x86_64.postinstall
b) Edit ``/install/custom/netboot/<ostype>/<profile>.postinstall`` and add below line in the end: ::
installroot=$1 ofeddir=/install/post/otherpkgs/<osver>/<arch>/ofed/ \
NODESETSTATE=genimage mlnxofed_options=--force /install/postscripts/mlnxofed_ib_install
3. Set the related osimage use the customized pkglist and customized compute.postinsall
* [RHEL/SLES] ::
chdef -t osimage -o <osver>-<arch>-netboot-compute \
pkglist=/install/custom/netboot/<ostype>/compute.<osver>.<arch>.pkglist \
postinstall=/install/custom/netboot/<ostype>/<profile>.postinstall
* [Ubuntu] ::
chdef -t osimage -o <osver>-<arch>-netboot-compute \
postinstall=/install/custom/netboot/<ostype>/<profile>.postinstall
4. Generate and package image for diskless installation ::
genimage <osver>-<arch>-netboot-compute
packimage <osver>-<arch>-netboot-compute
5. Install node ::
nodeset <nodename> osimage=<osver>-<arch>-netboot-compute
rsetboot <nodename> net
rpower <nodename> reset

42
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_preparation.rst Normal file
View File

@ -0,0 +1,42 @@
Preparation
===========
Obtain the Mellanox OFED ISO file from `Mellanox official site <http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>`_ and mount it onto suggested target location on the xCAT MN according your OS and ARCH: ::
mkdir -p /install/post/otherpkgs/<osver>/<arch>/ofed
mount -o loop MLNX_OFED_LINUX-<packver1>-<packver2>-<osver>-<arch>.iso \
/install/post/otherpkgs/<osver>/<arch>/ofed
Take sles11 sp1 for x86_64 as an example ::
mkdir -p /install/post/otherpkgs/sles11.1/x86_64/ofed/
mount -o loop MLNX_OFED_LINUX-1.5.3-3.0.0-sles11sp1-x86_64.iso \
/install/post/otherpkgs/sles11.1/x86_64/ofed/
Take Ubuntu14.4.1 for Power8 LE as an example ::
mkdir -p /install/post/otherpkgs/ubuntu14.04.1/ppc64el/ofed
mount -o loop MLNX_OFED_LINUX-2.3-1.0.1-ubuntu14.04-ppc64le.iso \
/install/post/otherpkgs/ubuntu14.04.1/ppc64el/ofed
**[NOTE]**
* Mellanox provides OFED files with **tarball** and **ISO** two format, but for xCAT, we just support **ISO** format right now.
Copy Sample script **mlnxofed_ib_install** shipped by xCAT into ``/install/postscripts`` before using, such as ::
cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install \
/install/postscripts/mlnxofed_ib_install
The **mlnxofed_ib_install** invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED ISO. If you want to pass the argument to ``mlnxofedinstall``, you set the argument to the environment variable ``mlnxofed_options`` which could be read by **mlnxofed_ib_install**. For example: PPE requires the 32-bit version of libibverbs, but the default **mlnxofed_ib_install** will remove all the old ib related packages at first including the 32-bit version of libibverbs. In this case, you can set the environment variable ``mlnxofed_options=--force`` when running the **mlnxofed_ib_install**. For diskful, you should put the environment variable ``mlnxofed_options=--force`` in mypostscript.tmpl. myposcript.tmpl is in ``/opt/xcat/share/xcat/templates/mypostscript/`` by default. When customize it, you should copy it into ``/install/postscripts/myposcript.tmpl`` ::
mlnxofed_options='--force'
export mlnxofed_options

9
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_usage.rst Normal file
View File

@ -0,0 +1,9 @@
Using mlnxofed_ib_install (drops support)
=========================================
.. toctree::
:maxdepth: 1
mlnxofed_ib_install_v1_preparation.rst
mlnxofed_ib_install_v1_diskful.rst
mlnxofed_ib_install_v1_diskless.rst

56
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskful.rst Normal file
View File

@ -0,0 +1,56 @@
Configuration for Diskful Installation
=======================================
1. Set script ``mlnxofed_ib_install`` as postbootscript ::
chdef <node> -p postbootscripts="mlnxofed_ib_install -p /install/<path>/<MLNX_OFED_LINUX.iso>"
2. Specify dependence package **[required for RHEL and SLES]**
a) Copy a correct pkglist file **shipped by xCAT** according your environment to the ``/install/custom/install/<ostype>/`` directory, these pkglist files are located under ``/opt/xcat/share/xcat/install/<ostype>/`` ::
cp /opt/xcat/share/xcat/install/<ostype>/compute.<osver>.<arch>.pkglist \
/install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist
b) Edit your ``/install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist`` and add one line
``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<osver>.<arch>.pkglist#``
You can check directory ``/opt/xcat/share/xcat/ib/netboot/<ostype>/`` and choose one correct ``ib.<osver>.<arch>.pkglist`` according your environment.
c) Make the related osimage use the customized pkglist ::
chdef -t osimage -o <osver>-<arch>-install-compute \
pkglist=/install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist
Take RHEL 6.4 on x86_64 for example ::
cp /opt/xcat/share/xcat/install/rh/compute.rhels6.x86_64.pkglist \
/install/custom/install/rh/compute.rhels6.x86_64.pkglist
Edit the ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` and add below line
``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#``
Then ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` looks like below ::
#Please make sure there is a space between @ and group name
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#
ntp
nfs-utils
net-snmp
rsync
yp-tools
openssh-server
util-linux-ng
Then modify related osimage ::
chdef -t osimage -o rhels6.4-x86_64-install-compute \
pkglist=/install/custom/install/rh/compute.rhels6.x86_64.pkglist
3. Install node ::
nodeset <node> osimage=<osver>-<arch>-install-compute
rsetboot <node> net
rpower <node> reset

77
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskless.rst Normal file
View File

@ -0,0 +1,77 @@
Configuration for Diskless Installation
=======================================
1. Specify dependence package **[required for RHEL and SLES]**
a) Copy a correct pkglist file **shipped by xCAT** according your environment to the ``/install/custom/netboot/<ostype>/`` directory ::
cp /opt/xcat/share/xcat/netboot/<ostype>/compute.<osver>.<arch>.pkglist \
/install/custom/netboot/<ostype>/compute.<osver>.<arch>.pkglist
b) Edit your ``/install/custom/netboot/<ostype>/<profile>.pkglist`` and add below line
``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<osver>.<arch>.pkglist#``
Take RHEL 6.4 on x86_64 for example ::
cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.pkglist \
/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist
Edit the ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` and add below line
``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#``
Then ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` looks like below ::
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#
bash
nfs-utils
openssl
dhclient
.....
2. Prepare postinstall scripts
a) Specify a correct postinstall script **shipped by xCAT** ::
mkdir -p /install/custom/netboot/<ostype>/
cp /opt/xcat/share/xcat/netboot/<ostype>/<profile>.postinstall \
/install/custom/netboot/<ostype>/
chmod +x /install/custom/netboot/<ostype>/<profile>.postinstall
Take RHEL 6.4 on x86_64 for example ::
mkdir -p /install/custom/netboot/rh/
cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.postinstall \
/install/custom/netboot/rh/
chmod +x /install/custom/netboot/rh/compute.rhels6.x86_64.postinstall
b) Edit ``/install/custom/netboot/<ostype>/<profile>.postinstall`` and add below line in the end ::
/install/postscripts/mlnxofed_ib_install \
-p /install/<path>/<MLNX_OFED_LINUX.iso> -i $1 -n genimage
3. Set the related osimage using the customized pkglist and compute.postinsall
* [RHEL/SLES] ::
chdef -t osimage -o <osver>-<arch>-netboot-compute \
pkglist=/install/custom/netboot/<ostype>/compute.<osver>.<arch>.pkglist \
postinstall=/install/custom/netboot/<ostype>/<profile>.postinstall
* [Ubuntu] ::
chdef -t osimage -o <osver>-<arch>-netboot-compute \
postinstall=/install/custom/netboot/<ostype>/<profile>.postinstall
4. Generate and package image for diskless installation ::
genimage <osver>-<arch>-netboot-compute
packimage <osver>-<arch>-netboot-compute
5. Install node ::
nodeset <nodename> osimage=<osver>-<arch>-netboot-compute
rsetboot <nodename> net
rpower <nodename> reset

34
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.rst Normal file
View File

@ -0,0 +1,34 @@
Preparation
===========
Obtain the Mellanox OFED ISO file from `Mellanox official site <http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>`_ and put it into one place under ``/install`` directory depending on your need.
**[NOTE]**
* Mellanox provides OFED drivers in **tarball** and **iso** formats. xCAT only supports the **iso** format at this time.
* Mellanox provides different OFED ISOs depending on operating system and machine architecture, named like MLNX_OFED_LINUX-<packver1>-<packver2>-<osver>-<arch>.iso, you should download correct one according your environment.
Copy **mlnxofed_ib_install.v2** into ``/install/postscripts`` and change name to **mlnxofed_ib_install** ::
cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install.v2 \
/install/postscripts/mlnxofed_ib_install
chmod +x /install/postscripts/mlnxofed_ib_install
``mlnxofed_ib_install`` has some options, **'-p' is always needed**.
Below are the details of these options:
* **-p**: [required]--the directory where the OFED iso file is located
* **-m**: [optional]--the mlnxofed_ib_install invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED iso. Use this option to pass arguments to the ``mlnxofedinstall``. You must include ``-end-`` at the completion of the options to distinguish the option list. if you don't pass any argument to ``mlnxofedinstall``, defualt value ``--without-32bit --without-fw-update --force`` will be passed to ``mlnxofedinstall`` by xCAT.
* **-i**: [required for diskless]--the image root path
* **-n**: [required for diskless]--nodeset status, the value is 'genimage'.
In general you can use ``mlnxofed_ib_install`` like below ::
mlnxofed_ib_install -p /install/<path>/<MLNX_OFED_LINUX.iso>
If need to pass ``--without-32bit --without-fw-update --add-kernel-support --force`` to ``mlnxofedinstall`` in diskless scenario, refer to below command ::
mlnxofed_ib_install -p /install/<path>/<MLNX_OFED_LINUX.iso> \
-m --without-32bit --without-fw-update --add-kernel-support --force -end- \
-i /<imagepath> -n genimage

10
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_usage.rst Normal file
View File

@ -0,0 +1,10 @@
Using mlnxofed_ib_install.v2 (Recommend)
=========================================
.. toctree::
:maxdepth: 1
mlnxofed_ib_install_v2_preparation.rst
mlnxofed_ib_install_v2_diskful.rst
mlnxofed_ib_install_v2_diskless.rst

34
docs/source/advanced/networks/infiniband/network_configuration.rst
View File

@ -1,2 +1,36 @@
IB Network Configuration
========================
xCAT provides a script ``configib`` to help configure the Infiniband adapters on the compute nodes.
The Infiniband adapter is considered an additional interface for xCAT. The process for configuring Infiniband adapters complies with the process of :doc:`Configure Additional Network Interfaces <../../../../guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/cfg_second_adapter>`.
Below are an simple example to configure Mellanox IB in Ubuntu 14.04.1 on Power8 LE
If your target Mellanox IB adapter has 2 ports, and you plan to give port ib0 4 different IPs, 2 are IPV4 (20.0.0.3 and 30.0.0.3) and another 2 are IPV6 (1:2::3 and 2:2::3).
1. Define your networks in networks table ::
chdef -t network -o ib0ipv41 net=20.0.0.0 mask=255.255.255.0 mgtifname=ib0
chdef -t network -o ib0ipv42 net=30.0.0.0 mask=255.255.255.0 mgtifname=ib0
chdef -t network -o ib0ipv61 net=1:2::/64 mask=/64 mgtifname=ib0 gateway=1:2::2
chdef -t network -o ib0ipv62 net=2:2::/64 mask=/64 mgtifname=ib0 gateway=
2. Define IPs for ib0 ::
chdef <node> nicips.ib0="20.0.0.3|30.0.0.3|1:2::3|2:2::3" \
nicnetworks.ib0="ib0ipv41|ib0ipv42|ib0ipv61|ib0ipv62" nictypes.ib0="Infiniband"
3. Configure ib0
* To configure during node installation ::
chdef <node> -p postscripts="confignics --ibaports=2"
nodeset <node> osimage=<osimagename>
rsetboot <node> net
rpower <node> reset
* To configure on a node which has had operating system ::
updatenode <node> -P "confignics --ibaports=2"

142
docs/source/advanced/networks/infiniband/switch_configuration.rst
View File

@ -1,2 +1,144 @@
IB Switch Configuration
=======================
Setup the xCAT Database
-----------------------
The Mellanox Switch is only supported in xCAT Release 2.7 or later.
Add the switch ip address in the ``/etc/hosts`` file
Define IB switch as a node: ::
chdef -t node -o mswitch groups=all nodetype=switch mgt=switch
Add the login user name and password to the switches table: ::
tabch switch=mswitch switches.sshusername=admin switches.sshpassword=admin switches.switchtype=MellanoxIB
The switches table will look like this: ::
#switch,...,sshusername,sshpassword,switchtype,....
"mswitch",,,,,,,"admin","admin","MellanoxIB",,
If there is only one admin and one password for all the switches then put the entry in the xCAT passwd table for the admin id and password to use to login. ::
tabch key=mswitch passwd.username=admin passwd.password=admin
The passwd table will look like this: ::
#key,username,password,cryptmethod,comments,disable
"mswitch","admin","admin",,,
Setup ssh connection to the Mellanox Switch
-------------------------------------------
To run commands like xdsh and script to the Mellanox Switch, we need to setup ssh to run without prompting for a password to the Mellanox Switch. To do this, first you must add a configuration file. This configuration file is NOT needed for xCAT 2.8 and later. ::
mkdir -p /var/opt/xcat/IBSwitch/Mellanox
cd /var/opt/xcat/IBSwitch/Mellanox
cp /opt/xcat/share/xcat/devicetype/IBSwitch/Mellanox/config .
The file contains the following: ::
[main]
[xdsh]
pre-command=cli
post-command=NULL
Then run the following: ::
rspconfig mswitch sshcfg=enable
**[Note]** For Mellanox switch in manufacturing defaults status, the user need to answer 'no' for the initial configuration wizard prompt as follows before run 'rspconfig'. ::
[s1mn][/](/)> ssh -l admin mswitch
Mellanox MLNX-OS Switch Management
Password:
Last login: Wed Feb 20 20:09:50 2013 from 1.2.3.4
Mellanox Switch
Mellanox configuration wizard
Do you want to use the wizard for initial configuration? **no**
To return to the wizard from the CLI, enter the "configuration jump-start"
command from configure mode. Launching CLI...
switch-xxxxxx [standalone: unknown] > exit
Setup syslog on the Switch
--------------------------
Use the following command to consolidate the syslog to the Management Node or Service Nodes, where ip is the addess of the MN or SN as known by the switch. ::
rspconfig mswitch logdest=<ip>
Configure xdsh for Mellanox Switch
----------------------------------
To run xdsh commands to the Mellanox Switch, you must use the --devicetype input flag to xdsh. In addition, for xCAT versions less than 2.8, you must add a configuration file, please see `Setup ssh connection to the Mellanox Switch`_ section.
For the Mellanox Switch the ``--devicetype`` is ``IBSwitch::Mellanox``. See :doc:`xdsh man page <../../../guides/admin-guides/references/man/xdsh.1.html>` for details.
Now you can run the switch commands from the mn using xdsh. For example: ::
xdsh mswitch -l admin --devicetype IBSwitch::Mellanox \
'enable;configure terminal;show ssh server host-keys'
Commands Supported for the Mellanox Switch
------------------------------------------
Setup the snmp alert destination: ::
rspconfig <switch> snmpdest=<ip> [remove]
where "remove" means to remove this ip from the snmp destination list.
Enable/disable setting the snmp traps. ::
rspconfig <switch> alert=enable/disable
Define the read only community for snmp version 1 and 2. ::
rspconfig <switch> community=<string>
Enable/disable snmp function on the swithc. ::
rspconfig <switch> snmpcfg=enable/disable
Enable/disable ssh-ing to the switch without password. ::
rspconfig <switch> sshcfg=enable/disable
Setup the syslog remove receiver for this switch, and also define the minimum level of severity of the logs that are sent. The valid levels are: emerg, alert, crit, err, warning, notice, info, debug, none, remove. "remove" means to remove the given ip from the receiver list. ::
rspconfig <switch> logdest=<ip> [<level>]
For doing other tasks on the switch, use xdsh. For example: ::
xdsh mswitch -l admin --devicetype IBSwitch::Mellanox 'show logging'
Interactive commands are not supported by xdsh. For interactive commands, use ssh.
Send SNMP traps to xCAT Management Node
---------------------------------------
First, get `http://www.mellanox.com/related-docs/prod_ib_switch_systems/MELLANOX-MIB.zip <http://www.mellanox.com/related-docs/prod_ib_switch_systems/MELLANOX-MIB.zip>`_ , unzip it. Copy the mib file MELLANOX-MIB.txt to ``/usr/share/snmp/mibs`` directory on the mn and sn (if the sn is the snmp trap destination.)
Then,
To configure, run: ::
monadd snmpmon
moncfg snmpmon <mswitch>
To start monitoring, run: ::
monstart snmpmon <mswitch>
To stop monitoring, run: ::
monstop snmpmon <mswitch>
To deconfigure, run: ::
mondecfg snmpmon <mswitch>
For more details on monitoring the cluster: TODO
`Monitoring_an_xCAT_Cluster/#snmp-monitoring <http://sourceforge.net/p/xcat/wiki/Monitoring_an_xCAT_Cluster/#snmp-monitoring>`_

76
docs/source/advanced/networks/infiniband/ufm_configuration.rst
View File

@ -1,2 +1,78 @@
UFM Configuration
=================
UFM server are just regular Linux boxes with UFM installed. xCAT can help install and configure the UFM servers. The XCAT mn can send remote command to UFM through xdsh. It can also collect SNMP traps and syslogs from the UFM servers.
Setup xdsh to UFM and backup
----------------------------
Assume we have two hosts with UFM installed, called host1 and host2. First define the two hosts in the xCAT cluster. Usually the network that the UFM hosts are in a different than the compute nodes, make sure to assign correct servicenode and xcatmaster in the noderes table. And also make sure to assign correct os and arch values in the nodetype table for the UFM hosts. For example: ::
mkdef -t node -o host1,host2 groups=ufm,all os=sles11.1 arch=x86_64 servicenode=10.0.0.1 xcatmaster=10.0.0.1
Then exchange the SSH key so that it can run xdsh. ::
xdsh host1,host2 -K
Now we can run xdsh on the UFM hosts. ::
xdsh ufm date
Consolidate syslogs
-------------------
Run the following command to make the UFM hosts to send the syslogs to the xCAT mn: ::
updatenode ufm -P syslog
To test, run the following commands on the UFM hosts and see if the xCAT MN receives the new messages in /var/log/messages ::
logger xCAT "This is a test"
Send SNMP traps to xCAT Management Node
---------------------------------------
You need to have the Advanced License for UFM in order to send SNMP traps.
1. Copy the mib file to ``/usr/share/snmp/mibs`` directory on the mn. ::
scp ufmhost:/opt/ufm/files/conf/vol_ufm3_0.mib /usr/share/snmp/mibs
Where ufmhost is the host where UFM is installed.
2. On the UFM host, open the /opt/ufm/conf/gv.cfg configuration file. Under the [Notifications] line, set ::
snmp_listeners = <IP Address 1>[:<port 1>][,<IP Address 2>[:<port 2>].]
The default port is 162. For example: ::
ssh ufmhost
vi /opt/ufm/conf/gv.cfg
....
[Notifications]
snmp_listeners = 10.0.0.1
Where 10.0.0.1 is the the ip address of the management node.
3. On the UFM host, restart the ufmd ::
service ufmd restart
4. From UFM GUI, click on the "Config" tab; bring up the "Event Management" Policy Table. Then select the SNMP check boxes for the events you are interested in to enable the system to send an SNMP traps for these events. Click "OK".
5. Make sure snmptrapd is up and running on mn and all monitoring servers.
It should have the '-m ALL' flag. ::
ps -ef |grep snmptrapd
root 31866 1 0 08:44 ? 00:00:00 /usr/sbin/snmptrapd -m ALL
If it is not running, then run the following commands: ::
monadd snmpmon
monstart snmpmon