From 8c3c9239b2327ec28d587e2356ee1d804f987bce Mon Sep 17 00:00:00 2001 From: huweihua Date: Wed, 14 Oct 2015 02:42:46 -0400 Subject: [PATCH 1/5] add IB docs under advanced directory --- .../infiniband/driver_and_installation.rst | 9 + .../networks/infiniband/firmware_updates.rst | 96 ++++++++ .../mlnxofed_ib_install_common_usage.rst | 227 ++++++++++++++++++ .../mlnxofed_ib_install_v1_usage.rst | 22 ++ .../mlnxofed_ib_install_v2_usage.rst | 22 ++ .../infiniband/network_configuration.rst | 35 +++ .../infiniband/switch_configuration.rst | 141 +++++++++++ .../networks/infiniband/ufm_configuration.rst | 76 ++++++ 8 files changed, 628 insertions(+) create mode 100644 docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst create mode 100644 docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_usage.rst create mode 100644 docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_usage.rst diff --git a/docs/source/advanced/networks/infiniband/driver_and_installation.rst b/docs/source/advanced/networks/infiniband/driver_and_installation.rst index 64afd2e61..f2c37b1b1 100644 --- a/docs/source/advanced/networks/infiniband/driver_and_installation.rst +++ b/docs/source/advanced/networks/infiniband/driver_and_installation.rst @@ -1,2 +1,11 @@ IB Driver Preparation and Installation ====================================== + +XCAT provides one sample postscript to help you install the **Mellanox OpenFabrics Enterprise Distribution** (OFED) Infiniband Driver. This shell script is ``/opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install``. You can use this script directly or just refer to it then change it to satisfy your own environment. From xCAT2.11, XCAT offers a new version of mlnxofed_ib_install(i.e. mlnxofed_ib_install.v2). From the perspective of function, the v2 is forward compatible with v1. But the v2 has different usage interface from v1. it becomes more flexible. we still ship v1 with XCAT **but stop support it from xCAT2.11**. We recommend to use mlnxofed_ib_install.v2. + +.. toctree:: + :maxdepth: 2 + + mlnxofed_ib_install_v2_usage.rst + mlnxofed_ib_install_v1_usage.rst + \ No newline at end of file diff --git a/docs/source/advanced/networks/infiniband/firmware_updates.rst b/docs/source/advanced/networks/infiniband/firmware_updates.rst index b8d3e298b..1f02c81eb 100644 --- a/docs/source/advanced/networks/infiniband/firmware_updates.rst +++ b/docs/source/advanced/networks/infiniband/firmware_updates.rst @@ -1,2 +1,98 @@ Firmware Updates ================ + + +Adapter Firmware Update +----------------------- + +Please download the OFED IB adapter firmware from the Mellanox site `http://www.mellanox.com/page/firmware_table_IBM `_ . + +Obtain device id: :: + + lspci | grep -i mel + +Check current installed fw level: :: + + mstflint -d 0002:01:00.0 q | grep FW + +Copy or mount firmware to host: + +Burn new firmware on each ibaX: :: + + mstflint -d 0002:01:00.0 -i b + +Note: if this is a PureFlex MezzanineP adapater then you must select the correct image for each ibaX device. Note the difference in the firmware image at end of filename: _0.bin (iba0/iba2) & _1.bin (iba1/iba3) + +Verify download successful: :: + + mstflint -d 0002:01:00.0 q + +Activate the new firmware: :: + + reboot the image + +Note: the above 0002:01:00.0 device location was used as an example only. it is not meant to imply that there is only one device location or that your device will have the same device location. + +Mellanox Switch Firmware Upgrade +-------------------------------- + +This section provides manual procedure to help update the firmware for Mellanox Infiniband (IB) Switches. You can down load IB switch firmware like IB6131 (image-PPC_M460EX-SX_3.2.xxx.img) from the Mellanox website `http://www.mellanox.com/page/firmware_table_IBM `_ and place into your XCAT Management Node or server that can communicate to Flex IB6131 switch module. There are two ways to update the MLNX-OS switch package. This process works regardless if updating an internal PureFlex chassis Infiniband switch (IB6131 or for an external Mellanox switch. + +Update via Browser +^^^^^^^^^^^^^^^^^^ + +This method is straight forward if your switches are on the public network or your browser is already capable to tunnel to the private address. If neither is the case then you may prefer to use option two. + +After logging into the switch (id=admin, pwd=admin) + +Select the "System" tab and then the "MLNX-OS Upgrade" option + +Under the "Install New Image", select the "Install via scp" +URL: scp://userid@fwhost/directoryofimage/imagename + +Select "Install Image" + +The image will then be downloaded to the switch and the installation process will begin. + +Once completed, the switch must be rebooted for the new package to be activate + +Update via CLI +^^^^^^^^^^^^^^ + +Login to the IB switch: :: + + ssh admin@ + enable (get into correct CLI mode. You can use en) + configure terminal (get into correct CLI mode. You can use co t) + +List current images and Remove older images to free up space: :: + + show image + image delete + (you can paste in ibimage name from show image for image delete) + +Get the new IB image using fetch with scp to a server that contains new IB image. An example of IB3161 image would be "image-PPC_M460EX-SX_3.2.0291.img" Admin can use different protocol . This image fetch scp command is about 4 minutes. :: + + image fetch ? + image fetch scp://userid:password@serveripddr/ + +Verify that new IB image is loaded, then install the new showIB image on IB switch. The install image process goes through 4 stages Verify image, Uncompress image, Create Filesystems, and Extract Image. This install process takes about 9 minutes. :: + + show image + image install + (you can paste in new IB image from "show image" to execute image install) + +Toggle boot partition to new IB image, verify image install is loaded , and that next boot setting is pointing to new IB image. :: + + image boot next + show image + +Save the changes made for new IB image: :: + + configuration write + +Activate the new IB image (reboot switch): :: + + reload + + diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst new file mode 100644 index 000000000..7b78c7cae --- /dev/null +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst @@ -0,0 +1,227 @@ +.. BEGIN_Preparation_For_V2 +Before start, here are some preparation and concept should be known. + +Obtain the Mellanox OFED ISO file from `Mellanox official site `_ and put it under the /install directory following the xCAT directory structure: ``/install/post/otherpkgs///ofed``. + +**[NOTE]** + +* Mellanox provides OFED files with **tarball** and **ISO** two format, but for XCAT, we just support **ISO** format right now. +* Mellanox provides different OFED ISOs depending on operating system and machine architecture, named like MLNX_OFED_LINUX----.iso, you should download correct one according your environment. + +Copy Sample script **mlnxofed_ib_install.v2** into ``/install/postscripts`` and change name to **mlnxofed_ib_install** before using, such as :: + + cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install.v2 /install/postscripts/mlnxofed_ib_install + chmod +x /install/postscripts/mlnxofed_ib_install + +Some options of mlnxofed_ib_install.v2 should be assigned values in the command line argument way when mlnxofed_ib_install.v2 is invoked. +These options are: + +* **-ofeddir** : the directory where OFED ISO file is saved. this is the necessary option +* **-ofedname**: the name of OFED ISO file. this is the necessary option +* **-passmlnxofedoptions**: The mlnxofed_ib_install.v2 invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED iso. Use this option to pass arguments to the ``mlnxofedinstall``. You must include ``-end-`` at the completion of the options to distinguish the option list. if you don't pass any argument to ``mlnxofedinstall``, defualt value ``--without-32bit --without-fw-update --force`` will be passed to ``mlnxofedinstall`` by XCAT. +* **-installroot**: the image root path. this is necessary attribute in diskless scenario +* **-nodesetstate**: nodeset status, the value is one of 'install', 'boot' and 'genimage'. this is necessary attribute in diskless scenario + +For example, if you use MLNX_OFED_LINUX-3.1-1.0.0-ubuntu14.04-ppc64le.iso and save it under ``/install/post/otherpkgs/ubuntu14.04.3/ppc64le/ofed/`` , you want to pass ``--without-32bit --without-fw-update --add-kernel-support --force`` to ``mlnxofedinstall``. you can use like below :: + + mlnxofed_ib_install -ofeddir /install/post/otherpkgs/ubuntu14.04.3/ppc64le/ofed/ -passmlnxofedoptions --without-32bit --without-fw-update --add-kernel-support --force -end- -ofedname MLNX_OFED_LINUX-3.1-1.0.0-ubuntu14.04-ppc64le.iso + +.. END_Preparation_For_V2 + +.. BEGIN_Diskfull_step_For_V2 + +Let's start configuration. + +1. Set script ``mlnxofed_ib_install`` as postbootscript :: + + chdef -p postbootscripts="mlnxofed_ib_install -ofeddir -passmlnxofedoptions -end- -ofedname " + +**[Note]** step 2-4 are only needed by RHEL and SLES + +2. Copy a correct pkglist file shipped by XCAT according your environment to the ``/install/custom/install//`` directory :: + + cp /opt/xcat/share/xcat/install//compute...pkglist /install/custom/install//compute...pkglist + +3. Edit your ``/install/custom/install//compute...pkglist`` and add one line:: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# + +**[NOTE]** You can check directory ``/opt/xcat/share/xcat/ib/netboot//`` and choose one correct ``ib...pkglist`` according your environment + +4. Make sure the related osimage use the customized pkglist :: + + lsdef -t osimage -o --install-compute + +If not, change it :: + + chdef -t osimage -o --install-compute pkglist=/install/custom/install//compute...pkglist + +5. Install node :: + + nodeset osimage=--install-compute + rsetboot net + rpower reset + +.. END_Diskfull_step_For_V2 + +.. BEGIN_Diskless_step_For_V2 +Let's start configuration. + +**[Note]** step 1 is only need by RHEL and SLES + +1. Copy a correct pkglist file shipped by XCAT according your environment to the ``/install/custom/netboot//`` directory :: + + cp /opt/xcat/share/xcat/netboot//compute...pkglist /install/custom/netboot//compute...pkglist + +Edit your ``/install/custom/netboot//.pkglist`` and add: :: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# + +Take sles11 sp1 on x86_64 for example, Edit the ``/install/custom/netboot/sles11.1/x86_64/compute/compute.sles11.1.x86_64.pkglist`` and add: :: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot/sles/ib.sles11.1.x86_64.pkglist# + +2. Prepare postinstall scripts :: + + mkdir -p /install/custom/netboot// + cp /opt/xcat/share/xcat/netboot//.postinstall /install/custom/netboot// + chmod +x /install/custom/netboot//.postinstall + +Edit ``/install/custom/netboot//.postinstall`` and add: :: + + /install/postscripts/mlnxofed_ib_install -ofeddir -ofedname -nodesetstate genimage -installroot $1 + +3. Set the related osimage using the customized pkglist and compute.postinsall :: + + chdef -t osimage -o --netboot-compute \ + pkglist=/install/custom/netboot//compute...pkglist \ + postinstall=/install/custom/netboot//.postinstall + +**[Note]** Ubuntu doesn't need pkglist attribute. + +4. Generate and package image for diskless installation :: + + genimage --netboot-compute + packimage --netboot-compute + +5. Install node :: + + nodeset osimage=--netboot-compute + rsetboot net + rpower reset + +.. END_Diskless_step_For_V2 + +.. BEGIN_Preparation_For_V1 +Obtain the Mellanox OFED ISO file from `Mellanox official site `_ and mount it onto suggested target location on the XCAT MN according your OS and ARCH: :: + + mkdir -p /install/post/otherpkgs///ofed + mount -o loop MLNX_OFED_LINUX----.iso /install/post/otherpkgs///ofed + +Take sles11 sp1 for x86_64 as an example :: + + mkdir -p /install/post/otherpkgs/sles11.1/x86_64/ofed/ + mount -o loop MLNX_OFED_LINUX-1.5.3-3.0.0-sles11sp1-x86_64.iso /install/post/otherpkgs/sles11.1/x86_64/ofed/ + +Take Ubuntu14.4.1 for p8le as an example :: + + mkdir -p /install/post/otherpkgs/ubuntu14.04.1/ppc64el/ofed + mount -o loop MLNX_OFED_LINUX-2.3-1.0.1-ubuntu14.04-ppc64le.iso /install/post/otherpkgs/ubuntu14.04.1/ppc64el/ofed + +**[NOTE]** + +* Mellanox provides OFED files with **tarball** and **ISO** two format, but for XCAT, we just support **ISO** format right now. + +Copy Sample script **mlnxofed_ib_install** shipped by XCAT into ``/install/postscripts`` before using, such as :: + + cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install /install/postscripts/mlnxofed_ib_install + +The **mlnxofed_ib_install** invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED ISO. If you want to pass the argument to ``mlnxofedinstall``, you set the argument to the environment variable ``mlnxofed_options`` which could be read by **mlnxofed_ib_install**. For example: PPE requires the 32-bit version of libibverbs, but the default **mlnxofed_ib_install** will remove all the old ib related packages at first including the 32-bit version of libibverbs. In this case, you can set the environment variable ``mlnxofed_options=--force`` when running the **mlnxofed_ib_install**. For diskfull, you should put the environment variable ``mlnxofed_options=--force`` in mypostscript.tmpl. myposcript.tmpl is in ``/opt/xcat/share/xcat/templates/mypostscript/`` by default. When customize it, you should copy it into ``/install/postscripts/myposcript.tmpl`` :: + + mlnxofed_options='--force' + export mlnxofed_options + +.. END_Preparation_For_V1 + +.. BEGIN_Diskfull_step_For_V1 +Let's start configuration. + +1. Set script ``mlnxofed_ib_install`` as postbootscript :: + + chdef -p postbootscripts=mlnxofed_ib_install + +**[Note]** step 2-4 are only needed by RHEL and SLES + +2. Copy the pkglist to the custom directory :: + + cp /opt/xcat/share/xcat/install//compute...pkglist /install/custom/install//compute...pkglist + +3. Edit your /install/custom/install//compute...pkglist and add :: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# + +4. Make sure the related osimage use the customized pkglist :: + + lsdef -t osimage -o --install-compute + +If not, change it :: + + chdef -t osimage -o --install-compute pkglist=/install/custom/install//compute...pkglist + +5. Install node :: + + nodeset osimage=--install-compute + rsetboot net + rpower reset + +.. END_Diskfull_step_For_V1 + +.. BEGIN_Diskless_step_For_V1 +Let's start configuration. + +**[Note]** step 1 is only need by RHEL and SLES + +1. Copy the pkglist to the custom directory :: + + cp /opt/xcat/share/xcat/netboot//compute...pkglist \ + /install/custom/netboot//compute...pkglist + +Edit your ``/install/custom/netboot//.pkglist`` and add: :: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# + +Take sles11 sp1 on x86_64 for example, Edit the ``/install/custom/netboot/sles11.1/x86_64/compute/compute.sles11.1.x86_64.pkglist`` and add: :: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot/sles/ib.sles11.1.x86_64.pkglist# + +2. Prepare postinstall scripts :: + + mkdir -p /install/custom/netboot// + cp /opt/xcat/share/xcat/netboot//.postinstall /install/custom/netboot// + chmod +x /install/custom/netboot//.postinstall + +Edit ``/install/custom/netboot//.postinstall`` and add: :: + + installroot=$1 ofeddir=/install/post/otherpkgs///ofed/ NODESETSTATE=genimage mlnxofed_options=--force /install/postscripts/mlnxofed_ib_install + +3. Set the related osimage use the customized pkglist and customized compute.postinsall :: + + chdef -t osimage -o --netboot-compute \ + pkglist=/install/custom/netboot//compute...pkglist \ + postinstall=/install/custom/netboot//.postinstall + +**[Note]** Ubuntu doesn't need pkglist attribute. + +4. Generate and package image for diskless installation :: + + genimage --netboot-compute + packimage --netboot-compute + +5. Install node :: + + nodeset osimage=--netboot-compute + rsetboot net + rpower reset + + +.. END_Diskless_step_For_V1 \ No newline at end of file diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_usage.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_usage.rst new file mode 100644 index 000000000..fe87b0d01 --- /dev/null +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_usage.rst @@ -0,0 +1,22 @@ +The Usage of mlnxofed_ib_install +================================ + +Configuration for Diskfull Installation +--------------------------------------- +.. include:: ./mlnxofed_ib_install_common_usage.rst + :start-after: BEGIN_Preparation_For_V1 + :end-before: END_Preparation_For_V1 + +.. include:: ./mlnxofed_ib_install_common_usage.rst + :start-after: BEGIN_Diskfull_step_For_V1 + :end-before: END_Diskfull_step_For_V1 + +Configuration for Diskless Installation +--------------------------------------- +.. include:: ./mlnxofed_ib_install_common_usage.rst + :start-after: BEGIN_Preparation_For_V1 + :end-before: END_Preparation_For_V1 + +.. include:: ./mlnxofed_ib_install_common_usage.rst + :start-after: BEGIN_Diskless_step_For_V1 + :end-before: END_Diskless_step_For_V1 \ No newline at end of file diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_usage.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_usage.rst new file mode 100644 index 000000000..2029425c8 --- /dev/null +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_usage.rst @@ -0,0 +1,22 @@ +The Usage of mlnxofed_ib_install.v2 (Recommend) +=============================================== + +Configuration for Diskfull Installation +--------------------------------------- +.. include:: ./mlnxofed_ib_install_common_usage.rst + :start-after: BEGIN_Preparation_For_V2 + :end-before: END_Preparation_For_V2 + +.. include:: ./mlnxofed_ib_install_common_usage.rst + :start-after: BEGIN_Diskfull_step_For_V2 + :end-before: END_Diskfull_step_For_V2 + +Configuration for Diskless Installation +--------------------------------------- +.. include:: ./mlnxofed_ib_install_common_usage.rst + :start-after: BEGIN_Preparation_For_V2 + :end-before: END_Preparation_For_V2 + +.. include:: ./mlnxofed_ib_install_common_usage.rst + :start-after: BEGIN_Diskless_step_For_V2 + :end-before: END_Diskless_step_For_V2 diff --git a/docs/source/advanced/networks/infiniband/network_configuration.rst b/docs/source/advanced/networks/infiniband/network_configuration.rst index 3fa97b26a..ce9707d02 100644 --- a/docs/source/advanced/networks/infiniband/network_configuration.rst +++ b/docs/source/advanced/networks/infiniband/network_configuration.rst @@ -1,2 +1,37 @@ IB Network Configuration ======================== + +XCAT provided two sample postscripts - configiba.1port and configiba.2ports to configure the IB adapter before XCAT 2.8, these tow scripts still work **but will be in maintenance mode**. + +A new postscript ``/install/postscripts/configib`` is shipped with XCAT 2.8, the ``configib`` postscript works with the new "nics" table and ``confignic`` postscript which is introduced in XCAT 2.8 also. XCAT recommends you to use new ``configib`` script from now on. + +IB Interface is a kind of additional adapters for XCAT, so the process of configuring Mellanox IB interface complies with the process of :doc:`Configure Additional Network Interfaces <../../../../guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/cfg_second_adapter>`. + +Below are an simple example to configure Mellanox IB in ubuntu14.4.1 on p8le + +If your target Mellanox IB adapter has 2 ports, and you plan to give port ib0 4 different IPs, 2 are IPV4 (20.0.0.3 and 30.0.0.3) and another 2 are IPV6 (1:2::3 and 2:2::3). + +1. Define your networks in networks table :: + + chdef -t network -o ib0ipv41 net=20.0.0.0 mask=255.255.255.0 mgtifname=ib0 + chdef -t network -o ib0ipv42 net=30.0.0.0 mask=255.255.255.0 mgtifname=ib0 + chdef -t network -o ib0ipv61 net=1:2::/64 mask=/64 mgtifname=ib0 gateway=1:2::2 + chdef -t network -o ib0ipv62 net=2:2::/64 mask=/64 mgtifname=ib0 gateway= + +2. Define IPs for ib0 :: + + chdef nicips.ib0="20.0.0.3|30.0.0.3|1:2::3|2:2::3" nicnetworks.ib0="ib0ipv41|ib0ipv42|ib0ipv61|ib0ipv62" nictypes.ib0="Infiniband" + +3. Configure ib0 + +Configure during node installation :: + + chdef -p postscripts="confignics --ibaports=2" + nodeset osimage= + rsetboot net + rpower reset + +Configure on a node which has have operating system :: + + updatenode -P "confignics --ibaports=2" + diff --git a/docs/source/advanced/networks/infiniband/switch_configuration.rst b/docs/source/advanced/networks/infiniband/switch_configuration.rst index 9cb975ec8..8905fddef 100644 --- a/docs/source/advanced/networks/infiniband/switch_configuration.rst +++ b/docs/source/advanced/networks/infiniband/switch_configuration.rst @@ -1,2 +1,143 @@ IB Switch Configuration ======================= + +Setup the XCAT Database +----------------------- + +The Mellanox Switch is only supported in XCAT Release 2.7 or later. + +Add the switch ip address in the ``/etc/hosts`` file + +Define IB switch as a node: :: + + chdef -t node -o mswitch groups=all nodetype=switch mgt=switch + +Add the login user name and password to the switches table: :: + + tabch switch=mswitch switches.sshusername=admin switches.sshpassword=admin switches.switchtype=MellanoxIB + +The switches table will look like this: :: + + #switch,...,sshusername,sshpassword,switchtype,.... + "mswitch",,,,,,,"admin","admin","MellanoxIB",, + +If there is only one admin and one password for all the switches then put the entry in the XCAT passwd table for the admin id and password to use to login. :: + + tabch key=mswitch passwd.username=admin passwd.password=admin + +The passwd table will look like this: :: + + #key,username,password,cryptmethod,comments,disable + "mswitch","admin","admin",,, + +Setup ssh connection to the Mellanox Switch +------------------------------------------- + +To run commands like xdsh and script to the Mellanox Switch, we need to setup ssh to run without prompting for a password to the Mellanox Switch. To do this, first you must add a configuration file. This configuration file is NOT needed for XCAT 2.8 and later. :: + + mkdir -p /var/opt/xcat/IBSwitch/Mellanox + cd /var/opt/xcat/IBSwitch/Mellanox + cp /opt/xcat/share/xcat/devicetype/IBSwitch/Mellanox/config . + +The file contains the following: :: + + [main] + [xdsh] + pre-command=cli + post-command=NULL + +Then run the following: :: + + rspconfig mswitch sshcfg=enable + +**[Note]** For Mellanox switch in manufacturing defaults status, the user need to answer 'no' for the initial configuration wizard prompt as follows before run 'rspconfig'. :: + + [s1mn][/](/)> ssh -l admin mswitch + Mellanox MLNX-OS Switch Management + Password: + Last login: Wed Feb 20 20:09:50 2013 from 1.2.3.4 + Mellanox Switch + Mellanox configuration wizard + Do you want to use the wizard for initial configuration? **no** + To return to the wizard from the CLI, enter the "configuration jump-start" + command from configure mode. Launching CLI... + switch-xxxxxx [standalone: unknown] > exit + +Setup syslog on the Switch +-------------------------- + +Use the following command to consolidate the syslog to the Management Node or Service Nodes, where ip is the addess of the MN or SN as known by the switch. :: + + rspconfig mswitch logdest= + +Configure xdsh for Mellanox Switch +---------------------------------- +To run xdsh commands to the Mellanox Switch, you must use the --devicetype input flag to xdsh. In addition, for XCAT versions less than 2.8, you must add a configuration file, please see **"Setup ssh connection to the Mellanox Switch"** section. + +For the Mellanox Switch the --devicetype is "IBSwitch::Mellanox". See xdsh man page: TODO `http://xcat.sourceforge.net/man1/xdsh.1.html `_ for details. + +Now you can run the switch commands from the mn using xdsh. For example: :: + + xdsh mswitch -l admin --devicetype IBSwitch::Mellanox 'enable;configure terminal;show ssh server host-keys' + +Commands Supported for the Mellanox Switch +------------------------------------------ + +Setup the snmp alert destination: :: + + rspconfig snmpdest= [remove] + +where "remove" means to remove this ip from the snmp destination list. + +Enable/disable setting the snmp traps. :: + + rspconfig alert=enable/disable + +Define the read only community for snmp version 1 and 2. :: + + rspconfig community= + +Enable/disable snmp function on the swithc. :: + + rspconfig snmpcfg=enable/disable + +Enable/disable ssh-ing to the switch without password. :: + + rspconfig sshcfg=enable/disable + +Setup the syslog remove receiver for this switch, and also define the minimum level of severity of the logs that are sent. The valid levels are: emerg, alert, crit, err, warning, notice, info, debug, none, remove. "remove" means to remove the given ip from the receiver list. :: + + rspconfig logdest= [] + +For doing other tasks on the switch, use xdsh. For example: :: + + xdsh mswitch -l admin --devicetype IBSwitch::Mellanox 'show logging' + +Interactive commands are not supported by xdsh. For interactive commands, use ssh. + +Send SNMP traps to XCAT Management Node +--------------------------------------- + +First, get `http://www.mellanox.com/related-docs/prod_ib_switch_systems/MELLANOX-MIB.zip `_ , unzip it. Copy the mib file MELLANOX-MIB.txt to ``/usr/share/snmp/mibs`` directory on the mn and sn (if the sn is the snmp trap destination.) + +Then, + +To configure, run: :: + + monadd snmpmon + moncfg snmpmon + +To start monitoring, run: :: + + monstart snmpmon + +To stop monitoring, run: :: + + monstop snmpmon + +To deconfigure, run: :: + + mondecfg snmpmon + +For more details on monitoring the cluster: TODO +`Monitoring_an_xCAT_Cluster/#snmp-monitoring `_ diff --git a/docs/source/advanced/networks/infiniband/ufm_configuration.rst b/docs/source/advanced/networks/infiniband/ufm_configuration.rst index 0f4eb6e16..d4dfd458a 100644 --- a/docs/source/advanced/networks/infiniband/ufm_configuration.rst +++ b/docs/source/advanced/networks/infiniband/ufm_configuration.rst @@ -1,2 +1,78 @@ UFM Configuration ================= + + +UFM server are just regular Linix boxes with UFM installed. XCAT can help install and configure the UFM servers. The XCAT mn can send remote command to UFM through xdsh. It can also collect SNMP traps and syslogs from the UFM servers. + +Setup xdsh to UFM and backup +---------------------------- + +Assume we have two hosts with UFM installed, called host1 and host2. First define the two hosts in the XCAT cluster. Usually the network that the UFM hosts are in a different than the compute nodes, make sure to assign correct servicenode and xcatmaster in the noderes table. And also make sure to assign correct os and arch values in the nodetype table for the UFM hosts. For example: :: + + mkdef -t node -o host1,host2 groups=ufm,all os=sles11.1 arch=x86_64 servicenode=10.0.0.1 xcatmaster=10.0.0.1 + +Then exchange the SSH key so that it can run xdsh. :: + + xdsh host1,host2 -K + +Now we can run xdsh on the UFM hosts. :: + + xdsh ufm date + +Consolidate syslogs +------------------- + +Run the following command to make the UFM hosts to send the syslogs to the XCAT mn: :: + + updatenode ufm -P syslog + +To test, run the following commands on the UFM hosts and see if the XCAT MN receives the new messages in /var/log/messages :: + + logger xCAT "This is a test" + + +Send SNMP traps to XCAT Management Node +--------------------------------------- + +You need to have the Advanced License for UFM in order to send SNMP traps. + +1. Copy the mib file to ``/usr/share/snmp/mibs`` directory on the mn. :: + + scp ufmhost:/opt/ufm/files/conf/vol_ufm3_0.mib /usr/share/snmp/mibs + +Where ufmhost is the host where UFM is installed. + +2. On the UFM host, open the /opt/ufm/conf/gv.cfg configuration file. Under the [Notifications] line, set :: + + snmp_listeners = [:][,[:].] + +The default port is 162. For example: :: + + ssh ufmhost + vi /opt/ufm/conf/gv.cfg + + .... + [Notifications] + snmp_listeners = 10.0.0.1 + +Where 10.0.0.1 is the the ip address of the management node. + +3. On the UFM host, restart the ufmd :: + + service ufmd restart + +4. From UFM GUI, click on the "Config" tab; bring up the "Event Management" Policy Table. Then select the SNMP check boxes for the events you are interested in to enable the system to send an SNMP traps for these events. Click "OK". + +5. Make sure snmptrapd is up and running on mn and all monitoring servers. + +It should have the '-m ALL' flag. :: + + ps -ef |grep snmptrapd + root 31866 1 0 08:44 ? 00:00:00 /usr/sbin/snmptrapd -m ALL + +If it is not running, then run the following commands: :: + + monadd snmpmon + monstart snmpmon + + \ No newline at end of file From 1e959ced054caa2af328e90b67c621d0785868da Mon Sep 17 00:00:00 2001 From: huweihua Date: Wed, 14 Oct 2015 03:01:25 -0400 Subject: [PATCH 2/5] fix some warning during make --- .../networks/infiniband/mlnxofed_ib_install_common_usage.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst index 7b78c7cae..2486b936f 100644 --- a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst @@ -1,4 +1,5 @@ .. BEGIN_Preparation_For_V2 + Before start, here are some preparation and concept should be known. Obtain the Mellanox OFED ISO file from `Mellanox official site `_ and put it under the /install directory following the xCAT directory structure: ``/install/post/otherpkgs///ofed``. @@ -65,6 +66,7 @@ If not, change it :: .. END_Diskfull_step_For_V2 .. BEGIN_Diskless_step_For_V2 + Let's start configuration. **[Note]** step 1 is only need by RHEL and SLES @@ -113,6 +115,7 @@ Edit ``/install/custom/netboot//.postinstall`` and add: :: .. END_Diskless_step_For_V2 .. BEGIN_Preparation_For_V1 + Obtain the Mellanox OFED ISO file from `Mellanox official site `_ and mount it onto suggested target location on the XCAT MN according your OS and ARCH: :: mkdir -p /install/post/otherpkgs///ofed @@ -144,6 +147,7 @@ The **mlnxofed_ib_install** invokes a script ``mlnxofedinstall`` shipped by Mell .. END_Preparation_For_V1 .. BEGIN_Diskfull_step_For_V1 + Let's start configuration. 1. Set script ``mlnxofed_ib_install`` as postbootscript :: @@ -177,6 +181,7 @@ If not, change it :: .. END_Diskfull_step_For_V1 .. BEGIN_Diskless_step_For_V1 + Let's start configuration. **[Note]** step 1 is only need by RHEL and SLES From 9adbc0cea9155cee8bcdb41ea4cdd677385512f3 Mon Sep 17 00:00:00 2001 From: huweihua Date: Sun, 18 Oct 2015 05:03:50 -0400 Subject: [PATCH 3/5] delete for useless file --- .../mlnxofed_ib_install_common_usage.rst | 232 ------------------ 1 file changed, 232 deletions(-) delete mode 100644 docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst deleted file mode 100644 index 2486b936f..000000000 --- a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_common_usage.rst +++ /dev/null @@ -1,232 +0,0 @@ -.. BEGIN_Preparation_For_V2 - -Before start, here are some preparation and concept should be known. - -Obtain the Mellanox OFED ISO file from `Mellanox official site `_ and put it under the /install directory following the xCAT directory structure: ``/install/post/otherpkgs///ofed``. - -**[NOTE]** - -* Mellanox provides OFED files with **tarball** and **ISO** two format, but for XCAT, we just support **ISO** format right now. -* Mellanox provides different OFED ISOs depending on operating system and machine architecture, named like MLNX_OFED_LINUX----.iso, you should download correct one according your environment. - -Copy Sample script **mlnxofed_ib_install.v2** into ``/install/postscripts`` and change name to **mlnxofed_ib_install** before using, such as :: - - cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install.v2 /install/postscripts/mlnxofed_ib_install - chmod +x /install/postscripts/mlnxofed_ib_install - -Some options of mlnxofed_ib_install.v2 should be assigned values in the command line argument way when mlnxofed_ib_install.v2 is invoked. -These options are: - -* **-ofeddir** : the directory where OFED ISO file is saved. this is the necessary option -* **-ofedname**: the name of OFED ISO file. this is the necessary option -* **-passmlnxofedoptions**: The mlnxofed_ib_install.v2 invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED iso. Use this option to pass arguments to the ``mlnxofedinstall``. You must include ``-end-`` at the completion of the options to distinguish the option list. if you don't pass any argument to ``mlnxofedinstall``, defualt value ``--without-32bit --without-fw-update --force`` will be passed to ``mlnxofedinstall`` by XCAT. -* **-installroot**: the image root path. this is necessary attribute in diskless scenario -* **-nodesetstate**: nodeset status, the value is one of 'install', 'boot' and 'genimage'. this is necessary attribute in diskless scenario - -For example, if you use MLNX_OFED_LINUX-3.1-1.0.0-ubuntu14.04-ppc64le.iso and save it under ``/install/post/otherpkgs/ubuntu14.04.3/ppc64le/ofed/`` , you want to pass ``--without-32bit --without-fw-update --add-kernel-support --force`` to ``mlnxofedinstall``. you can use like below :: - - mlnxofed_ib_install -ofeddir /install/post/otherpkgs/ubuntu14.04.3/ppc64le/ofed/ -passmlnxofedoptions --without-32bit --without-fw-update --add-kernel-support --force -end- -ofedname MLNX_OFED_LINUX-3.1-1.0.0-ubuntu14.04-ppc64le.iso - -.. END_Preparation_For_V2 - -.. BEGIN_Diskfull_step_For_V2 - -Let's start configuration. - -1. Set script ``mlnxofed_ib_install`` as postbootscript :: - - chdef -p postbootscripts="mlnxofed_ib_install -ofeddir -passmlnxofedoptions -end- -ofedname " - -**[Note]** step 2-4 are only needed by RHEL and SLES - -2. Copy a correct pkglist file shipped by XCAT according your environment to the ``/install/custom/install//`` directory :: - - cp /opt/xcat/share/xcat/install//compute...pkglist /install/custom/install//compute...pkglist - -3. Edit your ``/install/custom/install//compute...pkglist`` and add one line:: - - #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# - -**[NOTE]** You can check directory ``/opt/xcat/share/xcat/ib/netboot//`` and choose one correct ``ib...pkglist`` according your environment - -4. Make sure the related osimage use the customized pkglist :: - - lsdef -t osimage -o --install-compute - -If not, change it :: - - chdef -t osimage -o --install-compute pkglist=/install/custom/install//compute...pkglist - -5. Install node :: - - nodeset osimage=--install-compute - rsetboot net - rpower reset - -.. END_Diskfull_step_For_V2 - -.. BEGIN_Diskless_step_For_V2 - -Let's start configuration. - -**[Note]** step 1 is only need by RHEL and SLES - -1. Copy a correct pkglist file shipped by XCAT according your environment to the ``/install/custom/netboot//`` directory :: - - cp /opt/xcat/share/xcat/netboot//compute...pkglist /install/custom/netboot//compute...pkglist - -Edit your ``/install/custom/netboot//.pkglist`` and add: :: - - #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# - -Take sles11 sp1 on x86_64 for example, Edit the ``/install/custom/netboot/sles11.1/x86_64/compute/compute.sles11.1.x86_64.pkglist`` and add: :: - - #INCLUDE:/opt/xcat/share/xcat/ib/netboot/sles/ib.sles11.1.x86_64.pkglist# - -2. Prepare postinstall scripts :: - - mkdir -p /install/custom/netboot// - cp /opt/xcat/share/xcat/netboot//.postinstall /install/custom/netboot// - chmod +x /install/custom/netboot//.postinstall - -Edit ``/install/custom/netboot//.postinstall`` and add: :: - - /install/postscripts/mlnxofed_ib_install -ofeddir -ofedname -nodesetstate genimage -installroot $1 - -3. Set the related osimage using the customized pkglist and compute.postinsall :: - - chdef -t osimage -o --netboot-compute \ - pkglist=/install/custom/netboot//compute...pkglist \ - postinstall=/install/custom/netboot//.postinstall - -**[Note]** Ubuntu doesn't need pkglist attribute. - -4. Generate and package image for diskless installation :: - - genimage --netboot-compute - packimage --netboot-compute - -5. Install node :: - - nodeset osimage=--netboot-compute - rsetboot net - rpower reset - -.. END_Diskless_step_For_V2 - -.. BEGIN_Preparation_For_V1 - -Obtain the Mellanox OFED ISO file from `Mellanox official site `_ and mount it onto suggested target location on the XCAT MN according your OS and ARCH: :: - - mkdir -p /install/post/otherpkgs///ofed - mount -o loop MLNX_OFED_LINUX----.iso /install/post/otherpkgs///ofed - -Take sles11 sp1 for x86_64 as an example :: - - mkdir -p /install/post/otherpkgs/sles11.1/x86_64/ofed/ - mount -o loop MLNX_OFED_LINUX-1.5.3-3.0.0-sles11sp1-x86_64.iso /install/post/otherpkgs/sles11.1/x86_64/ofed/ - -Take Ubuntu14.4.1 for p8le as an example :: - - mkdir -p /install/post/otherpkgs/ubuntu14.04.1/ppc64el/ofed - mount -o loop MLNX_OFED_LINUX-2.3-1.0.1-ubuntu14.04-ppc64le.iso /install/post/otherpkgs/ubuntu14.04.1/ppc64el/ofed - -**[NOTE]** - -* Mellanox provides OFED files with **tarball** and **ISO** two format, but for XCAT, we just support **ISO** format right now. - -Copy Sample script **mlnxofed_ib_install** shipped by XCAT into ``/install/postscripts`` before using, such as :: - - cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install /install/postscripts/mlnxofed_ib_install - -The **mlnxofed_ib_install** invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED ISO. If you want to pass the argument to ``mlnxofedinstall``, you set the argument to the environment variable ``mlnxofed_options`` which could be read by **mlnxofed_ib_install**. For example: PPE requires the 32-bit version of libibverbs, but the default **mlnxofed_ib_install** will remove all the old ib related packages at first including the 32-bit version of libibverbs. In this case, you can set the environment variable ``mlnxofed_options=--force`` when running the **mlnxofed_ib_install**. For diskfull, you should put the environment variable ``mlnxofed_options=--force`` in mypostscript.tmpl. myposcript.tmpl is in ``/opt/xcat/share/xcat/templates/mypostscript/`` by default. When customize it, you should copy it into ``/install/postscripts/myposcript.tmpl`` :: - - mlnxofed_options='--force' - export mlnxofed_options - -.. END_Preparation_For_V1 - -.. BEGIN_Diskfull_step_For_V1 - -Let's start configuration. - -1. Set script ``mlnxofed_ib_install`` as postbootscript :: - - chdef -p postbootscripts=mlnxofed_ib_install - -**[Note]** step 2-4 are only needed by RHEL and SLES - -2. Copy the pkglist to the custom directory :: - - cp /opt/xcat/share/xcat/install//compute...pkglist /install/custom/install//compute...pkglist - -3. Edit your /install/custom/install//compute...pkglist and add :: - - #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# - -4. Make sure the related osimage use the customized pkglist :: - - lsdef -t osimage -o --install-compute - -If not, change it :: - - chdef -t osimage -o --install-compute pkglist=/install/custom/install//compute...pkglist - -5. Install node :: - - nodeset osimage=--install-compute - rsetboot net - rpower reset - -.. END_Diskfull_step_For_V1 - -.. BEGIN_Diskless_step_For_V1 - -Let's start configuration. - -**[Note]** step 1 is only need by RHEL and SLES - -1. Copy the pkglist to the custom directory :: - - cp /opt/xcat/share/xcat/netboot//compute...pkglist \ - /install/custom/netboot//compute...pkglist - -Edit your ``/install/custom/netboot//.pkglist`` and add: :: - - #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# - -Take sles11 sp1 on x86_64 for example, Edit the ``/install/custom/netboot/sles11.1/x86_64/compute/compute.sles11.1.x86_64.pkglist`` and add: :: - - #INCLUDE:/opt/xcat/share/xcat/ib/netboot/sles/ib.sles11.1.x86_64.pkglist# - -2. Prepare postinstall scripts :: - - mkdir -p /install/custom/netboot// - cp /opt/xcat/share/xcat/netboot//.postinstall /install/custom/netboot// - chmod +x /install/custom/netboot//.postinstall - -Edit ``/install/custom/netboot//.postinstall`` and add: :: - - installroot=$1 ofeddir=/install/post/otherpkgs///ofed/ NODESETSTATE=genimage mlnxofed_options=--force /install/postscripts/mlnxofed_ib_install - -3. Set the related osimage use the customized pkglist and customized compute.postinsall :: - - chdef -t osimage -o --netboot-compute \ - pkglist=/install/custom/netboot//compute...pkglist \ - postinstall=/install/custom/netboot//.postinstall - -**[Note]** Ubuntu doesn't need pkglist attribute. - -4. Generate and package image for diskless installation :: - - genimage --netboot-compute - packimage --netboot-compute - -5. Install node :: - - nodeset osimage=--netboot-compute - rsetboot net - rpower reset - - -.. END_Diskless_step_For_V1 \ No newline at end of file From f186f1e8380506829404901eaf0a52210d7ea473 Mon Sep 17 00:00:00 2001 From: huweihua Date: Sun, 18 Oct 2015 05:05:44 -0400 Subject: [PATCH 4/5] modify depending on comments --- .../infiniband/driver_and_installation.rst | 2 +- .../networks/infiniband/firmware_updates.rst | 2 +- .../advanced/networks/infiniband/index.rst | 2 + .../mlnxofed_ib_install_v1_diskful.rst | 28 ++++++++++ .../mlnxofed_ib_install_v1_diskless.rst | 52 ++++++++++++++++++ .../mlnxofed_ib_install_v1_preparation.rst | 42 +++++++++++++++ .../mlnxofed_ib_install_v1_usage.rst | 27 +++------- .../mlnxofed_ib_install_v2_diskful.rst | 28 ++++++++++ .../mlnxofed_ib_install_v2_diskless.rst | 54 +++++++++++++++++++ .../mlnxofed_ib_install_v2_preparation.rst | 34 ++++++++++++ .../mlnxofed_ib_install_v2_usage.rst | 28 +++------- .../infiniband/network_configuration.rst | 15 +++--- .../infiniband/switch_configuration.rst | 17 +++--- .../networks/infiniband/ufm_configuration.rst | 10 ++-- 14 files changed, 278 insertions(+), 63 deletions(-) create mode 100644 docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskful.rst create mode 100644 docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskless.rst create mode 100644 docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_preparation.rst create mode 100644 docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskful.rst create mode 100644 docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskless.rst create mode 100644 docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.rst diff --git a/docs/source/advanced/networks/infiniband/driver_and_installation.rst b/docs/source/advanced/networks/infiniband/driver_and_installation.rst index f2c37b1b1..1c22dfed9 100644 --- a/docs/source/advanced/networks/infiniband/driver_and_installation.rst +++ b/docs/source/advanced/networks/infiniband/driver_and_installation.rst @@ -1,7 +1,7 @@ IB Driver Preparation and Installation ====================================== -XCAT provides one sample postscript to help you install the **Mellanox OpenFabrics Enterprise Distribution** (OFED) Infiniband Driver. This shell script is ``/opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install``. You can use this script directly or just refer to it then change it to satisfy your own environment. From xCAT2.11, XCAT offers a new version of mlnxofed_ib_install(i.e. mlnxofed_ib_install.v2). From the perspective of function, the v2 is forward compatible with v1. But the v2 has different usage interface from v1. it becomes more flexible. we still ship v1 with XCAT **but stop support it from xCAT2.11**. We recommend to use mlnxofed_ib_install.v2. +xCAT provides sample postscripts to help you install the Mellanox OpenFabrics Enterprise Distribution (OFED) Infiniband Driver. These scripts are located in ``opt/xcat/share/xcat/ib/scripts/Mellanox/``. You can use these scripts directly or change them to satisfy your own environment. **xCAT 2.11 drops support of mlnxofed_ib_install and recommends using version 2 of the script: mlnxofed_ib_install.v2**. .. toctree:: :maxdepth: 2 diff --git a/docs/source/advanced/networks/infiniband/firmware_updates.rst b/docs/source/advanced/networks/infiniband/firmware_updates.rst index 1f02c81eb..b48296e8f 100644 --- a/docs/source/advanced/networks/infiniband/firmware_updates.rst +++ b/docs/source/advanced/networks/infiniband/firmware_updates.rst @@ -36,7 +36,7 @@ Note: the above 0002:01:00.0 device location was used as an example only. it is Mellanox Switch Firmware Upgrade -------------------------------- -This section provides manual procedure to help update the firmware for Mellanox Infiniband (IB) Switches. You can down load IB switch firmware like IB6131 (image-PPC_M460EX-SX_3.2.xxx.img) from the Mellanox website `http://www.mellanox.com/page/firmware_table_IBM `_ and place into your XCAT Management Node or server that can communicate to Flex IB6131 switch module. There are two ways to update the MLNX-OS switch package. This process works regardless if updating an internal PureFlex chassis Infiniband switch (IB6131 or for an external Mellanox switch. +This section provides manual procedure to help update the firmware for Mellanox Infiniband (IB) Switches. You can down load IB switch firmware like IB6131 (image-PPC_M460EX-SX_3.2.xxx.img) from the Mellanox website `http://www.mellanox.com/page/firmware_table_IBM `_ and place into your xCAT Management Node or server that can communicate to Flex IB6131 switch module. There are two ways to update the MLNX-OS switch package. This process works regardless if updating an internal PureFlex chassis Infiniband switch (IB6131 or for an external Mellanox switch. Update via Browser ^^^^^^^^^^^^^^^^^^ diff --git a/docs/source/advanced/networks/infiniband/index.rst b/docs/source/advanced/networks/infiniband/index.rst index 0de9e2243..1ef66d623 100644 --- a/docs/source/advanced/networks/infiniband/index.rst +++ b/docs/source/advanced/networks/infiniband/index.rst @@ -1,6 +1,8 @@ Infiniband (Mellanox) ===================== +xCAT offers a certain degree support for Mellanox infiniband product, it help you to configurate Mellanox infiniband products easily. + .. toctree:: :maxdepth: 2 diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskful.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskful.rst new file mode 100644 index 000000000..3eb63a99c --- /dev/null +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskful.rst @@ -0,0 +1,28 @@ +Configuration for Diskful Installation +====================================== + +1. Set script ``mlnxofed_ib_install`` as postbootscript :: + + chdef -p postbootscripts=mlnxofed_ib_install + +2. Specify dependence package **[required for RHEL and SLES]** + + a) Copy the pkglist to the custom directory :: + + cp /opt/xcat/share/xcat/install//compute...pkglist \ + /install/custom/install//compute...pkglist + + b) Edit your /install/custom/install//compute...pkglist and add one line:: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# + + c) Make the related osimage use the customized pkglist :: + + chdef -t osimage -o --install-compute \ + pkglist=/install/custom/install//compute...pkglist + +3. Install node :: + + nodeset osimage=--install-compute + rsetboot net + rpower reset diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskless.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskless.rst new file mode 100644 index 000000000..e56a719b4 --- /dev/null +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskless.rst @@ -0,0 +1,52 @@ +Configuration for Diskless Installation +======================================= + +1. Specify dependence package **[required for RHEL and SLES]** + + a) Copy the pkglist to the custom directory :: + + cp /opt/xcat/share/xcat/netboot//compute...pkglist \ + /install/custom/netboot//compute...pkglist + + b) Edit your ``/install/custom/netboot//.pkglist`` and add one line ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist#`` + +2. Prepare postinstall scripts + + a) Specify postinstall scripts:: + + mkdir -p /install/custom/netboot// + + cp /opt/xcat/share/xcat/netboot//.postinstall \ + /install/custom/netboot// + + chmod +x /install/custom/netboot//.postinstall + + b) Edit ``/install/custom/netboot//.postinstall`` and add below line in the end: :: + + installroot=$1 ofeddir=/install/post/otherpkgs///ofed/ \ + NODESETSTATE=genimage mlnxofed_options=--force /install/postscripts/mlnxofed_ib_install + + +3. Set the related osimage use the customized pkglist and customized compute.postinsall + +* [RHEL/SLES] :: + + chdef -t osimage -o --netboot-compute \ + pkglist=/install/custom/netboot//compute...pkglist \ + postinstall=/install/custom/netboot//.postinstall + +* [Ubuntu] :: + + chdef -t osimage -o --netboot-compute \ + postinstall=/install/custom/netboot//.postinstall + +4. Generate and package image for diskless installation :: + + genimage --netboot-compute + packimage --netboot-compute + +5. Install node :: + + nodeset osimage=--netboot-compute + rsetboot net + rpower reset diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_preparation.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_preparation.rst new file mode 100644 index 000000000..340871bef --- /dev/null +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_preparation.rst @@ -0,0 +1,42 @@ +Preparation +=========== + +Obtain the Mellanox OFED ISO file from `Mellanox official site `_ and mount it onto suggested target location on the xCAT MN according your OS and ARCH: :: + + mkdir -p /install/post/otherpkgs///ofed + + mount -o loop MLNX_OFED_LINUX----.iso \ + /install/post/otherpkgs///ofed + +Take sles11 sp1 for x86_64 as an example :: + + mkdir -p /install/post/otherpkgs/sles11.1/x86_64/ofed/ + + mount -o loop MLNX_OFED_LINUX-1.5.3-3.0.0-sles11sp1-x86_64.iso \ + /install/post/otherpkgs/sles11.1/x86_64/ofed/ + + +Take Ubuntu14.4.1 for Power8 LE as an example :: + + mkdir -p /install/post/otherpkgs/ubuntu14.04.1/ppc64el/ofed + + mount -o loop MLNX_OFED_LINUX-2.3-1.0.1-ubuntu14.04-ppc64le.iso \ + /install/post/otherpkgs/ubuntu14.04.1/ppc64el/ofed + + +**[NOTE]** + +* Mellanox provides OFED files with **tarball** and **ISO** two format, but for xCAT, we just support **ISO** format right now. + +Copy Sample script **mlnxofed_ib_install** shipped by xCAT into ``/install/postscripts`` before using, such as :: + + cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install \ + /install/postscripts/mlnxofed_ib_install + +The **mlnxofed_ib_install** invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED ISO. If you want to pass the argument to ``mlnxofedinstall``, you set the argument to the environment variable ``mlnxofed_options`` which could be read by **mlnxofed_ib_install**. For example: PPE requires the 32-bit version of libibverbs, but the default **mlnxofed_ib_install** will remove all the old ib related packages at first including the 32-bit version of libibverbs. In this case, you can set the environment variable ``mlnxofed_options=--force`` when running the **mlnxofed_ib_install**. For diskful, you should put the environment variable ``mlnxofed_options=--force`` in mypostscript.tmpl. myposcript.tmpl is in ``/opt/xcat/share/xcat/templates/mypostscript/`` by default. When customize it, you should copy it into ``/install/postscripts/myposcript.tmpl`` :: + + mlnxofed_options='--force' + export mlnxofed_options + + + \ No newline at end of file diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_usage.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_usage.rst index fe87b0d01..79577bdbd 100644 --- a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_usage.rst +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_usage.rst @@ -1,22 +1,9 @@ -The Usage of mlnxofed_ib_install -================================ +Using mlnxofed_ib_install (drops support) +========================================= -Configuration for Diskfull Installation ---------------------------------------- -.. include:: ./mlnxofed_ib_install_common_usage.rst - :start-after: BEGIN_Preparation_For_V1 - :end-before: END_Preparation_For_V1 +.. toctree:: + :maxdepth: 1 -.. include:: ./mlnxofed_ib_install_common_usage.rst - :start-after: BEGIN_Diskfull_step_For_V1 - :end-before: END_Diskfull_step_For_V1 - -Configuration for Diskless Installation ---------------------------------------- -.. include:: ./mlnxofed_ib_install_common_usage.rst - :start-after: BEGIN_Preparation_For_V1 - :end-before: END_Preparation_For_V1 - -.. include:: ./mlnxofed_ib_install_common_usage.rst - :start-after: BEGIN_Diskless_step_For_V1 - :end-before: END_Diskless_step_For_V1 \ No newline at end of file + mlnxofed_ib_install_v1_preparation.rst + mlnxofed_ib_install_v1_diskful.rst + mlnxofed_ib_install_v1_diskless.rst \ No newline at end of file diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskful.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskful.rst new file mode 100644 index 000000000..6c09b4043 --- /dev/null +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskful.rst @@ -0,0 +1,28 @@ +Configuration for Diskful Installation +======================================= + +1. Set script ``mlnxofed_ib_install`` as postbootscript :: + + chdef -p postbootscripts="mlnxofed_ib_install -p /install//" + +2. Specify dependence package **[required for RHEL and SLES]** + + a) Copy a correct pkglist file shipped by xCAT according your environment to the ``/install/custom/install//`` directory :: + + cp /opt/xcat/share/xcat/install//compute...pkglist \ + /install/custom/install//compute...pkglist + + b) Edit your ``/install/custom/install//compute...pkglist`` and add below line:: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# + + c) Make the related osimage use the customized pkglist :: + + chdef -t osimage -o --install-compute \ + pkglist=/install/custom/install//compute...pkglist + +3. Install node :: + + nodeset osimage=--install-compute + rsetboot net + rpower reset diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskless.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskless.rst new file mode 100644 index 000000000..68b9a2960 --- /dev/null +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskless.rst @@ -0,0 +1,54 @@ +Configuration for Diskless Installation +======================================= + +1. Specify dependence package **[required for RHEL and SLES]** + + a) Copy a correct pkglist file shipped by xCAT according your environment to the ``/install/custom/netboot//`` directory :: + + cp /opt/xcat/share/xcat/netboot//compute...pkglist \ + /install/custom/netboot//compute...pkglist + + b) Edit your ``/install/custom/netboot//.pkglist`` and add below line :: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# + +2. Prepare postinstall scripts + + a) Specify postinstall scripts :: + + mkdir -p /install/custom/netboot// + + cp /opt/xcat/share/xcat/netboot//.postinstall \ + /install/custom/netboot// + + chmod +x /install/custom/netboot//.postinstall + + b) Edit ``/install/custom/netboot//.postinstall`` and add below line in the end :: + + /install/postscripts/mlnxofed_ib_install \ + -p /install// -i $1 -n genimage + + +3. Set the related osimage using the customized pkglist and compute.postinsall + +* [RHEL/SLES] :: + + chdef -t osimage -o --netboot-compute \ + pkglist=/install/custom/netboot//compute...pkglist \ + postinstall=/install/custom/netboot//.postinstall + +* [Ubuntu] :: + + chdef -t osimage -o --netboot-compute \ + postinstall=/install/custom/netboot//.postinstall + +4. Generate and package image for diskless installation :: + + genimage --netboot-compute + packimage --netboot-compute + +5. Install node :: + + nodeset osimage=--netboot-compute + rsetboot net + rpower reset diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.rst new file mode 100644 index 000000000..b0294be20 --- /dev/null +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.rst @@ -0,0 +1,34 @@ +Preparation +=========== + +Obtain the Mellanox OFED ISO file from `Mellanox official site `_ and put it into one place under ``/install`` directory depending on your need. + +**[NOTE]** + +* Mellanox provides OFED drivers in **tarball** and **iso** formats. xCAT only supports the **iso** format at this time. +* Mellanox provides different OFED ISOs depending on operating system and machine architecture, named like MLNX_OFED_LINUX----.iso, you should download correct one according your environment. + +Copy **mlnxofed_ib_install.v2** into ``/install/postscripts`` and change name to **mlnxofed_ib_install** :: + + cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install.v2 \ + /install/postscripts/mlnxofed_ib_install + + chmod +x /install/postscripts/mlnxofed_ib_install + +Some options of mlnxofed_ib_install should be assigned values when it's used. +These options are: + +* **-p**: [required]--the directory where the OFED iso file is located +* **-m**: [optional]--the mlnxofed_ib_install invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED iso. Use this option to pass arguments to the ``mlnxofedinstall``. You must include ``-end-`` at the completion of the options to distinguish the option list. if you don't pass any argument to ``mlnxofedinstall``, defualt value ``--without-32bit --without-fw-update --force`` will be passed to ``mlnxofedinstall`` by xCAT. +* **-i**: [required for diskless]--the image root path +* **-n**: [required for diskless]--nodeset status, the value is 'genimage'. + +In general you can use ``mlnxofed_ib_install`` like below :: + + mlnxofed_ib_install -p /install// + +If need to pass ``--without-32bit --without-fw-update --add-kernel-support --force`` to ``mlnxofedinstall`` in diskless scenario, refer to below command :: + + mlnxofed_ib_install -p /install// \ + -m --without-32bit --without-fw-update --add-kernel-support --force -end- \ + -i / -n genimage diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_usage.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_usage.rst index 2029425c8..be8532360 100644 --- a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_usage.rst +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_usage.rst @@ -1,22 +1,10 @@ -The Usage of mlnxofed_ib_install.v2 (Recommend) -=============================================== +Using mlnxofed_ib_install.v2 (Recommend) +========================================= -Configuration for Diskfull Installation ---------------------------------------- -.. include:: ./mlnxofed_ib_install_common_usage.rst - :start-after: BEGIN_Preparation_For_V2 - :end-before: END_Preparation_For_V2 +.. toctree:: + :maxdepth: 1 -.. include:: ./mlnxofed_ib_install_common_usage.rst - :start-after: BEGIN_Diskfull_step_For_V2 - :end-before: END_Diskfull_step_For_V2 - -Configuration for Diskless Installation ---------------------------------------- -.. include:: ./mlnxofed_ib_install_common_usage.rst - :start-after: BEGIN_Preparation_For_V2 - :end-before: END_Preparation_For_V2 - -.. include:: ./mlnxofed_ib_install_common_usage.rst - :start-after: BEGIN_Diskless_step_For_V2 - :end-before: END_Diskless_step_For_V2 + mlnxofed_ib_install_v2_preparation.rst + mlnxofed_ib_install_v2_diskful.rst + mlnxofed_ib_install_v2_diskless.rst + diff --git a/docs/source/advanced/networks/infiniband/network_configuration.rst b/docs/source/advanced/networks/infiniband/network_configuration.rst index ce9707d02..bd8eca1db 100644 --- a/docs/source/advanced/networks/infiniband/network_configuration.rst +++ b/docs/source/advanced/networks/infiniband/network_configuration.rst @@ -1,13 +1,11 @@ IB Network Configuration ======================== -XCAT provided two sample postscripts - configiba.1port and configiba.2ports to configure the IB adapter before XCAT 2.8, these tow scripts still work **but will be in maintenance mode**. +xCAT provides a script ``configib`` to help configure the Infiniband adapters on the compute nodes. -A new postscript ``/install/postscripts/configib`` is shipped with XCAT 2.8, the ``configib`` postscript works with the new "nics" table and ``confignic`` postscript which is introduced in XCAT 2.8 also. XCAT recommends you to use new ``configib`` script from now on. +The Infiniband adapter is considered an additional interface for xCAT. The process for configuring Infiniband adapters complies with the process of :doc:`Configure Additional Network Interfaces <../../../../guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/cfg_second_adapter>`. -IB Interface is a kind of additional adapters for XCAT, so the process of configuring Mellanox IB interface complies with the process of :doc:`Configure Additional Network Interfaces <../../../../guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/cfg_second_adapter>`. - -Below are an simple example to configure Mellanox IB in ubuntu14.4.1 on p8le +Below are an simple example to configure Mellanox IB in Ubuntu 14.04.1 on Power8 LE If your target Mellanox IB adapter has 2 ports, and you plan to give port ib0 4 different IPs, 2 are IPV4 (20.0.0.3 and 30.0.0.3) and another 2 are IPV6 (1:2::3 and 2:2::3). @@ -20,18 +18,19 @@ If your target Mellanox IB adapter has 2 ports, and you plan to give port ib0 4 2. Define IPs for ib0 :: - chdef nicips.ib0="20.0.0.3|30.0.0.3|1:2::3|2:2::3" nicnetworks.ib0="ib0ipv41|ib0ipv42|ib0ipv61|ib0ipv62" nictypes.ib0="Infiniband" + chdef nicips.ib0="20.0.0.3|30.0.0.3|1:2::3|2:2::3" \ + nicnetworks.ib0="ib0ipv41|ib0ipv42|ib0ipv61|ib0ipv62" nictypes.ib0="Infiniband" 3. Configure ib0 -Configure during node installation :: +* To configure during node installation :: chdef -p postscripts="confignics --ibaports=2" nodeset osimage= rsetboot net rpower reset -Configure on a node which has have operating system :: +* To configure on a node which has had operating system :: updatenode -P "confignics --ibaports=2" diff --git a/docs/source/advanced/networks/infiniband/switch_configuration.rst b/docs/source/advanced/networks/infiniband/switch_configuration.rst index 8905fddef..96c8cd0a7 100644 --- a/docs/source/advanced/networks/infiniband/switch_configuration.rst +++ b/docs/source/advanced/networks/infiniband/switch_configuration.rst @@ -1,10 +1,10 @@ IB Switch Configuration ======================= -Setup the XCAT Database +Setup the xCAT Database ----------------------- -The Mellanox Switch is only supported in XCAT Release 2.7 or later. +The Mellanox Switch is only supported in xCAT Release 2.7 or later. Add the switch ip address in the ``/etc/hosts`` file @@ -21,7 +21,7 @@ The switches table will look like this: :: #switch,...,sshusername,sshpassword,switchtype,.... "mswitch",,,,,,,"admin","admin","MellanoxIB",, -If there is only one admin and one password for all the switches then put the entry in the XCAT passwd table for the admin id and password to use to login. :: +If there is only one admin and one password for all the switches then put the entry in the xCAT passwd table for the admin id and password to use to login. :: tabch key=mswitch passwd.username=admin passwd.password=admin @@ -33,7 +33,7 @@ The passwd table will look like this: :: Setup ssh connection to the Mellanox Switch ------------------------------------------- -To run commands like xdsh and script to the Mellanox Switch, we need to setup ssh to run without prompting for a password to the Mellanox Switch. To do this, first you must add a configuration file. This configuration file is NOT needed for XCAT 2.8 and later. :: +To run commands like xdsh and script to the Mellanox Switch, we need to setup ssh to run without prompting for a password to the Mellanox Switch. To do this, first you must add a configuration file. This configuration file is NOT needed for xCAT 2.8 and later. :: mkdir -p /var/opt/xcat/IBSwitch/Mellanox cd /var/opt/xcat/IBSwitch/Mellanox @@ -72,13 +72,14 @@ Use the following command to consolidate the syslog to the Management Node or Se Configure xdsh for Mellanox Switch ---------------------------------- -To run xdsh commands to the Mellanox Switch, you must use the --devicetype input flag to xdsh. In addition, for XCAT versions less than 2.8, you must add a configuration file, please see **"Setup ssh connection to the Mellanox Switch"** section. +To run xdsh commands to the Mellanox Switch, you must use the --devicetype input flag to xdsh. In addition, for xCAT versions less than 2.8, you must add a configuration file, please see `Setup ssh connection to the Mellanox Switch`_ section. -For the Mellanox Switch the --devicetype is "IBSwitch::Mellanox". See xdsh man page: TODO `http://xcat.sourceforge.net/man1/xdsh.1.html `_ for details. +For the Mellanox Switch the ``--devicetype`` is ``IBSwitch::Mellanox``. See :doc:`xdsh man page <../../../guides/admin-guides/references/man/xdsh.1.html>` for details. Now you can run the switch commands from the mn using xdsh. For example: :: - xdsh mswitch -l admin --devicetype IBSwitch::Mellanox 'enable;configure terminal;show ssh server host-keys' + xdsh mswitch -l admin --devicetype IBSwitch::Mellanox \ + 'enable;configure terminal;show ssh server host-keys' Commands Supported for the Mellanox Switch ------------------------------------------ @@ -115,7 +116,7 @@ For doing other tasks on the switch, use xdsh. For example: :: Interactive commands are not supported by xdsh. For interactive commands, use ssh. -Send SNMP traps to XCAT Management Node +Send SNMP traps to xCAT Management Node --------------------------------------- First, get `http://www.mellanox.com/related-docs/prod_ib_switch_systems/MELLANOX-MIB.zip `_ , unzip it. Copy the mib file MELLANOX-MIB.txt to ``/usr/share/snmp/mibs`` directory on the mn and sn (if the sn is the snmp trap destination.) diff --git a/docs/source/advanced/networks/infiniband/ufm_configuration.rst b/docs/source/advanced/networks/infiniband/ufm_configuration.rst index d4dfd458a..3c64c079e 100644 --- a/docs/source/advanced/networks/infiniband/ufm_configuration.rst +++ b/docs/source/advanced/networks/infiniband/ufm_configuration.rst @@ -2,12 +2,12 @@ UFM Configuration ================= -UFM server are just regular Linix boxes with UFM installed. XCAT can help install and configure the UFM servers. The XCAT mn can send remote command to UFM through xdsh. It can also collect SNMP traps and syslogs from the UFM servers. +UFM server are just regular Linux boxes with UFM installed. xCAT can help install and configure the UFM servers. The XCAT mn can send remote command to UFM through xdsh. It can also collect SNMP traps and syslogs from the UFM servers. Setup xdsh to UFM and backup ---------------------------- -Assume we have two hosts with UFM installed, called host1 and host2. First define the two hosts in the XCAT cluster. Usually the network that the UFM hosts are in a different than the compute nodes, make sure to assign correct servicenode and xcatmaster in the noderes table. And also make sure to assign correct os and arch values in the nodetype table for the UFM hosts. For example: :: +Assume we have two hosts with UFM installed, called host1 and host2. First define the two hosts in the xCAT cluster. Usually the network that the UFM hosts are in a different than the compute nodes, make sure to assign correct servicenode and xcatmaster in the noderes table. And also make sure to assign correct os and arch values in the nodetype table for the UFM hosts. For example: :: mkdef -t node -o host1,host2 groups=ufm,all os=sles11.1 arch=x86_64 servicenode=10.0.0.1 xcatmaster=10.0.0.1 @@ -22,16 +22,16 @@ Now we can run xdsh on the UFM hosts. :: Consolidate syslogs ------------------- -Run the following command to make the UFM hosts to send the syslogs to the XCAT mn: :: +Run the following command to make the UFM hosts to send the syslogs to the xCAT mn: :: updatenode ufm -P syslog -To test, run the following commands on the UFM hosts and see if the XCAT MN receives the new messages in /var/log/messages :: +To test, run the following commands on the UFM hosts and see if the xCAT MN receives the new messages in /var/log/messages :: logger xCAT "This is a test" -Send SNMP traps to XCAT Management Node +Send SNMP traps to xCAT Management Node --------------------------------------- You need to have the Advanced License for UFM in order to send SNMP traps. From e2563c1780865d0c1cd7cd8924f07e6da5833b29 Mon Sep 17 00:00:00 2001 From: huweihua Date: Mon, 19 Oct 2015 04:05:51 -0400 Subject: [PATCH 5/5] add some example --- .../mlnxofed_ib_install_v1_diskful.rst | 25 ++++++++++++ .../mlnxofed_ib_install_v1_diskless.rst | 29 +++++++++++++- .../mlnxofed_ib_install_v2_diskful.rst | 38 ++++++++++++++++--- .../mlnxofed_ib_install_v2_diskless.rst | 33 +++++++++++++--- .../mlnxofed_ib_install_v2_preparation.rst | 4 +- 5 files changed, 115 insertions(+), 14 deletions(-) diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskful.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskful.rst index 3eb63a99c..37da6def7 100644 --- a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskful.rst +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskful.rst @@ -21,6 +21,31 @@ Configuration for Diskful Installation chdef -t osimage -o --install-compute \ pkglist=/install/custom/install//compute...pkglist + Take RHEL 6.4 on x86_64 for example :: + + cp /opt/xcat/share/xcat/install/rh/compute.rhels6.x86_64.pkglist \ + /install/custom/install/rh/compute.rhels6.x86_64.pkglist + + Edit the ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` and add below line + ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#`` + + Then ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` looks like below :: + + #Please make sure there is a space between @ and group name + #INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist# + ntp + nfs-utils + net-snmp + rsync + yp-tools + openssh-server + util-linux-ng + + Then modify related osimage :: + + chdef -t osimage -o rhels6.4-x86_64-install-compute \ + pkglist=/install/custom/install/rh/compute.rhels6.x86_64.pkglist + 3. Install node :: nodeset osimage=--install-compute diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskless.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskless.rst index e56a719b4..fd1173186 100644 --- a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskless.rst +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v1_diskless.rst @@ -3,16 +3,34 @@ Configuration for Diskless Installation 1. Specify dependence package **[required for RHEL and SLES]** - a) Copy the pkglist to the custom directory :: + a) Copy a correct pkglist file **shipped by xCAT** according your environment to the ``/install/custom/netboot//`` directory :: cp /opt/xcat/share/xcat/netboot//compute...pkglist \ /install/custom/netboot//compute...pkglist b) Edit your ``/install/custom/netboot//.pkglist`` and add one line ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist#`` + Take RHEL 6.4 on x86_64 for example :: + + cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.pkglist \ + /install/custom/netboot/rh/compute.rhels6.x86_64.pkglist + + Edit the ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` and add below line + ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#`` + + Then ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` looks like below :: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist# + bash + nfs-utils + openssl + dhclient + ..... + + 2. Prepare postinstall scripts - a) Specify postinstall scripts:: + a) Specify postinstall script **shipped by xCAT** :: mkdir -p /install/custom/netboot// @@ -21,6 +39,13 @@ Configuration for Diskless Installation chmod +x /install/custom/netboot//.postinstall + Take RHEL 6.4 on x86_64 for example :: + + mkdir -p /install/custom/netboot/rh/ + cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.postinstall \ + /install/custom/netboot/rh/ + chmod +x /install/custom/netboot/rh/compute.rhels6.x86_64.postinstall + b) Edit ``/install/custom/netboot//.postinstall`` and add below line in the end: :: installroot=$1 ofeddir=/install/post/otherpkgs///ofed/ \ diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskful.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskful.rst index 6c09b4043..b51ae7299 100644 --- a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskful.rst +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskful.rst @@ -7,20 +7,48 @@ Configuration for Diskful Installation 2. Specify dependence package **[required for RHEL and SLES]** - a) Copy a correct pkglist file shipped by xCAT according your environment to the ``/install/custom/install//`` directory :: + a) Copy a correct pkglist file **shipped by xCAT** according your environment to the ``/install/custom/install//`` directory, these pkglist files are located under ``/opt/xcat/share/xcat/install//`` :: cp /opt/xcat/share/xcat/install//compute...pkglist \ /install/custom/install//compute...pkglist - b) Edit your ``/install/custom/install//compute...pkglist`` and add below line:: - - #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# - + b) Edit your ``/install/custom/install//compute...pkglist`` and add one line + + ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist#`` + + You can check directory ``/opt/xcat/share/xcat/ib/netboot//`` and choose one correct ``ib...pkglist`` according your environment. + + c) Make the related osimage use the customized pkglist :: chdef -t osimage -o --install-compute \ pkglist=/install/custom/install//compute...pkglist + + Take RHEL 6.4 on x86_64 for example :: + cp /opt/xcat/share/xcat/install/rh/compute.rhels6.x86_64.pkglist \ + /install/custom/install/rh/compute.rhels6.x86_64.pkglist + + Edit the ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` and add below line + ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#`` + + Then ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` looks like below :: + + #Please make sure there is a space between @ and group name + #INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist# + ntp + nfs-utils + net-snmp + rsync + yp-tools + openssh-server + util-linux-ng + + Then modify related osimage :: + + chdef -t osimage -o rhels6.4-x86_64-install-compute \ + pkglist=/install/custom/install/rh/compute.rhels6.x86_64.pkglist + 3. Install node :: nodeset osimage=--install-compute diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskless.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskless.rst index 68b9a2960..14aefcf76 100644 --- a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskless.rst +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskless.rst @@ -3,18 +3,34 @@ Configuration for Diskless Installation 1. Specify dependence package **[required for RHEL and SLES]** - a) Copy a correct pkglist file shipped by xCAT according your environment to the ``/install/custom/netboot//`` directory :: + a) Copy a correct pkglist file **shipped by xCAT** according your environment to the ``/install/custom/netboot//`` directory :: cp /opt/xcat/share/xcat/netboot//compute...pkglist \ /install/custom/netboot//compute...pkglist - b) Edit your ``/install/custom/netboot//.pkglist`` and add below line :: + b) Edit your ``/install/custom/netboot//.pkglist`` and add below line + ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist#`` - #INCLUDE:/opt/xcat/share/xcat/ib/netboot//ib...pkglist# + Take RHEL 6.4 on x86_64 for example :: + + cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.pkglist \ + /install/custom/netboot/rh/compute.rhels6.x86_64.pkglist + + Edit the ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` and add below line + ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#`` + + Then ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` looks like below :: + + #INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist# + bash + nfs-utils + openssl + dhclient + ..... 2. Prepare postinstall scripts - a) Specify postinstall scripts :: + a) Specify a correct postinstall script **shipped by xCAT** :: mkdir -p /install/custom/netboot// @@ -23,12 +39,19 @@ Configuration for Diskless Installation chmod +x /install/custom/netboot//.postinstall + Take RHEL 6.4 on x86_64 for example :: + + mkdir -p /install/custom/netboot/rh/ + cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.postinstall \ + /install/custom/netboot/rh/ + chmod +x /install/custom/netboot/rh/compute.rhels6.x86_64.postinstall + b) Edit ``/install/custom/netboot//.postinstall`` and add below line in the end :: /install/postscripts/mlnxofed_ib_install \ -p /install// -i $1 -n genimage - + 3. Set the related osimage using the customized pkglist and compute.postinsall * [RHEL/SLES] :: diff --git a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.rst b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.rst index b0294be20..fcec74f83 100644 --- a/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.rst +++ b/docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.rst @@ -15,8 +15,8 @@ Copy **mlnxofed_ib_install.v2** into ``/install/postscripts`` and change name to chmod +x /install/postscripts/mlnxofed_ib_install -Some options of mlnxofed_ib_install should be assigned values when it's used. -These options are: +``mlnxofed_ib_install`` has some options, **'-p' is always needed**. +Below are the details of these options: * **-p**: [required]--the directory where the OFED iso file is located * **-m**: [optional]--the mlnxofed_ib_install invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED iso. Use this option to pass arguments to the ``mlnxofedinstall``. You must include ``-end-`` at the completion of the options to distinguish the option list. if you don't pass any argument to ``mlnxofedinstall``, defualt value ``--without-32bit --without-fw-update --force`` will be passed to ``mlnxofedinstall`` by xCAT.