2
0
mirror of https://github.com/xcat2/xcat-core.git synced 2025-05-29 09:13:08 +00:00

Merge pull request #1945 from whowutwut/mellanox_documentation_improvements

Mellanox IB documentation clarification for OFED Installation Step
This commit is contained in:
Weihua Hu 2016-10-10 14:45:06 +08:00 committed by GitHub
commit 0c951db82b
14 changed files with 257 additions and 612 deletions

View File

@ -1,11 +0,0 @@
IB Driver Preparation and Installation
======================================
xCAT provides sample postscripts to help you install the Mellanox OpenFabrics Enterprise Distribution (OFED) InfiniBand Driver. These scripts are located in ``opt/xcat/share/xcat/ib/scripts/Mellanox/``. You can use these scripts directly or change them to satisfy your own environment. **xCAT 2.11 drops support of mlnxofed_ib_install and recommends using version 2 of the script: mlnxofed_ib_install.v2**.
.. toctree::
:maxdepth: 2
mlnxofed_ib_install_v2_usage.rst
mlnxofed_ib_install_v1_usage.rst

View File

@ -1,13 +1,18 @@
InfiniBand (Mellanox)
=====================
xCAT offers a certain degree support for Mellanox InfiniBand product, it help you to configurate Mellanox InfiniBand products easily. For more information about Mellanox InfiniBand, please refer to `Mellanox official site <http://www.mellanox.com/>`_.
xCAT has the ability to help with Mellanox InfiniBand (IB) adapter installation and network configuration as part of the node provisioning process.
.. toctree::
:maxdepth: 2
driver_and_installation.rst
mlnxofed_ib_install_v2.rst
network_configuration.rst
switch_configuration.rst
ufm_configuration.rst
firmware_updates.rst
For more information about Mellanox products, refer to http://www.mellanox.com.

View File

@ -0,0 +1,11 @@
Configuration
=============
The process to configure the osimage to install the Mellanox OFED Drivers for Diskful and Diskless scenarios are outlined below.
.. toctree::
:maxdepth: 2
mlnxofed_ib_install_v2_diskful.rst
mlnxofed_ib_install_v2_diskless.rst

View File

@ -1,53 +0,0 @@
Configuration for Diskful Installation
======================================
1. Set script ``mlnxofed_ib_install`` as postbootscript ::
chdef <node> -p postbootscripts=mlnxofed_ib_install
2. Specify dependence package **[required for RHEL and SLES]**
a) Copy the pkglist to the custom directory ::
cp /opt/xcat/share/xcat/install/<ostype>/compute.<osver>.<arch>.pkglist \
/install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist
b) Edit your /install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist and add one line::
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<osver>.<arch>.pkglist#
c) Make the related osimage use the customized pkglist ::
chdef -t osimage -o <osver>-<arch>-install-compute \
pkglist=/install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist
Take RHEL 6.4 on x86_64 for example ::
cp /opt/xcat/share/xcat/install/rh/compute.rhels6.x86_64.pkglist \
/install/custom/install/rh/compute.rhels6.x86_64.pkglist
Edit the ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` and add below line
``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#``
Then ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` looks like below ::
#Please make sure there is a space between @ and group name
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#
ntp
nfs-utils
net-snmp
rsync
yp-tools
openssh-server
util-linux-ng
Then modify related osimage ::
chdef -t osimage -o rhels6.4-x86_64-install-compute \
pkglist=/install/custom/install/rh/compute.rhels6.x86_64.pkglist
3. Install node ::
nodeset <node> osimage=<osver>-<arch>-install-compute
rsetboot <node> net
rpower <node> reset

View File

@ -1,77 +0,0 @@
Configuration for Diskless Installation
=======================================
1. Specify dependence package **[required for RHEL and SLES]**
a) Copy a correct pkglist file **shipped by xCAT** according your environment to the ``/install/custom/netboot/<ostype>/`` directory ::
cp /opt/xcat/share/xcat/netboot/<ostype>/compute.<osver>.<arch>.pkglist \
/install/custom/netboot/<ostype>/compute.<osver>.<arch>.pkglist
b) Edit your ``/install/custom/netboot/<ostype>/<profile>.pkglist`` and add one line ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<osver>.<arch>.pkglist#``
Take RHEL 6.4 on x86_64 for example ::
cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.pkglist \
/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist
Edit the ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` and add below line
``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#``
Then ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` looks like below ::
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#
bash
nfs-utils
openssl
dhclient
.....
2. Prepare postinstall scripts
a) Specify postinstall script **shipped by xCAT** ::
mkdir -p /install/custom/netboot/<ostype>/
cp /opt/xcat/share/xcat/netboot/<ostype>/<profile>.postinstall \
/install/custom/netboot/<ostype>/
chmod +x /install/custom/netboot/<ostype>/<profile>.postinstall
Take RHEL 6.4 on x86_64 for example ::
mkdir -p /install/custom/netboot/rh/
cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.postinstall \
/install/custom/netboot/rh/
chmod +x /install/custom/netboot/rh/compute.rhels6.x86_64.postinstall
b) Edit ``/install/custom/netboot/<ostype>/<profile>.postinstall`` and add below line in the end: ::
installroot=$1 ofeddir=/install/post/otherpkgs/<osver>/<arch>/ofed/ \
NODESETSTATE=genimage mlnxofed_options=--force /install/postscripts/mlnxofed_ib_install
3. Set the related osimage use the customized pkglist and customized compute.postinsall
* [RHEL/SLES] ::
chdef -t osimage -o <osver>-<arch>-netboot-compute \
pkglist=/install/custom/netboot/<ostype>/compute.<osver>.<arch>.pkglist \
postinstall=/install/custom/netboot/<ostype>/<profile>.postinstall
* [Ubuntu] ::
chdef -t osimage -o <osver>-<arch>-netboot-compute \
postinstall=/install/custom/netboot/<ostype>/<profile>.postinstall
4. Generate and package image for diskless installation ::
genimage <osver>-<arch>-netboot-compute
packimage <osver>-<arch>-netboot-compute
5. Install node ::
nodeset <nodename> osimage=<osver>-<arch>-netboot-compute
rsetboot <nodename> net
rpower <nodename> reset

View File

@ -1,42 +0,0 @@
Preparation
===========
Obtain the Mellanox OFED ISO file from `Mellanox official site <http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>`_ and mount it onto suggested target location on the xCAT MN according your OS and ARCH: ::
mkdir -p /install/post/otherpkgs/<osver>/<arch>/ofed
mount -o loop MLNX_OFED_LINUX-<packver1>-<packver2>-<osver>-<arch>.iso \
/install/post/otherpkgs/<osver>/<arch>/ofed
Take sles11 sp1 for x86_64 as an example ::
mkdir -p /install/post/otherpkgs/sles11.1/x86_64/ofed/
mount -o loop MLNX_OFED_LINUX-1.5.3-3.0.0-sles11sp1-x86_64.iso \
/install/post/otherpkgs/sles11.1/x86_64/ofed/
Take Ubuntu14.4.1 for Power8 LE as an example ::
mkdir -p /install/post/otherpkgs/ubuntu14.04.1/ppc64el/ofed
mount -o loop MLNX_OFED_LINUX-2.3-1.0.1-ubuntu14.04-ppc64le.iso \
/install/post/otherpkgs/ubuntu14.04.1/ppc64el/ofed
**[NOTE]**
* Mellanox provides OFED files with **tarball** and **ISO** two format, but for xCAT, we just support **ISO** format right now.
Copy Sample script **mlnxofed_ib_install** shipped by xCAT into ``/install/postscripts`` before using, such as ::
cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install \
/install/postscripts/mlnxofed_ib_install
The **mlnxofed_ib_install** invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED ISO. If you want to pass the argument to ``mlnxofedinstall``, you set the argument to the environment variable ``mlnxofed_options`` which could be read by **mlnxofed_ib_install**. For example: PPE requires the 32-bit version of libibverbs, but the default **mlnxofed_ib_install** will remove all the old ib related packages at first including the 32-bit version of libibverbs. In this case, you can set the environment variable ``mlnxofed_options=--force`` when running the **mlnxofed_ib_install**. For diskful, you should put the environment variable ``mlnxofed_options=--force`` in mypostscript.tmpl. myposcript.tmpl is in ``/opt/xcat/share/xcat/templates/mypostscript/`` by default. When customize it, you should copy it into ``/install/postscripts/myposcript.tmpl`` ::
mlnxofed_options='--force'
export mlnxofed_options

View File

@ -1,9 +0,0 @@
Using mlnxofed_ib_install (drops support)
=========================================
.. toctree::
:maxdepth: 1
mlnxofed_ib_install_v1_preparation.rst
mlnxofed_ib_install_v1_diskful.rst
mlnxofed_ib_install_v1_diskless.rst

View File

@ -0,0 +1,13 @@
Mellanox OFED Installation Script
=================================
Mellanox provides a tested and packaged version of the OpenFabrics Enterprise Distribution (OFED) driver, named Mellanox OFED (MLNX_OFED). To assist with the installation of the MLNX_OFED driver, xCAT provids a sample postscript: ``mlnxofed_ib_install.v2``.
.. toctree::
:maxdepth: 2
mlnxofed_ib_install_v2_preparation.rst
mlnxofed_configuration.rst
mlnxofed_ib_verified_scenario_matrix.rst
mlnxofed_ib_known_issue.rst

View File

@ -1,169 +1,44 @@
Configuration for Diskful Installation
=======================================
1. Set script ``mlnxofed_ib_install`` as ``postbootscripts`` or ``postscripts`` ::
chdef <node> -p postbootscripts="mlnxofed_ib_install -p /install/<subpath>/<MLNX_OFED_LINUX.iso>"
Or ::
chdef <node> -p postscripts="mlnxofed_ib_install -p /install/<subpath>/<MLNX_OFED_LINUX.iso>"
xCAT simulates completely the way Mellanox scripts work by using ``postbootscripts``. This way need to reboot after drive installation to make Mellanox drivers work reliably just like Mellanox suggested. If you want to use the reboot after operating system installation to avoid reboot twice, you can using ``postscripts`` attribute to install Mellanox drivers. This way has been verified in limited scenarios. For more information please refer to :doc:`The Scenarios Have Been Verified </advanced/networks/infiniband/mlnxofed_ib_verified_scenario_matrix>`. You can try this way in other else scenarios if you needed.
2. Specify dependency package
Some dependencies need to be installed before running Mellanox scripts. These dependencies are different between different scenario. xCAT configurates these dependency packages by using ``pkglist`` attribute of ``osimage`` definition. Please refer to :doc:`Add Additional Software Packages </guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/additional_pkg>` for more information::
# lsdef -t osimage <os>-<arch>-install-compute
Object name: <os>-<arch>-install-compute
imagetype=linux
....
pkgdir=/<os packages directory>
pkglist=/<os packages list directory>/compute.<os>.<arch>.pkglist
....
You can append the ib dependency packages list in the end of ``/<os packages list directory>/compute.<os>.<arch>.pkglist`` directly like below: ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
@base
@x11
openssl
ntp
rsyn
#ib part
createrepo
kernel-devel
kernel-source
....
Or if you want to isolate InfiniBand dependency packages list into a separate file, after you edit this file, you can append the file in ``/<os packages list directory>/compute.<os>.<arch>.pkglist`` like below way: ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
@base
@x11
openssl
ntp
rsyn
#INCLUDE:/<ib pkglist path>/<you ib pkglist file>#
xCAT has shipped some ib pkglist files under ``/opt/xcat/share/xcat/ib/netboot/<ostype>/``, these pkglist files have been verified in sepecific scenario. Please refer to :doc:`The Scenarios Have Been Verified </advanced/networks/infiniband/mlnxofed_ib_verified_scenario_matrix>` to judge if you can use it directly in your environment. If so, you can use it like below: ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
@base
@x11
openssl
ntp
rsyn
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<os>.<arch>.pkglist#
Take rhels7.2 on ppc64le for example: ::
# lsdef -t osimage rhels7.2-ppc64le-install-compute
Object name: rhels7.2-ppc64le-install-compute
imagetype=linux
osarch=ppc64le
osdistroname=rhels7.2-ppc64le
osname=Linux
osvers=rhels7.2
otherpkgdir=/install/post/otherpkgs/rhels7.2/ppc64le
pkgdir=/install/rhels7.2/ppc64le
pkglist=/install/custom/install/rh/compute.rhels7.ib.pkglist
profile=compute
provmethod=install
template=/opt/xcat/share/xcat/install/rh/compute.rhels7.tmpl
**[Note]**: If the osimage definition was generated by xCAT command ``copycds``, default value ``/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist`` was assigned to ``pkglist`` attribute. ``/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist`` is the sample pkglist shipped by xCAT, recommend to make a copy of this sample and using the copy in real environment. In the above example, ``/install/custom/install/rh/compute.rhels7.ib.pkglist`` is a copy of ``/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist``. ::
# cat /install/custom/install/rh/compute.rhels7.ib.pkglist
#Please make sure there is a space between @ and group name
wget
ntp
nfs-utils
net-snmp
rsync
yp-tools
openssh-server
util-linux
net-tools
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels7.ppc64le.pkglist#
3. Install node ::
nodeset <node> osimage=<osver>-<arch>-install-compute
rsetboot <node> net
rpower <node> reset
After steps above, you can login target node and find the Mellanox InfiniBand drives are located under ``/lib/modules/<kernel_version>/extra/``.
Issue ``ibv_devinfo`` command you can get the InfiniBand apater information ::
# ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 10.14.2036
node_guid: f452:1403:0076:10e0
sys_image_guid: f452:1403:0076:10e0
vendor_id: 0x02c9
vendor_part_id: 4113
hw_ver: 0x0
board_id: IBM1210111019
phys_port_cnt: 2
Device ports:
port: 1
state: PORT_INIT (2)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 65535
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 65535
port_lmc: 0x00
link_layer: InfiniBand
Using ``service openibd status`` to verify if openibd works well. Below is the output in rhels7.2. ::
# service openibd status
HCA driver loaded
Configured IPoIB devices:
ib0 ib1
Currently active IPoIB devices:
Configured Mellanox EN devices:
Currently active Mellanox devices:
The following OFED modules are loaded:
rdma_ucm
rdma_cm
ib_addr
ib_ipoib
mlx4_core
mlx4_ib
mlx4_en
mlx5_core
mlx5_ib
ib_uverbs
ib_umad
ib_ucm
ib_sa
ib_cm
ib_mad
ib_core
Diskful Installation
====================
#. Prepare dependency packages in the pkglist
In order for the Mellanox installation script to execute successfully, certain dependency packages are required to be installed on the compute node. xCAT provides sample package list files to help resolve these dependencies. The samples are located at ``/opt/xcat/share/xcat/ib/netboot/<os>/``.
To use the ``/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels7.ppc64le.pkglist``, edit your existing ``pkglist`` file for the target osimage and add the following at the bottom: ::
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels7.ppc64le.pkglist#
#. Configure the ``mlnxofed_ib_install`` script to install the MNLX_OFED drivers
xCAT has a concept of postscripts that can be used to customize the node after the operating system is installed.
Mellanox recommends that the operating system is rebooted after the drivers are installed, so xCAT recommends using the ``postscripts`` attribute to avoid the need for a second reboot. To invoke the ``mlnxofed_ib_install`` as a postscript ::
chdef -t node -o <node_name> \
-p postscripts="mlnxofed_ib_install -p /install/<path-to>/<MLNX_OFED_LINUX.iso>"
**[kernel mismatch issue]** The Mellanox OFED ISO is built againt a series of specific kernel version. If the version of the linux kernel does not match any of the Mellanox offered pre-built kernel modules, you can pass the ``--add-kernel-support`` argument to the Mellanox installation script to build the kernel modules based on the version you are using. ::
chdef -t node -o <node_name> \
-p postscripts="mlnxofed_ib_install -p /install/<path-to>/<MLNX_OFED_LINUX.iso> \
-m --add-kernel-support -end-"
#. Provision the node
#. Verification
* The Mellanox IB drivers are located at: ``/lib/modules/<kernel_version>/extra/``
* Use the ``ibv_devinfo`` comamnd to obtain information about the InfiniBand adapter
* Check the status of ``openibd`` service
sysVinit: ::
service openibd status
systemd: ::
systemctl status openibd.service

View File

@ -1,169 +1,64 @@
Configuration for Diskless Installation
=======================================
1. Specify dependency package
Some dependencies need to be installed before running Mellanox scripts. These dependencies are different among different scenarios. xCAT can help user to install these dependency packages by adding these package names to the file specified by the ``pkglist`` attribute of the ``osimage`` definition. Please refer to :doc:`Add Additional Software Packages </guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/additional_pkg>` for more information::
# lsdef -t osimage <osver>-<arch>-netboot-compute
Object name: <osver>-<arch>-netboot-compute
imagetype=linux
....
pkgdir=/<os packages directory>
pkglist=/<os packages list directory>/compute.<os>.<arch>.pkglist
....
You can append the ib dependency packages list in the end of ``/<os packages list directory>/compute.<os>.<arch>.pkglist`` directly like below: ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
bash
nfs-utils
openssl
dhclient
kernel
.....
#ib part
createrepo
kernel-devel
kernel-source
....
Or if you want to isolate InfiniBand dependency packages list into a separate file, after you edit this file, you can append the file in ``/<os packages list directory>/compute.<os>.<arch>.pkglist`` like below way: ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
bash
nfs-utils
openssl
dhclient
kernel
.....
#INCLUDE:/<ib pkglist path>/<you ib pkglist file>#
xCAT ships some InfiniBand pkglist files under ``/opt/xcat/share/xcat/ib/netboot/<ostype>/``, these pkglist files have been verified in sepecific scenario. Please refer to :doc:`The Scenarios Have Been Verified </advanced/networks/infiniband/mlnxofed_ib_verified_scenario_matrix>` to judge if you can use it directly in your environment. If so, you can use it like below: ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
bash
nfs-utils
openssl
dhclient
kernel
.....
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<os>.<arch>.pkglist#
2. Prepare postinstall scripts
Edit ``postinstall`` script to trigger InfniBand drvices installation during ``genimage``. Using below command to find out where the ``postinstall`` script is defined. ::
# lsdef -t osimage <os>-<arch>-netboot-compute
Object name: <os>-<arch>-netboot-compute
....
postinstall=/<postinstall script path/compute.<os>.<arch>.postinstall
....
Edit ``/<postinstall script path/compute.<os>.<arch>.postinstall`` and add below line in the end. ::
/install/postscripts/mlnxofed_ib_install \
-p /install/<path>/<MLNX_OFED_LINUX.iso> -i $1 -n genimage
**[Note]** Mellanox OFED ISO was built against a series of certain kernael versions, If the version of linux kernel you are using does not match with any of the Mellanox offered pre-built kernel modules, you can pass ``--add-kernel-support`` command line argument to Mellanox OFED installation script to build these kernel modules base on the version of linux kernel you are using. The line added into ``<profile>.postinstall`` should like below. ::
/install/postscripts/mlnxofed_ib_install \
-p /install/<subpath>/<MLNX_OFED_LINUX.iso> -m --add-kernel-support -end- -i $1 -n genimage
Take rhels7.2 on ppc64le for example: ::
#lsdef -t osimage rhels7.2-ppc64le-netboot-compute
Object name: rhels7.2-ppc64le-netboot-compute
exlist=/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.exlist
imagetype=linux
osarch=ppc64le
osdistroname=rhels7.2-ppc64le
osname=Linux
osvers=rhels7.2
otherpkgdir=/install/post/otherpkgs/rhels7.2/ppc64le
permission=755
pkgdir=/install/rhels7.2/ppc64le
pkglist=/install/custom/netboot/rh/compute.rhels7.ppc64le.pkglist
postinstall=/install/custom/netboot/rh/compute.rhels7.ppc64le.ib.postinstall
profile=compute
provmethod=netboot
rootimgdir=/install/netboot/rhels7.2/ppc64le/compute
**[Note]**: If the osimage definition was generated by xCAT command ``copycds``, default value ``/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.pkglist`` was assigned to ``pkglist`` attribute. ``/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.pkglist`` is the sample pkglist shipped by xCAT, recommend to make a copy of this sample and using the copy in real environment. In the above example, ``/install/custom/netboot/rh/compute.rhels7.ppc64le.pkglist`` is a copy of ``/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.pkglist``. For the same reason, ``/install/custom/netboot/rh/compute.rhels7.ppc64le.ib.postinstall`` is a copy of ``/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.postinstall``.
``compute.rhels7.ppc64le.pkglist`` looks like below: ::
# cat /install/custom/netboot/rh/compute.rhels7.ppc64le.pkglist
bash
nfs-utils
openssl
dhclient
bc
......
lsvpd
irqbalance
procps-ng
parted
net-tools
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels7.ppc64le.pkglist#
``compute.rhels7.ppc64le.ib.postinstall`` looks like below: ::
# cat /install/custom/netboot/rh/compute.rhels7.ppc64le.ib.postinstall
#!/bin/sh
#-- Do not remove following line if you want to make use of CVS version tracking
.....
# [ -r $workdir/$profile.$ext ] && cat $workdir/$profile.$ext | grep -E '^[[:space:]]*#.*[[:space:]]\$Id' >> $installroot/etc/IMGVERSION
#done
/install/postscripts/mlnxofed_ib_install -p /install/ofed/MLNX_OFED_LINUX-3.2-2.0.0.0-rhel7.2-ppc64le.iso -i $1 -n genimage
3. Generate and package image for diskless installation ::
genimage <osver>-<arch>-netboot-compute
packimage <osver>-<arch>-netboot-compute
4. Install node ::
nodeset <nodename> osimage=<osver>-<arch>-netboot-compute
rsetboot <nodename> net
rpower <nodename> reset
After installation, you can login target ndoe and issue ``ibv_devinfo`` command to verify if your InfiniBand driver works well. if everything is fine, you can get the InfiniBand apater information. ::
# ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 10.14.2036
node_guid: f452:1403:0076:10e0
sys_image_guid: f452:1403:0076:10e0
vendor_id: 0x02c9
vendor_part_id: 4113
hw_ver: 0x0
board_id: IBM1210111019
phys_port_cnt: 2
Device ports:
port: 1
state: PORT_INIT (2)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 65535
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 65535
port_lmc: 0x00
link_layer: InfiniBand
Diskless Installation
=====================
#. Prepare dependency packages in the pkglist
In order for the Mellanox installation script to execute successfully, certain dependency packages are required to be installed on the compute node. xCAT provides sample package list files to help resolve these dependencies. The samples are located at ``/opt/xcat/share/xcat/ib/netboot/<os>/``.
To use the ``/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels7.ppc64le.pkglist``, edit your existing ``pkglist`` file for the target osimage and add the following at the bottom: ::
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels7.ppc64le.pkglist#
#. Configure the ``mlnxofed_ib_install`` script to install the MNLX_OFED drivers
Edit the ``postinstall`` script on the osimage to invoke the ``mlnxofed_ib_install`` install script.
For example, take ``rhels7.2-ppc64le-netboot-compute``:
#. Find the path to the ``postinstall`` script: ::
# lsdef -t osimage -o rhels7.2-ppc64le-netboot-compute -i postinstall
Object name: rhels7.2-ppc64le-netboot-compute
postinstall=/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.postinstall
#. Edit the ``/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.postinstall`` and add the following: ::
/install/postscripts/mlnxofed_ib_install \
-p /install/<path-to>/<MLNX_OFED_LINUX.iso> -i $1 -n genimage
*Note: The $1 is a argument that is passed to the the postinstall script at runtime.*
**[kernel mismatch issue]** The Mellanox OFED ISO is built againt a series of specific kernel version. If the version of the linux kernel does not match any of the Mellanox offered pre-built kernel modules, you can pass the ``--add-kernel-support`` argument to the Mellanox installation script to build the kernel modules based on the version you are using. ::
/install/postscripts/mlnxofed_ib_install \
-p /install/<path-to>/<MLNX_OFED_LINUX.iso> -m --add-kernel-support -end- \
-i $1 -n genimage
#. Generate the diskless image
Use the ``genimage`` command to generate the diskless image from the osimage definition ::
genimage <osimage>
Use the ``packimage`` command to pack the diskless image for deployment ::
packimage <osimage>
#. Provision the node
#. Verification
* The Mellanox IB drivers are located at: ``/lib/modules/<kernel_version>/extra/``
* Use the ``ibv_devinfo`` comamnd to obtain information about the InfiniBand adapter
* Check the status of ``openibd`` service
sysVinit: ::
service openibd status
systemd: ::
systemctl status openibd.service

View File

@ -1,46 +1,59 @@
Preparation
===========
Obtain the Mellanox OFED ISO
----------------------------
Obtain the Mellanox OFED ISO file from `Mellanox official Download Page <http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>`_ and put it into one place under ``/install`` directory depending on your need.
**[NOTE]**
* Mellanox provides OFED drivers in **tarball** and **ISO image** formats. xCAT only supports the **iso** format at this time.
* Mellanox provides different OFED ISOs depending on operating system and machine architecture, named like MLNX_OFED_LINUX-<packver1>-<packver2>-<osver>-<arch>.iso, you should download correct one according to your environment.
* Mellanox has some updates and known issues for echo OFED, please read `InfiniBand/VPI Software Overview <http://www.mellanox.com/page/software_overview_ib>`_ to understand these information.
* The Mellanox links offered above maybe outdate in future for Mellanox updates its web page, xCAT will keep updating for synchronization. If we don't update in time, please access `Mellanox web portal <http://www.mellanox.com>`_ to find ``Support/Education`` then ``InfiniBand/VPI Drivers`` lables.
Prepare Install Script
----------------------
**mlnxofed_ib_install.v2** is a sample script, its framework can help you install Mellanox drives easily. But in specific scenario, some detail need to be modified to meet requirement, such like dependency package list. It has been verified in limited scenarios and can work as solution in these scenarios. For these scenarions information please refer to :doc:`The Scenarios Have Been Verified </advanced/networks/infiniband/mlnxofed_ib_verified_scenario_matrix>`.
Copy **mlnxofed_ib_install.v2** into ``/install/postscripts`` and change name to **mlnxofed_ib_install** ::
cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install.v2 \
/install/postscripts/mlnxofed_ib_install
chmod +x /install/postscripts/mlnxofed_ib_install
``mlnxofed_ib_install`` has some options, **'-p' is always needed**.
Below are the details of these options:
* **-p**: [required]--the directory where the OFED iso file is located
* **-m**: [optional]--the mlnxofed_ib_install invokes a script ``mlnxofedinstall`` shipped by Mellanox OFED iso. Use this option to pass arguments to the ``mlnxofedinstall``. You must include ``-end-`` at the completion of the options to distinguish the option list. if you don't pass any argument to ``mlnxofedinstall``, **defualt value** ``--without-32bit --without-fw-update --force`` will be passed to ``mlnxofedinstall`` by xCAT.
* **-i**: [required for diskless]--the image root path
* **-n**: [required for diskless]--nodeset status, the value is 'genimage'
In general you can use ``mlnxofed_ib_install`` like below ::
mlnxofed_ib_install -p /install/<path>/<MLNX_OFED_LINUX.iso>
If need to pass ``--without-32bit --without-fw-update --add-kernel-support --force`` to ``mlnxofedinstall``, refer to below command. ::
mlnxofed_ib_install -p /install/<path>/<MLNX_OFED_LINUX.iso> \
-m --without-32bit --without-fw-update --add-kernel-support --force -end-
**[Note]** We recommend to update your firmware to the version Mellanox supported in its release notes to avoid unexpected problem when you install InfiniBand driver.
Preparation
===========
Download MLNX_OFED ISO
----------------------
**xCAT only supports installation using the ISO format.**
Download the Mellanox OFED ISO file `here (MLNX_OFED) <http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>`_.
Prepare Installation Script
---------------------------
The ``mlnxofed_ib_install.v2`` is a sample script intended to assist with the installation of the Mellanox OFED drivers. The following support matrix documents the limited number of scenarios that have been verified: :doc:`support matrix </advanced/networks/infiniband/mlnxofed_ib_verified_scenario_matrix>`.
#. Copy the ``mlnxofed_ib_install.v2`` to ``/install/postscripts``, renaming to ``mlnxofed_ib_install``. ::
cp /opt/xcat/share/xcat/ib/scripts/Mellanox/mlnxofed_ib_install.v2 \
/install/postscripts/mlnxofed_ib_install
# ensure the script has execute permission
chmod +x /install/postscripts/mlnxofed_ib_install
#. Familarize the options available for the xCAT ``mlnxofed_ib_install`` script.
+---------+------------------+----------------------------------------------------------+
| Option | Required | Description |
+=========+==================+==========================================================+
|``-p`` | Yes || The full path to the MLNX_OFED ISO image |
+---------+------------------+----------------------------------------------------------+
|``-m`` | No || Use this option to pass arguments to the Mellanox OFED |
| | || installation script ``mlnxofedinstall``. |
| | || |
| | || The special keyword ``-end-`` must be added to the end |
| | || of the string to mark the completion of the option list |
| | || option list. |
| | || |
| | || If nothing is specified, xCAT passes the the following |
| | || ``--without-32bit --with out-fw-update --force`` |
+---------+------------------+----------------------------------------------------------+
|``-i`` | For diskless || The image root path of the diskless image |
| | || |
+---------+------------------+----------------------------------------------------------+
|``-n`` | For diskless || nodeset status, value is ``genimage`` |
| | || |
+---------+------------------+----------------------------------------------------------+
A very basic usage of the install script: ::
/install/postscripts/mlnxofed_ib_install -p /install/<path-to>/<MLNX_OFED_LINUX.iso>
To pass the ``--add-kernel-support`` option to ``mlnxofedinstall``, use the following command: ::
/install/postscripts/mlnxofed_ib_install -p /install/<path-to>/<MLNX_OFED_LINUX.iso> \
-m --without-32bit --without-fw-update --add-kernel-support --force -end-

View File

@ -1,12 +0,0 @@
Using mlnxofed_ib_install.v2 (Recommend)
=========================================
.. toctree::
:maxdepth: 1
mlnxofed_ib_install_v2_preparation.rst
mlnxofed_ib_install_v2_diskful.rst
mlnxofed_ib_install_v2_diskless.rst
mlnxofed_ib_verified_scenario_matrix.rst
mlnxofed_ib_known_issue.rst

View File

@ -1,19 +1,19 @@
Known Issues
============
Preventing upgrade of the Mellanox Drivers
------------------------------------------
Known Issue 1
-------------
On RedHat operating systems, after the Mellanox drivers are installed, you may have a requirement to update your operating system to a later version.
Some operating systems may ship InfiniBand drivers that are higher version than the Mellanox drivers you have installed and therefor may update the existing drivers.
To prevent this from happening, add the following in the ``/etc/yum.conf`` ::
After you install mellanox derives in rhels7.2 successfully by xCAT, maybe you have new requirement to upgrade your operating system to higher version. In this case you probably hit such problem the IB adaptor drives shipped by operating system is higher than the Mellanox drives you have installed. That means the the Mellanox drives will be replaced by the drives shipped by operating system. If it's not the result you expect, you hope keep the Mellanox drives after operating system upgraded, please add below statement into ``/etc/yum.conf`` in your target node after you install mellanox derives successfully for the first time. ::
exclude=dapl* libib* ibacm infiniband* libmlx* librdma* opensm* ibutils*
Known Issue 2
-------------
Development packages in SLES
----------------------------
If you want to use ``--add-kernel-support`` attribute in sles12.1 and ppc64le scenario, you will find some dependency packages are not shipped by SLES Server DVDs, such like ``python-devel``, it's shipped in SDK DVDs. xCAT doesn't ship specific pkglist to support such scenario. If you have such requirement, please used ``otherpkglist`` and ``otherpkgs`` attributes to prepare dependency packages repository ahead. If you need help about ``otherpkglist`` and ``otherpkgs`` attributes, please refer to :doc:`Add Additional Software Packages </guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/additional_pkg>`.
If using the ``--add-kernel-support`` attribute on SLES operating systems, you may find problems with installing some dependency packages which are not shipped by the SLES server DVDs. The development rpms are provided by the SDK DVDs. Refer to :doc:`Add Additional Software Packages </guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/additional_pkg>` to configure the SDK repositories.

View File

@ -1,15 +1,52 @@
The Scenarios Have Been Verified
=================================
MLNX_OFED Support Matrix
========================
The following ISO images and attributs have been verified by the xCAT Team.
**RedHat Enterprise Linux**
+----------------------------------+-------------------------------------------------------------------+
| RHEL 7.2 (ppc64le) |
+==================================+===================================================================+
| **OFED ISO** | MLNX_OFED_LINUX-3.2-2.0.0.0-rhel7.2-ppc64le.iso |
+----------------------------------+-------------------------------------------------------------------+
| **Attribute Supported** | --without-32bit --without-fw-update --add-kernel-support --force |
+----------------------------------+-------------------------------------------------------------------+
| **IB.pkglist** | ib.rhels7.ppc64le.pkglist |
+----------------------------------+-------------------------------------------------------------------+
+----------------------------------+-------------------------------------------------------------------+
| RHEL 7.1 (ppc64) |
+==================================+===================================================================+
| **OFED ISO** | MLNX_OFED_LINUX-3.2-2.0.0.0-rhel7.1-ppc64.iso |
+----------------------------------+-------------------------------------------------------------------+
| **Attribute Supported** | --without-32bit --without-fw-update --add-kernel-support --force |
+----------------------------------+-------------------------------------------------------------------+
| **IB.pkglist** | ib.rhels7.ppc64.pkglist |
+----------------------------------+-------------------------------------------------------------------+
+---------------+---------+---------------------------------------------------+------------------------------------------------------------------+---------------------------+
| OS version | Arch | Ofed version | Attribute supported by mlnx | IB.pkglist |
+===============+=========+===================================================+==================================================================+===========================+
| rhels7.1 | ppc64 | MLNX_OFED_LINUX-3.2-2.0.0.0-rhel7.1-ppc64.iso |--without-32bit --without-fw-update --add-kernel-support --force | ib.rhels7.ppc64.pkglist |
+---------------+---------+---------------------------------------------------+------------------------------------------------------------------+---------------------------+
| rhels7.2 | ppc64le | MLNX_OFED_LINUX-3.2-2.0.0.0-rhel7.2-ppc64le.iso |--without-32bit --without-fw-update --add-kernel-support --force | ib.rhels7.ppc64le.pkglist |
+---------------+---------+---------------------------------------------------+------------------------------------------------------------------+---------------------------+
| sles12.1 | ppc64le |MLNX_OFED_LINUX-3.2-2.0.0.0-sles12sp1-ppc64le.iso |--without-32bit --without-fw-update --force | ib.sles12.ppc64le.pkglist |
+---------------+---------+---------------------------------------------------+------------------------------------------------------------------+---------------------------+
| ubuntu14.04.3 | ppc64le |MLNX_OFED_LINUX-3.2-2.0.0.0-ubuntu14.04-ppc64le.iso|--without-32bit --without-fw-update --add-kernel-support --force |ib.ubuntu14.ppc64le.pkglist|
+---------------+---------+---------------------------------------------------+------------------------------------------------------------------+---------------------------+
**Suse Linux Enterprise Server**
+----------------------------------+-------------------------------------------------------------------+
| SLES 12 (ppc64le) |
+==================================+===================================================================+
| **OFED ISO** | MLNX_OFED_LINUX-3.2-2.0.0.0-sles12sp1-ppc64le.iso |
+----------------------------------+-------------------------------------------------------------------+
| **Attribute Supported** | --without-32bit --without-fw-update --force |
+----------------------------------+-------------------------------------------------------------------+
| **IB.pkglist** | ib.sles12.ppc64le.pkglist |
+----------------------------------+-------------------------------------------------------------------+
**Ubuntu**
+----------------------------------+-------------------------------------------------------------------+
| Ubuntu14.04.3 (ppc64le) |
+==================================+===================================================================+
| **OFED ISO** | MLNX_OFED_LINUX-3.2-2.0.0.0-ubuntu14.04-ppc64le.iso |
+----------------------------------+-------------------------------------------------------------------+
| **Attribute Supported** | --without-32bit --without-fw-update --add-kernel-support --force |
+----------------------------------+-------------------------------------------------------------------+
| **IB.pkglist** | ib.ubuntu14.ppc64le.pkglist |
+----------------------------------+-------------------------------------------------------------------+