xCAT/xcat-core
2
0
Fork 0
mirror of https://github.com/xcat2/xcat-core.git synced 2025-06-12 00:00:12 +00:00
Code Issues Releases Wiki Activity

Merge pull request #912 from hu-weihua/ibdoc

Browse Source 
Update IB docs for bug fix
This commit is contained in:
neo954
2016-04-18 12:48:27 +08:00
parent e56c35c3cd 49a85041fe
commit 3d96f7701b
8 changed files with 336 additions and 172 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/source/advanced/networks/infiniband/driver_and_installation.rst
View File

@ -1,11 +1,11 @@
IB Driver Preparation and Installation
======================================
xCAT provides sample postscripts to help you install the Mellanox OpenFabrics Enterprise Distribution (OFED) Infiniband Driver. These scripts are located in ``opt/xcat/share/xcat/ib/scripts/Mellanox/``. You can use these scripts directly or change them to satisfy your own environment. **xCAT 2.11 drops support of mlnxofed_ib_install and recommends using version 2 of the script: mlnxofed_ib_install.v2**.
xCAT provides sample postscripts to help you install the Mellanox OpenFabrics Enterprise Distribution (OFED) InfiniBand Driver. These scripts are located in ``opt/xcat/share/xcat/ib/scripts/Mellanox/``. You can use these scripts directly or change them to satisfy your own environment. **xCAT 2.11 drops support of mlnxofed_ib_install and recommends using version 2 of the script: mlnxofed_ib_install.v2**.
.. toctree::
:maxdepth: 2
mlnxofed_ib_install_v2_usage.rst
mlnxofed_ib_install_v1_usage.rst

6
docs/source/advanced/networks/infiniband/index.rst
View File

@ -1,8 +1,8 @@
Infiniband (Mellanox)
InfiniBand (Mellanox)
=====================
xCAT offers a certain degree support for Mellanox infiniband product, it help you to configurate Mellanox infiniband products easily.
xCAT offers a certain degree support for Mellanox InfiniBand product, it help you to configurate Mellanox InfiniBand products easily. For more information about Mellanox InfiniBand, please refer to `Mellanox official site <http://www.mellanox.com/>`_.
.. toctree::
:maxdepth: 2

218
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskful.rst
View File

@ -1,95 +1,169 @@
Configuration for Diskful Installation
=======================================
1. Set script ``mlnxofed_ib_install`` as postbootscript ::
1. Set script ``mlnxofed_ib_install`` as ``postbootscripts`` or ``postscripts`` ::
chdef <node> -p postbootscripts="mlnxofed_ib_install -p /install/<path>/<MLNX_OFED_LINUX.iso>"
chdef <node> -p postbootscripts="mlnxofed_ib_install -p /install/<subpath>/<MLNX_OFED_LINUX.iso>"
Or ::
chdef <node> -p postscripts="mlnxofed_ib_install -p /install/<subpath>/<MLNX_OFED_LINUX.iso>"
xCAT simulates completely the way Mellanox scripts work by using ``postbootscripts``. This way need to reboot after drive installation to make Mellanox drivers work reliably just like Mellanox suggested. If you want to use the reboot after operating system installation to avoid reboot twice, you can using ``postscripts`` attribute to install Mellanox drivers. This way has been verified in limited scenarios. For more information please refer to :doc:`The Scenarioes Have Been Verified </advanced/networks/infiniband/mlnxofed_ib_verified_scenario_matrix>`. You can try this way in other else scenarios if you needed.
2. Specify dependence package **[required for RHEL and SLES]**
2. Specify dependency package
a) Copy a correct pkglist file **shipped by xCAT** according your environment to the ``/install/custom/install/<ostype>/`` directory, these pkglist files are located under ``/opt/xcat/share/xcat/install/<ostype>/`` ::
Some dependencies need to be installed before running Mellanox scripts. These dependencies are different between different scenario. xCAT configurates these dependency packages by using ``pkglist`` attribute of ``osimage`` definition. Please refer to :doc:`Add Additional Software Packages </guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/additional_pkg>` for more information::
cp /opt/xcat/share/xcat/install/<ostype>/compute.<osver>.<arch>.pkglist \
/install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist
# lsdef -t osimage <os>-<arch>-install-compute
Object name: <os>-<arch>-install-compute
imagetype=linux
....
pkgdir=/<os packages directory>
pkglist=/<os packages list directory>/compute.<os>.<arch>.pkglist
....
b) Edit your ``/install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist`` and add one line
``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<osver>.<arch>.pkglist#``
You can check directory ``/opt/xcat/share/xcat/ib/netboot/<ostype>/`` and choose one correct ``ib.<osver>.<arch>.pkglist`` according your environment.
c) Make the related osimage use the customized pkglist ::
You can append the ib dependency packages list in the end of ``/<os packages list directory>/compute.<os>.<arch>.pkglist`` directly like below: ::
chdef -t osimage -o <osver>-<arch>-install-compute \
pkglist=/install/custom/install/<ostype>/compute.<osver>.<arch>.pkglist
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
@base
@x11
openssl
ntp
rsyn
#ib part
createrepo
kernel-devel
kernel-source
....
Or if you want to isolate InfiniBand dependency packages list into a separate file, after you edit this file, you can append the file in ``/<os packages list directory>/compute.<os>.<arch>.pkglist`` like below way: ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
@base
@x11
openssl
ntp
rsyn
#INCLUDE:/<ib pkglist path>/<you ib pkglist file>#
xCAT has shipped some ib pkglist files under ``/opt/xcat/share/xcat/ib/netboot/<ostype>/``, these pkglist files have been verified in sepecific scenarion. Please refer to :doc:`The Scenarioes Have Been Verified </advanced/networks/infiniband/mlnxofed_ib_verified_scenario_matrix>` to judge if you can use it directly in your environment. If so, you can use it like below: ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
@base
@x11
openssl
ntp
rsyn
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<os>.<arch>.pkglist#
Take rhels7.2 on ppc64le for example: ::
# lsdef -t osimage rhels7.2-ppc64le-install-compute
Object name: rhels7.2-ppc64le-install-compute
imagetype=linux
osarch=ppc64le
osdistroname=rhels7.2-ppc64le
osname=Linux
osvers=rhels7.2
otherpkgdir=/install/post/otherpkgs/rhels7.2/ppc64le
pkgdir=/install/rhels7.2/ppc64le
pkglist=/install/custom/install/rh/compute.rhels7.ib.pkglist
profile=compute
provmethod=install
template=/opt/xcat/share/xcat/install/rh/compute.rhels7.tmpl
Take RHEL 6.4 on x86_64 for example ::
cp /opt/xcat/share/xcat/install/rh/compute.rhels6.x86_64.pkglist \
/install/custom/install/rh/compute.rhels6.x86_64.pkglist
**[Note]**: If the osimage definition was generated by xCAT command ``copycds``, default value ``/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist`` was assigned to ``pkglist`` attribute. ``/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist`` is the sample pkglist shipped by xCAT, recommend to make a copy of this sample and using the copy in real environment. In the above example, ``/install/custom/install/rh/compute.rhels7.ib.pkglist`` is a copy of ``/opt/xcat/share/xcat/install/rh/compute.rhels7.pkglist``. ::
# cat /install/custom/install/rh/compute.rhels7.ib.pkglist
#Please make sure there is a space between @ and group name
wget
ntp
nfs-utils
net-snmp
rsync
yp-tools
openssh-server
util-linux
net-tools
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels7.ppc64le.pkglist#
Edit the ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` and add below line
``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#``
Then ``/install/custom/install/rh/compute.rhels6.x86_64.pkglist`` looks like below ::
#Please make sure there is a space between @ and group name
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#
ntp
nfs-utils
net-snmp
rsync
yp-tools
openssh-server
util-linux-ng
Then modify related osimage ::
chdef -t osimage -o rhels6.4-x86_64-install-compute \
pkglist=/install/custom/install/rh/compute.rhels6.x86_64.pkglist
3. Install node ::
nodeset <node> osimage=<osver>-<arch>-install-compute
rsetboot <node> net
rpower <node> reset
**[Note]**:
* In RHEL7.x, after performing all steps above, ``openibd`` doesn't work well. you can resolve this problem depending on `Mellanox OFED Linux Release Notes <http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_Release_Notes_3_1-1_0_5.pdf>`_ by yourself. But reboot one more time can resolve all of these complex issues. so **we strongly recommend reboot machine again to avoid unexpected problem in RHEL7.x.**
After steps above, you can login target node and find the Mellanox InfiniBand drives are located under ``/lib/modules/<kernel_version>/extra/``.
* If you performed firmware updates, i.e. you didn't pass ``--without-fw-update`` to option ``-m`` of ``mlnxofed_ib_install``, **reboot machine for all distro**
Issue ``ibv_devinfo`` command you can get the InfiniBand apater information ::
After steps above, you can login target ndoe and find the Mellanox IB drives are located under ``/lib/modules/<kernel_version>/extra/``.
# ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 10.14.2036
node_guid: f452:1403:0076:10e0
sys_image_guid: f452:1403:0076:10e0
vendor_id: 0x02c9
vendor_part_id: 4113
hw_ver: 0x0
board_id: IBM1210111019
phys_port_cnt: 2
Device ports:
port: 1
state: PORT_INIT (2)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 65535
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 65535
port_lmc: 0x00
link_layer: InfiniBand
Using ``service openibd status`` to verify if openibd works well. Below is the output in rhels7.2. ::
# service openibd status
HCA driver loaded
Configured IPoIB devices:
ib0 ib1
Currently active IPoIB devices:
Configured Mellanox EN devices:
Currently active Mellanox devices:
The following OFED modules are loaded:
rdma_ucm
rdma_cm
ib_addr
ib_ipoib
mlx4_core
mlx4_ib
mlx4_en
mlx5_core
mlx5_ib
ib_uverbs
ib_umad
ib_ucm
ib_sa
ib_cm
ib_mad
ib_core
Issue ``ibstat`` command you can get the IB apater information ::
[root@server ~]# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.11.500
Hardware version: 0
Node GUID: 0x5cf3fc000004ec02
System image GUID: 0x5cf3fc000004ec05
Port 1:
State: Initializing
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02594868
Port GUID: 0x5cf3fc000004ec03
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Disabled
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02594868
Port GUID: 0x5cf3fc000004ec04
Link layer: InfiniBand

225
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_diskless.rst
View File

@ -1,128 +1,169 @@
Configuration for Diskless Installation
=======================================
1. Specify dependence package **[required for RHEL and SLES]**
1. Specify dependency package
a) Copy a correct pkglist file **shipped by xCAT** according your environment to the ``/install/custom/netboot/<ostype>/`` directory ::
Some dependencies need to be installed before running Mellanox scripts. These dependencies are different among different scenarios. xCAT can help user to install these dependency packages by adding these package names to the file specified by the ``pkglist`` attribute of the ``osimage`` definition. Please refer to :doc:`Add Additional Software Packages </guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/additional_pkg>` for more information::
cp /opt/xcat/share/xcat/netboot/<ostype>/compute.<osver>.<arch>.pkglist \
/install/custom/netboot/<ostype>/compute.<osver>.<arch>.pkglist
# lsdef -t osimage <osver>-<arch>-netboot-compute
Object name: <osver>-<arch>-netboot-compute
imagetype=linux
....
pkgdir=/<os packages directory>
pkglist=/<os packages list directory>/compute.<os>.<arch>.pkglist
....
b) Edit ``/install/custom/netboot/<ostype>/<profile>.pkglist`` and add ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<osver>.<arch>.pkglist#``
You can append the ib dependency packages list in the end of ``/<os packages list directory>/compute.<os>.<arch>.pkglist`` directly like below: ::
For example, on RHEL 6.4 (x86_64): ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
bash
nfs-utils
openssl
dhclient
kernel
.....
cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.pkglist \
/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist
Edit ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` and add ``#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#``
Then ``/install/custom/netboot/rh/compute.rhels6.x86_64.pkglist`` looks like below ::
#ib part
createrepo
kernel-devel
kernel-source
....
Or if you want to isolate InfiniBand dependency packages list into a separate file, after you edit this file, you can append the file in ``/<os packages list directory>/compute.<os>.<arch>.pkglist`` like below way: ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
bash
nfs-utils
openssl
dhclient
kernel
.....
#INCLUDE:/<ib pkglist path>/<you ib pkglist file>#
xCAT ships some InfiniBand pkglist files under ``/opt/xcat/share/xcat/ib/netboot/<ostype>/``, these pkglist files have been verified in sepecific scenarion. Please refer to :doc:`The Scenarioes Have Been Verified </advanced/networks/infiniband/mlnxofed_ib_verified_scenario_matrix>` to judge if you can use it directly in your environment. If so, you can use it like below: ::
#cat /<os packages list directory>/compute.<os>.<arch>.pkglist
bash
nfs-utils
openssl
dhclient
kernel
.....
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/<ostype>/ib.<os>.<arch>.pkglist#
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels6.x86_64.pkglist#
bash
nfs-utils
openssl
dhclient
.....
2. Prepare postinstall scripts
a) Specify a correct postinstall script **shipped by xCAT** ::
mkdir -p /install/custom/netboot/<ostype>/
cp /opt/xcat/share/xcat/netboot/<ostype>/<profile>.postinstall \
/install/custom/netboot/<ostype>/
chmod +x /install/custom/netboot/<ostype>/<profile>.postinstall
Edit ``postinstall`` script to trigger InfniBand drvices installation during ``genimage``. Using below command to find out where the ``postinstall`` script is defined. ::
# lsdef -t osimage <os>-<arch>-netboot-compute
Object name: <os>-<arch>-netboot-compute
....
postinstall=/<postinstall script path/compute.<os>.<arch>.postinstall
....
Take RHEL 6.4 on x86_64 for example ::
mkdir -p /install/custom/netboot/rh/
cp /opt/xcat/share/xcat/netboot/rh/compute.rhels6.x86_64.postinstall \
/install/custom/netboot/rh/
chmod +x /install/custom/netboot/rh/compute.rhels6.x86_64.postinstall
b) Edit ``/install/custom/netboot/<ostype>/<profile>.postinstall`` and add below line in the end ::
Edit ``/<postinstall script path/compute.<os>.<arch>.postinstall`` and add below line in the end ::
/install/postscripts/mlnxofed_ib_install \
-p /install/<path>/<MLNX_OFED_LINUX.iso> -i $1 -n genimage
**[Note]** If you want to customized kernel version (i.e the kernel version of the diskless image you want to generate is different with the kernel version of you management node), you need to pass ``--add-kernel-support`` attribute to Mellanox. the line added into ``<profile>.postinstall`` should like below ::
**[Note]** Mellanox OFED ISO was built against a series of certain kernael versions, If the version of linux kernel you are using does not match with any of the Mellanox offered pre-built kernel modules, you can pass ``--add-kernel-support`` command line argument to Mellanox OFED installation script to build these kernel modules base on the version of linux kernel you are using. The line added into ``<profile>.postinstall`` should like below ::
/install/postscripts/mlnxofed_ib_install \
-p /install/<path>/<MLNX_OFED_LINUX.iso> -m --add-kernel-support -end- -i $1 -n genimage
-p /install/<subpath>/<MLNX_OFED_LINUX.iso> -m --add-kernel-support -end- -i $1 -n genimage
Below steps maybe helpful for you to do judgment if you belong to this situation.
Get the kernel version of your management node ::
uname -r
Get the kernel version of target image. take generating a diskless image of rhels7.0 on x86_64 for example ::
[root@server]# lsdef -t osimage rhels7.0-x86_64-install-compute -i pkgdir
Object name: rhels7.0-x86_64-install-compute
pkgdir=/install/rhels7.0/x86_64
[root@server]# ls -l /install/rhels7.0/x86_64/Packages/ |grep kernel*
.......
-r--r--r-- 1 root root 30264588 May 5 2014 kernel-3.10.0-123.el7.x86_64.rpm
.......
3. Set the related osimage using the customized pkglist and compute.postinsall
Take rhels7.2 on ppc64le for example: ::
* [RHEL/SLES] ::
#lsdef -t osimage rhels7.2-ppc64le-netboot-compute
Object name: rhels7.2-ppc64le-netboot-compute
exlist=/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.exlist
imagetype=linux
osarch=ppc64le
osdistroname=rhels7.2-ppc64le
osname=Linux
osvers=rhels7.2
otherpkgdir=/install/post/otherpkgs/rhels7.2/ppc64le
permission=755
pkgdir=/install/rhels7.2/ppc64le
pkglist=/install/custom/netboot/rh/compute.rhels7.ppc64le.pkglist
postinstall=/install/custom/netboot/rh/compute.rhels7.ppc64le.ib.postinstall
profile=compute
provmethod=netboot
rootimgdir=/install/netboot/rhels7.2/ppc64le/compute
chdef -t osimage -o <osver>-<arch>-netboot-compute \
pkglist=/install/custom/netboot/<ostype>/compute.<osver>.<arch>.pkglist \
postinstall=/install/custom/netboot/<ostype>/<profile>.postinstall
* [Ubuntu] ::
**[Note]**: If the osimage definition was generated by xCAT command ``copycds``, default value ``/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.pkglist`` was assigned to ``pkglist`` attribute. ``/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.pkglist`` is the sample pkglist shipped by xCAT, recommend to make a copy of this sample and using the copy in real environment. In the above example, ``/install/custom/netboot/rh/compute.rhels7.ppc64le.pkglist`` is a copy of ``/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.pkglist``. For the same reason, ``/install/custom/netboot/rh/compute.rhels7.ppc64le.ib.postinstall`` is a copy of ``/opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.postinstall``. ::
chdef -t osimage -o <osver>-<arch>-netboot-compute \
postinstall=/install/custom/netboot/<ostype>/<profile>.postinstall
``compute.rhels7.ppc64le.pkglist`` looks like below: ::
4. Generate and package image for diskless installation ::
# cat /install/custom/netboot/rh/compute.rhels7.ppc64le.pkglist
bash
nfs-utils
openssl
dhclient
bc
......
lsvpd
irqbalance
procps-ng
parted
net-tools
#INCLUDE:/opt/xcat/share/xcat/ib/netboot/rh/ib.rhels7.ppc64le.pkglist#
``compute.rhels7.ppc64le.ib.postinstall`` looks like below: ::
# cat /install/custom/netboot/rh/compute.rhels7.ppc64le.ib.postinstall
#!/bin/sh
#-- Do not remove following line if you want to make use of CVS version tracking
.....
# [ -r $workdir/$profile.$ext ] && cat $workdir/$profile.$ext | grep -E '^[[:space:]]*#.*[[:space:]]\$Id' >> $installroot/etc/IMGVERSION
#done
/install/postscripts/mlnxofed_ib_install -p /install/ofed/MLNX_OFED_LINUX-3.2-2.0.0.0-rhel7.2-ppc64le.iso -i $1 -n genimage
3. Generate and package image for diskless installation ::
genimage <osver>-<arch>-netboot-compute
packimage <osver>-<arch>-netboot-compute
5. Install node ::
4. Install node ::
nodeset <nodename> osimage=<osver>-<arch>-netboot-compute
rsetboot <nodename> net
rpower <nodename> reset
After installation, you can login target ndoe and issue ``ibstat`` command to verify if your IB driver works well. if everything is fine, you can get the IB apater information ::
[root@server ~]# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.11.500
Hardware version: 0
Node GUID: 0x5cf3fc000004ec02
System image GUID: 0x5cf3fc000004ec05
Port 1:
State: Initializing
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02594868
Port GUID: 0x5cf3fc000004ec03
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Disabled
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02594868
Port GUID: 0x5cf3fc000004ec04
Link layer: InfiniBand
After installation, you can login target ndoe and issue ``ibv_devinfo`` command to verify if your InfiniBand driver works well. if everything is fine, you can get the InfiniBand apater information ::
# ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 10.14.2036
node_guid: f452:1403:0076:10e0
sys_image_guid: f452:1403:0076:10e0
vendor_id: 0x02c9
vendor_part_id: 4113
hw_ver: 0x0
board_id: IBM1210111019
phys_port_cnt: 2
Device ports:
port: 1
state: PORT_INIT (2)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 65535
port_lmc: 0x00
link_layer: InfiniBand
port: 2
state: PORT_DOWN (1)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 0
port_lid: 65535
port_lmc: 0x00
link_layer: InfiniBand

19
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_preparation.rst
View File

@ -1,12 +1,22 @@
Preparation
===========
Obtain the Mellanox OFED ISO file from `Mellanox official site <http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>`_ and put it into one place under ``/install`` directory depending on your need.
Obtain the Mellanox OFED ISO
----------------------------
Obtain the Mellanox OFED ISO file from `Mellanox official Download Page <http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers>`_ and put it into one place under ``/install`` directory depending on your need.
**[NOTE]**
* Mellanox provides OFED drivers in **tarball** and **iso** formats. xCAT only supports the **iso** format at this time.
* Mellanox provides different OFED ISOs depending on operating system and machine architecture, named like MLNX_OFED_LINUX-<packver1>-<packver2>-<osver>-<arch>.iso, you should download correct one according your environment.
* Mellanox provides OFED drivers in **tarball** and **ISO image** formats. xCAT only supports the **iso** format at this time.
* Mellanox provides different OFED ISOs depending on operating system and machine architecture, named like MLNX_OFED_LINUX-<packver1>-<packver2>-<osver>-<arch>.iso, you should download correct one according to your environment.
* Mellanox has some updates and known issues for echo OFED, please read `InfiniBand/VPI Software Overview <http://www.mellanox.com/page/software_overview_ib>`_ to understand these information.
* The Mellanox links offered above maybe outdate in future for Mellanox updates its web page, xCAT will keep updating for synchronization. If we don't update in time, please access `Mellanox web portal <http://www.mellanox.com>`_ to find ``Support/Education`` then ``InfiniBand/VPI Drivers`` lables.
Prepare Install Script
----------------------
**mlnxofed_ib_install.v2** is a sample script, its framework can help you install Mellanox drives easily. But in specific scenario, some detail need to be modified to meet requirement, such like dependency package list. It has been verified in limited scenarios and can work as solution in these scenarios. For these scenarions information please refer to :doc:`The Scenarioes Have Been Verified </advanced/networks/infiniband/mlnxofed_ib_verified_scenario_matrix>`.
Copy **mlnxofed_ib_install.v2** into ``/install/postscripts`` and change name to **mlnxofed_ib_install** ::
@ -31,3 +41,6 @@ If need to pass ``--without-32bit --without-fw-update --add-kernel-support --for
mlnxofed_ib_install -p /install/<path>/<MLNX_OFED_LINUX.iso> \
-m --without-32bit --without-fw-update --add-kernel-support --force -end-
**[Note]** We recommend to update your firmware to the version Mellanox supported in its release notes to avoid unexpected problem when you install InfiniBand driver.

2
docs/source/advanced/networks/infiniband/mlnxofed_ib_install_v2_usage.rst
View File

@ -7,4 +7,6 @@ Using mlnxofed_ib_install.v2 (Recommend)
mlnxofed_ib_install_v2_preparation.rst
mlnxofed_ib_install_v2_diskful.rst
mlnxofed_ib_install_v2_diskless.rst
mlnxofed_ib_verified_scenario_matrix.rst
mlnxofed_ib_known_issue.rst

19
docs/source/advanced/networks/infiniband/mlnxofed_ib_known_issue.rst Normal file
View File

@ -0,0 +1,19 @@
Known Issues
============
Known Issue 1
-------------
After you install mellanox derives in rhels7.2 successfully by xCAT, maybe you have new requirement to upgrade your operating system to higher version. In this case you probably hit such problem the IB adaptor drives shipped by operating system is higher than the Mellanox drives you have installed. That means the the Mellanox drives will be replaced by the drives shipped by operating system. If it's not the result you expect, you hope keep the Mellanox drives after operating system upgraded, please add below statement into ``/etc/yum.conf`` in your target node after you install mellanox derives successfully for the first time. ::
exclude=dapl* libib* ibacm infiniband* libmlx* librdma* opensm* ibutils*
Known Issue 2
-------------
If you want to use ``--add-kernel-support`` attribute in sles12.1 and ppc64le scenario, you will find some dependency packages are not shipped by SLES Server DVDs, such like ``python-devel``, it's shipped in SDK DVDs. xCAT doesn't ship specific pkglist to support such scenario. If you have such requirement, please used ``otherpkglist`` and ``otherpkgs`` attributes to prepare dependency packages repository ahead. If you need help about ``otherpkglist`` and ``otherpkgs``attributes, please refer to :doc:`Add Additional Software Packages </guides/admin-guides/manage_clusters/ppc64le/diskful/customize_image/additional_pkg>`.

15
docs/source/advanced/networks/infiniband/mlnxofed_ib_verified_scenario_matrix.rst Normal file
View File

@ -0,0 +1,15 @@
The Scenarioes Have Been Verified
=================================
+---------------+---------+---------------------------------------------------+------------------------------------------------------------------+---------------------------+
| OS version | Arch | Ofed version | Attribute supported by mlnx | IB.pkglist |
+===============+=========+===================================================+==================================================================+===========================+
| rhels7.1 | ppc64 | MLNX_OFED_LINUX-3.2-2.0.0.0-rhel7.1-ppc64.iso |--without-32bit --without-fw-update --add-kernel-support --force | ib.rhels7.ppc64.pkglist |
+---------------+---------+---------------------------------------------------+------------------------------------------------------------------+---------------------------+
| rhels7.2 | ppc64le | MLNX_OFED_LINUX-3.2-2.0.0.0-rhel7.2-ppc64le.iso |--without-32bit --without-fw-update --add-kernel-support --force | ib.rhels7.ppc64le.pkglist |
+---------------+---------+---------------------------------------------------+------------------------------------------------------------------+---------------------------+
| sles12.1 | ppc64le |MLNX_OFED_LINUX-3.2-2.0.0.0-sles12sp1-ppc64le.iso |--without-32bit --without-fw-update --force | ib.sles12.ppc64le.pkglist |
+---------------+---------+---------------------------------------------------+------------------------------------------------------------------+---------------------------+
| ubuntu14.04.3 | ppc64le |MLNX_OFED_LINUX-3.2-2.0.0.0-ubuntu14.04-ppc64le.iso|--without-32bit --without-fw-update --add-kernel-support --force |ib.ubuntu14.ppc64le.pkglist|
+---------------+---------+---------------------------------------------------+------------------------------------------------------------------+---------------------------+