mirror of
https://github.com/xcat2/xcat-core.git
synced 2025-08-19 01:30:21 +00:00
Merge pull request #2458 from whowutwut/onie
Refactor the ONIE documentation
This commit is contained in:
yangsong
GitHub
@@ -1,7 +0,0 @@
|
||||
ONIE Compatible Bare Metal Switches
|
||||
===================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
onie_switches.rst
|
||||
@@ -1,135 +0,0 @@
|
||||
ONIE compatible bare metal switch
|
||||
=================================
|
||||
|
||||
The ONIE [1]_. compatible bare metal switches(abbreviated as "ONIE switch") from vendors such as Mellanox or Edgecore are often used as top-of-rack switches in the cluster. Usually, the switches are shipped with a Cumulus Network OS(https://cumulusnetworks.com) and a license pre-installed. In some cases, user may get whitebox switch hardware with a standalone Cumulus installer and license file. This documentation presents a typical workflow on how to setup ONIE switch from white box, then configure and manage the switch with xCAT.
|
||||
|
||||
.. [1] Open Network Install Environment: Created by Cumulus Networks, Inc. in 2012, the Open Network Install Environment (ONIE) Project is a small operating system, pre-installed as firmware on bare metal network switches, that provides an environment for automated operating system provisioning.
|
||||
|
||||
Create an ONIE switch object
|
||||
-------------------------------
|
||||
|
||||
The ONIE switch object can be created with the "onieswitch" template shipped in xCAT, the ip address and mac of the switch management ethernet port should be specified : ::
|
||||
|
||||
mkdef edgecoresw1 --template onieswitch arch=armv71 ip=192.168.5.191 mac=8C:EA:1B:12:CA:40
|
||||
|
||||
Provision the Cumulus OS on ONIE switch
|
||||
---------------------------------------
|
||||
|
||||
To provision Cumulus OS, the Cumulus installation file, a binary shipped with the switch, should be saved in a directory exported in the http server.
|
||||
|
||||
Run ``chdef`` to specify the "provmethod" attribute of the switch object to the full path of the installation file: ::
|
||||
|
||||
chdef edgecoresw1 netboot=onie provmethod="/install/custom/sw/edgecore/cumulus-linux-3.1.0-bcm-armel-1471981017.dc7e2adzfb43f6b.bin"
|
||||
|
||||
Run ``makedhcp`` to prepare the DHCP/BOOTP lease. ::
|
||||
|
||||
makedhcp -a edgecoresw1
|
||||
|
||||
The command or operation to start the provision dependes on the status of switch:
|
||||
|
||||
1. If the switch is a white box without Cumulus OS installed, simply connect the management ethernet port of the switch to xCAT management node, then power on the switch.
|
||||
|
||||
2. If a Cumulus OS has been installed on the switch, you need to login to the switch(the default user is ``cumulus`` and the password is ``CumulusLinux!``) and run a batch of commands: ::
|
||||
|
||||
sudo onie-select -i
|
||||
sudo reboot
|
||||
|
||||
If the passwordless-ssh of "root" has been enabled, the commands can be issued with: ::
|
||||
|
||||
xdsh edgecoresw1 "/usr/cumulus/bin/onie-select -i -f;reboot"
|
||||
|
||||
After reboot, the switch will enter ONIE install mode and begin the installation. The provision might take about 50 minutes.
|
||||
|
||||
|
||||
Switch Configuration
|
||||
--------------------
|
||||
|
||||
Enable the passwordless ssh for "root"
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In a newly installed Cumulus OS, a default user ``cumulus`` will be created, the switch can be accessed via ssh with the default password ``CumulusLinux!``.
|
||||
|
||||
The passwordless ssh access of "root" should be enabled with the script ``/opt/xcat/share/xcat/scripts/configonie`` ::
|
||||
|
||||
/opt/xcat/share/xcat/scripts/configonie --switches edgecoresw1 --ssh
|
||||
|
||||
After the passwordless access for "root" is setup successfully, the switch can be managed with the node management commands such as ``xdsh``, ``xdcp`` and ``updatenode``, etc.
|
||||
|
||||
Licence file installation
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
On the newly installed switch, only the serial console and the management ethernet port are enabled. To activate the data ports, the licence file shipped with the switch should be installed: ::
|
||||
|
||||
xdcp edgecoresw1 /install/custom/sw/edgecore/licensefile.txt /tmp
|
||||
xdsh edgecoresw1 "/usr/cumulus/bin/cl-license -i /tmp/licensefile.txt"
|
||||
|
||||
To check whether the license file is installed successfully: ::
|
||||
|
||||
~: xdsh edgecoresw1 /usr/cumulus/bin/cl-license
|
||||
edgecoresw1: xxx@xx.com|xxxxxxxxxxxxxxx
|
||||
|
||||
Reboot the switch to apply the licence file: ::
|
||||
|
||||
xdsh edgecoresw1 reboot
|
||||
|
||||
Enable SNMP
|
||||
~~~~~~~~~~~
|
||||
|
||||
The snmpd in the switch is not enabled by default, xCAT ships a postscript to enable it: ::
|
||||
|
||||
updatenode edgecoresw1 -P enablesnmp
|
||||
|
||||
|
||||
Switch Discovery
|
||||
----------------
|
||||
|
||||
The ONIE switch can be scaned and discovered with ``switchdiscover`` ::
|
||||
|
||||
~: switchdiscover --range 192.168.23.1-10
|
||||
Discovering switches using nmap for 192.168.23.1-10. It may take long time...
|
||||
ip name vendor mac
|
||||
------------ ------------ ------------ ------------
|
||||
192.168.23.1 edgecoresw1 Edgecore switch 8C:EA:1B:12:CA:40
|
||||
Switch discovered: edgecoresw1
|
||||
|
||||
Once SNMP on the ONIE switch is enabled, the ONIE switch can be discovered with "snmp" method: ::
|
||||
|
||||
~: switchdiscover --range 192.168.23.1-10 -s snmp
|
||||
Discovering switches using snmpwalk for 192.168.23.1-10 ....
|
||||
ip name vendor mac
|
||||
------------ ------------ ------------ ------------
|
||||
192.168.23.1 edgecoresw1 Linux edgecoresw1 4.1.0-cl-2-iproc #1 SMP Debian 4.1.25-1+cl3u4 (2016-08-13) armv7l 8c:ea:1b:12:ca:40
|
||||
Switch discovered: edgecoresw1
|
||||
|
||||
|
||||
Switch Management
|
||||
-----------------
|
||||
|
||||
File Dispatch
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The files can be dispatched to ONIE switches with ``xdcp`` ::
|
||||
|
||||
xdcp edgecoresw1 <path of file to dispatch> <destination path of the file on switch>
|
||||
|
||||
Refer to :doc:`xdcp manpage </guides/admin-guides/references/man1/xdcp.1>` for details.
|
||||
|
||||
Remote Commands
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
Commands can be run on ONIE switches remotely with ``xdsh`` ::
|
||||
|
||||
xdsh edgecoresw1 <remote commands>
|
||||
|
||||
Refer to :doc:`xdsh manpage </guides/admin-guides/references/man1/xdsh.1>` for details.
|
||||
|
||||
Run scripts remotely
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The scripts under "/install/postscripts" can be run on ONIE switches with ``updatenode -P`` ::
|
||||
|
||||
updatenode edgecoresw1 -P <script name>
|
||||
|
||||
Refer to :doc:`updatenode manpage </guides/admin-guides/references/man1/updatenode.1>` for details.
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ Networks
|
||||
:maxdepth: 2
|
||||
|
||||
ethernet_switches/index.rst
|
||||
onie_switches/index.rst
|
||||
switchdiscover/index.rst
|
||||
infiniband/index.rst
|
||||
ipv6/index.rst
|
||||
edgecore_switches/index.rst
|
||||
|
||||
12
docs/source/advanced/networks/onie_switches/index.rst
Normal file
12
docs/source/advanced/networks/onie_switches/index.rst
Normal file
@@ -0,0 +1,12 @@
|
||||
Open Network Install Environment Switches
|
||||
=========================================
|
||||
|
||||
The Open Network Install Environment, or "ONIE" [1]_. is an open source project definining an **install environment** for bare metal switches. This environment allows choice for the end users when selecting a network operating system to install onto these bare metal switches.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
os_cumulus/index.rst
|
||||
|
||||
|
||||
.. [1] Open Network Install Environment: Created by Cumulus Networks, Inc. in 2012, the Open Network Install Environment (ONIE) Project is a small operating system, pre-installed as firmware on bare metal network switches, that provides an environment for automated operating system provisioning.
|
||||
@@ -0,0 +1,12 @@
|
||||
Cumulus Linux OS
|
||||
================
|
||||
|
||||
This documentation will focus on installing the Cumulus Network Operating System (https://www.cumulusnetworks.com/) onto a "white-box" Edgecore switch but the same concepts should apply to other ONIE enabled switches using other network operating systems.
|
||||
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
prepare.rst
|
||||
install.rst
|
||||
manage.rst
|
||||
@@ -0,0 +1,112 @@
|
||||
Installation and Configuration
|
||||
==============================
|
||||
|
||||
Cumulus OS Installtion
|
||||
----------------------
|
||||
|
||||
**Note:** *The following assumes that the physical switches have power and have obtained a DHCP IP address from the xCAT open range.*
|
||||
|
||||
xCAT provides support for detecting and installing the Cumulus Linux OS into ONIE enabled switches by utilizing DHCP to detect "**onie_vendor**" from the ``vendor-class-identifier`` string and then send it the Cumulus Linux OS installer.
|
||||
|
||||
#. Create a pre-defined switch definition for the ONIE switch using the ``onieswitch`` template.
|
||||
|
||||
The mac address of the switch management port is required for xCAT to configure the DHCP information and send over the OS to install on the switch.
|
||||
|
||||
**[small clusters]** If you know the mac address of the management port on the switch, create the pre-defined switch defintion providing the mac address. ::
|
||||
|
||||
mkdef frame01sw1 --template onieswitch arch=armv71 \
|
||||
ip=192.168.1.1 mac="aa:bb:cc:dd:ee:ff"
|
||||
|
||||
**[large clusters]** xCAT's :doc:`switchdiscover </guides/admin-guides/references/man1/switchdiscover.1>` command can be used to discover the mac address and fill in the predefined switch definitions based on the switch/switchport mapping.
|
||||
|
||||
|
||||
#. Define all the switch objects providing the switch/switchport mapping: ::
|
||||
|
||||
mkdef frame01sw1 --template onieswitch arch=armv71 \
|
||||
ip=192.168.1.1 switch=coresw1 switchport=1
|
||||
mkdef frame02sw1 --template onieswitch arch=armv71 \
|
||||
ip=192.168.2.1 switch=coresw1 switchport=2
|
||||
mkdef frame03sw1 --template onieswitch arch=armv71 \
|
||||
ip=192.168.3.1 switch=coresw1 switchport=3
|
||||
mkdef frame04sw1 --template onieswitch arch=armv71 \
|
||||
ip=192.168.4.1 switch=coresw1 switchport=4
|
||||
...
|
||||
|
||||
#. Leverage ``switchdiscover`` over the DHCP range to automatically detect the MAC address and write them into the predefined swtiches above. ::
|
||||
|
||||
switchdiscover --range <IP range>
|
||||
|
||||
|
||||
#. Set the ``provmethod`` attribute of the target switch(es) to the Cumulus Linux install image: ::
|
||||
|
||||
chdef frame[01-04]sw1 \
|
||||
provmethod="/install/custom/sw_os/cumulus/cumulus-linux-3.1.0-bcm-armel.bin"
|
||||
|
||||
#. Run ``makedhcp`` to prepare the DHCP/BOOTP lease information for the switch: ::
|
||||
|
||||
makedhcp -a frame[01-04]sw1
|
||||
|
||||
Executing the ``makedhcp`` command will kick off the network install of the ONIE enabled switch. If there is no OS pre-loaded on the switch, the switch continues to send a DHCPREQUEST out to the network. After ``makedhcp`` is run against the switch, an entry is added to the leases file that will respond to the request with the Cumulus Linux installer file. ::
|
||||
|
||||
host frame1sw1 {
|
||||
dynamic;
|
||||
hardware ethernet 8c:ea:1b:12:ca:40;
|
||||
fixed-address 192.168.3.200;
|
||||
supersede server.ddns-hostname = "frame1sw1";
|
||||
supersede host-name = "frame1sw1";
|
||||
if substring (option vendor-class-identifier, 0, 11) = "onie_vendor" {
|
||||
supersede www-server = "http://192.168.27.1/install/custom/sw_os/cumulus/cumulus-linux-3.1.0-bcm-armel.bin";
|
||||
}
|
||||
}
|
||||
|
||||
*Typical installation time is around 1 hour*
|
||||
|
||||
|
||||
Configure xCAT Remote Commands
|
||||
------------------------------
|
||||
|
||||
After Cumulus Linux OS is installed, a default user ``cumulus`` will be created with default password: ``CumulusLinux!``.
|
||||
|
||||
To ease in the management of the switch, xCAT provides a script to help configure password-less ssh as the ``root`` user. This script sends over the xCAT ssh keys so that the xCAT remote commands (``xdsh``, ``xdcp``, etc) can be run against the ONIE switches.
|
||||
|
||||
Execute the following to sync the xCAT keys to the switch: ::
|
||||
|
||||
/opt/xcat/share/xcat/scripts/configonie --switches frame01sw1 --ssh
|
||||
|
||||
Validate the ssh keys are correctly configured by running a ``xdsh`` command: ::
|
||||
|
||||
xdsh frame01sw1 uptime
|
||||
|
||||
|
||||
Activate the License
|
||||
--------------------
|
||||
|
||||
After Cumulus Linux OS is installed onto the ONIE switch, only the serial port console and the management ethernet port is enabled. To activate the rest of the switch ports, the license file needs to be installed onto the switch.
|
||||
|
||||
#. Copy the license file to the switch: ::
|
||||
|
||||
xdcp frame01sw1 /install/custom/sw_os/cumulus/licensefile.txt /root/
|
||||
|
||||
#. Activate the license: ::
|
||||
|
||||
xdsh frame01sw1 "/usr/cumulus/bin/cl-license -i /root/licensefile.txt"
|
||||
|
||||
#. Verify that the license file is successfully installed: ::
|
||||
|
||||
xdsh frame01sw1 /usr/cumulus/bin/cl-license
|
||||
|
||||
Output should be similar to: ``frame01sw1 xxx@xx.com|xxxxxxxxxxxxxxx``
|
||||
|
||||
#. Reboot the switch to apply the license file: ::
|
||||
|
||||
xdsh frame01sw1 reboot
|
||||
|
||||
|
||||
Enable SNMP (optional)
|
||||
----------------------
|
||||
|
||||
In order to utilize ``xcatprobe switch_macmap``, snmp needs to be enabled. To enable, run the ``enablesnmp`` postscript on the switch: ::
|
||||
|
||||
updatenode frame01sw1 -P enablesnmp
|
||||
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
Switch Management
|
||||
=================
|
||||
|
||||
VLAN Configuration
|
||||
------------------
|
||||
|
||||
xCAT ships a simple configuration script that will set all the ports on the switch to be part of VLAN 1. See the Cumulus Networks documentation for more information regarding advanced networking configuration. ::
|
||||
|
||||
updatenode <switch> -P configinterface
|
||||
|
||||
|
||||
Re-install OS
|
||||
-------------
|
||||
|
||||
There may be occasions where a re-install of the OS is required. Assuming the files are available on the xCAT management node, the following commands will invoke the install process:
|
||||
|
||||
* **[use xCAT]** ``xdsh`` can be used to invoke the reinstall of the OS: ::
|
||||
|
||||
xdsh <switch> "/usr/cumulus/bin/onie-select -i -f;reboot"
|
||||
|
||||
# to clear out all the previous configuration, use the -k option
|
||||
xdsh <switch> "/usr/cumulus/bin/onie-select -k -f;reboot"
|
||||
|
||||
* **[manually]** Log into the Cumulus OS switch and run the following commands: ::
|
||||
|
||||
sudo onie-select -i
|
||||
sudo reboot
|
||||
@@ -0,0 +1,19 @@
|
||||
Preparation
|
||||
===========
|
||||
|
||||
Prepare the Cumulus Linux files on the xCAT Management Node.
|
||||
|
||||
#. Obtain a valid Cumulus Linux License and download the Cumulus Linux OS installer.
|
||||
|
||||
#. Copy the above files into a location under the xCAT ``/install`` directory. ::
|
||||
|
||||
# Create a directory to hold the cumulus linux files
|
||||
mkdir -p /install/custom/sw_os/cumulus/
|
||||
|
||||
# copy the license file
|
||||
cp licensefile.txt /install/custom/sw_os/cumulus/
|
||||
|
||||
# copy the installer
|
||||
cp cumulus-linux-3.1.0-bcm-armel.bin /install/custom/sw_os/cumulus/
|
||||
|
||||
|
||||
Reference in New Issue
Block a user