mirror of
https://github.com/xcat2/xcat-core.git
synced 2025-06-17 03:40:39 +00:00
Merge pull request #2155 from immarvin/oncumulus
refine the cumulus support documentation
This commit is contained in:
cxhong
GitHub
@ -1,129 +0,0 @@
|
||||
Edgecore Switch
|
||||
===============
|
||||
|
||||
The Edgecore switch from Mellanox is 1Gb top-of-rack switch. It's coming with ONIE installer. Open Network Install Environment (ONIE)is an open source initiative that defines an open "install environment" for bare metal network switches. For more information, see http://onie.opencompute.org/. Mellanox will ship the switch with Cumulus Network OS and along with a license file installed. In some case, user may get whitebox without OS and licenses. Since edgecore switch has different configuration than other switches that xCAT supports, xCAT handles edgecore switch differently.
|
||||
|
||||
|
||||
ONIE Mode
|
||||
---------
|
||||
|
||||
If the switch arrives without an OS pre-loaded, the ONIE installer and management port is the only thing enabled on the switch. Once the switch connects to the xCAT network, the switch should get a dynamic IP address. The xCAT DHCP server will get requests from the onie-installer from the switch and attempt to find an OS binary file to execute. The following messages will be logged in /var/log/messages on the management node. ::
|
||||
|
||||
Info: Fetching http://172.1.0.1/onie-installer-arm-accton_as4610_54-r0 ...
|
||||
Info: Fetching http://172.1.0.1/onie-installer-arm-accton_as4610_54 ...
|
||||
Info: Fetching http://172.1.0.1/onie-installer-accton_as4610_54 ...
|
||||
Info: Fetching http://172.1.0.1/onie-installer-arm ...
|
||||
Info: Fetching http://172.1.0.1/onie-installer .
|
||||
|
||||
|
||||
To remove the installed Cumulus Linux OS to boot back to ONIE mode, connect to the switch via serial-port or ssh and execute the following commands: ::
|
||||
|
||||
ssh cumulus@172.1.0.1
|
||||
#clean up all the configuration
|
||||
sudo onie-select -k
|
||||
sudo reboot
|
||||
#boot back to onie mode
|
||||
sudo onie-select -i
|
||||
sudo reboot
|
||||
|
||||
|
||||
After switch reboots, it will enter ONIE mode and send DHCP request to attempt to fetch the OS binary file.
|
||||
|
||||
|
||||
Discover Edgecore Switch
|
||||
------------------------
|
||||
|
||||
ONIE supports a number of methods for locating OS binary file. xCAT choose to use a DHCP server to provide specific information to the switch.
|
||||
|
||||
* IP address of the switch
|
||||
* URL of the OS binary file on the Web server
|
||||
|
||||
With the xCAT DHCP configuration, ONIE picks up an IP address and downloads the URL specified by the user and start to install of the OS. The steps take to discover the edgecore switch and process request from ONIE installer as follows:
|
||||
|
||||
#. Pre-define switch object into xCAT db, make sure ip adress, netboot and provemethod are set, also define core switch and port number where edgecore switch connect to. ::
|
||||
|
||||
|
||||
#lsdef edgecoresw1
|
||||
Object name: edgecoresw1
|
||||
groups=switch
|
||||
ip=192.168.23.1
|
||||
mgt=switch
|
||||
netboot=onie
|
||||
nodetype=switch
|
||||
postbootscripts=otherpkgs
|
||||
postscripts=syslog,remoteshell,syncfiles
|
||||
provmethod=/install/custom/sw/edgecore/cumulus-linux-3.1.0-bcm-armel-1471981017.dc7e2adzfb43f6b.bin
|
||||
switch=switch-10-5-23-1
|
||||
switchport=1
|
||||
|
||||
#makehosts edgecoresw1
|
||||
|
||||
|
||||
#. Run ``switchdiscover`` command, it will find edgecore switch and update mac address on pre-defined switch node definition. ::
|
||||
|
||||
#switchdiscover --range 192.168.5.170-190 -s nmap
|
||||
#lsdef edgecoresw1
|
||||
Object name: edgecoresw1
|
||||
groups=switch
|
||||
ip=192.168.23.1
|
||||
mac=8C:EA:1B:12:CA:40
|
||||
mgt=switch
|
||||
netboot=onie
|
||||
nodetype=switch
|
||||
postbootscripts=otherpkgs
|
||||
postscripts=syslog,remoteshell,syncfiles
|
||||
provmethod=/install/custom/sw/edgecore/cumulus-linux-3.1.0-bcm-armel-1471981017.dc7e2adzfb43f6b.bin
|
||||
status=Matched
|
||||
switch=switch-10-5-23-1
|
||||
switchport=1
|
||||
switchtype=cumulus
|
||||
usercomment=Edgecore switch
|
||||
|
||||
|
||||
#. Run ``makedhcp`` after edgecore switch discovered, it will update ``dhcpd.conf`` and response the DHCP request from the onie-installer. ::
|
||||
|
||||
#makedhcp -n
|
||||
#makedhcp -a edgecoresw1
|
||||
|
||||
|
||||
#. Installation of the Cumulus Linux OS takes about 50 minutes. Monitor the /var/log/messages to check the status of the installation. ::
|
||||
|
||||
|
||||
Oct 27 15:28:08 fs4 dhcpd: DHCPDISCOVER from 8c:ea:1b:12:ca:40 via enP4p1s0f2
|
||||
Oct 27 15:28:08 fs4 dhcpd: DHCPOFFER on 192.168.23.1 to 8c:ea:1b:12:ca:40 via enP4p1s0f2
|
||||
Oct 27 15:28:08 fs4 dhcpd: DHCPREQUEST for 192.168.23.1 (192.168.3.25) from 8c:ea:1b:12:ca:40 via enP4p1s0f2
|
||||
Oct 27 15:28:08 fs4 dhcpd: DHCPACK on 192.168.23.1 to 8c:ea:1b:12:ca:40 via enP4p1s0f2
|
||||
|
||||
|
||||
#. Once installation finished, the pre-defined switch name and IP address will be configured on edgecore switch. ::
|
||||
|
||||
cumulus@edgecoresw1:~$ ifconfig
|
||||
eth0 Link encap:Ethernet HWaddr 8c:ea:1b:12:ca:40
|
||||
inet addr:192.168.23.1 Bcast:192.168.255.255 Mask:255.255.0.0
|
||||
inet6 addr: fe80::8eea:1bff:fe12:ca40/64 Scope:Link
|
||||
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
|
||||
cumulus@edgecoresw1:~$ hostname
|
||||
edgecoresw1
|
||||
|
||||
|
||||
Configure Edgecore Switch
|
||||
-------------------------
|
||||
|
||||
xCAT provides a script ``/opt/xcat/share/xcat/script/configcumulus`` to configure attributes in the Cumulus Switch. Use the ``--help`` option to see more details. ::
|
||||
|
||||
#configcumulus --help
|
||||
Usage:
|
||||
configcumulus [-?│-h│--help]
|
||||
configcumulus [--switches switchnames] [--all]
|
||||
configcumulus [--switches switchnames] [--ssh]
|
||||
configcumulus [--switches switchnames] [--license filename ]
|
||||
configcumulus [--switches switchnames] [--snmp] [--user snmp_user] [--password snmp_password]
|
||||
configcumulus [--switches switchnames] [--ntp]
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
Edgecore Switches
|
||||
=================
|
||||
ONIE Compatible Bare Metal Switches
|
||||
===================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
edgecore_switches.rst
|
||||
onie_switches.rst
|
||||
|
||||
@ -0,0 +1,135 @@
|
||||
ONIE compatible bare metal switch
|
||||
=================================
|
||||
|
||||
The ONIE [1]_. compatible bare metal switches(abbreviated as "ONIE switch") from vendors such as Mellanox or Edgecore are often used as top-of-rack switches in the cluster. Usually, the switches are shipped with a Cumulus Network OS(https://cumulusnetworks.com) and a license pre-installed. In some cases, user may get whitebox switch hardware with a standalone Cumulus installer and license file. This documentation presents a typical workflow on how to setup ONIE switch from white box, then configure and manage the switch with xCAT.
|
||||
|
||||
.. [1] Open Network Install Environment: Created by Cumulus Networks, Inc. in 2012, the Open Network Install Environment (ONIE) Project is a small operating system, pre-installed as firmware on bare metal network switches, that provides an environment for automated operating system provisioning.
|
||||
|
||||
Create an ONIE switch object
|
||||
-------------------------------
|
||||
|
||||
The ONIE switch object can be created with the "onieswitch" template shipped in xCAT, the ip address and mac of the switch management ethernet port should be specified : ::
|
||||
|
||||
mkdef edgecoresw1 --template onieswitch arch=armv71 ip=192.168.5.191 mac=8C:EA:1B:12:CA:40
|
||||
|
||||
Provision the Cumulus OS on ONIE switch
|
||||
---------------------------------------
|
||||
|
||||
To provision Cumulus OS, the Cumulus installation file, a binary shipped with the switch, should be saved in a directory exported in the http server.
|
||||
|
||||
Run ``chdef`` to specify the "provmethod" attribute of the switch object to the full path of the installation file: ::
|
||||
|
||||
chdef edgecoresw1 netboot=onie provmethod="/install/custom/sw/edgecore/cumulus-linux-3.1.0-bcm-armel-1471981017.dc7e2adzfb43f6b.bin"
|
||||
|
||||
Run ``makedhcp`` to prepare the DHCP/BOOTP lease. ::
|
||||
|
||||
makedhcp -a edgecoresw1
|
||||
|
||||
The command or operation to start the provision dependes on the status of switch:
|
||||
|
||||
1. If the switch is a white box without Cumulus OS installed, simply connect the management ethernet port of the switch to xCAT management node, then power on the switch.
|
||||
|
||||
2. If a Cumulus OS has been installed on the switch, you need to login to the switch(the default user is ``cumulus`` and the password is ``CumulusLinux!``) and run a batch of commands: ::
|
||||
|
||||
sudo onie-select -i
|
||||
sudo reboot
|
||||
|
||||
If the passwordless-ssh of "root" has been enabled, the commands can be issued with: ::
|
||||
|
||||
xdsh edgecoresw1 "/usr/cumulus/bin/onie-select -i -f;reboot"
|
||||
|
||||
After reboot, the switch will enter ONIE install mode and begin the installation. The provision might take about 50 minutes.
|
||||
|
||||
|
||||
Switch Configuration
|
||||
--------------------
|
||||
|
||||
Enable the passwordless ssh for "root"
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In a newly installed Cumulus OS, a default user ``cumulus`` will be created, the switch can be accessed via ssh with the default password ``CumulusLinux!``.
|
||||
|
||||
The passwordless ssh access of "root" should be enabled with the script ``/opt/xcat/share/xcat/scripts/configcumulus`` ::
|
||||
|
||||
/opt/xcat/share/xcat/scripts/configcumulus --switches edgecoresw1 --ssh
|
||||
|
||||
After the passwordless access for "root" is setup successfully, the switch can be managed with the node management commands such as ``xdsh``, ``xdcp`` and ``updatenode``, etc.
|
||||
|
||||
Licence file installation
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
On the newly installed switch, only the serial console and the management ethernet port are enabled. To activate the data ports, the licence file shipped with the switch should be installed: ::
|
||||
|
||||
xdcp edgecoresw1 /install/custom/sw/edgecore/licensefile.txt /tmp
|
||||
xdsh edgecoresw1 "/usr/cumulus/bin/cl-license -i /tmp/licensefile.txt"
|
||||
|
||||
To check whether the license file is installed successfully: ::
|
||||
|
||||
~: xdsh edgecoresw1 /usr/cumulus/bin/cl-license
|
||||
edgecoresw1: xxx@xx.com|xxxxxxxxxxxxxxx
|
||||
|
||||
Reboot the switch to apply the licence file: ::
|
||||
|
||||
xdsh edgecoresw1 reboot
|
||||
|
||||
Enable SNMP
|
||||
~~~~~~~~~~~
|
||||
|
||||
The snmpd in the switch is not enabled by default, xCAT ships a postscript to enable it: ::
|
||||
|
||||
updatenode edgecoresw1 -P enablesnmp
|
||||
|
||||
|
||||
Switch Discovery
|
||||
----------------
|
||||
|
||||
The ONIE switch can be scaned and discovered with ``switchdiscover`` ::
|
||||
|
||||
~: switchdiscover --range 192.168.23.1-10
|
||||
Discovering switches using nmap for 192.168.23.1-10. It may take long time...
|
||||
ip name vendor mac
|
||||
------------ ------------ ------------ ------------
|
||||
192.168.23.1 edgecoresw1 Edgecore switch 8C:EA:1B:12:CA:40
|
||||
Switch discovered: edgecoresw1
|
||||
|
||||
Once SNMP on the ONIE switch is enabled, the ONIE switch can be discovered with "snmp" method: ::
|
||||
|
||||
~: switchdiscover --range 192.168.23.1-10 -s snmp
|
||||
Discovering switches using snmpwalk for 192.168.23.1-10 ....
|
||||
ip name vendor mac
|
||||
------------ ------------ ------------ ------------
|
||||
192.168.23.1 edgecoresw1 Linux edgecoresw1 4.1.0-cl-2-iproc #1 SMP Debian 4.1.25-1+cl3u4 (2016-08-13) armv7l 8c:ea:1b:12:ca:40
|
||||
Switch discovered: edgecoresw1
|
||||
|
||||
|
||||
Switch Management
|
||||
-----------------
|
||||
|
||||
File Dispatch
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The files can be dispatched to ONIE switches with ``xdcp`` ::
|
||||
|
||||
xdcp edgecoresw1 <path of file to dispatch> <destination path of the file on switch>
|
||||
|
||||
Refer to :doc:`xdcp manpage </guides/admin-guides/references/man1/xdcp.1>` for details.
|
||||
|
||||
Remote Commands
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
Commands can be run on ONIE switches remotely with ``xdsh`` ::
|
||||
|
||||
xdsh edgecoresw1 <remote commands>
|
||||
|
||||
Refer to :doc:`xdsh manpage </guides/admin-guides/references/man1/xdsh.1>` for details.
|
||||
|
||||
Run scripts remotely
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The scripts under "/install/postscripts" can be run on ONIE switches with ``updatenode -P`` ::
|
||||
|
||||
updatenode edgecoresw1 -P <script name>
|
||||
|
||||
Refer to :doc:`updatenode manpage </guides/admin-guides/references/man1/updatenode.1>` for details.
|
||||
|
||||
|
||||
26
xCAT/postscripts/enablesnmp
Executable file
26
xCAT/postscripts/enablesnmp
Executable file
@ -0,0 +1,26 @@
|
||||
#!/bin/bash
|
||||
#the postscript to enable snmpd in ONIE compatible switches
|
||||
|
||||
if ! cat /etc/os-release |grep -i '^NAME=[ "]*Cumulus Linux[ "]*$' >/dev/null 2>&1 ; then
|
||||
echo "This script is only supported on Cumulus OS in ONIE switch"
|
||||
exit 2
|
||||
fi
|
||||
snmp_conf="/etc/snmp/snmpd.conf"
|
||||
|
||||
[ -f "$snmp_conf" ] || exit -1
|
||||
|
||||
[ -z "$snmp_user" ] && snmp_user="xcatadmin"
|
||||
[ -z "$snmp_passwd" ] && snmp_passwd="xcatpassw0rd"
|
||||
|
||||
[ -f ${snmp_conf}.orig ] || cp $snmp_conf ${snmp_conf}.orig
|
||||
sed -i "1i\#xCAT customized snmpd configuration" $snmp_conf
|
||||
sed -i "/^\s*agentAddress/s/^/#/" $snmp_conf
|
||||
sed -i '/agentAddress udp\:161\,udp6\:\[\:\:1\]\:161/s/^#//' $snmp_conf
|
||||
sed -i "/rocommunity\s*public\s*default/s/^#//" $snmp_conf
|
||||
sed -i "/rocommunity\s*public\s*default\s*-V\s*systemonly/s/^#//" $snmp_conf
|
||||
sed -i "/#\s*createUser\s*authPrivUser\s*SHA/{n;s/^.*$/createUser $snmp_user SHA $snmp_passwd/}" $snmp_conf
|
||||
sed -i "/#rwuser\s*authPrivUser\s*priv/{n;s/^.*$/rwuser $snmp_user/}" $snmp_conf
|
||||
|
||||
systemctl enable snmpd
|
||||
systemctl restart snmpd
|
||||
|
||||
@ -1,14 +1,14 @@
|
||||
# <the template for PowerLE NV node definition>
|
||||
# <the template for ONIE switch definition>
|
||||
|
||||
cumulusswitch:
|
||||
onieswitch:
|
||||
objtype=node
|
||||
arch="OPTIONAL: the arch of the switch management service processor, such as armv71"
|
||||
groups=switch
|
||||
ip="MANDATORY:the ip address of the management interface"
|
||||
mac="MANDATORY:the mac of the management interface"
|
||||
ip="OPTIONAL:the ip address of the management interface"
|
||||
mac="OPTIONAL:the mac of the management interface"
|
||||
mgt=switch
|
||||
netboot=onie
|
||||
nodetype=switch
|
||||
switchtype="OPTIONAL: the manufacturer of switch"
|
||||
switchtype=onie
|
||||
provmethod="OPTIONAL: the full path of the cumulus installer"
|
||||
usercomment="the template for cumulus switch definition"
|
||||
Reference in New Issue
Block a user