From 4f771e3dc3441bd763614987d8f873265667f390 Mon Sep 17 00:00:00 2001 From: immarvin Date: Thu, 10 Nov 2016 22:34:56 -0500 Subject: [PATCH 1/3] refine the cumulus support doc --- .../edgecore_switches/edgecore_switches.rst | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst b/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst index db9a3cc11..63b60c24c 100644 --- a/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst +++ b/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst @@ -1,7 +1,18 @@ Edgecore Switch =============== -The Edgecore switch from Mellanox is 1Gb top-of-rack switch. It's coming with ONIE installer. Open Network Install Environment (ONIE)is an open source initiative that defines an open "install environment" for bare metal network switches. For more information, see http://onie.opencompute.org/. Mellanox will ship the switch with Cumulus Network OS and along with a license file installed. In some case, user may get whitebox without OS and licenses. Since edgecore switch has different configuration than other switches that xCAT supports, xCAT handles edgecore switch differently. +The Edgecore switch from Mellanox is 1Gb top-of-rack switch. Usually, Mellanox ships the switch with Cumulus Network OS(https://cumulusnetworks.com) and along with a license file installed. In some case, user may get whitebox switch hardware without OS and license installed. Since edgecore switch is different from other traditional switches that xCAT supports, xCAT supports edgecore switch in a different way. Currently, the features provided by xCAT includes: :: + + 1) Cumulus Network OS provision + 2) switch discovery + 3) switch configuration: + (a) enable root-passwordless ssh + (b) install licence file + (c) enable snmp + 4) distribute files with ``xdcp`` + 5) invoke commands or scripts remotely with `xdsh`` + 6) + ONIE Mode From 2e1e2732670de8c8ad50981d6331d68ed82b8f42 Mon Sep 17 00:00:00 2001 From: immarvin Date: Tue, 15 Nov 2016 02:27:58 -0500 Subject: [PATCH 2/3] refine cumulus support doc --- .../edgecore_switches/edgecore_switches.rst | 20 +++++++++++++++---- .../objects/node/cumulusswitch.stanza | 6 +++--- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst b/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst index 63b60c24c..31e1bd238 100644 --- a/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst +++ b/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst @@ -1,7 +1,9 @@ Edgecore Switch =============== -The Edgecore switch from Mellanox is 1Gb top-of-rack switch. Usually, Mellanox ships the switch with Cumulus Network OS(https://cumulusnetworks.com) and along with a license file installed. In some case, user may get whitebox switch hardware without OS and license installed. Since edgecore switch is different from other traditional switches that xCAT supports, xCAT supports edgecore switch in a different way. Currently, the features provided by xCAT includes: :: +The Edgecore switch from Mellanox is 1Gb top-of-rack switch. Usually, Mellanox ships the switch with Cumulus Network OS(https://cumulusnetworks.com) and along with a license file installed. In some case, user may get whitebox switch hardware without OS and license installed. + +Since edgecore switch is different from other traditional switches that xCAT supports, xCAT supports edgecore switch in a different way. Currently, the features provided by xCAT include: :: 1) Cumulus Network OS provision 2) switch discovery @@ -9,11 +11,21 @@ The Edgecore switch from Mellanox is 1Gb top-of-rack switch. Usually, Mellanox s (a) enable root-passwordless ssh (b) install licence file (c) enable snmp - 4) distribute files with ``xdcp`` - 5) invoke commands or scripts remotely with `xdsh`` - 6) + 4) distribute files to switches with ``xdcp`` + 5) invoke remote commands or scripts on switches with `xdsh`` + 6) configure switches with ``updatenode`` + +This documentation presents a typical workflow on how to setup a edgecore switch from white box, then configure and manage the switch with xCAT. +Create a edgecore switch object +------------------------------- + +If you have the information of the ip and mac information of the switch, the edgecore switch object definition can be created with the "cumulusswitch" template shipped in xCAT : :: + + mkdef edgecoresw1 --template cumulusswitch arch=armv71 ip=192.168.5.191 mac=8C:EA:1B:12:CA:40 + +Discover the switch ONIE Mode --------- diff --git a/xCAT/templates/objects/node/cumulusswitch.stanza b/xCAT/templates/objects/node/cumulusswitch.stanza index 48575c911..cff7d05ac 100644 --- a/xCAT/templates/objects/node/cumulusswitch.stanza +++ b/xCAT/templates/objects/node/cumulusswitch.stanza @@ -4,11 +4,11 @@ cumulusswitch: objtype=node arch="OPTIONAL: the arch of the switch management service processor, such as armv71" groups=switch - ip="MANDATORY:the ip address of the management interface" - mac="MANDATORY:the mac of the management interface" + ip="OPTIONAL:the ip address of the management interface" + mac="OPTIONAL:the mac of the management interface" mgt=switch netboot=onie nodetype=switch - switchtype="OPTIONAL: the manufacturer of switch" + switchtype=edgecore provmethod="OPTIONAL: the full path of the cumulus installer" usercomment="the template for cumulus switch definition" From 5c9a3624d6800a79a160aeccb02ddcb8a55950d3 Mon Sep 17 00:00:00 2001 From: immarvin Date: Fri, 16 Dec 2016 09:42:50 -0500 Subject: [PATCH 3/3] refine the doc on cumulus switch --- .../edgecore_switches/edgecore_switches.rst | 152 ------------------ .../networks/edgecore_switches/index.rst | 6 +- .../edgecore_switches/onie_switches.rst | 135 ++++++++++++++++ xCAT/postscripts/enablesnmp | 26 +++ ...cumulusswitch.stanza => onieswitch.stanza} | 6 +- 5 files changed, 167 insertions(+), 158 deletions(-) delete mode 100644 docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst create mode 100644 docs/source/advanced/networks/edgecore_switches/onie_switches.rst create mode 100755 xCAT/postscripts/enablesnmp rename xCAT/templates/objects/node/{cumulusswitch.stanza => onieswitch.stanza} (82%) diff --git a/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst b/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst deleted file mode 100644 index 31e1bd238..000000000 --- a/docs/source/advanced/networks/edgecore_switches/edgecore_switches.rst +++ /dev/null @@ -1,152 +0,0 @@ -Edgecore Switch -=============== - -The Edgecore switch from Mellanox is 1Gb top-of-rack switch. Usually, Mellanox ships the switch with Cumulus Network OS(https://cumulusnetworks.com) and along with a license file installed. In some case, user may get whitebox switch hardware without OS and license installed. - -Since edgecore switch is different from other traditional switches that xCAT supports, xCAT supports edgecore switch in a different way. Currently, the features provided by xCAT include: :: - - 1) Cumulus Network OS provision - 2) switch discovery - 3) switch configuration: - (a) enable root-passwordless ssh - (b) install licence file - (c) enable snmp - 4) distribute files to switches with ``xdcp`` - 5) invoke remote commands or scripts on switches with `xdsh`` - 6) configure switches with ``updatenode`` - -This documentation presents a typical workflow on how to setup a edgecore switch from white box, then configure and manage the switch with xCAT. - - -Create a edgecore switch object -------------------------------- - -If you have the information of the ip and mac information of the switch, the edgecore switch object definition can be created with the "cumulusswitch" template shipped in xCAT : :: - - mkdef edgecoresw1 --template cumulusswitch arch=armv71 ip=192.168.5.191 mac=8C:EA:1B:12:CA:40 - -Discover the switch - -ONIE Mode ---------- - -If the switch arrives without an OS pre-loaded, the ONIE installer and management port is the only thing enabled on the switch. Once the switch connects to the xCAT network, the switch should get a dynamic IP address. The xCAT DHCP server will get requests from the onie-installer from the switch and attempt to find an OS binary file to execute. The following messages will be logged in /var/log/messages on the management node. :: - - Info: Fetching http://172.1.0.1/onie-installer-arm-accton_as4610_54-r0 ... - Info: Fetching http://172.1.0.1/onie-installer-arm-accton_as4610_54 ... - Info: Fetching http://172.1.0.1/onie-installer-accton_as4610_54 ... - Info: Fetching http://172.1.0.1/onie-installer-arm ... - Info: Fetching http://172.1.0.1/onie-installer . - - -To remove the installed Cumulus Linux OS to boot back to ONIE mode, connect to the switch via serial-port or ssh and execute the following commands: :: - - ssh cumulus@172.1.0.1 - #clean up all the configuration - sudo onie-select -k - sudo reboot - #boot back to onie mode - sudo onie-select -i - sudo reboot - - -After switch reboots, it will enter ONIE mode and send DHCP request to attempt to fetch the OS binary file. - - -Discover Edgecore Switch ------------------------- - -ONIE supports a number of methods for locating OS binary file. xCAT choose to use a DHCP server to provide specific information to the switch. - -* IP address of the switch -* URL of the OS binary file on the Web server - -With the xCAT DHCP configuration, ONIE picks up an IP address and downloads the URL specified by the user and start to install of the OS. The steps take to discover the edgecore switch and process request from ONIE installer as follows: - -#. Pre-define switch object into xCAT db, make sure ip adress, netboot and provemethod are set, also define core switch and port number where edgecore switch connect to. :: - - - #lsdef edgecoresw1 - Object name: edgecoresw1 - groups=switch - ip=192.168.23.1 - mgt=switch - netboot=onie - nodetype=switch - postbootscripts=otherpkgs - postscripts=syslog,remoteshell,syncfiles - provmethod=/install/custom/sw/edgecore/cumulus-linux-3.1.0-bcm-armel-1471981017.dc7e2adzfb43f6b.bin - switch=switch-10-5-23-1 - switchport=1 - - #makehosts edgecoresw1 - - -#. Run ``switchdiscover`` command, it will find edgecore switch and update mac address on pre-defined switch node definition. :: - - #switchdiscover --range 192.168.5.170-190 -s nmap - #lsdef edgecoresw1 - Object name: edgecoresw1 - groups=switch - ip=192.168.23.1 - mac=8C:EA:1B:12:CA:40 - mgt=switch - netboot=onie - nodetype=switch - postbootscripts=otherpkgs - postscripts=syslog,remoteshell,syncfiles - provmethod=/install/custom/sw/edgecore/cumulus-linux-3.1.0-bcm-armel-1471981017.dc7e2adzfb43f6b.bin - status=Matched - switch=switch-10-5-23-1 - switchport=1 - switchtype=cumulus - usercomment=Edgecore switch - - -#. Run ``makedhcp`` after edgecore switch discovered, it will update ``dhcpd.conf`` and response the DHCP request from the onie-installer. :: - - #makedhcp -n - #makedhcp -a edgecoresw1 - - -#. Installation of the Cumulus Linux OS takes about 50 minutes. Monitor the /var/log/messages to check the status of the installation. :: - - - Oct 27 15:28:08 fs4 dhcpd: DHCPDISCOVER from 8c:ea:1b:12:ca:40 via enP4p1s0f2 - Oct 27 15:28:08 fs4 dhcpd: DHCPOFFER on 192.168.23.1 to 8c:ea:1b:12:ca:40 via enP4p1s0f2 - Oct 27 15:28:08 fs4 dhcpd: DHCPREQUEST for 192.168.23.1 (192.168.3.25) from 8c:ea:1b:12:ca:40 via enP4p1s0f2 - Oct 27 15:28:08 fs4 dhcpd: DHCPACK on 192.168.23.1 to 8c:ea:1b:12:ca:40 via enP4p1s0f2 - - -#. Once installation finished, the pre-defined switch name and IP address will be configured on edgecore switch. :: - - cumulus@edgecoresw1:~$ ifconfig - eth0 Link encap:Ethernet HWaddr 8c:ea:1b:12:ca:40 - inet addr:192.168.23.1 Bcast:192.168.255.255 Mask:255.255.0.0 - inet6 addr: fe80::8eea:1bff:fe12:ca40/64 Scope:Link - UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 - cumulus@edgecoresw1:~$ hostname - edgecoresw1 - - -Configure Edgecore Switch -------------------------- - -xCAT provides a script ``/opt/xcat/share/xcat/script/configcumulus`` to configure attributes in the Cumulus Switch. Use the ``--help`` option to see more details. :: - - #configcumulus --help - Usage: - configcumulus [-?│-h│--help] - configcumulus [--switches switchnames] [--all] - configcumulus [--switches switchnames] [--ssh] - configcumulus [--switches switchnames] [--license filename ] - configcumulus [--switches switchnames] [--snmp] [--user snmp_user] [--password snmp_password] - configcumulus [--switches switchnames] [--ntp] - - - - - - - - diff --git a/docs/source/advanced/networks/edgecore_switches/index.rst b/docs/source/advanced/networks/edgecore_switches/index.rst index 6d3bdf038..0c897d340 100644 --- a/docs/source/advanced/networks/edgecore_switches/index.rst +++ b/docs/source/advanced/networks/edgecore_switches/index.rst @@ -1,7 +1,7 @@ -Edgecore Switches -================= +ONIE Compatible Bare Metal Switches +=================================== .. toctree:: :maxdepth: 2 - edgecore_switches.rst + onie_switches.rst diff --git a/docs/source/advanced/networks/edgecore_switches/onie_switches.rst b/docs/source/advanced/networks/edgecore_switches/onie_switches.rst new file mode 100644 index 000000000..dd41a8c71 --- /dev/null +++ b/docs/source/advanced/networks/edgecore_switches/onie_switches.rst @@ -0,0 +1,135 @@ +ONIE compatible bare metal switch +================================= + +The ONIE [1]_. compatible bare metal switches(abbreviated as "ONIE switch") from vendors such as Mellanox or Edgecore are often used as top-of-rack switches in the cluster. Usually, the switches are shipped with a Cumulus Network OS(https://cumulusnetworks.com) and a license pre-installed. In some cases, user may get whitebox switch hardware with a standalone Cumulus installer and license file. This documentation presents a typical workflow on how to setup ONIE switch from white box, then configure and manage the switch with xCAT. + +.. [1] Open Network Install Environment: Created by Cumulus Networks, Inc. in 2012, the Open Network Install Environment (ONIE) Project is a small operating system, pre-installed as firmware on bare metal network switches, that provides an environment for automated operating system provisioning. + +Create an ONIE switch object +------------------------------- + +The ONIE switch object can be created with the "onieswitch" template shipped in xCAT, the ip address and mac of the switch management ethernet port should be specified : :: + + mkdef edgecoresw1 --template onieswitch arch=armv71 ip=192.168.5.191 mac=8C:EA:1B:12:CA:40 + +Provision the Cumulus OS on ONIE switch +--------------------------------------- + +To provision Cumulus OS, the Cumulus installation file, a binary shipped with the switch, should be saved in a directory exported in the http server. + +Run ``chdef`` to specify the "provmethod" attribute of the switch object to the full path of the installation file: :: + + chdef edgecoresw1 netboot=onie provmethod="/install/custom/sw/edgecore/cumulus-linux-3.1.0-bcm-armel-1471981017.dc7e2adzfb43f6b.bin" + +Run ``makedhcp`` to prepare the DHCP/BOOTP lease. :: + + makedhcp -a edgecoresw1 + +The command or operation to start the provision dependes on the status of switch: + +1. If the switch is a white box without Cumulus OS installed, simply connect the management ethernet port of the switch to xCAT management node, then power on the switch. + +2. If a Cumulus OS has been installed on the switch, you need to login to the switch(the default user is ``cumulus`` and the password is ``CumulusLinux!``) and run a batch of commands: :: + + sudo onie-select -i + sudo reboot + +If the passwordless-ssh of "root" has been enabled, the commands can be issued with: :: + + xdsh edgecoresw1 "/usr/cumulus/bin/onie-select -i -f;reboot" + +After reboot, the switch will enter ONIE install mode and begin the installation. The provision might take about 50 minutes. + + +Switch Configuration +-------------------- + +Enable the passwordless ssh for "root" +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In a newly installed Cumulus OS, a default user ``cumulus`` will be created, the switch can be accessed via ssh with the default password ``CumulusLinux!``. + +The passwordless ssh access of "root" should be enabled with the script ``/opt/xcat/share/xcat/scripts/configcumulus`` :: + + /opt/xcat/share/xcat/scripts/configcumulus --switches edgecoresw1 --ssh + +After the passwordless access for "root" is setup successfully, the switch can be managed with the node management commands such as ``xdsh``, ``xdcp`` and ``updatenode``, etc. + +Licence file installation +~~~~~~~~~~~~~~~~~~~~~~~~~ + +On the newly installed switch, only the serial console and the management ethernet port are enabled. To activate the data ports, the licence file shipped with the switch should be installed: :: + + xdcp edgecoresw1 /install/custom/sw/edgecore/licensefile.txt /tmp + xdsh edgecoresw1 "/usr/cumulus/bin/cl-license -i /tmp/licensefile.txt" + +To check whether the license file is installed successfully: :: + + ~: xdsh edgecoresw1 /usr/cumulus/bin/cl-license + edgecoresw1: xxx@xx.com|xxxxxxxxxxxxxxx + +Reboot the switch to apply the licence file: :: + + xdsh edgecoresw1 reboot + +Enable SNMP +~~~~~~~~~~~ + +The snmpd in the switch is not enabled by default, xCAT ships a postscript to enable it: :: + + updatenode edgecoresw1 -P enablesnmp + + +Switch Discovery +---------------- + +The ONIE switch can be scaned and discovered with ``switchdiscover`` :: + + ~: switchdiscover --range 192.168.23.1-10 + Discovering switches using nmap for 192.168.23.1-10. It may take long time... + ip name vendor mac + ------------ ------------ ------------ ------------ + 192.168.23.1 edgecoresw1 Edgecore switch 8C:EA:1B:12:CA:40 + Switch discovered: edgecoresw1 + +Once SNMP on the ONIE switch is enabled, the ONIE switch can be discovered with "snmp" method: :: + + ~: switchdiscover --range 192.168.23.1-10 -s snmp + Discovering switches using snmpwalk for 192.168.23.1-10 .... + ip name vendor mac + ------------ ------------ ------------ ------------ + 192.168.23.1 edgecoresw1 Linux edgecoresw1 4.1.0-cl-2-iproc #1 SMP Debian 4.1.25-1+cl3u4 (2016-08-13) armv7l 8c:ea:1b:12:ca:40 + Switch discovered: edgecoresw1 + + +Switch Management +----------------- + +File Dispatch +~~~~~~~~~~~~~ + +The files can be dispatched to ONIE switches with ``xdcp`` :: + + xdcp edgecoresw1 + +Refer to :doc:`xdcp manpage ` for details. + +Remote Commands +~~~~~~~~~~~~~~~ + +Commands can be run on ONIE switches remotely with ``xdsh`` :: + + xdsh edgecoresw1 + +Refer to :doc:`xdsh manpage ` for details. + +Run scripts remotely +~~~~~~~~~~~~~~~~~~~~ + +The scripts under "/install/postscripts" can be run on ONIE switches with ``updatenode -P`` :: + + updatenode edgecoresw1 -P