2
0
mirror of https://github.com/xcat2/xcat-core.git synced 2025-05-21 19:22:05 +00:00

Check in init version of cumulus support[Do Not Merge] (#2020)

* Check in init version of cumulus support

* Add cumulus installation doc

* modify victor's comments.
This commit is contained in:
cxhong 2016-11-07 21:21:56 -05:00 committed by yangsong
parent 3b18361833
commit 1cd4b6446c
6 changed files with 596 additions and 18 deletions

View File

@ -0,0 +1,129 @@
Edgecore Switch
===============
The Edgecore switch from Mellanox is 1Gb top-of-rack switch. It's coming with ONIE installer. Open Network Install Environment (ONIE)is an open source initiative that defines an open "install environment" for bare metal network switches. For more information, see http://onie.opencompute.org/. Mellanox will ship the switch with Cumulus Network OS and along with a license file installed. In some case, user may get whitebox without OS and licenses. Since edgecore switch has different configuration than other switches that xCAT supports, xCAT handles edgecore switch differently.
ONIE Mode
---------
If the switch arrives without an OS pre-loaded, the ONIE installer and management port is the only thing enabled on the switch. Once the switch connects to the xCAT network, the switch should get a dynamic IP address. The xCAT DHCP server will get requests from the onie-installer from the switch and attempt to find an OS binary file to execute. The following messages will be logged in /var/log/messages on the management node. ::
Info: Fetching http://172.1.0.1/onie-installer-arm-accton_as4610_54-r0 ...
Info: Fetching http://172.1.0.1/onie-installer-arm-accton_as4610_54 ...
Info: Fetching http://172.1.0.1/onie-installer-accton_as4610_54 ...
Info: Fetching http://172.1.0.1/onie-installer-arm ...
Info: Fetching http://172.1.0.1/onie-installer .
To remove the installed Cumulus Linux OS to boot back to ONIE mode, connect to the switch via serial-port or ssh and execute the following commands: ::
ssh cumulus@172.1.0.1
#clean up all the configuration
sudo onie-select -k
sudo reboot
#boot back to onie mode
sudo onie-select -i
sudo reboot
After switch reboots, it will enter ONIE mode and send DHCP request to attempt to fetch the OS binary file.
Discover Edgecore Switch
------------------------
ONIE supports a number of methods for locating OS binary file. xCAT choose to use a DHCP server to provide specific information to the switch.
* IP address of the switch
* URL of the OS binary file on the Web server
With the xCAT DHCP configuration, ONIE picks up an IP address and downloads the URL specified by the user and start to install of the OS. The steps take to discover the edgecore switch and process request from ONIE installer as follows:
#. Pre-define switch object into xCAT db, make sure ip adress, netboot and provemethod are set, also define core switch and port number where edgecore switch connect to. ::
#lsdef edgecoresw1
Object name: edgecoresw1
groups=switch
ip=192.168.23.1
mgt=switch
netboot=onie
nodetype=switch
postbootscripts=otherpkgs
postscripts=syslog,remoteshell,syncfiles
provmethod=/install/custom/sw/edgecore/cumulus-linux-3.1.0-bcm-armel-1471981017.dc7e2adzfb43f6b.bin
switch=switch-10-5-23-1
switchport=1
#makehosts edgecoresw1
#. Run ``switchdiscover`` command, it will find edgecore switch and update mac address on pre-defined switch node definition. ::
#switchdiscover --range 192.168.5.170-190 -s nmap
#lsdef edgecoresw1
Object name: edgecoresw1
groups=switch
ip=192.168.23.1
mac=8C:EA:1B:12:CA:40
mgt=switch
netboot=onie
nodetype=switch
postbootscripts=otherpkgs
postscripts=syslog,remoteshell,syncfiles
provmethod=/install/custom/sw/edgecore/cumulus-linux-3.1.0-bcm-armel-1471981017.dc7e2adzfb43f6b.bin
status=Matched
switch=switch-10-5-23-1
switchport=1
switchtype=cumulus
usercomment=Edgecore switch
#. Run ``makedhcp`` after edgecore switch discovered, it will update ``dhcpd.conf`` and response the DHCP request from the onie-installer. ::
#makedhcp -n
#makedhcp -a edgecoresw1
#. Installation of the Cumulus Linux OS takes about 50 minutes. Monitor the /var/log/messages to check the status of the installation. ::
Oct 27 15:28:08 fs4 dhcpd: DHCPDISCOVER from 8c:ea:1b:12:ca:40 via enP4p1s0f2
Oct 27 15:28:08 fs4 dhcpd: DHCPOFFER on 192.168.23.1 to 8c:ea:1b:12:ca:40 via enP4p1s0f2
Oct 27 15:28:08 fs4 dhcpd: DHCPREQUEST for 192.168.23.1 (192.168.3.25) from 8c:ea:1b:12:ca:40 via enP4p1s0f2
Oct 27 15:28:08 fs4 dhcpd: DHCPACK on 192.168.23.1 to 8c:ea:1b:12:ca:40 via enP4p1s0f2
#. Once installation finished, the pre-defined switch name and IP address will be configured on edgecore switch. ::
cumulus@edgecoresw1:~$ ifconfig
eth0 Link encap:Ethernet HWaddr 8c:ea:1b:12:ca:40
inet addr:192.168.23.1 Bcast:192.168.255.255 Mask:255.255.0.0
inet6 addr: fe80::8eea:1bff:fe12:ca40/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
cumulus@edgecoresw1:~$ hostname
edgecoresw1
Configure Edgecore Switch
-------------------------
xCAT provides a script ``/opt/xcat/share/xcat/script/configcumulus`` to configure attributes in the Cumulus Switch. Use the ``--help`` option to see more details. ::
#configcumulus --help
Usage:
configcumulus [-?│-h│--help]
configcumulus [--switches switchnames] [--all]
configcumulus [--switches switchnames] [--ssh]
configcumulus [--switches switchnames] [--license filename ]
configcumulus [--switches switchnames] [--snmp] [--user snmp_user] [--password snmp_password]
configcumulus [--switches switchnames] [--ntp]

View File

@ -0,0 +1,7 @@
Edgecore Switches
=================
.. toctree::
:maxdepth: 2
edgecore_switches.rst

View File

@ -8,3 +8,4 @@ Networks
switchdiscover/index.rst
infiniband/index.rst
ipv6/index.rst
edgecore_switches/index.rst

View File

@ -526,7 +526,7 @@ sub addnode
{
$guess_next_server = 1;
}
if ($nrent->{netboot} and $nrent->{netboot} eq 'petitboot') {
if ($nrent->{netboot} and ($nrent->{netboot} eq 'petitboot' or $nrent->{netboot} eq 'onie' )) {
if ($guess_next_server) {
my $node_server = undef;
if ($nrent->{xcatmaster}) {
@ -693,6 +693,8 @@ sub addnode
$lstatements = 'filename = \"/boot/grub2/grub2-' . $node . '\";' . $lstatements;
} elsif ($nrent and $nrent->{netboot} and $nrent->{netboot} eq 'petitboot') {
$lstatements = 'option conf-file \"http://' . $nxtsrv . '/tftpboot/petitboot/' . $node . '\";' . $lstatements;
} elsif ($nrent and $nrent->{netboot} and $nrent->{netboot} eq 'onie') {
$lstatements = 'if option vendor-class-identifier = \"onie_vendor:arm-accton_as4610_54-r0\" { option www-server = \"http://' . $nxtsrv . $ntent->{provmethod} . '\";}' . $lstatements;
} elsif ($nrent and $nrent->{netboot} and $nrent->{netboot} eq 'nimol') {
$lstatements = 'supersede server.filename=\"/vios/nodes/' . $node . '\";' . $lstatements;
}
@ -1965,7 +1967,7 @@ sub process_request
my $nodetypetab;
$nodetypetab = xCAT::Table->new('nodetype', -create => 0);
if ($nodetypetab) {
$nodetypeents = $nodetypetab->getNodesAttribs($req->{node}, [qw(os)]);
$nodetypeents = $nodetypetab->getNodesAttribs($req->{node}, [qw(os provmethod)]);
}
my $iscsitab = xCAT::Table->new('iscsi', -create => 0);
if ($iscsitab) {
@ -2921,6 +2923,7 @@ sub newconfig
push @dhcpconf, "option tcode \"" . $::XCATSITEVALS{timezone} . "\";\n";
}
push @dhcpconf, "option gpxe.no-pxedhcp 1;\n";
push @dhcpconf, "option www-server code 114 = string;\n";
push @dhcpconf, "\n";
push @dhcpconf, "omapi-port 7911;\n"; #Enable omapi...
push @dhcpconf, "key xcat_key {\n";

View File

@ -32,6 +32,14 @@ my %global_scan_type = (
snmp => "snmp_scan"
);
my %global_mac_identity = (
"a8:97:dc" => "BNT G8052 switch",
"6c:ae:8b" => "BNT G8264-T switch",
"fc:cf:62" => "BNT G8124 switch",
"7c:fe:90" => "Mellanox IB switch",
"8c:ea:1b" => "Edgecore switch"
);
my %global_switch_type = (
Juniper => "Juniper",
juniper => "Juniper",
@ -44,7 +52,8 @@ my %global_switch_type = (
Mellanox => "Mellanox",
mellanox => "Mellanox",
MLNX => "Mellanox",
MELLAN => "Mellanox"
MELLAN => "Mellanox",
Edgecore => "cumulus"
);
@ -727,28 +736,42 @@ sub nmap_scan {
if ($vendor) {
my $search_string = join '|', keys(%global_switch_type);
if ($vendor =~ /($search_string)/) {
$switches->{$mac}->{ip} = $ip;
$switches->{$mac}->{vendor} = $vendor;
$switches->{$mac}->{name} = $host->{hostname};
$found = 1;
if (exists($globalopt{verbose})) {
send_msg($request, 0, "FOUND Switch: ip=$ip, mac=$mac, vendor=$vendor\n");
}
}
}
if ( ($found == 0) && ($type eq "mac") ) {
push(@$osguess_ips, $ip);
} # end nmap osscan command
my $search_string = join '|', keys(%global_mac_identity);
if ($mac =~ /($search_string)/i) {
my $key = $1;
$vendor = $global_mac_identity{lc $key};
$found = 1;
}
# still not found, used nmap osscan command
if ( $found == 0) {
push(@$osguess_ips, $ip);
}
}
if ($found == 1) {
$switches->{$mac}->{ip} = $ip;
$switches->{$mac}->{vendor} = $vendor;
$switches->{$mac}->{name} = $host->{hostname};
if (exists($globalopt{verbose})) {
send_msg($request, 0, "FOUND Switch: found = $found, ip=$ip, mac=$mac, vendor=$vendor\n");
}
}
} #end for each address
}
} #end for each host
}
}
my $guess_switches = nmap_osguess($request, $osguess_ips);
foreach my $guess_mac ( keys %$guess_switches ) {
$switches->{$guess_mac}->{ip} = $guess_switches->{$guess_mac}->{ip};;
$switches->{$guess_mac}->{vendor} = $guess_switches->{$guess_mac}->{vendor};
if (!$osguess_ips) {
my $guess_switches = nmap_osguess($request, $osguess_ips);
foreach my $guess_mac ( keys %$guess_switches ) {
$switches->{$guess_mac}->{ip} = $guess_switches->{$guess_mac}->{ip};;
$switches->{$guess_mac}->{vendor} = $guess_switches->{$guess_mac}->{vendor};
}
}
return $switches;
@ -1406,9 +1429,13 @@ sub switchsetup {
my $config_script = "$::XCATROOT/share/xcat/scripts/config".$mytype;
if (-r -x $config_script) {
my $switches = join(",",@{${nodes_to_config}->{$mytype}});
send_msg($request, 0, "call to config $mytype switches $switches\n");
my $out = `$config_script --switches $switches --all`;
send_msg($request, 0, "output = $out\n");
if ($mytype eq "cumulus") {
send_msg($request, 0, "Cumulus switch needs to take 50 mins to install, please run /opt/xcat/share/xcat/script/configcumulus after cumulus OS installed on switch\n");
} else {
send_msg($request, 0, "call to config $mytype switches $switches\n");
my $out = `$config_script --switches $switches --all`;
send_msg($request, 0, "output = $out\n");
}
} else {
send_msg($request, 0, "the switch type $mytype is not support yet\n");
}

View File

@ -0,0 +1,411 @@
#!/usr/bin/env perl
#---------------------------------------------------------
# Configure Ethnet BNT switches
#---------------------------------------------------------
BEGIN
{
$::XCATROOT = $ENV{'XCATROOT'} ? $ENV{'XCATROOT'} : '/opt/xcat';
$::XCATDIR = $ENV{'XCATDIR'} ? $ENV{'XCATDIR'} : '/etc/xcat';
}
use lib "$::XCATROOT/lib/perl";
use strict;
use Socket;
use Getopt::Long;
use Expect;
use Net::Ping;
use xCAT::Usage;
use xCAT::NodeRange;
use xCAT::NetworkUtils;
use xCAT::Utils;
use xCAT::Table;
use xCAT::MsgUtils;
Getopt::Long::Configure("bundling");
$Getopt::Long::ignorecase = 0;
#global variables
my @nodes;
my @filternodes;
#---------------------------------------------------------
#Main
# parse the options
if (
!GetOptions(
'h|help' => \$::HELP,
'switches=s' => \$::SWITCH,
'port=s' => \$::PORT,
'vlan=s' => \$::VLAN,
'user=s' => \$::USER,
'password=s' => \$::PASSWORD,
'snmp' => \$::SNMP,
'ssh' => \$::SSH,
'license=s' => \$::LICENSE,
'ntp' => \$::NTP,
'all' => \$::ALL,
)
)
{
&usage;
exit(1);
}
# display the usage if -h or --help is specified
if ($::HELP)
{
&usage;
exit(0);
}
if ($::SWITCH) {
my @filternodes = xCAT::NodeRange::noderange( $::SWITCH );
if (nodesmissed) {
my $nodenotdefined = join(',', nodesmissed);
xCAT::MsgUtils->message("I","The following nodes are not defined in xCAT DB: $nodenotdefined");
}
# check switch type
my $switchestab = xCAT::Table->new('switches');
my $switches_hash = $switchestab->getNodesAttribs(\@filternodes,['switchtype']);
foreach my $fsw (@filternodes) {
if (($switches_hash->{$fsw}->[0]->{switchtype}) =~ /cumulus/) {
push @nodes, $fsw;
} else {
xCAT::MsgUtils->message("E","The $fsw is not cumulus switch, will not config");
}
}
unless (@nodes) {
xCAT::MsgUtils->message("E","No Valid Switch to process");
exit(1);
}
} else {
xCAT::MsgUtils->message("E","Invalid flag, please provide switches with --switches");
&usage;
exit(1);
}
my $switches = join(",",@nodes);
my $cmd;
my $vlan;
my $port;
my $sub_req;
my $rc;
if (($::SSH) || ($::ALL))
{
config_ssh();
}
if (($::LICENSE) || ($::ALL))
{
install_license();
}
if (($::SNMP) || ($::ALL))
{
config_snmp();
}
if (($::NTP) || ($::ALL))
{
config_ntp();
}
if ($::VLAN)
{
#config_vlan();
}
sub config_ssh {
my $password = "CumulusLinux!";
my $userid = "cumulus";
my $timeout = 10;
my $keyfile = "/root/.ssh/id_rsa.pub";
my $rootkey = `cat /root/.ssh/id_rsa.pub`;
my $cmd;
my @config_switches;
foreach my $switch (@nodes) {
#remove old host key from /root/.ssh/known_hosts
$cmd = `ssh-keygen -R $switch`;
my ($exp, $errstr) = cumulus_connect($switch, $userid, $password, $timeout);
if (!defined $exp) {
print ("connect failed $errstr\n");
next;
}
my $ret;
my $err;
($ret, $err) = cumulus_exec($exp, "mkdir -p /root/.ssh");
($ret, $err) = cumulus_exec($exp, "chmod 700 /root/.ssh");
($ret, $err) = cumulus_exec($exp, "echo \"$rootkey\" >/root/.ssh/authorized_keys");
($ret, $err) = cumulus_exec($exp, "chmod 644 /root/.ssh/authorized_keys");
$exp->hard_close();
push (@config_switches, $switch);
}
if (@config_switches) {
#update switch status
my $csw = join(",",@config_switches);
$cmd = "chdef $csw status=ssh_configed ";
$rc= xCAT::Utils->runcmd($cmd, 0);
print "ssh configured for $csw\n";
}
}
sub cumulus_connect {
my $server = shift;
my $userid = shift;
my $password = shift;
my $timeout = shift;
my $ssh = Expect->new;
my $command = 'ssh';
my @parameters = ($userid . "@" . $server);
$ssh->debug(0);
$ssh->log_stdout(0); # suppress stdout output..
$ssh->slave->stty(qw(sane -echo));
unless ($ssh->spawn($command, @parameters))
{
my $err = $!;
$ssh->soft_close();
my $rsp;
return(undef, "unable to run command $command $err\n");
}
$ssh->expect($timeout,
[ "-re", qr/WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED/, sub {die "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!\n"; } ],
[ "-re", qr/\(yes\/no\)\?\s*$/, sub { $ssh->send("yes\n"); exp_continue; } ],
[ "-re", qr/ password:/, sub {$ssh->send("$password\n"); exp_continue; } ],
[ "-re", qr/:~\$/, sub { $ssh->send("sudo su\n"); exp_continue; } ],
[ "-re", qr/ password for cumulus:/, sub { $ssh->send("$password\n"); exp_continue; } ],
[ "-re", qr/.*\/home\/cumulus#/, sub { $ssh->clear_accum(); } ],
[ timeout => sub { die "No login.\n"; } ]
);
$ssh->clear_accum();
return ($ssh);
}
sub cumulus_exec {
my $exp = shift;
my $cmd = shift;
my $timeout = shift;
my $prompt = shift;
$timeout = 10 unless defined $timeout;
$prompt = qr/.*\/home\/cumulus#/ unless defined $prompt;
$exp->clear_accum();
$exp->send("$cmd\n");
my ($mpos, $merr, $mstr, $mbmatch, $mamatch) = $exp->expect(6, "-re", $prompt);
if (defined $merr) {
return(undef,$merr);
}
return($mbmatch);
}
# for cumulus switch, need to set the license file
sub install_license {
my @config_switches;
print "install_license\n";
my $license_file;
my $file_name = "/root/license.txt";
if ($::LICENSE) {
$license_file = $::LICENSE;
}
print "file = $license_file\n";
if (!(-e $license_file) ) {
print "$license_file is not exist\n";
}
foreach my $switch (@nodes) {
my $cmd = "xdcp $switch $license_file $file_name";
$rc= xCAT::Utils->runcmd($cmd, 0);
if ($::RUNCMD_RC != 0) {
xCAT::MsgUtils->message("E","Failed to xscp $license_file to $switch");
next;
}
$cmd = "xdsh $switch '/usr/cumulus/bin/cl-license -i $file_name' ";
$rc= xCAT::Utils->runcmd($cmd, 0);
if ($::RUNCMD_RC != 0) {
xCAT::MsgUtils->message("E","Failed to $cmd to $switch");
next;
}
push (@config_switches, $switch);
}
if (@config_switches) {
my $csw = join(",",@config_switches);
print "license is installed on $csw\n";
}
}
#setup secure SNMP v3
sub config_snmp {
my $snmp_user;
my $snmp_passwd;
my @config_switches;
my $cmd;
print "start to config_snmp\n";
if ($::USER) {
$snmp_user = $::USER;
} else {
$snmp_user = "xcatadmin";
}
if ($::PASSWORD) {
$snmp_passwd = $::PASSWORD;
} else {
$snmp_passwd = "xcatpassw0rd";
}
my $file = "temp.txt";
open(FILE , ">$file")
or die "cannot open file $file\n";
print FILE "#xCAT modify following line\n";
print FILE "agentAddress udp:161,udp6:[::1]:161\n";
print FILE "rocommunity public default\n";
print FILE "rocommunity public default -V systemonly\n";
print FILE "createUser $snmp_user SHA $snmp_passwd\n";
print FILE "rwuser $snmp_user\n";
foreach my $switch (@nodes) {
#check if xdsh works
$cmd = "xdsh $switch date";
$rc= xCAT::Utils->runcmd($cmd, 0);
if ($::RUNCMD_RC != 0) {
xCAT::MsgUtils->message("E","xdsh command to $switch failed");
next;
}
my $cmd_line = "sed -i 's/^agentAddress/#agentAddress/g' /etc/snmp/snmpd.conf";
$cmd = "xdsh $switch $cmd_line";
$rc= xCAT::Utils->runcmd($cmd, 0);
$cmd = "xdcp $switch $file";
$rc= xCAT::Utils->runcmd($cmd, 0);
$cmd = "xdsh $switch 'cat $file >> /etc/snmp/snmpd.conf;rm -fr $file;systemctl restart snmpd;systemctl enable snmpd' ";
$rc= xCAT::Utils->runcmd($cmd, 0);
if ($::RUNCMD_RC != 0) {
xCAT::MsgUtils->message("E","Failed to update snmpd.conf for $switch");
next;
}
push (@config_switches, $switch);
}
close FILE;
$cmd = `rm -rf $file`;
if (@config_switches) {
#update switch status
my $csw = join(",",@config_switches);
$cmd = "chdef $csw status=snmp_configed snmpversion=3 snmpauth=sha snmpusername=$snmp_user snmppassword=$snmp_passwd";
$rc= xCAT::Utils->runcmd($cmd, 0);
}
}
sub config_ntp {
my @config_switches;
my $cmd;
my $master = `hostname -i`;
my $file = "temp.txt";
open(FILE , ">$file")
or die "cannot open file $file\n";
print FILE "#This file is created by xCAT \n";
print FILE "driftfile /var/lib/ntp/drift\n";
print FILE "disable auth\n";
print FILE "restrict 127.0.0.1\n";
print FILE "server $master iburst\n";
print FILE "interface listen eth0\n";
foreach my $switch (@nodes) {
#check if xdsh works
$cmd = "xdsh $switch date";
$rc= xCAT::Utils->runcmd($cmd, 0);
if ($::RUNCMD_RC != 0) {
xCAT::MsgUtils->message("E","xdsh command to $switch failed");
next;
}
my $cmd_line = "echo 'US/Eastern'>/etc/timezone;dpkg-reconfigure --frontend noninteractive tzdata";
print "$cmd_line\n";
$cmd = "xdsh $switch $cmd_line";
if ($::RUNCMD_RC != 0) {
print "Failed to update ntp timezone\n";
xCAT::MsgUtils->message("E","Failed to update ntp timezone for $switch");
next;
}
print "$cmd\n";
$cmd = "xdcp $switch $file";
$rc= xCAT::Utils->runcmd($cmd, 0);
$cmd = "xdsh $switch 'cp /etc/ntp.conf /etc/ntp.conf.orig;cp $file /etc/ntp.conf;rm -fr $file;systemctl restart ntp;systemctl enable ntp' ";
$rc= xCAT::Utils->runcmd($cmd, 0);
if ($::RUNCMD_RC != 0) {
xCAT::MsgUtils->message("E","Failed to update ntp for $switch");
next;
}
push (@config_switches, $switch);
}
close FILE;
$cmd = `rm -rf $file`;
if (@config_switches) {
#update switch status
my $csw = join(",",@config_switches);
$cmd = "chdef $csw status=ntp_configed";
$rc= xCAT::Utils->runcmd($cmd, 0);
}
}
sub config_vlan {
if ($::PORT) {
$port = $::PORT;
} else {
&usage;
exit(1);
}
$vlan = $::VLAN;
print "Tagging VLAN=$vlan for $switches port $port\n";
#create vlan, tagged vlan
#$cmd = `xdsh $switches --devicetype EthSwitch::BNT "enable;configure terminal;vlan $vlan;exit;interface port $port;switchport mode trunk;switchport trunk allowed vlan $vlan;write memory;exit;exit"`;
}
#---------------------------------------------------------
=head3 usage
Displays message for -h option
=cut
#---------------------------------------------------------
sub usage
{
print "Usage:
configBNT [-?│-h│--help]
configBNT [--switches switchnames] [--all]
configBNT [--switches switchnames] [--ssh]
configBNT [--switches switchnames] [--license filename ]
configBNT [--switches switchnames] [--snmp] [--user snmp_user] [--password snmp_password] [--group snmp_group]
configBNT [--switches switchnames] [--ntp]
configBNT [--switches switchnames] [--port port] [--vlan vlan]
\n";
}