2
0
mirror of https://github.com/xcat2/xcat-core.git synced 2025-06-13 09:50:19 +00:00

The tool to get xCAT node range operation performance baseline (#2681) (#2916)

* Feature #2681, to add a script in xCAT-test to get the performance baseline for node range operation.

* adopt review comments in PR#2916:
  1, add the rst file for xcatperftest (#2681)
  2, using /bin/bash as script executor
  3, FILES in manpage should not be in the same line
  4, minor adjustment if PERF_NOCREATE/PERF_DRYRUN is set
This commit is contained in:
Bin Xu
2017-05-09 17:49:15 +08:00
committed by yangsong
parent a9844c9aa4
commit 8b8a82627b
6 changed files with 674 additions and 1 deletions

View File

@ -0,0 +1,213 @@
##############
xcatperftest.1
##############
.. highlight:: perl
****
NAME
****
\ **xcatperftest**\ - Run xCAT command performance baseline testing on fake nodes.
********
SYNOPSIS
********
\ **xcatperftest**\ [\ **-?|-h**\ ]
[\ **PERF_DRYRUN**\ =y] [\ **PERF_NOCREATE**\ =y] \ **xcatperftest**\ <total> [\ *command-list-file*\ ]
***********
DESCRIPTION
***********
The xcatperftest command runs commandes defined in a command list file and get their execution response time baseline for performance purpose.
The xcatperftest command is part of the xCAT package xCAT-test, and you can run it standalone or leverage it to build up your automation test cases.
Any commands could be defined in the command list file, however, it is recommended that the one-time initial configuration are well prepared prior to run xcatperftest command.
For example, the network object, osdistor and osimage image objects.
Follow the below steps to run xcatperftest command:
1, Install xCAT-test on a xCAT management nodes.
2, Prepare a command list in which the commands are what you want to messure.
3, Prepare the initial configuration based on the command list to make sure all commands could be executed in techinal.
4, Run xcatperftest with the total fake nodes number and the above command list file.
Node: It is suggested to run the command in background as it normally takes long time to finish all the performanc testing with large amount of fake nodes.
*******
OPTIONS
*******
\ **-?|-h**\
Display usage message.
<command-list-file>
Specifies the command list file with full-path. xCAT supports an example command file: /opt/xcat/share/xcat/tools/autotest/perfcmds.lst
<total>
Total number of fake nodes will be defined during the testing.
************
RETURN VALUE
************
0 The command completed successfully.
1 An error has occurred.
*****************
COMMAND LIST FILE
*****************
The command list file is in flat text format, the testing framework will parse the file line by line, here is an example of the commannd list file:
.. code-block:: perl
#SERIES# 1,50,100,250,500,1000,2500,5000
mkdef -z -f < #STANZ#
lsdef #NODES#
makehosts #NODES#
makedns -n #NODES#
makedhcp #NODES#
makeknownhosts #NODES#
nodech #NODES# groups,=group1
nodels #NODES# noderes
nodeset #NODES# osimage=rhels7.3-GA-ppc64le-install-compute
chdef -t node -o #NODES# postscripts="fake" profile="install" netboot="grub2"
rmdef -t node #PERFGRP#
mkdef -z < #STANZ#
noderm #PERFGRP#
\ **Note**\ : Each line defines one command, and the commands dependency should be handled by the line order.
If you define a node range series line (started with #SERIES#) in this file, xcatperftest will run the command for each node range defined in series line.
\ **#SERIES#**\ To define a node range series, and the series should be an comma split incremental number sequence.
\ **#STANZ#**\ It will be replaced with real stanz file path when this command line runs.
\ **#NODES#**\ It will be replaced with real node range defined in #SERIES# line when this command line runs. If no series line, the node group will be used.
\ **#PERFGRP#**\ It will be replaced with node group when this command line runs.
********************
ENVIRONMENT VARIABLE
********************
The xcatperftest command supports be customized by some environment variables.
\ **FAKE_NODE_PREFIX**\
Optional, the prefix of the fake compute node name. By default, the value is 'fake'
\ **FAKE_NODE_GROUP**\
# Optional, the group name of all the fake compute nodes. By default, the value is 'perftest'
\ **FAKE_NETWORK_PRO**\
Mandatory, the Provision network for all the fake compute nodes. By default, the value is '192.168'.
It must be a string like 'A.B', and be matched with \`tabdump networks\`
\ **FAKE_NETWORK_BMC**\
Mandatory, the BMC network for all the fake compute nodes. By default, the value is '192.168'. Note: It could not be the same subnet as 'FAKE_NETWORK_PRO'
It must be a string like 'A.B' and no need to be defined in 'networks' table.
\ **PERF_NODETEMPL**\
Optional, The node template name used for generating fake nodes. By default, it will be auto-detected according to the current arch.
\ **PERF_DRYRUN**\
Optional, Indicate no real commands will be executed if the environment variable is set.
\ **PERF_NOCREATE**\
Optional, Indicate no new fake nodes will be created if the environment variable is set.
********
EXAMPLES
********
1.
To run the performance testing for the commands defined in /tmp/cmd.lst on 5000 fake nodes:
.. code-block:: perl
xcatperftest 5000 /tmp/cmd.lst
2.
To generate an xCAT node object stanz file for 10000 nodes in subnet 10.100.0.0:
.. code-block:: perl
FAKE_NETWORK_PRO=10.100 FAKE_NETWORK_BMC=10.200 xcatperftest 10000
3.
To run the performance testing for the commands defined in /opt/xcat/share/xcat/tools/autotest/perfcmds.lst on 5000 existing fake nodes:
.. code-block:: perl
PERF_NOCREATE=y xcatperftest 5000 /opt/xcat/share/xcat/tools/autotest/perfcmds.lst
*****
FILES
*****
/opt/xcat/bin/xcatperftest
/opt/xcat/share/xcat/tools/autotest/perfcmds.lst

View File

@ -0,0 +1,13 @@
#SERIES# 1,50,100,250,500,1000,2500,5000,10000
lsdef #NODES#
makehosts #NODES#
makedns -n #NODES#
makedhcp #NODES#
makeknownhosts #NODES#
nodech #NODES# groups,=group1
nodels #NODES# noderes
nodeset #NODES# osimage=rhels7.3-GA-ppc64le-install-compute
chdef -t node -o #NODES# postscripts="fake" profile="install" netboot="grub2"
rmdef -t node #PERFGRP#
mkdef -z < #STANZ#
noderm #PERFGRP#

294
xCAT-test/bin/xcatperftest Executable file
View File

@ -0,0 +1,294 @@
#!/bin/bash
# IBM(c) 2017 EPL license http://www.eclipse.org/legal/epl-v10.html
#(C)IBM Corp
#
###################################################################
#
# Description:
# This script is used for performance testing purpose. It could
# generate n*250 fake nodes based on a predefined template (/opt/xcat/share/xcat/templates/objects/node/),
# and then run the performance testing on a batch of xCAT commands.
#
# Note: It is availabe for the commands which require management node only.
#
# Syntax:
# $prog <total> [command-list-file]
#
###################################################################
if [ -z $LC_ALL ]; then
export LC_ALL=C
fi
# Used for number parameter validation
isNumber()
{
expr $1 + 0 &>/dev/null
}
# Give a simple usage
if [ -z "$1" ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
echo "[PERF_DRYRUN=y] [PERF_NOCREATE=y] $0 <total> [command-list-file]"
exit
else
isNumber $1
if [ ! $? -eq 0 ]; then
echo "You must input an numeric string for total nodes number."
exit -1
fi
fi
# Source xCAT profile to make sure the xCAT commands could be executed without absolute path.
if [ -z $XCATROOT ]; then
if [ -r /etc/profile.d/xcat.sh ]; then
. /etc/profile.d/xcat.sh
fi
fi
# Mandatory, to specify the number of total fake nodes which will be created for testing
FAKE_NODE_TOTAL=$1
# Optional, the prefix of the fake compute node name.
# By default, it is 'fake' but it could be changed when you set environment variable `FAKE_NODE_PREFIX`
if [ -z $FAKE_NODE_PREFIX ]; then
FAKE_NODE_PREFIX='fake'
fi
# Optional, the group name of all the fake compute nodes.
# By default, it is 'perftest' but it could be changed when you set environment variable `FAKE_NODE_GROUP`
if [ -z $FAKE_NODE_GROUP ]; then
FAKE_NODE_GROUP='perftest'
fi
# Mandatory, the Provision network for all the fake compute nodes. It must be a string like 'A.B', and be matched with `tabdump networks`
# By default, it is '192.168' but it could be changed when you set environment variable `FAKE_NETWORK_PRO`
if [ -z $FAKE_NETWORK_PRO ]; then
FAKE_NETWORK_PRO='192.168'
fi
# Mandatory, the BMC network for all the fake compute nodes. It must be a string like 'A.B' and no need to be defined in 'networks' table.
# By default, it is '192.169' but it could be changed when you set environment variable `FAKE_NETWORK_BMC`
# Note: it could not be the same subnet as 'FAKE_NETWORK_PRO'
if [ -z $FAKE_NETWORK_BMC ]; then
FAKE_NETWORK_BMC='192.169'
fi
# Optional, The node template name used for generating fake nodes.
# By default, it is '<arch>-template' but it could be changed when you set environment variable `FAKE_NODE_GROUP`
if [ -z $PERF_NODETEMPL ]; then
PERF_NODETEMPL="`arch`-template"
fi
# IP address assinged to node will be in [1-250]
NODE_PER_ROW=250
PERFORMANCE_DIR=/tmp/xcat-performance
PERFORMANCE_NODE_TMPL=$PERFORMANCE_DIR/node.tmpl
PERFORMANCE_REPORT=$PERFORMANCE_DIR/report-$FAKE_NODE_TOTAL.$$
PERFORMANCE_STANZ=$PERFORMANCE_DIR/stanz-$FAKE_NODE_TOTAL.$$
# If the command list file is not specified, the tool will only create the stanz file for fake nodes.
# If it is specified but not exists, the tool will exit with error.
if [ ! -z $2 ]; then
if [ -f $2 ]; then
RUN_CMD_LIST=$2
else
echo "ERROR: The command list file you specified does not exist."
exit -1
fi
fi
# Get a random MAC address
genMAC()
{
printf '00:60:2F:%02X:%02X:%02X' $[RANDOM%256] $[RANDOM%256] $[RANDOM%256]
}
# Generate stanz file for all fake nodes
genStanz()
{
if [ ! -z $PERF_DRYRUN ]; then
echo $1, $2, $3, $4
else
echo -n .
fi
sed -e '/Object name:/c \'"$1"':\n objtype=node' \
-e '/ip=/c \ ip='"$3"'' \
-e '/mac=/c \ mac='"$2"'' \
-e '/bmc=/c \ bmc='"$4"'' \
-e '/bmcusername=/c \ bmcusername=fake' \
-e '/bmcpassword=/c \ bmcpassword=fake' \
-e '/groups=/c \ groups=all,'"$FAKE_NODE_GROUP"'' \
-e '/postscripts=/c \ postscripts=mypostboot' \
-e '/postbootscripts=/c \ postbootscripts=mypostboot' \
$PERFORMANCE_NODE_TMPL >> $PERFORMANCE_STANZ
}
# Create a fake xCAT node definition
fakeNode()
{
# TODO: support regular expression for IP
genStanz $FAKE_NODE_PREFIX$1 $(genMAC) $FAKE_NETWORK_PRO.$2.$3 $FAKE_NETWORK_BMC.$2.$3
#mkdef -f -t node $FAKE_NODE_PREFIX$1 --template $PERF_NODETEMPL ip=$FAKE_NETWORK_PRO.$2.$3 mac=$(genMAC) \
# bmc=$FAKE_NETWORK_BMC.$2.$3 bmcpassword=fake bmcusername=fake groups=all,performance > /dev/null 2>&1
}
# Create batch fake nodes stanz file for testing
bootstrap()
{
declare -i count=0
[ $rack = 0 ] && rack=1
for i in $(seq 0 $(expr $rack - 1))
do
for j in $(seq 1 $NODE_PER_ROW)
do
count+=1
fakeNode $count $i $j
[ $(($count % $1)) = 0 ] && echo
[ "x$count" == "x$FAKE_NODE_TOTAL" ] && break
done
done
}
# Get current time
getTime()
{
date +%s -d "$1"
}
# Executing the testing on specific commands defined in command list file
# All MACROs defined in command list file will be replaced with the real value
runTest()
{
cmd=$1
if [[ $cmd =~ '#STANZ#' ]]; then
#mkdef -z
execCmd "${cmd/\#STANZ\#/$PERFORMANCE_STANZ}" "$FAKE_NODE_TOTAL"
elif [[ $cmd =~ '#NODES#' ]]; then
#noderange operation
if [ -z "$2" ]; then
# No SERIES defined, run command on the whole group
execCmd "${cmd/\#NODES\#/$FAKE_NODE_GROUP}" "$FAKE_NODE_TOTAL"
else
# run the command for each number in SERIES
for num in $2
do
isNumber $num || continue
if [[ $num -le $FAKE_NODE_TOTAL ]]; then
cmd=$1
execCmd "${cmd/\#NODES\#/$FAKE_NODE_PREFIX[1-$num]}" "$num"
fi
done
fi
elif [[ $cmd =~ '#PERFGRP#' ]]; then
execCmd "${cmd/\#PERFGRP\#/$FAKE_NODE_GROUP}" "$FAKE_NODE_TOTAL"
else
execCmd "$cmd" "N/A"
fi
}
# Output performance result for each command.
printResult()
{
cmd=`echo "$1" | awk '{print $1}'`
result=$([[ $4 = 0 ]] && echo "SUCESS" || echo "FAIL")
# TOTAL, CMD, NODERANGE, TIME, SUCESS, FULL COMMAND
echo "$FAKE_NODE_TOTAL", "$cmd", "$2", "$3", $result, \"$1\" >> $PERFORMANCE_REPORT
}
# Executing each command and print the result to report file
execCmd()
{
echo "Testing for [ $1 ] ..."
if [ ! -z $PERF_DRYRUN ]; then
return
fi
starttime=`date +'%Y-%m-%d %H:%M:%S'`
start=$(getTime "$starttime")
eval "$1" > /dev/null 2>&1
retval=$?
endtime=`date +'%Y-%m-%d %H:%M:%S'`
end=$(getTime "$endtime")
printResult "$1" "$2" "$(($end-$start))" "$retval"
}
#################################################
# Main Loop of the performance baseline testing #
#################################################
lsxcatd -a
if [ 0 != $? ]; then
echo "ERROR: xCAT daemon is not running"
#exit 99
fi
mkdir -p /tmp/xcat-performance
lsdef -t node --template $PERF_NODETEMPL > $PERFORMANCE_NODE_TMPL 2>/dev/null
if [ 0 != $? ]; then
echo "ERROR: Cannot find the default template for `arch`, make sure it exists and rerun the script."
exit 99
fi
rack=$(expr $FAKE_NODE_TOTAL / $NODE_PER_ROW)
echo "=================================================="
# Starting to add fake nodes
starttime=`date +'%Y-%m-%d %H:%M:%S'`
start=$(getTime "$starttime")
bootstrap 50
endtime=`date +'%Y-%m-%d %H:%M:%S'`
end=$(getTime "$endtime")
echo
echo "=================================================="
#echo "It takes $(($end-$start)) seconds to create $FAKE_NODE_TOTAL nodes"
#echo "$FAKE_NODE_TOTAL", "$(($end-$start))", "mkdef" >> $PERFORMANCE_REPORT
if [ -z $RUN_CMD_LIST ]; then
echo "Done. Check the stanz file in $PERFORMANCE_STANZ"
exit 0
fi
echo "Continue the performance testing for commands in $RUN_CMD_LIST "
echo "=================================================="
# Initial Populate the fake nodes into DB
if [ -z $PERF_NOCREATE ]; then
execCmd "mkdef -z -f < $PERFORMANCE_STANZ" "$FAKE_NODE_TOTAL"
fi
series=`grep '^#SERIES#' $RUN_CMD_LIST | awk '{print $2}'`
if [ ! -z $series ]; then
series=${series//,/ }
fi
#echo $series
cmdlist=`cat $RUN_CMD_LIST`
IFS_BAK=$IFS
IFS=$'\n'
for line in $cmdlist
do
[ "x${line:0:1}" = "x#" ] && continue
# begin to run the command
IFS=$IFS_BAK
runTest "$line" "$series"
IFS=$'\n'
done
IFS=$IFS_BAK
IFS_BAK=
rm -f $PERFORMANCE_NODE_TMPL
rm -f $PERFORMANCE_STANZ
if [ -z $PERF_DRYRUN ]; then
echo
echo "Done. Check the performance result in $PERFORMANCE_REPORT"
fi

View File

@ -1,4 +1,5 @@
xcattest opt/xcat/bin
bin/xcatperftest opt/xcat/bin
share/man/man1/* opt/xcat/share/man/man1
share/doc/man1/* opt/xcat/share/doc/man1
autotest opt/xcat/share/xcat/tools
autotest opt/xcat/share/xcat/tools

View File

@ -0,0 +1,151 @@
=head1 NAME
B<xcatperftest> - Run xCAT command performance baseline testing on fake nodes.
=head1 SYNOPSIS
B<xcatperftest> [B<-?|-h>]
[B<PERF_DRYRUN>=y] [B<PERF_NOCREATE>=y] B<xcatperftest> <total> [I<command-list-file>]
=head1 DESCRIPTION
The xcatperftest command runs commandes defined in a command list file and get their execution response time baseline for performance purpose.
The xcatperftest command is part of the xCAT package xCAT-test, and you can run it standalone or leverage it to build up your automation test cases.
Any commands could be defined in the command list file, however, it is recommended that the one-time initial configuration are well prepared prior to run xcatperftest command.
For example, the network object, osdistor and osimage image objects.
Follow the below steps to run xcatperftest command:
1, Install xCAT-test on a xCAT management nodes.
2, Prepare a command list in which the commands are what you want to messure.
3, Prepare the initial configuration based on the command list to make sure all commands could be executed in techinal.
4, Run xcatperftest with the total fake nodes number and the above command list file.
Node: It is suggested to run the command in background as it normally takes long time to finish all the performanc testing with large amount of fake nodes.
=head1 OPTIONS
=over 10
=item B<-?|-h>
Display usage message.
=item <command-list-file>
Specifies the command list file with full-path. xCAT supports an example command file: /opt/xcat/share/xcat/tools/autotest/perfcmds.lst
=item <total>
Total number of fake nodes will be defined during the testing.
=back
=head1 RETURN VALUE
0 The command completed successfully.
1 An error has occurred.
=head1 COMMAND LIST FILE
The command list file is in flat text format, the testing framework will parse the file line by line, here is an example of the commannd list file:
#SERIES# 1,50,100,250,500,1000,2500,5000
mkdef -z -f < #STANZ#
lsdef #NODES#
makehosts #NODES#
makedns -n #NODES#
makedhcp #NODES#
makeknownhosts #NODES#
nodech #NODES# groups,=group1
nodels #NODES# noderes
nodeset #NODES# osimage=rhels7.3-GA-ppc64le-install-compute
chdef -t node -o #NODES# postscripts="fake" profile="install" netboot="grub2"
rmdef -t node #PERFGRP#
mkdef -z < #STANZ#
noderm #PERFGRP#
B<Note>: Each line defines one command, and the commands dependency should be handled by the line order.
If you define a node range series line (started with #SERIES#) in this file, xcatperftest will run the command for each node range defined in series line.
B<#SERIES#> To define a node range series, and the series should be an comma split incremental number sequence.
B<#STANZ#> It will be replaced with real stanz file path when this command line runs.
B<#NODES#> It will be replaced with real node range defined in #SERIES# line when this command line runs. If no series line, the node group will be used.
B<#PERFGRP#> It will be replaced with node group when this command line runs.
=head1 ENVIRONMENT VARIABLE
The xcatperftest command supports be customized by some environment variables.
B<FAKE_NODE_PREFIX>
Optional, the prefix of the fake compute node name. By default, the value is 'fake'
B<FAKE_NODE_GROUP>
# Optional, the group name of all the fake compute nodes. By default, the value is 'perftest'
B<FAKE_NETWORK_PRO>
Mandatory, the Provision network for all the fake compute nodes. By default, the value is '192.168'.
It must be a string like 'A.B', and be matched with `tabdump networks`
B<FAKE_NETWORK_BMC>
Mandatory, the BMC network for all the fake compute nodes. By default, the value is '192.168'. Note: It could not be the same subnet as 'FAKE_NETWORK_PRO'
It must be a string like 'A.B' and no need to be defined in 'networks' table.
B<PERF_NODETEMPL>
Optional, The node template name used for generating fake nodes. By default, it will be auto-detected according to the current arch.
B<PERF_DRYRUN>
Optional, Indicate no real commands will be executed if the environment variable is set.
B<PERF_NOCREATE>
Optional, Indicate no new fake nodes will be created if the environment variable is set.
=head1 EXAMPLES
=over 4
=item 1.
To run the performance testing for the commands defined in /tmp/cmd.lst on 5000 fake nodes:
xcatperftest 5000 /tmp/cmd.lst
=item 2.
To generate an xCAT node object stanz file for 10000 nodes in subnet 10.100.0.0:
FAKE_NETWORK_PRO=10.100 FAKE_NETWORK_BMC=10.200 xcatperftest 10000
=item 3.
To run the performance testing for the commands defined in /opt/xcat/share/xcat/tools/autotest/perfcmds.lst on 5000 existing fake nodes:
PERF_NOCREATE=y xcatperftest 5000 /opt/xcat/share/xcat/tools/autotest/perfcmds.lst
=back
=head1 FILES
/opt/xcat/bin/xcatperftest
/opt/xcat/share/xcat/tools/autotest/perfcmds.lst

View File

@ -42,6 +42,7 @@ mkdir -p $RPM_BUILD_ROOT/%{prefix}/share/xcat/tools/autotest
mkdir -p $RPM_BUILD_ROOT/%{prefix}/share/man/man1
mkdir -p $RPM_BUILD_ROOT/%{prefix}/share/doc/man1
cp bin/* $RPM_BUILD_ROOT/%{prefix}/bin
cp xcattest $RPM_BUILD_ROOT/%{prefix}/bin
chmod 755 $RPM_BUILD_ROOT/%{prefix}/bin/*