Table of Contents
{{:Design Warning}}
Mini-design of Trace Facility for xCAT
Overview
In the present code of xCAT that no common approach for developer to generate the debug trace. Generally, developer adds the 'print/Dumper/MsgUtils' code inside the xcatd/plugins to display some run-time variables/status in the development phase and remove/comment out them for the release version. The operations of adding/removing debug trace have to be done for every debugging, it's especially hard from the perspective of FVT or customer to get the trace log.
In this design, I'll add two types of trace facilities for xCAT:
Enable the subroutine calling trace
Display the calling trace for the subroutines. The trace message includes: The called subroutine name; The arguments which passed to the called subroutine; The calling stack of the subroutine. By default, the trace will be enabled to all the subroutines in xcatd and plugin modules. The subroutine list also can be configured by configuration file or through command line.
Enable the commented debug trace in debug mode
We always concern about the trace code in the source code that it impacts the performance or even causes the possible defects for a release version. In this design I'll introduce an approach that only enable the trace code in the debug mode. At common mode, the trace code existed as comments.
External Interface
Offer a command named 'xcatdebug' (It already existed, adding the new functions) to enable/disable and configure the trace facility.
xcatdebug [-f enable|disable [-c configuration file| subroutine list]
Enable/Disable subroutine calling trace. If no -c flag is specified, the enable/disable will impact to all the subroutines which defined in the xcatd and plugins.
'-c' flag can be used to specify a 'configuration file' or 'subroutine list':
'SUBROUTINE_DEFINITION': To make user easier to define the subroutine list, I defined this object. The format of it can be pkgname. If [pkgname] is ignored, that means the subsequence subroutines belongs to the 'xcatd' (main process).
'configuration file': Multiple entries of SUBROUTINE_DEFINITION;
e.g. -c /tmp/xcattrace.cfg. The format inside the configuration file:
(daemonize,do_installm_service,do_udp_service)
xCAT::Utils(isMN,Version)
xCAT_plugin::DBobjectdefs(defls,process_request)
'subroutine list': Multiple SUBROUTINE_DEFINITION split with '|'.
e.g. -c (daemonize,do_installm_service,do_udp_service)|
xCAT::Utils(isMN,Version)|
xCAT_plugin::DBobjectdefs(defls,process_request)
Note: The xcatd processes will not be restarted when execute the [-f enable|disable].
xcatdebug [-d enable|disable]
Enable/Disbale the trace code in the source code of xCAT project.
Note: Since the enable process will be done before the compiling of source code, the xcatd process have be to restarted when executing the enable/disable operation.
The trace format
A typical trace log includes three parts: Subroutine name; Arguments; Calling stack. Following is a log example of the calling of xCAT_plugin::DBobjectdefs::process_request subroutine:
Calling *xCAT_plugin::DBobjectdefs::process_request; Argus[{'_xcat_clientfqdn' => ['localhost'],'_xcat_clientport' => [41891] ...; Calling stack [ at /opt/xcat/sbin/xcatd line 785#012#011main::ANON('HASH(0x32c9038)', 'CODE(0x2ed1b38)', 'CODE(0x2ee1820)') called at /opt/xcat/sbin/xcatd line 1539#012#011main::dispatch_request('HASH(0x32c9038)', 'CODE(0x2ed1b38)', 'DBobjectdefs') called at /opt/xcat/sbin/xcatd line 1349#012#011eval {...} called at /opt/xcat/sbin/xcatd line 1348#012#011main::plugin_command('HASH(0x32c9038)', 'IO::Socket::SSL=GLOB(0x30cefc0)', 'CODE(0x2ed1b38)') called at /opt/xcat/sbin/xcatd line 1842#012#011eval {...} called at /opt/xcat/sbin/xcatd line 1803#012#011main::service_connection('IO::Socket::SSL=GLOB(0x30cefc0)', 'root', 'localhost', 'localhost') called at /opt/xcat/sbin/xcatd line 1073#012]
main::dispatch_request('HASH(0x32c9038)', 'CODE(0x2ed1b38)', 'DBobjectdefs')
xCAT_plugin::DBobjectdefs::process_request({'_xcat_clientfqdn' => ['localhost'],'_xcat_clientport' => [41891] ...)
IO::Socket::SSL=GLOB(0x30cefc0)', 'CODE(0x2ed1b38)')
IO::Socket::SSL=GLOB(0x30cefc0)', 'CODE(0x2ed1b38)')
main::service_connection('IO::Socket::SSL=GLOB(0x30cefc0)', 'root', 'localhost', 'localhost')
The location of trace log
The trace log (inside ## DEBUG_BEGIN - ## DEBUG_END) which added by the developer in the subroutine can be displayed to any place (output in cli; syslog, log file) base on how the trace code is written.
The trace log of subroutines calling trace can be put to syslog or log file; The current decision is put the log in the /var/log/xcat/subcallingtrace.
Internal implementation
Enable the subroutine calling trace
Enable/Disable the function - define two signal handle subroutine for signal (NUM50 - Enable; NUM51 - Disable) in the xcatd. From the xcatdebug to send signal NUM50/NUM51 to enable or disable the functions.
Enable: Get all the symbols from the symbol table (includes xcatd and plugin modules) and figure out the subroutines that really defined and has been specified in the enable list, then replace the subroutine with a new one that added the debug trace. And store the original one in a global hash for the restoring.
Disable: Restore the original definition of the subroutines.
Enable the debug trace only in debug mode
Add the following part of code in the BEGIN{} section to do the filter thing during the loading of source file that change some code in the source file.
if (defined $ENV{ENABLE_TRACE_CODE}) {
use xCAT::Enabletrace qw(loadtrace filter);
loadtrace();
}
Added a new xCAT module /opt/xcat/lib/perl/xCAT/Enabletrace.pm to enable the commented trace log.
Note: This section has to be added to any package or xcatd if you want to use this function.
And add the trace code as following format in any place (xcat/plugin modules)
Trace section
## TRACE_BEGIN
# print "In the debug\n";
## TRACE_END
Trace in a single line
## TRACE_LINE print "In the trace line\n";
In common mode, the code keeps as above. But in debug mode, the first # of all the lines between the ' ## DEBUG_BEGIN' and '## DEBUG_END' (e.g. '# logger -t xcat "debug in plugin_command";' will be removed.
News
- Apr 22, 2016: xCAT 2.11.1 released.
- Mar 11, 2016: xCAT 2.9.3 (AIX only) released.
- Dec 11, 2015: xCAT 2.11 released.
- Nov 11, 2015: xCAT 2.9.2 (AIX only) released.
- Jul 30, 2015: xCAT 2.10 released.
- Jul 30, 2015: xCAT migrates from sourceforge to github
- Jun 26, 2015: xCAT 2.7.9 released.
- Mar 20, 2015: xCAT 2.9.1 released.
- Dec 12, 2014: xCAT 2.9 released.
- Sep 5, 2014: xCAT 2.8.5 released.
- May 23, 2014: xCAT 2.8.4 released.
- Jan 24, 2014: xCAT 2.7.8 released.
- Nov 15, 2013: xCAT 2.8.3 released.
- Jun 26, 2013: xCAT 2.8.2 released.
- May 17, 2013: xCAT 2.7.7 released.
- May 10, 2013: xCAT 2.8.1 released.
- Feb 28, 2013: xCAT 2.8 released.
- Nov 30, 2012: xCAT 2.7.6 released.
- Oct 29, 2012: xCAT 2.7.5 released.
- Aug 27, 2012: xCAT 2.7.4 released.
- Jun 22, 2012: xCAT 2.7.3 released.
- May 25, 2012: xCAT 2.7.2 released.
- Apr 20, 2012: xCAT 2.7.1 released.
- Mar 19, 2012: xCAT 2.7 released.
- Mar 15, 2012: xCAT 2.6.11 released.
- Jan 23, 2012: xCAT 2.6.10 released.
- Nov 15, 2011: xCAT 2.6.9 released.
- Sep 30, 2011: xCAT 2.6.8 released.
- Aug 26, 2011: xCAT 2.6.6 released.
- May 20, 2011: xCAT 2.6 released.
- Feb 14, 2011: Watson plays on Jeopardy and is managed by xCAT!
- xCAT Release Notes Summary
- xCAT OS And Hw Support Matrix
- xCAT Test Environment Summary
History
- Oct 22, 2010: xCAT 2.5 released.
- Apr 30, 2010: xCAT 2.4 is released.
- Oct 31, 2009: xCAT 2.3 released.
xCAT's 10 year anniversary! - Apr 16, 2009: xCAT 2.2 released.
- Oct 31, 2008: xCAT 2.1 released.
- Sep 12, 2008: Support for xCAT 2
can now be purchased! - June 9, 2008: xCAT breaths life into
(at the time) the fastest
supercomputer on the planet - May 30, 2008: xCAT 2.0 for Linux
officially released! - Oct 31, 2007: IBM open sources
xCAT 2.0 to allow collaboration
among all of the xCAT users. - Oct 31, 1999: xCAT 1.0 is born!
xCAT started out as a project in
IBM developed by Egan Ford. It
was quickly adopted by customers
and IBM manufacturing sites to
rapidly deploy clusters.