xCAT/xcat-core
2
0
Fork 0
mirror of https://github.com/xcat2/xcat-core.git synced 2024-11-24 02:19:57 +00:00
Code Issues Releases Wiki Activity
Page: XCAT_Cluster_in_SoftLayer
Pages
2014_10_01_Bash_vulnerability 2014_11_04_Poodle_Attack AIX_NIM_NFSv4_support Add_Extra_Parameters_for_confignics_Postscript CRHS like_function CRHS like_function_enhancements CSM_CFM like_function_to_sync_config_files_to_the_nodes Capture_Linux_Image_from_running_node_to_create_a_stateless statelite_image Chef_databag_support_for_deploying_openstack_in_xCAT Cluster_config_file Conserver_Replacement Coordinated_Cluster_Shutdown_and_Bringup Copycds_enhancement Coral PDU supports Create_osimage_definition_for_Management_Node Cumulus_OS_Upgrade_Support_Test DFM_Service_Node_Hierarchy_support DHCPv6_management DNS_Enhancments DNS_master_slave_support Def_commands_to_display set_nic_attributes_more_easily Deploying_OpenStack_Baremetal_Nodes_Using_xCAT Disable_ssh_root_passwordless_node to node_access Document_large_cluster_tunables,_etc. Dynamic_node_groups_support Energy_Management_for_Flex Energy_management_support_for_blade Enhance_NTP_setup_support Enhancement_of_lsslp_ flexdiscover Enhancement_on_makeroutes_and_snmove_commands Exporting_more_table_attributes_to_nodes Filling_the_osimage_attributes_with_the_files_that_are_found_in_the_search_path Full_Automation_of_DB2_Setup_on_Service_Nodes Function Verification Test Design of xCAT High Availability GPFS_statelite_nodes Getmacs_and_rnetboot_enhancement Getnodetype_performance_imprvoments_ _phase_I HA_NFS_on_AIX_service_nodes HA_Service_Nodes_for_AIX HA_Service_Nodes_for_AIX_ _using_External_NFS_Server HFI_MAC_address_failover HPC_Integration_Phase_1 Hardware_Inventory_for_Node Health_Check_Script_Framework Hierarchical_Design Home Image_Provisioning Image_import_and_export_enhancement_for_Linux Improve_perl_array_search_efficiency_for_scalability Improvement_of_pre creating_mypostscripts_on_linux Initrd_Enhancement_separate_initrd_hacking_from_noteset_get_drivers_from_os_update_distro Kdump_for_Linux_diskless_nodes Linux_AIX_mixed_Clusters_postscripts_cleanup Local_Disk_support_for_statelite Lstree_for_Servicenode_Hardware_and_VM_hierarchies Management_Node_Pool Management_Node_Setup_Verification Managing_Multiple_Sites_and_Clusters Mellanox_Switch_Support Mini design for openbmc python framework Mini design_of_updating_pre defined_groups_for_nodes Minimize_the_install setup_steps_of_xcat_on_the_MN Mkvlan_enhancements Multiple_Zone_Support2 Multiple_pkgdir_support_with_full_installation NFS_Redundancy_Investigation NFS_redundancy New_Cloud_Tables_in_xCAT Node_Status_and_Application_Status Node_definition_installation_service_node_setup_verification Nodeset_defaults_when_node_provmethod=osimage_is_set OS_Deployment_Monitoring_And_Automatic_Retry OS_Distro_Update_xCAT_command_osdistroupdate OpenStack Chef Cookbook_xCAT_Integration OpenStack_Deployment_using_Chef OpenStack_Deployment_using_Puppet Ospkgs_enhancements_to_install upgrade_OS_packages_from_Updates Overview_of_HA_Services_for_Diskless_AIX_Nodes Power8_BE_support Predict_long_nicname_for_RH7 Proposal_for_docker_Support_in_xCAT RHEL7_Support RHEV_support Rcons_hierachy_support Redfish_support Refine_the_usage_of_installnic_and_primarynic_attributes Release processs automation investigation Resolving_Nodenames_for_System_p_HW_Ctrl Run_parallel_commands_for_Ethernet_switches Run_postbootscripts_after_each_diskful_reboot SLES12_Support SLES_11_SP1_SDK_support Secure root support Sequential_Discovery Sequential_Discovery_phase_2 Service_Node_Scenarios_and_Retry_Logic_in_Postscript_Process Service_node_manual_takeover_on_AIX Service_node_take_over Setup_IP_routes_between_the_MN_and_the_CNs Shared data based xCAT MN HA mini design Shared data based xCAT MN HA user case Shared data based xCAT MN HA Specify_the_disk_partitions_during_diskfull_install Steps to install xcat_openbmc_py3 on RH8 Support uploading openbmc firmware in parallel Support_Management_Node_in_the_servicenode_table Support_for_Redundant_BPAs_and_FSPs Support_multiple_hardware_control_points Support_osimage.pkgdir_with_multiple_paths_in_genimage_and_diskfull_installation Support_postscripts_associated_with_rsync_files_to_be_run_after_rsync_in_xdcp Support_to_inject_drivers_from_Driver_RPM_Package Support_xcatd_running_user_defined_sql_scripts_on_start_and_foreign_keys Supporting_driver_update_disk_for_Linux Supporting_the_scalable_functions_for_HX5_blade Switch_Discovery_And_Switch_Management System_p_Discovery_Enhancements System_z_hierarchical_tree Test Design of HA Test Design of TestCase Label Test Design of `goconserver` Regression Testing Test Design of `goconserver` Support Test Design of configib Test Design of cumulus osimage Test Design of cumulus_osimage Test Design of gonconserver redirect logs Test Design of hardware control commands regression Test Design of makegocons option l Test Design of rbeacon python version Test Design of rcons based goconserver Test Design of revenlog python version Test Design of rflash python version Test Design of rinv python version Test Design of rpower python version Test Design of rsetboot python version Test Design of rspconfig python version Test Design of rvitals python version Test Design of syncfiles Test Design of xcat inventory diff Test Design of xcat inventory export import performance Test Design of xcat inventory option e Test Design of xcat inventory osimage environment variables export part Test Design of xcat inventory osimage environment variables Test Design of xcat inventory osimage phase2 Test Design of xcat inventory osimage support share file directory Test Design of xcat inventory osimage Test Design of xcat inventory phase 1 Test Design of xcat inventory regular expression Test Design of xcat inventory save keys Test Design of xcat inventory user case Test Design of xcat terraform provider phase1 Test Report of `goconserver` Test cases against nodeset error handling Test_Case_Designs_FVT The mini design of osimage version control(initial draft) The mini design of syncfiles for cumulus The mini design of using xcat inventory templates to setup xCAT cluster with ansible The_remote_console_support_of_hmc Trace_Facility_for_xCAT Ubuntu_supports_internet_repository Updatenode_Performance_Enhancements Updatenode_Sync_Service_Nodes_for_install Updatenode_enhancement_for_security_keys_and_certificates Updatenode_support_to_run_mypostscript_sections Updatenode_to_the_Management_Node Updates_to_xcatsetup_and_Cluster_Config_File Windows_Multiple_Disks_Partitions_Configuration Windows_Multiple_Nics_Configuration Windows_Multiple_Winpes Windows_Support_to_Run_Postscripts_Postbootscripts_in_Windows_Deployment Wish_List_for_xCAT_2 Wish_List_for_xCAT_FVT XCAT 2 Mini Designs for New Features XCAT_2.10_Release_Notes XCAT_2.11.1_Release_Notes XCAT_2.11_ESP_Release_Notes XCAT_2.11_Release_Notes XCAT_2.12.1_Release_Notes XCAT_2.12.2_Release_Notes XCAT_2.12.3_Release_Notes XCAT_2.12.4_Release_Notes XCAT_2.12.5_Release_Notes XCAT_2.12_Release_Notes XCAT_2.13.10_Release_Notes XCAT_2.13.11_Release_Notes XCAT_2.13.1_Release_Notes XCAT_2.13.2_Release_Notes XCAT_2.13.3_Release_Notes XCAT_2.13.4_Release_Notes XCAT_2.13.5_Release_Notes XCAT_2.13.6_Release_Notes XCAT_2.13.7_Release_Notes XCAT_2.13.8_Release_Notes XCAT_2.13.9_Release_Notes XCAT_2.13_Release_Notes XCAT_2.14.1_Release_Notes XCAT_2.14.2_Release_Notes XCAT_2.14.3_Release_Notes XCAT_2.14.4_Release_Notes XCAT_2.14.5_Release_Notes XCAT_2.14.6_Release_Notes XCAT_2.14_Release_Notes XCAT_2.15.1_Release_Notes XCAT_2.15_Release_Notes XCAT_2.16.1_Release_Notes XCAT_2.16.2_Release_Notes XCAT_2.16.3_Release_Notes XCAT_2.16.4_Release_Notes XCAT_2.16.5_Release_Notes XCAT_2.16_Release_Notes XCAT_2.17_Release_Notes XCAT_2.9.2_Release_Notes XCAT_2.9.2_aix_Release_Notes XCAT_2.9.3_Release_Notes XCAT_2.9.4_Release_Notes XCAT_Cluster_in_SoftLayer XCAT_HPC_Integration_Files_per_OS,_Architecture,_Product XCAT_Health_Check XCAT_IPv6_support_on_AIX XCAT_IPv6_support_on_Linux XCAT_Mini Design_Template XCAT_Monitoring_Failure_Analysis_Conditions_Batched_Events XCAT_Postscripts_framework_enhancements XCAT_Provisioning_OS_on_consistent_disk XCAT_REST_API_enhancements XCAT_Support_for_Kits_and_Kit_Components XCAT_probe XCAT_support_for_multiple_network_domains Xcatexport_and_xcatimport_commands Xcatmon_and_appstatus_for_HPC_products Xeon_Phi_support cofignetwork_for_RH7 confignetwork integrate confignics_configeth_configib functions usage determine_install_disk diskdiscover_and_configraid docker_instances_lifecycle_management enhancement_for_rinstall_command logging pre script to make debug easy makedhcp_append_statement_automatically_rather_than_get_statement_from_nodeset mini design of ZTP based onie switch discovery and configuration minidesign Switch based switch discovery minidesign_xcatprobe node status enhancement ntp_enhancement openbmc_support post.xcat_restructure rflash_streamline_procedure rscan command for hypervisor rspconfig network rspconfig ntpservers rspconfig password scaleup_basedon_xcat_in_openshift_cluster supporting_partitionfile_for_Ubuntu supporting_persistent_kernel_cmdline_after_diskful_installation test_sidebar the mini design of xcat inventory export and import for osimage use fifo pipe to enhance getadapter xCAT Build Schedule xCAT_2.12_ReleaseInformation xCAT_2.13_ReleaseInformation xCAT_2.14_ReleaseInformation xCAT_2.15_ReleaseInformation xCAT_Docker_Image xcatprobe_framework xcatprobe_requests
16 XCAT_Cluster_in_SoftLayer
ligc edited this page 2015-07-30 04:22:49 -04:00
Table of Contents

Table of Contents

Design_Warning

The purpose of this feature is to support xCAT clusters with a SoftLayer cloud. This request is from GTS to provide opensource HPC Cloud offerings to their customers.

Here are the aspects of the propsal:

  • Support an xCAT cluster totally resident in a SoftLayer Cloud
  • Cluster system management will be predominantly driven from the xCAT management node. There may be certain functions that will require direct interaction with the SoftLayer portal. These should be limited as much as possible.
  • customer uses SoftLayer portal to request nodes, and then xcat discovers/manages those nodes as part of the cluster
  • consider baremetal nodes
  • consider virtual machines
  • xcat uses the SoftLayer APIs to request nodes and automatically adds them to the xCAT cluster
  • Support a hybrid cluster comprised of an existing customer's on-premise xCAT cluster being extended with nodes from a SoftLayer Cloud

xCAT in SoftLayer

The customer uses the SoftLayer portal to request the nodes and other elements of the cloud.

The customer installs the xCAT management node software on one of those nodes

A new xCAT script calls the SL API to query the information for all the bare metal nodes in that softlayer account, and produces a stanza file that can be piped into mkdef -z.

 The key attributes this script gets are: ipmi.bmc, ipmi.password, ipmi.username, mac.mac,hosts.ip, and the hostname and root pw (which it puts in nodelist.comment)

An OS image is deployed to those nodes with the required HPC software stack. Additional software can be installed as needed. This assumes full-disk install on the nodes. We do not see a requirement for diskless node support at this time.

The end user documentation for using xCAT on SoftLayer is: [Using_xCAT_in_SoftLayer]

How do we do OS deployment? There are several options discussed in the following sections.

Use SoftLayer's Bare Metal OS Provisioning

Through the softlayer portal, you can request that it put one of the following OSes on the nodes: centos/rhel, debian/ubuntu, freebsd, windows. If one of these is that OS you want, you can have SL do that and then just have xCAT manage OS upgrades, additional rpms, and HPC software through updatenode (ospkgs, otherpkgs). The drawback is that this can take SL 2 to 4 hours to complete provision the OS.

Have xCAT Provision the OS

If the OS you need is not in the list above (e.g. SLES), then you need xCAT to do the bare metal provisioning of the OS. Using pxe and dhcp is problematic, because the nodes can often be on different vlans, and the delay when bringing nics up will cause it to time out even on the same vlan (see next point).

SoftLayer has a pretty sophisticated private network that allows them to on-demand configure private vlans that only a single tenant can communicate on. But one of the side effects of this is that when a bare metal svr's nics is brought up on one of the private vlans, it takes about 60 seconds before that nic can successfully send an ip packet. This is much longer than any of the default timeouts used in the network installation process, so we have to insert waits in key points during the process in order for it to succeed. Several things are needed:

  • instead of pxe booting the node, we take advantage of the fact that SL always puts on OS image on the node (even if it isn't the version we want). So we use a new command called pushinitrd to copy the initrd, kernel, and kernel params to grub on all of the nodes. (This script uses modifygrub under the covers.)
  • to avoid additional dhcp requests, we also use a static ip for the node:
    • the pushinitrd script above sets the ip/submask/gateway/hostname/dns info as kernel parameters
    • for scripted install, we provide a new compute.sles11.softlayer.tmpl file that sets the dns info and static nic ip info
    • for sysclone, genesis is changed to use static ip if appropriate kernel parms are set, and we make sure systemimage (SI) doesn't try to use dhcp or restart the network in the genesis initrd by setting kernel parms like IPADDR, NETMASK, etc.
  • to deal with delay when the nic is brought up:
    • for scripted install:
      • in place of post.sles and post.sles.common, we provide post.sles.softlayer.common which avoids (what i think is) an unnecessary reboot and skips some dhcp config
      • pushinitrd hacks the autoinst file (after nodeset has built it) to insert code that will hack the /etc/init.d/network script on the node so that it will not return until it is able to ping successfully on the install nic to the mn. (There are files/dirs specifically for doing things when the nics are brought up (in sles /etc/sysconfig/network/if-up.d/, in rhel /sbin/ifup-local), but these aren't run by autoyast right after the reboot when it is doing its configuration.)
      • in the real OS, the nics are always configured in a bond (one for the private network and one for the public network) so that when the system is booted for real (after the autoyast config stage), the nics come up quickly. (This is what SL does when they install an os.)
    • for sysclone install:
      • after configuring the install nic with a static ip, genesis pings the mn until it can reach it.
      • in the real OS, the nics are always configured in a bond (one for the private network and one for the public network) so that when the system is booted for real (after the autoyast config stage), the nics come up quickly. (This is what SL does when they install an os.)

The main changes/additions needed to support centos/rhel (in addition to the current sles support) are:

  • for scripted install:
    • tmpl file: use static ip info instead of dhcp, and set route for private network. See my compute.sles11.softlayer.tmp as an example.
    • pushinitrd: insert /etc/init.d/network wait hack in different place in autoinst file
    • modifygrub: change the kernel parameter names to be what kickstart needs to use static ip in its initrd
  • for sysclone:
    • 15all.configefi: add code to modify grub the redhat way
    • 16all.updatenetwork: this has the code for redhat, but needs to be tested
    • 20all.mkinitrd_for_suse: change the name to more generic and add support for dracut

Use xCAT to Provision the OS via the Out-of-Band Network

Investigate whether SuperMicro hardware has support to load the initrd over the out-of-band hdwctrl network, similar to what flexcat does for fsm. SuperMicro's BMC web interface does have an option for remote usb or cd, so this looks promising. (You get to it through the bmc web interface, Remote Contronl, Console Redirection, when the console window comes up, choose the Virtural Media->Virtual Storage menu.)

Hybrid xCAT SoftLayer Cluster

There are many issues with trying to directly extend a customer's local cluster with additional nodes from a SoftLayer cloud. The largest issue is networking. The SL nodes will be on external networks not directly connected to the xCAT management node.

One thought was to investigate installing a service node in the SL cloud extension to manage those nodes, but the SN would need remote data base access to the xCAT management node, and that most likely will not have adequate performance across a remote network.

Another consideration is to introduce a new concept of xCAT "remote nodes":

 \- the customer has an existing xCAT cluster running in their local on-premise environment 
 \- the customer sets up another xCAT cluster in SoftLayer with its own management node as described above 
 \- the customer also adds those SL nodes to the local xCAT management node as "remote nodes", whose service node is defined as the remote SL management node (with a new "remote server" attr set) 
 \- when commands are run on the local MN, they can be run for a mix of local and remote nodes. The "pre-process" function will need to check the new "remote server" attr and not set the "pre-processed" flag in the request. The request will then be sent to the xcatd on the remote server, and processed there, returning data back to the local xcatd. 

 NOTE: This will take more internal design to work through. Concerns of SSL keys, credentials, etc., properly handling the requests in the xcatd daemons, making sure data gets sent back correctly, timeouts on the remote requests, etc.

Staging

We should address this work in several stages (each stage building on the previous):

1. minimal proof-of concept

 \- understand SoftLayer, learn how to use SL portal, request machines, experiment with SL API 
 \- manually step through the process of "first implementation" below

2. first implementation with limitations and restrictions

 some possible restrictions may include: 

 \- customer requests all cluster nodes through SoftLayer portal, and uses SL to provision the initial OS. 
 \- xCAT will use SL API to query hardware assigned to customer, add nodes to xCAT database, and apply required HPC software through updatenode 
 \- Customer/GTS provides all opensource HPC software rpms, customization scripts and other setup for applications. 
 \- assumes that xCAT will require minimal additional development work to configure the application networks (if not already done as part of SL OS provisioning) 
 \- since SL is provisioning the base OS, both baremetal and VM nodes should look the same to xCAT for updatenode 
 \- no xCAT service nodes for hierarchy

3. second implementation

 same restrictions as above, except that now: 

 \- xCAT interacts with SL APIs to request the machines 
 \- SL will provision a specific base OS as requested by xCAT 
 \- xCAT will scp a kernel initrd into that SL OS to boot the customer requested OS managed by xCAT (this allows provisioning OS's not available through SL)

4. third implementation (not sure if this will be possible)

 performance enhancements: 

 \- xCAT does not rely on SL OS provisioning, will require investigation into possible ways to do this as out-of-band

5. hybrid solution for remote node support

 \- requires architectural changes to xCAT cluster support

Other Design Considerations

  • Required reviewers:
  • Required approvers: Guang Cheng Li
  • Database schema changes: N/A
  • Affect on other components: N/A
  • External interface changes, documentation, and usability issues: N/A
  • Packaging, installation, dependencies: N/A
  • Portability and platforms (HW/SW) supported: N/A
  • Performance and scaling considerations: N/A
  • Migration and coexistence: N/A
  • Serviceability: N/A
  • Security: N/A
  • NLS and accessibility: N/A
  • Invention protection: N/A

News

History

  • Oct 22, 2010: xCAT 2.5 released.
  • Apr 30, 2010: xCAT 2.4 is released.
  • Oct 31, 2009: xCAT 2.3 released.
    xCAT's 10 year anniversary!
  • Apr 16, 2009: xCAT 2.2 released.
  • Oct 31, 2008: xCAT 2.1 released.
  • Sep 12, 2008: Support for xCAT 2
    can now be purchased!
  • June 9, 2008: xCAT breaths life into
    (at the time) the fastest
    supercomputer on the planet
  • May 30, 2008: xCAT 2.0 for Linux
    officially released!
  • Oct 31, 2007: IBM open sources
    xCAT 2.0 to allow collaboration
    among all of the xCAT users.
  • Oct 31, 1999: xCAT 1.0 is born!
    xCAT started out as a project in
    IBM developed by Egan Ford. It
    was quickly adopted by customers
    and IBM manufacturing sites to
    rapidly deploy clusters.