xCAT/xcat-core
2
0
Fork 0
mirror of https://github.com/xcat2/xcat-core.git synced 2024-11-23 18:10:14 +00:00
Code Issues Releases Wiki Activity
Page: XCAT_IPv6_support_on_AIX
Pages
2014_10_01_Bash_vulnerability 2014_11_04_Poodle_Attack AIX_NIM_NFSv4_support Add_Extra_Parameters_for_confignics_Postscript CRHS like_function CRHS like_function_enhancements CSM_CFM like_function_to_sync_config_files_to_the_nodes Capture_Linux_Image_from_running_node_to_create_a_stateless statelite_image Chef_databag_support_for_deploying_openstack_in_xCAT Cluster_config_file Conserver_Replacement Coordinated_Cluster_Shutdown_and_Bringup Copycds_enhancement Coral PDU supports Create_osimage_definition_for_Management_Node Cumulus_OS_Upgrade_Support_Test DFM_Service_Node_Hierarchy_support DHCPv6_management DNS_Enhancments DNS_master_slave_support Def_commands_to_display set_nic_attributes_more_easily Deploying_OpenStack_Baremetal_Nodes_Using_xCAT Disable_ssh_root_passwordless_node to node_access Document_large_cluster_tunables,_etc. Dynamic_node_groups_support Energy_Management_for_Flex Energy_management_support_for_blade Enhance_NTP_setup_support Enhancement_of_lsslp_ flexdiscover Enhancement_on_makeroutes_and_snmove_commands Exporting_more_table_attributes_to_nodes Filling_the_osimage_attributes_with_the_files_that_are_found_in_the_search_path Full_Automation_of_DB2_Setup_on_Service_Nodes Function Verification Test Design of xCAT High Availability GPFS_statelite_nodes Getmacs_and_rnetboot_enhancement Getnodetype_performance_imprvoments_ _phase_I HA_NFS_on_AIX_service_nodes HA_Service_Nodes_for_AIX HA_Service_Nodes_for_AIX_ _using_External_NFS_Server HFI_MAC_address_failover HPC_Integration_Phase_1 Hardware_Inventory_for_Node Health_Check_Script_Framework Hierarchical_Design Home Image_Provisioning Image_import_and_export_enhancement_for_Linux Improve_perl_array_search_efficiency_for_scalability Improvement_of_pre creating_mypostscripts_on_linux Initrd_Enhancement_separate_initrd_hacking_from_noteset_get_drivers_from_os_update_distro Kdump_for_Linux_diskless_nodes Linux_AIX_mixed_Clusters_postscripts_cleanup Local_Disk_support_for_statelite Lstree_for_Servicenode_Hardware_and_VM_hierarchies Management_Node_Pool Management_Node_Setup_Verification Managing_Multiple_Sites_and_Clusters Mellanox_Switch_Support Mini design for openbmc python framework Mini design_of_updating_pre defined_groups_for_nodes Minimize_the_install setup_steps_of_xcat_on_the_MN Mkvlan_enhancements Multiple_Zone_Support2 Multiple_pkgdir_support_with_full_installation NFS_Redundancy_Investigation NFS_redundancy New_Cloud_Tables_in_xCAT Node_Status_and_Application_Status Node_definition_installation_service_node_setup_verification Nodeset_defaults_when_node_provmethod=osimage_is_set OS_Deployment_Monitoring_And_Automatic_Retry OS_Distro_Update_xCAT_command_osdistroupdate OpenStack Chef Cookbook_xCAT_Integration OpenStack_Deployment_using_Chef OpenStack_Deployment_using_Puppet Ospkgs_enhancements_to_install upgrade_OS_packages_from_Updates Overview_of_HA_Services_for_Diskless_AIX_Nodes Power8_BE_support Predict_long_nicname_for_RH7 Proposal_for_docker_Support_in_xCAT RHEL7_Support RHEV_support Rcons_hierachy_support Redfish_support Refine_the_usage_of_installnic_and_primarynic_attributes Release processs automation investigation Resolving_Nodenames_for_System_p_HW_Ctrl Run_parallel_commands_for_Ethernet_switches Run_postbootscripts_after_each_diskful_reboot SLES12_Support SLES_11_SP1_SDK_support Secure root support Sequential_Discovery Sequential_Discovery_phase_2 Service_Node_Scenarios_and_Retry_Logic_in_Postscript_Process Service_node_manual_takeover_on_AIX Service_node_take_over Setup_IP_routes_between_the_MN_and_the_CNs Shared data based xCAT MN HA mini design Shared data based xCAT MN HA user case Shared data based xCAT MN HA Specify_the_disk_partitions_during_diskfull_install Steps to install xcat_openbmc_py3 on RH8 Support uploading openbmc firmware in parallel Support_Management_Node_in_the_servicenode_table Support_for_Redundant_BPAs_and_FSPs Support_multiple_hardware_control_points Support_osimage.pkgdir_with_multiple_paths_in_genimage_and_diskfull_installation Support_postscripts_associated_with_rsync_files_to_be_run_after_rsync_in_xdcp Support_to_inject_drivers_from_Driver_RPM_Package Support_xcatd_running_user_defined_sql_scripts_on_start_and_foreign_keys Supporting_driver_update_disk_for_Linux Supporting_the_scalable_functions_for_HX5_blade Switch_Discovery_And_Switch_Management System_p_Discovery_Enhancements System_z_hierarchical_tree Test Design of HA Test Design of TestCase Label Test Design of `goconserver` Regression Testing Test Design of `goconserver` Support Test Design of configib Test Design of cumulus osimage Test Design of cumulus_osimage Test Design of gonconserver redirect logs Test Design of hardware control commands regression Test Design of makegocons option l Test Design of rbeacon python version Test Design of rcons based goconserver Test Design of revenlog python version Test Design of rflash python version Test Design of rinv python version Test Design of rpower python version Test Design of rsetboot python version Test Design of rspconfig python version Test Design of rvitals python version Test Design of syncfiles Test Design of xcat inventory diff Test Design of xcat inventory export import performance Test Design of xcat inventory option e Test Design of xcat inventory osimage environment variables export part Test Design of xcat inventory osimage environment variables Test Design of xcat inventory osimage phase2 Test Design of xcat inventory osimage support share file directory Test Design of xcat inventory osimage Test Design of xcat inventory phase 1 Test Design of xcat inventory regular expression Test Design of xcat inventory save keys Test Design of xcat inventory user case Test Design of xcat terraform provider phase1 Test Report of `goconserver` Test cases against nodeset error handling Test_Case_Designs_FVT The mini design of osimage version control(initial draft) The mini design of syncfiles for cumulus The mini design of using xcat inventory templates to setup xCAT cluster with ansible The_remote_console_support_of_hmc Trace_Facility_for_xCAT Ubuntu_supports_internet_repository Updatenode_Performance_Enhancements Updatenode_Sync_Service_Nodes_for_install Updatenode_enhancement_for_security_keys_and_certificates Updatenode_support_to_run_mypostscript_sections Updatenode_to_the_Management_Node Updates_to_xcatsetup_and_Cluster_Config_File Windows_Multiple_Disks_Partitions_Configuration Windows_Multiple_Nics_Configuration Windows_Multiple_Winpes Windows_Support_to_Run_Postscripts_Postbootscripts_in_Windows_Deployment Wish_List_for_xCAT_2 Wish_List_for_xCAT_FVT XCAT 2 Mini Designs for New Features XCAT_2.10_Release_Notes XCAT_2.11.1_Release_Notes XCAT_2.11_ESP_Release_Notes XCAT_2.11_Release_Notes XCAT_2.12.1_Release_Notes XCAT_2.12.2_Release_Notes XCAT_2.12.3_Release_Notes XCAT_2.12.4_Release_Notes XCAT_2.12.5_Release_Notes XCAT_2.12_Release_Notes XCAT_2.13.10_Release_Notes XCAT_2.13.11_Release_Notes XCAT_2.13.1_Release_Notes XCAT_2.13.2_Release_Notes XCAT_2.13.3_Release_Notes XCAT_2.13.4_Release_Notes XCAT_2.13.5_Release_Notes XCAT_2.13.6_Release_Notes XCAT_2.13.7_Release_Notes XCAT_2.13.8_Release_Notes XCAT_2.13.9_Release_Notes XCAT_2.13_Release_Notes XCAT_2.14.1_Release_Notes XCAT_2.14.2_Release_Notes XCAT_2.14.3_Release_Notes XCAT_2.14.4_Release_Notes XCAT_2.14.5_Release_Notes XCAT_2.14.6_Release_Notes XCAT_2.14_Release_Notes XCAT_2.15.1_Release_Notes XCAT_2.15_Release_Notes XCAT_2.16.1_Release_Notes XCAT_2.16.2_Release_Notes XCAT_2.16.3_Release_Notes XCAT_2.16.4_Release_Notes XCAT_2.16.5_Release_Notes XCAT_2.16_Release_Notes XCAT_2.17_Release_Notes XCAT_2.9.2_Release_Notes XCAT_2.9.2_aix_Release_Notes XCAT_2.9.3_Release_Notes XCAT_2.9.4_Release_Notes XCAT_Cluster_in_SoftLayer XCAT_HPC_Integration_Files_per_OS,_Architecture,_Product XCAT_Health_Check XCAT_IPv6_support_on_AIX XCAT_IPv6_support_on_Linux XCAT_Mini Design_Template XCAT_Monitoring_Failure_Analysis_Conditions_Batched_Events XCAT_Postscripts_framework_enhancements XCAT_Provisioning_OS_on_consistent_disk XCAT_REST_API_enhancements XCAT_Support_for_Kits_and_Kit_Components XCAT_probe XCAT_support_for_multiple_network_domains Xcatexport_and_xcatimport_commands Xcatmon_and_appstatus_for_HPC_products Xeon_Phi_support cofignetwork_for_RH7 confignetwork integrate confignics_configeth_configib functions usage determine_install_disk diskdiscover_and_configraid docker_instances_lifecycle_management enhancement_for_rinstall_command logging pre script to make debug easy makedhcp_append_statement_automatically_rather_than_get_statement_from_nodeset mini design of ZTP based onie switch discovery and configuration minidesign Switch based switch discovery minidesign_xcatprobe node status enhancement ntp_enhancement openbmc_support post.xcat_restructure rflash_streamline_procedure rscan command for hypervisor rspconfig network rspconfig ntpservers rspconfig password scaleup_basedon_xcat_in_openshift_cluster supporting_partitionfile_for_Ubuntu supporting_persistent_kernel_cmdline_after_diskful_installation test_sidebar the mini design of xcat inventory export and import for osimage use fifo pipe to enhance getadapter xCAT Build Schedule xCAT_2.12_ReleaseInformation xCAT_2.13_ReleaseInformation xCAT_2.14_ReleaseInformation xCAT_2.15_ReleaseInformation xCAT_Docker_Image xcatprobe_framework xcatprobe_requests
7 XCAT_IPv6_support_on_AIX
ligc edited this page 2015-07-30 04:22:49 -04:00
Unescape Escape
Table of Contents
This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Table of Contents

{{:Design Warning}}

XCAT IPv6 support on AIX

This documentation illustrates how to configure and use the xCAT in IPv6 environment on AIX. The xCAT IPv6 configuration on Linux will be described in a different document.

We have tried to support all the xCAT features in IPv6 environment, but it turned out that some of the xCAT features can not work in IPv6 environment due to various reasons, such as the xCAT prerequisite library does not support IPv6; we have tried to keep the IPv6 configuration steps to be as similar as the IPv4 configuration, but there are still some differences. This documentation will focus on: 1) xCAT features limitation in IPv6 environment. 2) the differences between IPv4 configuration and IPv6 configuration. We assume the readers for this documentation are familiar with general xCAT configuration and have xCAT knowledge in IPv4 environment, the same configurations steps for both the IPv4 and IPv6 environment will NOT be covered in this documentation.

1. Feature restrictions in IPv6 environment

1)Hardware discovery: the openslp package being used by xCAT does not support IPv6, the first openslp version that supports IPv6 is openslp 1.3.0, however, xCAT is still using openslp 1.2.1, porting the xCAT patches for openslp to a newer openslp version needs a huge amount of work, we could not cover this in xCAT 2.5, will leave the hardware discovery support in IPv6 environment on xCAT 2.6 or later.

2)renergy: renergy relies on several open source libraries...

3)Infiniband support: needs to make sure the IB switches support IPv6, but some of the IB scripts will be run on compute nodes and the IB scripts are written in Perl. Note: Brian Croswell mentioned that some other teams are using IB in IPv6 environment, so there should not be significant problems.

4)autogpfsd: Not sure about the background and how to use this add-on, so do not change this add-on to support IPv6.

5)getmacs –arp is not supported in IPv6 environment, the arp is only for IPv4, the protocol NDP(Neighbor Discovery Protocol) can be used to show the mapping between MAC address and IPv6 address. Maybe will add the ndp as the getmacs method for IPv6 in the future, but not in the first pass. Here is an example: -bash-3.2# ndp -a e115v3p6n04 (4000::186) at link#3 0:14:5e:c8:5c:8c [ethernet] hv32lpar06 (fe80::221:5eff:fea3:76b0) at link#3 0:21:5e:a3:76:b0 [ethernet] e96m5sq02 (fe80::bc38:50ff:fe00:2005) at link#1 0:0:0:0:0:0 [loopback] permanent -bash-3.2#

6)IPv6 on Power blades are not supported in this first pass??

7)hierarchy will not be supported in xCAT 2.5, the mysql does not support IPv6 and we are investigating to use postgresql as an alternative for mysq.

8)Cross subnet NIM installation will not be supported in xCAT 2.5

9)The management node has to use the same IP version to communicate with all the compute nodes, i.e., IPv4 or IPv6, the managment node can not communicate with some compute nodes through IPv4 and communicate with some other nodes through IPv6. However, the compute nodes can have secondary adapters configured with different IP versions than the adapter communicating with the managment node, the secondary ip addresses can be configured using postscripts such as configeth.

Note: AIX user in xCAT 2.8.3 or later should use postscript configeth_aix instead of configeth.

2. External Differences

2.1 IP addresses allocation

AIX team indicates that if the NIM master and the NIM clients are in the same subnet, then only the link local address will work. And it seems for me that we can not have both link local address and global unicast ip address configured on the same interface. So it implies that we can only use link local address in cluster. The link local addresses starts with binary 1111 1110 10. The link local addresses can be configured automatically using command autoconf6 -6i <interface_name>, and the link local address can be calculated using MAC address, a script /opt/xcat/share/xcat/tools/mac2linklocal is now being shipped with xCAT that can be used to calculate the link local address from mac address. Here is the IPv6 assignment table from whitepaper “IPv6 on AIX 5L”:

Since the link local address is calculated from the MAC address, so the administrator can not assign the link local addresses in advance. Calculating and inputting the link local addresses one by one is not easy and is not an acceptable way. A new flag “-m” will be added to the makehost command, the makehosts -m will pdates /etc/hosts file with IPv6 link local addresses, and the link local addresses are generated from the mac addresses stored in mac table.

2.2 bootp boot vs tftp boot

In IPv4 environment, the openfirmware uses bootp protocol to boot the node through network; however, in IPv6 environment, openfirmware uses tftp boot, all the parameters client ip address, server ip address, gateway ip address and the tftp file need to be specified with the openfirmware boot command or through SMS, the openfirmware will not send bootp request to the NIM master. Here is an example of IPv6 tftp boot setup in SMS:

1. Client IP Address [FE80::221:5EFF:fEA3:76B0]

2. Server IP Address [FE80::BC38:50FF:FE00:2005]

3. Gateway Link Local IP Address [FE80::BC38:50FF:FE00:2005]

4. Full TFTP Filename [hv32lpar06]

Please be aware that the tftp boot gateway must be an IPv6 link local address, so when creating the networks table, the gateway should be an IPv6 link local address also.

2.3 NIM master initialization

Unlike the NIM master initialization in IPv4 environment, the NIM master initialization in IPv6 environment is more complicated, the script nim_master_setup does not support IPv6, we need to use several separate AIX and NIM commands to initialize the NIM master in IPv6 environment. The mknimimage command will take care of the IPv6 NIM master initialization, if you would like to initialize the IPv6 NIM master manually, refer to the following steps:

a.Start ndp service:

  1. startsrc -s ndpd-host

b.Configure nfs domain for nfs version 4

  1. chnfsdom cluster.com
  2. stopsrc -g nfs
  3. startsrc -g nfs

c.Install filesets bos.sysmgt.nim.master bos.sysmgt.nim.spot

  1. installp -FacqXY -d /61H bos.sysmgt.nim.master bos.sysmgt.nim.spot

d. Initiate nim master

  1. nimconfig -aplatform=chrp -anetboot_kernel=64 -acable_type=N/A -a netname=master -apif_name=en0

e.Configure nim master

  1. nim -o change -a global_export=yes master
  2. nim -o change -a nfs_domain=clusters.com master

f.Create IPv6 network objects

  1. nim -o define -t ent6 -a net_addr=2000:: -a routing1="default fe80::21a:64ff:fe45:e950" network6

g.Add an IPv6 interface to master

  1. nim -o change -a if2="network6 hv32lpar06 fe80::21a:64ff:fe45:e950" -a cable_type2=N/A master

2.4 NIM resources

The concepts of the NIM resources in IPv6 environment are the same as the IPv4 configuration, but there are some configuration differences, the NIM resources in IPv6 environment need to support NFS version 4.

To create the NIM resources with NFSv4 capability, one additional attribute “-a nfs_vers=4” is required when creating the NIM resources, I have tried to change the existing NIM resources to support NFS version 4 through SMIT and NIM commands but failed, the resources are shown with nfs_vers=4 but the NIM client can not mount these resources exports, I am not exactly sure if it is an AIX problem or more configuration steps are needed, so let's specify the “-a nfs_vers=4” when creating any NIM resources in IPv6 environment. The NIM resources are created in several ways, the lpp_source, spot, bosinst_data are created by mknimimage command, the xcataixscript is created by nimnodeset command, and the installp_bundle xCATaixSSL and xCATaixSSL are created manually.

mknimimage will take care of the NFS version 4 support for resources lpp_source, spot, bosinst_data and NIM script. And the nimnodeset will take care of NFS version 4 support for resource xcataixscript. The following commands can be used to enable the NFS version 4 support for installp_bundle xCATaixSSL and xCATaixSSL.

nim -o define -t installp_bundle -a server=master -a nfs_vers=4 -a location=/install/nim/installp_bundle/xCATaixSSL.bnd xCATaixSSL

nim -o define -t installp_bundle -a server=master -a nfs_vers=4 -a location=/install/nim/installp_bundle/xCATaixSSH.bnd xCATaixSSH__

A new flag --nfsv4 will be added to the mknimimage command, if the --nfsv4 is specified, the NIM resources will be created with NFS v4 support. The nimnodeset command will determine the NFS version based on the nodes' ip address.

2.5 sshd configuration

The default sshd configuration on AIX does not enable IPv6 support, the line "ListenAddress ::" has to been uncommented in /etc/ssh/sshd_config to enable the IPv6 support with sshd. If you want to enable both IPv4 and IPv6 for sshd configuration, the line "ListenAddress 0.0.0.0" also needs to be uncommented.

2.6 networks table

The IPv6 network address and subnet are different with the IPv4, here is an example of IPv6 entry in networks table: netname,net,mask,mgtifname,gateway,dhcpserver,tftpserver,nameservers,ntpservers,logservers,dynamicrange,nodehostname,comments,disable "ipv6net","fe80::","64","en1","fe80::bc38:50ff:fe00:2003",,,,,,,,,

The "fe80::" is the network address, the "64" is the prefix length. We are not using the IPv6 netmask such as "ffff:ffff:ffff:ffff:0000:0000:0000:0000" because the IPv6 prefix length makes more sense from user interface perspective, the xCAT code will do the prefixlength to netmask conversion internally if necessary.

2.7 Considerations for IPv4 and IPv6 mixed cluster

It is true that most of the customers could not use pure IPv6 environment in the near future due to IPv6 support issues with software or hardware, so xCAT needs to provide flexibilities to support IPv4 and IPv6 mixed cluster.

I am not seeing obvious problems if the users only want to use xCAT to perform routine system management tasks in IPv4 and IPv6 mixed cluster, but for operating systems deployment, I do not think we could support IPv4 and IPv6 mixed cluster, there are too many network services do not support IPv4 and IPv6 mixed environment such as dhcp/dhcpv6 and nfsv3/nfsv4.

So only one IP version stack should be used to perform operating system deployment, after the operating system deployment is done, then the xCAT postscripts or distributed shell commands can be used to do further network configurations such as configure IPv4 and IPv6 mixed cluster.

Since the xCAT management node may have both IPv4 and IPv6 addresses configured, xCAT needs to know which IP version should be used, for most of the commands against specific nodes such as pping <node_range>, xCAT could know the IP version according to the node's ip address, if the node's ip address is an IPv4 address, then should use IPv4 tools(e.g., ping); if the node's ip address is an IPv6 address, then the IPv6 tools(e.g., ping6) should be used. However, there are also some xCAT commands are not for specific nodes, such as mknimimage, makedhcp -a and makenetworks, for such kind of xCAT commands, we need some way to identify the IP version.

Adding a flag to all of these commands that are not for specific nodes to identify the IP version is a clear and easy way, but not an elegant and transparent way. Using a big switch such as “ipv6cluster” in site table should be able to work, but we want to make the IPv4 and IPv6 differences as transparent as possible to the users, we will try to not add new flag to commands, but if it is absolutely necessary, adding new flags to commands will be a proper way.

3. References

1.Whitepaper “IPV6 on AIX 5L” . http://www-03.ibm.com/systems/resources/systems_p_os_aix_whitepapers_pdf_aix_ipv6.pdf

2.HPC IPv6 White Paper. https://rs6000.pok.ibm.com/afs/apd/project/security/ipv6/hpc.IPv6.wp.v1.5.doc

3.Open Firmware Recommended Practice: iSCSI and TFTP Booting Extensions

News

History

  • Oct 22, 2010: xCAT 2.5 released.
  • Apr 30, 2010: xCAT 2.4 is released.
  • Oct 31, 2009: xCAT 2.3 released.
    xCAT's 10 year anniversary!
  • Apr 16, 2009: xCAT 2.2 released.
  • Oct 31, 2008: xCAT 2.1 released.
  • Sep 12, 2008: Support for xCAT 2
    can now be purchased!
  • June 9, 2008: xCAT breaths life into
    (at the time) the fastest
    supercomputer on the planet
  • May 30, 2008: xCAT 2.0 for Linux
    officially released!
  • Oct 31, 2007: IBM open sources
    xCAT 2.0 to allow collaboration
    among all of the xCAT users.
  • Oct 31, 1999: xCAT 1.0 is born!
    xCAT started out as a project in
    IBM developed by Egan Ford. It
    was quickly adopted by customers
    and IBM manufacturing sites to
    rapidly deploy clusters.