xCAT/xcat-core
2
0
Fork 0
mirror of https://github.com/xcat2/xcat-core.git synced 2025-06-12 00:00:12 +00:00
Code Issues Releases Wiki Activity

Merge pull request #2358 from bybai/statelitedoc

Browse Source 
add statelite document
This commit is contained in:
Victor Hu
2017-01-25 14:31:14 -05:00
committed by GitHub
parent 050feda877 a7af376da2
commit b6be4d0c1c
6 changed files with 746 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/source/guides/admin-guides/manage_clusters/ppc64le/index.rst
View File

@ -13,6 +13,7 @@ These are machines use the IBM Power Architecture and is **IPMI** managed.
management.rst
diskful/index.rst
diskless/index.rst
statelite/index.rst
updatenode.rst
parallel_cmd.rst
virtual_machines/index.rst

266
docs/source/guides/admin-guides/manage_clusters/ppc64le/statelite/advanced_features.rst Normal file
View File

@ -0,0 +1,266 @@
Advanced features
=================
Both directory and its child items coexist in litefile table
------------------------------------------------------------
As described in the above chapters, we can add the files/directories to litefile table. Sometimes, it is necessary to put one directory and also its child item(s) into the litefile table. Due to the implementation of the statelite on Linux, some scenarios works, but some doesn't work.
Here are some examples of both directory and its child items coexisting:
Both the parent directory and the child file coexist: ::
"ALL","/root/testblank/",,,
"ALL","/root/testblank/tempfschild","tempfs",,
One more complex example: ::
"ALL","/root/",,,
"ALL","/root/testblank/tempfschild","tempfs",,
Another more complex example, but we don't intend to support such one scenario: ::
"ALL","/root/",,,
"ALL","/root/testblank/",,,
"ALL","/root/testblank/tempfschild","tempfs",,
For example, in scenario 1, the parent is ``/root/testblank/``, and the child is ``/root/testblank/tempfschild``.
In scenario 2, the parent is ``/root/``, and the child is ``/root/testblank/tempfschild``.
In order to describe the hierarchy scenarios we can use , ``P`` to denote parent, and ``C`` to denote child.
+--------------+-----------------------------------------------------+-------------------------------------------------+
| Option | Example | Remarks |
+==============+=====================================================+=================================================+
| P:tmpfs | "ALL","/root/testblank/",,, | Both the parent and the child are mounted to |
| | "ALL","/root/testblanktempfschild","tempfs",, | tmpfs on the booted node following their |
| | | respective options. Only the parent are mounted |
| | | to the local file system. |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:tmpfs | "ALL","/root/testblank/",,, | Both parent and child are mounted to tmpfs |
| C:persistent | "ALL","/root/testblank/testpersfile","persistent",, | on the booted node following their respective |
| | | options. Only the parent is mounted to the local|
| | | file |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:persistent | "ALL","/root/testblank/","persistent",, | Not permitted now. But plan to support it. |
| C:tmpfs | "ALL","/root/testblank/tempfschild",,, | |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:persistent | "ALL","/root/testblank/","persistent",, | Both parent and child are mounted to tmpfs |
| C:persistent | "ALL","/root/testblank/testpersfile","persistent",, | on the booted node following their respective |
| | | options. Only the parent is mounted to local |
| | | file system. |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:ro C:any | | Not permitted |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:tmpfs C:ro | | Both parent and child are mounted to tmpfs |
| | | on the booted node following their respective |
| | | options. Only the parent is mounted to local |
| | | file system. |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:tmpfs | | Both parent and child are mounted to tmpfs |
| C:con | | on the booted node following their respective |
| | | options. Only the parent is mounted to local |
| | | file system. |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:link | "ALL","/root/testlink/","link",, | Both parent and child are created in tmpfs |
| C:link | "ALL","/root/testlink/testlinkchild","link",, | on the booted node following their respective |
| | | options; there's only one symbolic link of |
| | | the parent is created in the local file system. |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P: link C: | "ALL","/root/testlinkpers/","link",, | Both parent and child are created in tmpfs |
| link, | "ALL","/root/testlink/testlinkchild",, | on the booted node following their respective |
| persistent | "link,persistent" | options; there's only one symbolic link of |
| | | the parent is created in the local file system. |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:link, | "ALL","/root/testlinkpers/","link,persistent",, | NOT permitted |
| persistent | | |
| C: link | "ALL","/root/testlink/testlinkchild","link" | |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:link, | "ALL","/root/testlinkpers/","link,persistent",, | Both parent and child are created in tmpfs |
| persistent | "ALL","/root/testlink | on the booted node following "link,persistent" |
| C:link, | | way; there's only one symbolic link of the |
| persistent | | parent is created in the local file system. |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:link | "ALL","/root/testlink/","link",, | Both parent and child are created in tmpfs |
| C:link,ro | "ALL","/root/testlink/testlinkro","link,ro",, | on the booted node, there's only one symbolic |
| | | link of the parent is created in the local |
| | | file system. |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:link | "ALL","/root/testlink/","link",, | Both parent and child are created in tmpfs |
| C:link,con | "ALL","/root/testlink/testlinkconchild","link,con",,| on the booted node, there's only one symbolic |
| | | link of the parent in the local file system. |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:link, | | NOT Permitted |
| persistent | | |
| C:link,ro | | |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:link, | | NOT Permitted |
| persistent | | |
| C:link,con | | |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:tmpfs | | NOT Permitted |
| C:link | | |
+--------------+-----------------------------------------------------+-------------------------------------------------+
| P:link | | NOT Permitted |
| C:persistent | | |
+--------------+-----------------------------------------------------+-------------------------------------------------+
litetree table
--------------
The litetree table controls where the initial content of the files in the litefile table come from, and the long term content of the ``ro`` files. When a node boots up in statelite mode, it will by default copy all of its tmpfs files from the ``/.default`` directory of the root image, so there is not requirement to setup a litetree table. If you decide that you want some of the files pulled from different locations that are different per node, you can use this table.
See litetree man page for description of attributes.
For example, a user may have two directories with a different ``/etc/motd`` that should be used for nodes in two locations: ::
10.0.0.1:/syncdirs/newyork-590Madison/rhels5.4/x86_64/compute/etc/motd
10.0.0.1:/syncdirs/shanghai-11foo/rhels5.4/x86_64/compute/etc/motd
You can specify this in one row in the litetree table: ::
1,,10.0.0.1:/syncdirs/$nodepos.room/$nodetype.os/$nodetype.arch/$nodetype.profile
When each statelite node boots, the variables in the litetree table will be substituted with the values for that node to locate the correct directory to use. Assuming that ``/etc/motd`` was specified in the litefile table, it will be searched for in all of the directories specified in the litetree table and found in this one.
You may also want to look by default into directories containing the node name first: ::
$noderes.nfsserver:/syncdirs/$node
The litetree prioritizes where node files are found. The first field is the priority. The second field is the image name (ALL for all images) and the final field is the mount point.
Our example is as follows: ::
1,,$noderes.nfsserver:/statelite/$node
2,,cnfs:/gpfs/dallas/
The two directories ``/statelite/$node`` on the node's $noderes.nfsserver and the ``/gpfs/dallas`` on the node cnfs contain root tree structures that are sparsely populated with files that we want to place in those nodes. If files are not found in the first directory, it goes to the next directory. If none of the files can be found in the litetree hierarchy, then they are searched for in ``/.default`` on the local image.
Installing a new Kernel in the statelite image
----------------------------------------------
Obtain you new kernel and kernel modules on the MN, for example here we have a new SLES kernel.
#. Copy the kernel into /boot : ::
cp **vmlinux-2.6.32.10-0.5-ppc64**/boot
#. Copy the kernel modules into ``/lib/modules/<new kernel directory>`` ::
/lib/modules # ls -al
total 16
drwxr-xr-x 4 root root 4096 Apr 19 10:39 .
drwxr-xr-x 17 root root 4096 Apr 13 08:39 ..
drwxr-xr-x 3 root root 4096 Apr 13 08:51 2.6.32.10-0.4-ppc64
**drwxr-xr-x 4 root root 4096 Apr 19 10:12 2.6.32.10-0.5-ppc64**
#. Run genimage to update the statelite image with the new kernel ::
genimage -k 2.6.32.10-0.5-ppc64 <osimage_name>
#. Then after a nodeset command and netbooti, shows the new kernel::
uname -a
Enabling the localdisk Option
-----------------------------
``Note``: You can skip this section if not using the ``localdisk`` option in your litefile table.
Several things need to be done to enable the 'localdisk' support:
Define how to partition the local disk
``````````````````````````````````````
When a node is deployed, the local hard disk needs to be partitioned and formatted before it can be used. This section explains how provide a configuration file that tells xCAT to partition a local disk and make it ready to use for the directories listed in the litefile table with the ``localdisk`` option.
The configuration file needs to be specified in the ``partitionfile`` attribute of the osimage definition. The configuration file includes several parts:
* Global parameters to control enabling or disabling the function
* [disk] part to control the partitioning of the disk
* [localspace] part to control which partition will be used to store the localdisk directories listed in the litefile table
* [swapspace] part to control the enablement of the swap space for the node.
An example localdisk configuration file: ::
enable=yes
enablepart=no
[disk]
dev=/dev/sdb
clear=yes
parts=100M-200M,1G-2G
[disk]
dev=/dev/sda
clear=yes
parts=10,20,30
[disk]
dev=/dev/sdc
clear=yes
parts=10,20,30
[localspace]
dev=/dev/sda1
fstype=ext3
[swapspace]
dev=/dev/sda2
The two global parameters ``enable`` and ``enablepart`` can be used to control the enabling/disabling of the functions:
* enable: The localdisk feature only works when ``enable`` is set to *yes*. If it is set to *no*, the localdisk configuration will not be run.
* enablepart: The partition action (refer to the ``[disk]`` section) will be run only when ``enablepart=yes``.
The ``[disk]`` section is used to configure how to partition a hard disk:
* dev: The path of the device file.
* clear: If set to ``yes`` it will clear all the existing partitions on this disk.
* fstype: The file system type for the new created partitions. ``ext3`` is the default value if not set.
* parts: A comma separated list of space ranges, one for each partition that will be created on the device. The valid format for each space range is ``<startpoint>-<endpoint>`` or ``<percentage of the disk>``. For example, you could set it to ``100M-10G`` or ``50``. If you set it to ``50``, that means 50% of the disk space will be assigned to that partition.
The ``[localspace]`` section is used to specify which partition will be used as local storage for the node.
* dev: The path of the partition.
* fstype: The file system type on the partition.
the ``[swapspace]`` section is used to configure the swap space for the statelite node.
* dev: The path of the partition file which will be used as the swap space.
To enable the local disk capability, create the configuration file (for example in ``/install/custom``) and set the path in the partitionfile attribute for the osimage: ::
chdef -t osimage partitionfile=/install/custom/cfglocaldisk
Now all nodes that use this osimage (i.e. have their provmethod attribute set to this osimage definition name), will have its local disk configured.
Configure the files in the litefile table
`````````````````````````````````````````
For the files/directories that you would like xCAT to store on the local disk, add an entry in the litefile table like this: ::
"ALL","/tmp/","localdisk",,
``Note``: you do not need to specify the swap space in the litefile table. Just putting it in the partitionfile config file is enough.
Add an entry in policy table to permit the running of the ``getpartitioin`` command from the node ::
chtab priority=7.1 policy.commands=getpartition policy.rule=allow
If Using the RAMdisk-based Image
````````````````````````````````
If you want to use the local disk option with a RAMdisk-based image, remember to follow the instructions in :doc:`Switch to the RAMdisk based solution <./provision_statelite>`.
If your reason for using a RAMdisk image is to avoid compute node runtime dependencies on the service node or management node, then the only entries you should have in the litefile table should be files/dirs that use the localdisk option.
Debugging techniques
--------------------
When a node boots up in statelite mode, there is a script that runs called statelite that is in the root directory of ``$imageroot/etc/init.d/statelite``. This script is not run as part of the rc scripts, but as part of the pre-switch root environment. Thus, all the linking is done in this script. There is a ``set x`` near the top of the file. You can uncomment it and see what the script runs. You will then see lots of mkdirs and links on the console.
You can also set the machine to shell. Just add the word ``shell`` on the end of the pxeboot file of the node in the append line. This will make the init script in the initramfs pause 3 times before doing a switch_root.
When all the files are linked they are logged in ``/.statelite/statelite.log`` on the node. You can get into the node after it has booted and look in the ``/.statelite`` directory.

134
docs/source/guides/admin-guides/manage_clusters/ppc64le/statelite/config_statelite.rst Normal file
View File

@ -0,0 +1,134 @@
Configuration
=============
Statelite configuration is done using the following tables in xCAT:
* litefile
* litetree
* statelite
* policy
* noderes
litefile table
--------------
The litefile table specifies the directories and files on the statelite nodes that should be read/write, persistent, or read-only overlay. All other files in the statelite nodes come from the read-only statelite image.
#. The first column in the litefile table is the image name this row applies to. It can be an exact osimage definition name, an osimage group (set in the groups attribute of osimages), or the keyword ``ALL``.
#. The second column in the litefile table is the full path of the directory or file on the node that you are setting options for.
#. The third column in the litefile table specifies options for the directory or file:
#. tmpfs - It provides a file or directory for the node to use when booting, its permission will be the same as the original version on the server. In most cases, it is read-write; however, on the next statelite boot, the original version of the file or directory on the server will be used, it means it is non-persistent. This option can be performed on files and directories.
#. rw - Same as Above.Its name "rw" does NOT mean it always be read-write, even in most cases it is read-write. Do not confuse it with the "rw" permission in the file system.
#. persistent - It provides a mounted file or directory that is copied to the xCAT persistent location and then over-mounted on the local file or directory. Anything written to that file or directory is preserved. It means, if the file/directory does not exist at first, it will be copied to the persistent location. Next time the file/directory in the persistent location will be used. The file/directory will be persistent across reboots. Its permission will be the same as the original one in the statelite location. It requires the statelite table to be filled out with a spot for persistent statelite. This option can be performed on files and directories.
#. con - The contents of the pathname are concatenated to the contents of the existing file. For this directive the searching in the litetree hierarchy does not stop when the first match is found. All files found in the hierarchy will be concatenated to the file when found. The permission of the file will be "-rw-r--r--", which means it is read-write for the root user, but readonly for the others. It is non-persistent, when the node reboots, all changes to the file will be lost. It can only be performed on files. Do not use it for one directory.
#. ro - The file/directory will be overmounted read-only on the local file/directory. It will be located in the directory hierarchy specified in the litetree table. Changes made to this file or directory on the server will be immediately seen in this file/directory on the node. This option requires that the file/directory to be mounted must be available in one of the entries in the litetree table. This option can be performed on files and directories.
#. tmpfs,rw - Only for compatibility it is used as the default option if you leave the options column blank. It has the same semantics with the link option, so when adding new items into the _litefile table, the link option is recommended.
#. link - It provides one file/directory for the node to use when booting, it is copied from the server, and will be placed in tmpfs on the booted node. In the local file system of the booted node, it is one symbolic link to one file/directory in tmpfs. And the permission of the symbolic link is "lrwxrwxrwx", which is not the real permission of the file/directory on the node. So for some application sensitive to file permissions, it will be one issue to use "link" as its option, for example, "/root/.ssh/", which is used for SSH, should NOT use "link" as its option. It is non-persistent, when the node is rebooted, all changes to the file/directory will be lost. This option can be performed on files and directories.
#. link,ro - The file is readonly, and will be placed in tmpfs on the booted node. In the local file system of the booted node, it is one symbolic link to the tmpfs. It is non-persistent, when the node is rebooted, all changes to the file/directory will be lost. This option requires that the file/directory to be mounted must be available in one of the entries in the litetree table. The option can be performed on files and directories.
#. link,con - Similar to the "con" option. All the files found in the litetree hierarchy will be concatenated to the file when found. The final file will be put to the tmpfs on the booted node. In the local file system of the booted node, it is one symbolic link to the file/directory in tmpfs. It is non-persistent, when the node is rebooted, all changes to the file will be lost. The option can only be performed on files.
#. link,persistent - It provides a mounted file or directory that is copied to the xCAT persistent location and then over-mounted to the tmpfs on the booted node, and finally the symbolic link in the local file system will be linked to the over-mounted tmpfs file/directory on the booted node. The file/directory will be persistent across reboots. The permission of the file/directory where the symbolic link points to will be the same as the original one in the statelite location. It requires the statelite table to be filled out with a spot for persistent statelite. The option can be performed on files and directories.
#. localdisk - The file or directory will be stored in the local disk of the statelite node. Refer to the section To enable the localdisk option to enable the 'localdisk' support.
Currently, xCAT does not handle the relative links very well. The relative links are commonly used by the system libraries, for example, under ``/lib/`` directory, there will be one relative link matching one ``.so`` file. So, when you add one relative link to the litefile table (Not recommend), make sure the real file also be included, or put its directory name into the litefile table.
**Note**: It is recommended that you specify at least the entries listed below in the litefile table, because most of these files need to be writeable for the node to boot up successfully. When any changes are made to their options, make sure they won't affect the whole system.
Sample Data for Redhat statelite setup
``````````````````````````````````````
This is the minimal list of files needed, you can add additional files to the litefile table. ::
#image,file,options,comments,disable
"ALL","/etc/adjtime","tmpfs",,
"ALL","/etc/securetty","tmpfs",,
"ALL","/etc/lvm/","tmpfs",,
"ALL","/etc/ntp.conf","tmpfs",,
"ALL","/etc/rsyslog.conf","tmpfs",,
"ALL","/etc/rsyslog.conf.XCATORIG","tmpfs",,
"ALL","/etc/udev/","tmpfs",,
"ALL","/etc/ntp.conf.predhclient","tmpfs",,
"ALL","/etc/resolv.conf","tmpfs",,
"ALL","/etc/yp.conf","tmpfs",,
"ALL","/etc/resolv.conf.predhclient","tmpfs",,
"ALL","/etc/sysconfig/","tmpfs",,
"ALL","/etc/ssh/","tmpfs",,
"ALL","/etc/inittab","tmpfs",,
"ALL","/tmp/","tmpfs",,
"ALL","/var/","tmpfs",,
"ALL","/opt/xcat/","tmpfs",,
"ALL","/xcatpost/","tmpfs",,
"ALL","/etc/systemd/system/multi-user.target.wants/","tmpfs",,
"ALL","/root/.ssh/","tmpfs",,
"ALL","/etc/rc3.d/","tmpfs",,
"ALL","/etc/rc2.d/","tmpfs",,
"ALL","/etc/rc4.d/","tmpfs",,
"ALL","/etc/rc5.d/","tmpfs",,
Sample Data for SLES statelite setup
````````````````````````````````````
This is the minimal list of files needed, you can add additional files to the litefile table. ::
#image,file,options,comments,disable
"ALL","/etc/lvm/","tmpfs",,
"ALL","/etc/ntp.conf","tmpfs",,
"ALL","/etc/ntp.conf.org","tmpfs",,
"ALL","/etc/resolv.conf","tmpfs",,
"ALL","/etc/ssh/","tmpfs",,
"ALL","/etc/sysconfig/","tmpfs",,
"ALL","/etc/syslog-ng/","tmpfs",,
"ALL","/etc/inittab","tmpfs",,
"ALL","/tmp/","tmpfs",,
"ALL","/etc/init.d/rc3.d/","tmpfs",,
"ALL","/etc/init.d/rc5.d/","tmpfs",,
"ALL","/var/","tmpfs",,
"ALL","/etc/yp.conf","tmpfs",,
"ALL","/etc/fstab","tmpfs",,
"ALL","/opt/xcat/","tmpfs",,
"ALL","/xcatpost/","tmpfs",,
"ALL","/root/.ssh/","tmpfs",,
litetree table
--------------
The litetree table controls where the initial content of the files in the litefile table come from, and the long term content of the ``ro`` files. When a node boots up in statelite mode, it will by default copy all of its tmpfs files from the ``.default`` directory of the root image, for example ``/install/netboot/rhels7.3/x86_64/compute/rootimg/.default``, so there is not required to set up a litetree table. If you decide that you want some of the files pulled from different locations that are different per node, you can use this table.
You can choose to use the defaults and not set up a litetree table.
statelite table
---------------
The statelite table specifies location on an NFS server where a nodes persistent files are stored. This is done by entering the information into the statelite table.
In the statelite table, the node or nodegroups in the table must be unique; that is a node or group should appear only once in the first column table. This makes sure that only one statelite image can be assigned to a node. An example would be: ::
"compute",,"<nfssvr_ip>:/gpfs/state",,
Any nodes in the compute node group will have their state stored in the ``/gpfs/state`` directory on the machine with ``<nfssvr_ip>`` as its IP address.
When the node boots up, then the value of the ``statemnt`` attribute will be mounted to ``/.statelite/persistent``. The code will then create the following subdirectory ``/.statelite/persistent/<nodename>``, if there are persistent files that have been added in the litefile table. This directory will be the root of the image for this node's persistent files. By default, xCAT will do a hard NFS mount of the directory. You can change the mount options by setting the mntopts attribute in the statelite table.
Also, to set the ``statemnt`` attribute, you can use variables from xCAT database. It follows the same grammar as the litetree table. For example: ::
#node,image,statemnt,mntopts,comments,disable
"cn1",,"$noderes.nfsserver:/lite/state/$nodetype.profile","soft,timeo=30",,
``Note``: Do not name your persistent storage directory with the node name, as the node name will be added in the directory automatically. If you do, then a directory named ``/state/cn1`` will have its state tree inside ``/state/cn1/cn1``.
Policy
------
Ensure policies are set up correctly in the Policy Table. When a node boots up, it queries the xCAT database to get the litefile and litetree table information. In order for this to work, the commands (of the same name) must be set in the policy table to allow nodes to request it. This should happen automatically when xCAT is installed, but you may want to verify that the following lines are in the policy table: ::
chdef -t policy -o 4.7 commands=litefile rule=allow
chdef -t policy -o 4.8 commands=litetree rule=allow
noderes
-------
``noderes.nfsserver`` attribute can be set for the NFSroot server. If this is not set, then the default is the Management Node.
``noderes.nfsdir`` can be set. If this is not set, the the default is ``/install``

43
docs/source/guides/admin-guides/manage_clusters/ppc64le/statelite/hierarchy_support.rst Normal file
View File

@ -0,0 +1,43 @@
Hierarchy Support
-----------------
In the ``statelite`` environment, the service node needs to provide NFS service for the compute node with ``statelite``, the service nodes must to be setup with diskfull installation.
Setup the diskfull service node
```````````````````````````````
#. Setup one diskfull service node at first.
#. Since statelite is a kind of NFS-hybrid method, you should remove the installloc attribute in the site table. This makes sure that the service node does not mount the ``/install`` directory from the management node on the service node.
Generate the statelite image
````````````````````````````
To generate the statelite image for your own profile follow instructions in :doc:`Customize your statelite osimage <./provision_statelite>`.
``NOTE``: if the NFS directories defined in the litetree table are on the service node, it is better to setup the NFS directories in the service node following the chapter.
Sync the ``/install`` directory
```````````````````````````````
The command prsync is used to sync the ``/install`` directory to the service nodes.
Run the following: ::
cd /
prsync install <sn>:/
``<sn>`` is the hostname of the service node you defined.
Since the ``prsync`` command will sync all the contents in the ``/install`` directory to the service nodes, the first time will take a long time. But after the first time, it will take very short time to sync.
``NOTE``: if you make any changes in the ``/install`` directory on the management node, and the changes can affect the statelite image, you need to sync the ``/install`` directory to the service node again.
Set the boot state to statelite
```````````````````````````````
You can now deploy the node: ::
rinstall <noderange> osimage=rhel5.3-x86_64-statelite-compute
This will create the necessary files in ``/tftpboot`` for the node to boot correctly.

45
docs/source/guides/admin-guides/manage_clusters/ppc64le/statelite/index.rst Normal file
View File

@ -0,0 +1,45 @@
Statelite Installation
======================
**Overview**
This document details the design and setup for the statelite solution of xCAT. **Statelite** is an intermediate mode between **diskful** and **diskless**.
Statelite provides two kinds of efficient and flexible solutions, most of the OS image can be NFS mounted read-only, or the OS image can be in the ramdisk with tmpfs type. Different from the stateless solution, statelite provides a configurable list of directories and files that can be read-write. These read-write directories and files can be configured to either persist or not persist across reboots.
**Solutions**
There are two solutions: ``NFSROOT-based`` and ``RAMdisk-based``.
#. NFSROOT-based(default):
#. rootfstype in the osimage xCAT data objects is left as blank, or set to ``nfs``, the ``NFSROOT-base`` statelite solution will be enabled.
#. the ROOTFS is NFS mounted read-only.
#. RAMdisk-based:
#. rootfstype in the osimage xCAT data objects is set to ``ramdisk``.
#. one image file will be downloaded when the node is booting up, and the file will be extracted to the ramdisk, and used as the ROOTFS.
**Advantages**
``Statelite`` offers the following advantages over xCAT's stateless (RAMdisk) implementation:
#. Some files can be made persistent over reboot. This is useful for license files or database servers where some state is needed. However, you still get the advantage of only having to manage a single image.
#. Changes to hundreds of machines can take place instantly, and automatically, by updating one main image. In most cases, machines do not need to reboot for these changes to take affect. This is only for the ``NFSROOT-based`` solution.
#. Ease of administration by being able to lock down an image. Many parts of the image can be read-only, so no modifications can transpire without updating the central image.
#. Files can be managed in a hierarchical manner. For example: Suppose you have a machine that is in one lab in Tokyo and another in London. You could set table values for those machines in the xCAT database to allow machines to sync from different places based on their attributes. This allows you to have one base image with multiple sources of file overlay.
#. Ideal for virtualization. In a virtual environment, you may not want a disk image (neither stateless nor stateful) on every virtual node as it consumes memory and disk. Virtualizing with the statelite approach allows for images to be smaller, easier to manage, use less disk, less memory, and more flexible.
**Disadvantages**
However, there're still several disadvantages, especially for the ``NFSROOT-based`` solution.
#. NFS Root requires more network traffic to run as the majority of the disk image runs over NFS. This may depend on your workload, but can be minimized. Since the bulk of the image is read-only, NFS caching on the server helps minimize the disk access on the server, and NFS caching on the client helps reduce the network traffic.
#. NFS Root can be complex to set up. As more files are created in different places, there are greater chances for failures. This flexibility is also one of the great virtues of Statelite. The image can work in nearly any environment.
.. toctree::
:maxdepth: 2
config_statelite.rst
provision_statelite.rst
hierarchy_support.rst
advanced_features.rst

257
docs/source/guides/admin-guides/manage_clusters/ppc64le/statelite/provision_statelite.rst Normal file
View File

@ -0,0 +1,257 @@
Provision statelite
===================
Show current provisioning method
--------------------------------
To determine the current provisioning method of your node, execute: ::
lsdef <noderange> -i provmethod
``Note``: syncfiles is not currently supported for statelite nodes.
Generate default statelite image from distoro media
---------------------------------------------------
In this example, we are going to create a new compute node osimage for ``rhels7.3`` on ``ppc64le``. We will set up a test directory structure that we can use to create our image. Later we can just move that into production.
Use the copycds command to copy the appropriate iso image into the ``/install`` directory for xCAT. The copycds commands will copy the contents to ``/install/rhels7.3/<arch>``. For example: ::
copycds RHEL-7.3-20161019.0-Server-ppc64le-dvd1.iso
The contents are copied into ``/install/rhels7.3/ppc64le/``
The configuration files pointed to by the attributes are the defaults shipped with xCAT. We will want to copy them to the ``/install`` directory, in our example the ``/install/test`` directory and modify them as needed.
Statelite Directory Structure
-----------------------------
Each statelite image will have the following directories: ::
/.statelite/tmpfs/
/.default/
/etc/init.d/statelite
All files with link options, which are symbolic links, will link to ``/.statelite/tmpfs``.
tmpfs files that are persistent link to ``/.statelite/persistent/<nodename>/``, ``/.statelite/persistent/<nodename>`` is the directory where the node's individual storage will be mounted to.
``/.default`` is where default files will be copied to from the image to tmpfs if the files are not found in the litetree hierarchy.
Customize your statelite osimage
--------------------------------
Create the osimage definition
`````````````````````````````
Setup your osimage/linuximage tables with new test image name, osvers,osarch, and paths to all the files for building and installing the node. So using the above generated ``rhels7.3-ppc64le-statelite-compute`` as an example, I am going to create my own image. The value for the provisioning method attribute is osimage in my example.::
mkdef rhels7.3-custom-statelite -u profile=compute provmethod=statelite
Check your setup: ::
lsdef -t osimage rhels7.3-custom-statelite
Customize the paths to your ``pkglist``, ``syncfile``, etc to the osimage definition, that you require. ``Note``, if you modify the files on the ``/opt/xcat/share/...`` path then copy to the appropriate ``/install/custom/...`` path. Remember all files must be under ``/install`` if using hierarchy (service nodes).
Copy the sample ``*list`` files and modify as needed: ::
mkdir -p /install/test/netboot/rh
cp -p /opt/xcat/share/xcat/netboot/rh/compute.rhels7.ppc64le.pkglist \
/install/test/netboot/rh/compute.rhels7.ppc64le.pkglist
cp -p /opt/xcat/share/xcat/netboot/rh/compute.exlist \
/install/test/netboot/rh/compute.exlist
chdef -t osimage -o rhels7.3-custom-statelite \
pkgdir=/install/rhels7.3/ppc64le \
pkglist=/install/test/netboot/rh/compute.rhels7.ppc64le.pkglist \
exlist=/install/test/netboot/rh/compute.exlist \
rootimgdir=/install/test/netboot/rh/ppc64le/compute
Setup pkglists
``````````````
In the above example, you have defined your pkglist to be in ``/install/test/netboot/rh/compute.rhels7.ppc64le.pkglist``.
Edit ``compute.rhels7.ppc64le.pkglist`` and ``compute.exlist`` as needed. ::
vi /install/test/netboot/rh/compute.rhels7.ppc64le.pkglist
vi /install/test/netboot/rh/compute.exlist
Make sure nothing is excluded in compute.exlist that you need.
Install other specific packages
```````````````````````````````
Make the directory to hold additional rpms to install on the compute node. ::
mkdir -p /install/test/post/otherpkgs/rh/ppc64le
Now copy all the additional OS rpms you want to install into ``/install/test/post/otherpkgs/rh/ppc64le``.
At first you need to create one text file which contains the complete list of files to include in the repository. The name of the text file is rpms.list and must be in ``/install/test/post/otherpkgs/rh/ppc64le`` directory. Create rpms.list: ::
cd /install/test/post/otherpkgs/rh/ppc64le
ls *.rpm > rpms.list
Then, run the following command to create the repodata for the newly-added packages: ::
createrepo -i rpms.list /install/test/post/otherpkgs/rh/ppc64le
The ``createrepo`` command with -i rpms.list option will create the repository for the rpm packages listed in the rpms.list file. It won't destroy or affect the rpm packages that are in the same directory, but have been included into another repository.
Or, if you create a sub-directory to contain the rpm packages, for example, named other in ``/install/test/post/otherpkgs/rh/ppc64le``. Run the following command to create repodata for the directory ``/install/test/post/otherpkgs/rh/ppc64le``. ::
createrepo /install/post/otherpkgs/<os>/<arch>/**other**
``Note``: Replace other with your real directory name.
Define the location of of your otherpkgs in your osimage: ::
chdef -t osimage -o rhels7.3-custom-statelite \
otherpkgdir=/install/test/post/otherpkgs/rh/ppc64le \
otherpkglist=/install/test/netboot/rh/compute.otherpkgs.pkglist
There are examples under ``/opt/xcat/share/xcat/netboot/<platform>`` of typical ``*otherpkgs.pkglist`` files that can used as an example of the format.
Set up Post scripts for statelite
`````````````````````````````````
The rules to create post install scripts for statelite image is the same as the rules for stateless/diskless install images.
There're two kinds of postscripts for statelite (also for stateless/diskless).
The first kind of postscript is executed at genimage time, it is executed again the image itself on the MN . It was setup in The postinstall file section before the image was generated.
The second kind of postscript is the script that runs on the node during node deployment time. During init.d timeframe, ``/etc/init.d/gettyset`` calls ``/opt/xcat/xcatdsklspost`` that is in the image. This script uses wget to get all the postscripts under ``mn:/install/postscripts`` and copy them to the ``/xcatpost`` directory on the node. It uses openssl or stunnel to connect to the xcatd on the mn to get all the postscript names for the node from the postscripts table. It then runs the postscripts for the node.
Setting up postinstall files (optional)
```````````````````````````````````````
Using postinstall files is optional. There are some examples shipped in ``/opt/xcat/share/xcat/netboot/<platform>``.
If you define a postinstall file to be used by genimage, then ::
chdef -t osimage -o rhels7.3-custom-statelite postinstall=<your postinstall file path>.
Generate the image
------------------
Run the following command to generate the image based on your osimage named ``rhels7.3-custom-statelite``. Adjust your genimage parameters to your architecture and network settings. See man genimage. ::
genimage rhels7.3-custom-statelite
The genimage will create a default ``/etc/fstab`` in the image, if you want to change the defaults, on the management node, edit fstab in the image: ::
cd /install/netboot/rhels7/ppc64le/compute/rootimg/etc
cp fstab fstab.ORIG
vi fstab
``Note``: adding ``/tmp`` and ``/var/tmp`` to ``/etc/fstab`` is optional, most installations can simply use ``/``. It was documented her to show that you can restrict the size of filesystems, if you need to. The indicated values are just and example, and you may need much bigger filessystems, if running applications like OpenMPI.
Pack the image
--------------
Execute liteimg ::
liteimg rhels7.3-custom-statelite
Boot the statelite node
-----------------------
Execute ``rinstall`` ::
rinstall node1 osimage=rhels7.3-custom-statelite
Switch to the RAMdisk based solution
------------------------------------
It is optional, if you want to use RAMdisk-based solution, follow this section.
Set rootfstype
``````````````
If you want the node to boot with a RAMdisk-based image instead of the NFS-base image, set the rootfstype attribute for the osimage to ``ramdisk``. For example: ::
chdef -t osimage -o rhels7.3-custom-statelite rootfstype=ramdisk
Run liteimg command
```````````````````
The ``liteimg`` command will modify your statelite image (the image that ``genimage`` just created) by creating a series of links. Once you are satisfied with your image contains what you want it to, run ``liteimg <osimagename>``: ::
liteimg rhels7.3-custom-statelite
For files with link options, the ``liteimg`` command creates two levels of indirection, so that files can be modified while in their image state as well as during runtime. For example, a file like ``$imageroot/etc/ntp.conf`` with link option in the litefile table, will have the following operations done to it:
In our case ``$imageroot`` is ``/install/netboot/rhels5.3/x86_64/compute/rootimg``
The ``liteimg`` script, for example, does the following to create the two levels of indirection. ::
mkdir -p $imageroot/.default/etc
mkdir -p $imageroot/.statelite/tmpfs/etc
mv $imgroot/etc/ntp.conf $imgroot/.default/etc
cd $imgroot/.statelite/tmpfs/etc
ln -sf ../../../.default/etc/ntp.conf .
cd $imgroot/etc
ln -sf ../.statelite/tmpfs/etc/ntp.conf .
When finished, the original file will reside in ``$imgroot/.default/etc/ntp.conf``. ``$imgroot/etc/ntp.conf`` will link to ``$imgroot/.statelite/tmpfs/etc/ntp.conf`` which will in turn link to ``$imgroot/.default/etc/ntp.conf``.
But for files without link options, the ``liteimg`` command only creates clones in ``$imageroot/.default/`` directory, when the node is booting up, the mount command with ``--bind`` option will get the corresponding files from the ``litetree`` places or ``.default`` directory to the sysroot directory.
``Note``: If you make any changes to your litefile table after running ``liteimg`` then you will need to rerun ``liteimg`` again. This is because files and directories need to have the two levels of redirects created.
Boot the statelite node
```````````````````````
Make sure you have set up all the attributes in your node definitions correctly following the node installation instructions corresponding to your hardware:
You can now deploy the node by running the following commmands: ::
rinstall <noderange>
You can then use ``rcons`` or ``wcons`` to watch the node boot up.
Adding/updating software and files for the running nodes
--------------------------------------------------------
Make changes to the files which configured in the litefile table
````````````````````````````````````````````````````````````````
During the preparation or booting of node against statelite mode, there are specific processes to handle the files which configured in the litefile table. The following operations need to be done after made changes to the statelite files.
#. Run ``liteimg`` against the osimage and reboot the node : Added, removed or changed the entries in the litefile table.
#. Reboot the node :
* Changed the location directory in the litetree table.
* Changed the location directory in the statelite table.
* Changed, removed the original files in the location of litetree or statelite table.
``Note``: Thing should not do:
* When there are node running on the nfs-based statelite osimage, do not run the packimage against this osimage.
Make changes to the common files
````````````````````````````````
Because most of system files for the nodes are NFS mounted on the Management Node with read-only option, installing or updating software and files should be done to the image. The image is located under ``/install/netboot/<os>/<arch>/<profile>/rootimg`` directory.
To install or update an rpm, do the following:
* Install the rpm package into rootimg ::
rpm --root /install/netboot/<os>/<arch>/<profile>/rootimg -ivh rpm_name
* Restart the software application on the nodes ::
xdsh <noderange> <restart_this_software_command>
It is recommended to follow the section (Adding third party softeware) to add the new rpm to the otherpkgs.pkglist file, so that the rpm will get installed into the new image next time the image is rebuilt.
``Note``: The newly added rpms are not shown when running ``rpm -qa`` on the nodes although the rpm is installed. It will shown next time the node is rebooted.
To create or update a file for the nodes, just modify the file in the image and restart any application that uses the file.
For the ramdisk-based node, you need to reboot the node to take the changes.