Rocks Cluster Distribution: Users Guide: 
| ||
|---|---|---|
| Prev | Chapter 4. Cluster Services | Next |
The 411 Secure Information Service provides NIS-like functionality for Rocks clusters. It is named after the common "411" code for information in the phone system. We use 411 to securely distribute password files, user and group configuration files and the like.
411 uses Public Key Cryptography to protect files' contents. It operates on a file level, rather than the RPC-based per-line maps of NIS. 411 does not rely on RPC, and instead distributes the files themselves using HTTP (web service). Its central task is to securely maintain critical login/password files on the worker nodes of a cluster. It does this by implementing a file-based distributed database with weak consistency semantics. The design goals of 411 include scalability, security, low-latency when changes occur, and resilience to failures.
 | Beginning with the Rocks 3.1.0 Matterhorn release, 411 replaces NIS as the default method of distributing /etc/passwd and other login files. We no longer support NIS. |
The 411 system intentionally mimics the NIS interface for system administrators. Of course there are elements in 411 which are not present in NIS, namely RSA public and private cryptographic keys. However we have attempted to make 411 as easy to use in the NIS capacity as possible.
Files listed in /var/411/Files.mk are automatically serviced by 411. This means that any file listed there will be kept up to date by the 411 agents on all compute nodes in your cluster. This is done using the makefile /var/411/Makefile in a similar fashion to NIS. To force the 411 system to flush all changes, execute the following on the frontend node:
# make -C /var/411 |
Note that this command is run by cron every hour on the frontend to propagate password changes, etc to compute nodes. New files can be added to Files.mk as necessary for custom services on the cluster.
To force all 411 files to be re-encrypted and change alerts sent to all compute nodes, run this on the frontend
# make -C /var/411 force |
| The 411 service uses IP broadcast messages on your cluster's private network to achieve optimal performance. To force all compute nodes to retrieve the latest files from the frontend, execute:
|
Client nodes listen on the IP broadcast channel for "411alert" messages from the master (the frontend). The master will send 411alerts during a 411put operation, just after it has encrypted a 411 file. The alert message serves as a cue to the client nodes that a file has changed and needs to be retrieved. In this way the 411 system generally achieves a low-latency response to changes. 411 alerts are resent periodically to compensate for lost messages.
Figure: The 411 listener architecture. When the frontend changes a login file, the 411 makefile sends out a broadcast alert to the cluster nodes notifying of the event. Nodes then pull the file from the frontend via HTTP.
To prevent flooding the master server with requests, the listeners keep an estimate of the cluster size, and use it calibrate a random backoff for their requests. A client does not immediately request the changed file, but waits some amount of time before asking for it.
 | 411 is akin to a distributed database, and is not a centralized lookup service like NIS. While scalable, 411 does not provide instantaneous distribution of new files. The delay between running the 411 makefile and all nodes receiving the changed file depends on cluster size. A large password file on a cluster with many nodes can take up to a minute to fully synchronize on all nodes. |
In addition to the 411 listener agent on nodes, another agent retrieves all messages from the frontend at a regular interval, regardless of whether the files have changed or not.
The polling interval at nodes is set to 5 hours by default. To change it, set the "interval" option in the 411 configuration file on nodes.
/etc/411.conf: ... <interval sec="300"> ... |
There is automatically some randomness introduced to the polling interval to avoid storms on the network.
The poller is implemented as a greceptor event module, and relies on the operation of that daemon. 411 Pollers obtain their master servers by reading a configuration file on their local disk. This file, written in XML, is generated automatically by the 411 listener.
 | Because the 411 polling daemon runs as root on client nodes, it is essential that the 411 http directory on a master server be writable only by root to avoid any chance of privilege elevation. This is the default in Rocks. |
See the 411 paper in the Bibliography section for details of 411 security mechanisms.
Beginning in Rocks 3.3.0, 411 has the ability to send messages to subsets of the cluster. This facility, called 411 groups, allows us to distribute different files to nodes depending on their type. The group mechanism depends on the client nodes specifying group names in their local 411 configuration file; these are called the client's "registered" groups.
| There is no per-group key in 411. The groups mechanism is only a convenience feature, without strong security to enforce it. Specifically, a node can eavesdrop on messages for a foreign group that it is not a member of. |
Group names are multi-level, and resemble file paths. By default, every node is a member of the '/' group (corresponding to the traditional top-level 411 group), and the '/Membership' group, where membership is the node membership in the frontend database, such as "Compute" or "NAS".
| A special Makefile called /var/411/Group.mk is available to help you setup and maintain 411 groups. After editing this file to specify which files go to which group, run
To activate the 411 group makefile actions. |
By default, nodes are members of a group with the same name as their Membership. For example compute nodes are automatically a member of the group "Compute". A sample 411.conf file with several groups looks like:
<!-- Configuration file for the 411 Information Service --> <config> <master url="http://10.1.1.1/411.d/" score="0"/> <group>/light/blue</group> <group>Compute</group> </config> |
Multi-element group names have a simple inheritance model: specific groups imply more general ones. For example, if you are a member of the group /compute/light, you will automatically be interested in messages in group "/compute/light" and "/compute". You will not be interested in messages from group "/compute/heavy". In this case "/compute/light" is the specific group, and "/compute" is the more general one.
Figure: 411 groups. The client uses registered groups from its local configuration file to filter a stream of offered messages from the master. The messages with the dashed border represent newly changed 411 files on the master, the solid messages at the bottom have been chosen by the client. Note that group "/compute/light" implies "/compute".
411get [--all] [--master=url] [--conf] [--pub] [--shared] [--local] [file] |
Retrieves and decrypts 411 messages. Prints resulting file to standard out. When invoked with no files, 411get will list the available 411 messages.
The following options are available:
--all Retrieves and writes all available 411 messages from the most attractive master. Does not print output to stdout, nor ask for confirmation before overwriting files.
--master The url of a 411 master server to use. Defaults to "http://10.1.1.1/411.d/" or whatever is present in "/etc/411.conf". If given, this master takes precedence over those listed in the configuration file.
--file, --local Assume the file is local, ie present in the current directory. Does not use http to retrieve the file. Decrypts and prints the file contents.
--conf The configuration file to use. Defaults to "/etc/411.conf".
--pub The location of the cluster public RSA key. Defaults to "/etc/security/cluster-public-key.pem".
--shared The location of the cluster shared key. Defaults to "/etc/411-security/shared.key"
The master servers, along with their quality score, are listed in the "/etc/411.conf" file on compute nodes.
411put [--411dir=dir] [--urldir=dir] [--see] [--noalert] [--alert=channel] [--411name] [--pub] [--priv] [--comment=char] [--chroot=dir] [--chroot-here] [--group=group] file1 file2 ... |
Encrypts and publishes files using the 411 secure information service. Will send a broadcast message to client nodes by default, alerting them of a changed file.
The following options are available:
--chroot=dir Turn "dir" into the root directory of the destination file. This allows files to be located in a different place on the master and clients.
Example: 411put --chroot=/var/411/groups/compute /var/411/groups/compute/etc/passwd |
Will put "/var/411/groups/compute/etc/passwd" on compute nodes as "/etc/passwd".
--chroot-here A convenience option, equivalent to --chroot=$PWD.
--group=name A 411 group for this file. Clients will ignore 411 messages in groups which they are not a part of. Allows 411 files to be published to a subset of the cluster. Name is path-like: "Compute/green", or "/Compute/green". Spaces are ok: "a space/yellow" is a valid group name as well.
--comment The comment character for this file. Used to place a descriptive header without disrupting normal operations. Often set to "#". Default is none.
--411dir The local directory to place encrypted 411 messages. Defaults to "/etc/411.d/". Be careful about the permissions of this directory.
--urldir The web directory where 411 messages are available. Defaults to "/411.d/".
--see Shows the encrypted file contents on stdout.
--noalert Suppresses alert message.
--alert Specifies the alert channel, which can be multicast or unicast UDP. Defaults to the IP broadcast channel (255.255.255.255).
--411name Prints the 411 message name for the file. Provided for convenience.
--pub The location of the 411 master public RSA key. Defaults to a 1024 bit key in "/etc/411-security/master.pub". This file should have permissions 0444 (read by all) and be owned by root.
--priv The location of the 411 master private RSA key. Defaults to a 1024 bit key in "/etc/411-security/master.key". This file should exist only on the master node and be owned by root and have permissions 0400 (read only by root).
--make-shared-key Generate a new random shared key. The key is a 256 random number encoded in base64.