.ifi init_plm "FS-00" .srv draft "" .srv draft_date "" .srv section %Arg1% .pdl 66 .ifi l0h "File System Mechanisms" Various mechanisms common within the file system will be presented. In the process of explaining these mechanisms, some of the operation of directory control will presented. This section discusses mechanisms mostly internal to directory control; the description of the more external functions of the file system (entry creation, for example) will be found under the description of the various file system primitives. .ifi l1h "Locating and Holding Directories and Directory Entries" Finding a directory involves bringing that directory into the process' .ifi hit "K|directory~locating" address space (if it is not already there) and determining the identity of the directory within the address space (its segment number or pointer). Finding a directory entry involves finding the directory and then finding the entry within the directory. Various mechanisms exist to perform these operations in various ways. It is also necessary to use certain mechanisms to keep the directories and directory entries from disappearing or changing during the time in which they are being manipulated. The official rules state that all finding of directories and directory entries (other than parents of already validated entries) must be performed by the module dc_find, so that it may enforce the .ifi hit "K|dc_find" .ifi hit "S|security policy~see also dc_find" security policy of the system. This sub-section describes the method by which dc_find locates directories and directory entries. This sub-section also describes the method by which dc_find, and callers of dc_find, maintain the validity of the pointers they have to the directories and directory entries. .ifi l2h "Locating Directories" The object of locating a directory involves taking the "name" of a directory and mapping it into the "address" (segment number or pointer) of the directory within the address space (and bringing the directory into the address space, if necessary). The "name" of a directory is either its pathname or its UID pathname (UID of the directory and all superior directories back to the root). The process of finding a directory, given its pathname, in its basic (overly simplified) form, is an iterative process performed by dc_find's .ifi hit "K|pathname to entry translation" internal routine find_dirsegno. The process is to start with the root and then find the various subordinate directories, in turn, until the desired directory is encountered. The root is inherently known. (Address and name space management can initiate the root by simply declaring a segment whose directory entry pointer is null; segment control special cases this to mean the root.) Once the "address" of the root is known, the next subordinate directory (given the next entryname stripped from the pathname) can be found. (The directory entry for the subordinate directory is found as described in the next section. Address and name space management uses this directory entry pointer (and the UID obtained from it) to initiate the subordinate directory.) This process is repeated until the desired directory is found. It is not allowed to bring AIM isolated directories into the address .ifi hit "K|AIM~isolation of directories" space since by so doing the user is informed of their existence. This is not allowed since the names of objects subordinate to AIM isolated directories are also AIM isolated information. find_dirsegno checks directories for AIM isolation and stops (with an audit) when such a directory is encountered. Within the above process, it is possible that a link will be encountered. If this happens, that portion of the pathname already resolved is replaced by .ifi hit "K|link chasing" the link target and the process starts over. (Say that >a>b>c>d is being located and that >a>b>c is a link to >e>f. Then, once >a>b has been located, we will see that the directory entry for c describes it as a link to >e>f. The pathname >e>f>d will be formed, the directories >a and >a>b will no longer be needed, and the process of finding the directory will start again looking for >e>f>d.) Locating a directory given its UID path is similar to the process for .ifi hit "K|UID path to entry translation" locating a directory given its pathname. This is done by the internal routine uid_path_util within dc_find (not to be confused with the file system primitive of the same name). In the UID path case, instead of searching through each directory found for the directory entry describing the next subordinate directory name, a search is made for the directory entry having the UID of the next subordinate directory. There is no need to worry about handling links in the process. .ifi l2h "Locating Directory Entries" The object of locating a directory entry is to take the "name" of a file .ifi hit "K|directory~entry~locating" system object and find the directory entry corresponding to this object. The "name" in this case could be the pathname of the object, the UID path of the object, or a pointer to the object. The process of finding the directory entry given a pathname is basically to find the containing directory (given the directory portion of the pathname) and then to look up the entry describing the entryname portion. Looking up an entry, given a name, is done by find_entry (within dc_find). find_entry uses the directory name hash table (actually, it uses hash$search) to find the entry. find_entry takes a name and a type desired for the object to be found. The type is a four bit string with a bit meaning each of directory, segment, link or nothing. Each one bit means that the caller will allow the found object to be of the type corresponding to that one bit. (A type of nothing means that the caller will allow the object to not exist.) The various uses of the type field will be described under the description of dc_find. find_entry is responsible for checking the basic validity of the directory entry found, and auditing attempts to look up a name of the wrong type. find_entry also makes the check for an entry being security-out-of-service. When finding a directory entry given a pathname, it is possible that the final target is a link. (Links within the directory portion of the pathname .ifi hit "K|link chasing" would have been chased by find_dirsegno.) The chasing of final target links, if desired, is performed by find_ (within dc_find). Chasing links simply involves taking the link target found and repeating the directory entry locating function repeatedly. A counter is kept here, as in all link chasing functions, to keep this search from proceeding indefinitely. Searching for a directory entry given its UID path was mostly described under locating a directory, above. The search for a given entry given a UID must be done by hand (unlike the search given an entryname) since no hash table is maintained in the directory for UIDs. Locating a directory entry given a pointer to the owning object is .ifi hit "K|segment pointer to entry translation" performed by sum (segment utility module). This module is used by dc_find when given the pointer to an object. It is also commonly used throughout directory control, when a program has a pointer to a directory and wishes to find the directory entry describing this directory. sum has two entries of use here, getbranch and getbranch_root_my. From the point of this discussion, the only difference between sum$getbranch and sum$getbranch_root_my is that the later will return a null entry pointer and error_table_$root when the supplied object pointer points to the root whereas the former returns error_table_$noentry (it considers this an error). The directory entry pointer is found from the KST entry for the object; this will be described within the description of sum under directory control primitives. .ifi l2h "Keeping Valid Directory and Entry Pointers" The above descriptions of locating directories and directory entries was overly simplified. In actuality, these various programs (and their callers) must concern themselves with maintaining the validity of the pointers that they generate to the directories and directory entries. In particular, they must make sure that address and name space management does not remove any of these directories from the address space while they are being manipulated. Also, they must make sure that no other process is modifying these directories while they are being searched or in other ways manipulated. .ifi l3h "Keeping Directories from being Terminated" The mechanics of keeping these directories from being removed from the address space (i.e., made unknown) by address and name space management is discussed further under that section. For now, we will consider that there are two rules: a directory will be protected from being made unknown in the .ifi hit "K|usage count" .ifi hit "K|making unknown~directories" address space if it has a non-zero usage count (in the KST entry) or a directory or segment subordinate to it has a non-zero usage count. That is, either the directory must be marked as in use or some subordinate directory must be in use. (It is required that all directories superior to a directory within the address space must also be within the address space.) Having address and name space management bring a directory into the address space automatically increments the usage count by one. Thus, programs need not worry about these directories disappearing within the address space from under them. However, it is necessary to clean up these directories for .ifi hit "K|directory~releasing" fear of filling the address space with useless directories. This is done through segno_usage$decrement. Thus, an expanded view of the process of locating a directory follows. The root is made known (and its usage count incremented as a result). The first level directory's entry is found. Given this entry, the first level directory itself is made known (and its usage count incremented as a result). Now that the root will be "held" by this first level directory, the root's usage count is decremented. The second level directory's entry is found within the first level directory. This second level directory is made known (and its usage count incremented as a result). The usage count of the first level directory is decremented. And so on. When the desired directory is reached, only this last directory will have its usage count incremented. The calling program is guaranteed that this directory will remain in the address space. When the calling program is done with the directory, decrementing the usage count for it is sufficient to free all superior directories (unless, of course, they have non-zero usage counts because of some other operation proceeding at the time). Refer to the usage of dc_find under directory control primitives for more details. .ifi l3h "Keeping Directories from being Modified" Perhaps the most important part of the mechanism to find directory entries is missing from the above: directory locking. When searching a directory for an entry, or when operating upon the entry .ifi hit "K|directory~locks~rules" found (or any other contents of a directory), it is necessary to have the directory locked to prevent modifications to the directory by other processes. Directory locking is described in another section. From the point of view of a given directory control program, the various entry pointers that program keeps will only stay valid (continue to point to the desired entries) as long as the containing directory is locked. As such, a further expansion of the mechanism needed to find a directory is the following. The root is found. It is locked, so that the first level directory entry can be found. Once this directory is known, the root can be unlocked. This first level directory is then locked so that the second level directory's entry can be found and then made known. The first level directory can then be unlocked. And so on. dc_find will return pointers to directories that are locked. The calling programs must unlock them when done. Refer to the usage of dc_find under directory control primitives for more details. .ifi l2h "The Pathname Associative Memory (PAM)" Having to walk down all of the directories from the root every time that .ifi hit "K|pathname to entry translation" .ifi hit "K|PAM" a directory is to be found would be very inefficient. In a normal process, there is a tendency to refer to the same set of directories over and over. For this reason, each process maintains a pathname associative memory (by the program pathname_am) that maps pathnames to directory segment numbers (and vice versa). .ifi l3h "Usage of the PAM" The most common usage of the PAM is in find_dirsegno to optimize the locating of a directory. The PAM based directory finding mechanism is as follows. Start with the desired pathname. See if it is in the PAM (pathname_am$get_segno). If so, its segment number is already known; this can .ifi hit "K|pathname_am" be returned. If not, look to see if the parent directory's pathname is in the PAM. And so on. Walking up the pathname in this way will stop when either the root is encountered (at which time the previously described mechanism comes into play) or a directory is found in the PAM. If a PAM match is found, find_dirsegno can simply walk down the hierarchy from there, instead of from the root. pathname_am$get_segno increments the usage count for the directory found just as if the directory were made known in the usual way. The directory so found will need its usage count decremented when done. For each directory that find_dirsegno finds while walking down the hierarchy from the root or the PAM found directory, find_dirsegno places this pathname/segment number pair into the PAM (pathname_am$set). This will help find_dirsegno out the next time it wants to find this pathname. The other usage of the PAM is by get_pathname_. get_pathname_ takes a .ifi hit "K|get_pathname_" segment number and returns its pathname. This operation is described under address and name space management. It uses the PAM (pathname_am$get_path) as a shortcut to walking up the hierarchy to find the pathname. Also, after it has expended its effort to find a pathname, it sets this into the PAM for later use. .ifi l3h "Operation of the PAM" The pathname associative memory is maintained within PDS by pathname_am. The PAM consists of a threaded list of entries mapping directory pathname to segment number. The list is threaded with the most recently used at the head .ifi hit "K|PAM~operation" of the list. When a new pathname is added to the PAM, the tail entry (least recently used) is deleted and this new pathname added as the head. Performing a match within the PAM (either during a get_path or a get_segno operation) causes the matching entry to be rethreaded to the head of the list. Note that multiple pathnames may refer to the same segment number. When a pathname becomes invalid by virtue of the target's being deleted or made unknown, it is removed from the PAM (clear entrypoint). It is important to note that pathnames in the PAM are not protected from being made unknown by address and name space management (but address and name space management will properly remove such a directory from the PAM when it is made unknown). The PAM pathnames are protected when a get_segno operation succeeds by virtue of the incrementing of the usage count mentioned above. The process of removing the directory from the address space during KST garbage collection will properly clear the pathname from the PAM. The operation of the PAM is vastly complicated by the question of the renaming and deleting of directories (especially by other processes). If a directory is renamed, it is necessary to invalidate all pathnames for that directory, as well as any directory subordinate to that directory within the .ifi hit "K|deletion" .ifi hit "K|renaming" PAM. Since other processes do not wish to walk through the PAM of all other processes when a directory rename is done, a mechanism has been devised to require maintaining the minimum amount of data across processes. This mechanism uses active_hardcore_data$pam_flush_buffer (indexed, circularly, by active_hardcore_data$pam_flush_level) to keep track of the extent to which all processes must flush their PAM of directories affected by some process. The basic operation is for the PAM to keep track of the UID pathnames of the directories within the PAM (via the UID field in the KST entry for the directory), and to use pam_flush_buffer to show what UIDs need flushing. That is, if some process renames the directory with UID N, that process informs other processes to flush all PAM entries for whom the corresponding directory's UID pathname (derived by walking down the KST) contains N. The mechanism to keep track of this single piece of information is very simple, making this cross-process passing of UIDs desirable. The workings of this revolves around ahd$pam_flush_level and the user's pam.flush_level. Before the user updates the PAM, the process flushes its PAM as required by the active_hardcore_data (ahd) information as described below. pam.flush_level is set to ahd$pam_flush_level when a PAM update is finished. After this time, ahd$pam_flush_level is incremented by one each time some process declares a need to flush PAMs in other processes. When the process looks at the PAM next, if pam.flush_level is equal to ahd$pam_flush_level, fine. If not, then the next N (ahd$pam_flush_level - pam.flush_level) slots in the circular ahd$pam_flush_buffer queue contain UIDs of directories to flush from the PAM. If N is greater than the size of ahd$pam_flush_buffer, it follows that the process lost track of what UIDs to flush, and must therefore flush all of them. Also, when setting a value into ahd$pam_flush_buffer, if the process notices that other setters of ahd$pam_flush_buffer caught up with it (wrapped around this circular buffer), the process forces the last slot of ahd$pam_flush_buffer to 777777777777 (the root) to force everyone to flush all of their PAM's the next time around. This examining and setting of ahd$pam_flush_level is done with appropriate hardware locking instructions. It was deemed undesirable to maintain the UID pathname for each PAM entry within the PAM (for size reasons); it is also undesirable to compute this for each PAM entry each time there is a UID to flush. So, an optimization is used. If the UID to flush does not correspond to a directory within this process (as determined via the UID hash kept in the KST), which is most likely to be the case, the UID does not need to be flushed. Only if the UID (from ahd$pam_flush_buffer) exists within the process is it necessary to look for PAM entries that correspond. .ifi l2h "Summary" File system programs must take certain precautions to keep the directory and directory entry pointers they possess valid. dc_find performs the necessary functions to make the returned pointers stay valid. It is necessary, though, for the calling programs to clean this up. If the directory control program (that which calls dc_find) has a pointer to a directory entry, the directory pointer itself is found (ptr (ep, 0)). Given this pointer to the directory, the directory control program must unlock the directory. If the directory was found given a pointer to the object, this is all that is necessary. (In such cases, sum$getbranch was called to find the directory. The directory's usage count was not incremented since it is being "held" by the inferior object.) Otherwise, the directory must be dereferenced (its usage count decremented). The functions of unlocking and dereferencing are done by calling dc_find$finished. This entry unlocks the directory and dereferences it, on the basis of an argument supplied to it. .ifi l1h "Re-locating Directories and Directory Entries" It is sometimes not possible to keep a directory locked throughout the series of events relating to the directory. For instance, address and name space management maintains the directory entry pointer for each object in the address space so that segment control can activate and deactivate them. These directories clearly can not be kept locked during the run of the process. Also, even during relatively short directory control sequences, it is necessary to unlock directories to keep from violating some other system locking rule (an example is given later in this section). Thus, a mechanism must exist so that a directory can be unlocked so that it can easily be re-locked and the process' assumptions re-validated about it. .ifi l2h "Re-validating Directory Entry Pointers" When an object is brought into the address space, address and name space management must place the directory entry pointer for the object in its KST entry. (This information is needed by seg_fault at activation time.) If the directory containing this entry is to be unlocked, the next time around this .ifi hit "K|directory salvager~compactor" directory entry pointer may be invalid, due to salvaging (compaction) of the directory. The way to re-validate this entry pointer (or to make it valid), .ifi hit "K|sum" .ifi hit "K|validate_entryp" and, indeed, the method used by sum, is to call validate_entryp. validate_entryp performs a set of checks given the entry pointer (with the .ifi hit "K|directory~locks~relocking" directory locked, of course) to see if this pointer still indicates the entry in question. If these checks fail, a search of the directory for the entry having the UID from the KST entry is done. If this fails, the segment must have been deleted and an error is returned. Refer to the translation of segment pointers to directory entries for more details. .ifi l2h "Validating Directory Contents" If a directory must be unlocked between two operations, it is necessary to determine if the contents of the directory had changed. This is done as follows. The program inspects dir.change_pclock. This value is incremented by one each time a process modifies the contents of the directory. If this value hasn't changed since it was last locked, it wasn't modified. .ifi l2h "Return Area Management" The normal sequence of events when returning information about a file .ifi hit "K|area return mechanism" system object to the user ring is the following. The directory is found, locked and the directory entry found. The data is copied into the a temporary data space. The directory is unlocked. The data is then copied into the user's data space, taking whatever faults may arise. When the user's data is being copied into a directory, it is copied before finding and locking the directory, to take any potential faults at this time. It is undesirable for faults to occur with directories locked. A special mechanism is used, though, when the amount of data to be copied out is large, in particular, when it is too large (or variable) to be copied into a ring 0 temporary area. Examples of this are when returning the ACL of an object, or the names within a directory. These operations must copy their data into the user ring within the loop processing the directory. They must be careful since a fault could occur during this copying. Worse yet is that they need to allocate the space (normally) for these return values. This creates a special problem. A directory control operation of this type will walk down the directory first to determine how much data must be returned. The allocation of this data, though, can not occur with the directory locked since this allocation may extend an extensible area which would not be possible if the area were immediately subordinate to the directory. So, the directory must be unlocked, the area in the user ring allocated, and the directory relocked. After this, the change_pclock comparison described above determines if the counts possessed of the data to return has been invalidated. [It is not necessary to recalculate access to the directory at this time .ifi hit "K|access modes~recomputation" to determine if the operation is still allowed. This is because the process is guaranteed that the directory didn't change if the pclock test succeeds. (If it failed, it must recalculate access.) Since the directory didn't change, even if some other process did change the process' access to the directory, the information that is about to be returned is the same that it just saw, and is data it did have access to and could have just as well copied out before the access was changed.] The relocking of the directory is done with a seg_fault_error handler. (Locking the directory references the directory header.) This is done because the directory may be deleted with the directory unlocked. (Directories are prevented from deletion while locked.) .ifi l2h "Summary" If a directory must be unlocked, it is necessary to revalidate any assumptions about it when it is relocked. The main question is one of access. Does this process still have access to the object? When the operation being performed modifies the object, it must use care. The possibility exists that some other process may delete this process' access to the directory, and that other process would be surprised to see the directory's contents modified after the access was deleted. However, if the operation to be performed merely returns information, an optimization can be used. If the directory's contents can be shown to have not changed since the access check was made (by dc_find), then, even if some other process deleted this process' access to the directory, this does not change what this process' is allowed to see at the time of the access check. Since it could just as well have returned the information at that time, it might as well still return it now. If the process was holding a pointer to a directory, then relocking the directory must ensure that the process still has access to the directory and that any assumptions based on its contents have not changed. This is done by checking dir.change_pclock. If dir.change_pclock hasn't changed, neither has the directory and so no loss of access matters. A change of dir.change_pclock requires that dc_find be rerun. Note that the directory can be deleted while it is unlocked. If the process was holding a pointer to a directory entry, then relocking the parent must refind the entry pointer (validate_entryp), recheck access on the parent and make sure no assumptions about the entry have changed. This is normally done by checking dir.change_pclock. No change implies no change to the entry and so any access loss does not matter. The dtem field in the entry can also be used to simply ensure that the process still has access to the entry, since dtem is guaranteed to be advanced by at least one for any access change. Checking entry.dtem, however, is not sufficient when the operation to be performed modifies the entry since a change of access to the parent of this entry by some other process would not have changed the dtem of this entry. If the process was holding a pointer to a segment, then relocking the parent must ensure that the segment held is still the segment desired, that access still exists on the parent, and that no assumptions about the segment have changed. Any subsequent reference to the segment will validate that it still exists and that its UID matches what was desired (via seg_fault). entry.dtem or dir.change_pclock can be examined to validate the other conditions. (Again, entry.dtem is not a sufficient check if the segment is to be modified.) If it is necessary to rerun dc_find, it is necessary to make sure that the object found is the same as the one for which any assumptions had been made. (If no assumptions are maintained across the relocking, this can be ignored.) It is possible that two names were swapped in the parent and so the pointer dc_find returns to the object named "foo" points to a different object than was "foo" the last time around. It is easy to tell if this "foo" is the same as the previous "foo"; simply check its UID. A check of entry.dtem is not necessary since dc_find would have revalidated access unless some assumptions about "foo" could have been affected during the unlocking. .ifi l1h "Modifying Directories" Whenever a directory control program modifies a directory, several operations must be performed. .ifi l2h "Signalling that a Modification is in Progress" When a process starts a modification, the process must lock the directory for writing. Also, the process id of that process must be recorded in .ifi hit "K|directory~modification in progress" dir.modify. In this way, crawlouts will detect that the contents of the directory are in question and that a directory salvage is to be performed. When the modification is done (and notified, as explained below), the dir.modify field is reset to zero and the directory unlocked. .ifi l2h "Recording Access and Attribute Changes - dtem" If the modification being performed to a directory entry involves potentially changing the access some process has to the object, this must be reflected to those processes. First, setfaults must be called to force all processes to recompute their access to the object. Secondly, change_dtem must be called. change_dtem will change the date-time entry modified field for this entry .ifi hit "K|change_dtem" .ifi hit "K|access modes~recomputation" to the current time. It does this in a way so as to guarantee that the dtem is incremented by at least one (dtem is only accurate to 1/16 sec.). It also keeps dtem from getting too far in the future. Since change_dtem must sometimes wait for time to go by so as to properly set the dtem, change_dtem must be called judiciously. The dtem must be modified, though, when access has been changed. Calling change_dtem with respect to a directory is the equivalent of performing a setfaults on the directory. This is explained in the section on access control. When an attribute of an entry is changed, change_dtem is still called. In this case, though, the dtem is not critical; it is just for user's information. As such, change_dtem is not called if the dtem matches the current time. .ifi l2h "Recording Directory Contents Modified - dtcm" The date-time contents modified (dtcm) field for a segment (which is maintained in the VTOCE/ASTE) is exactly that. It is maintained with extensive mechanism described in the Storage System PLM. The dtcm field for a directory has a different meaning. The dtcm for a directory is advanced to the current time if a branch anywhere subordinate to the directory is modified or if directory control explicitly declares the directory as modified. When directory control .ifi hit "K|directory~modification" operations such as the salvager are running, it is desirable to not let the directory be flagged as modified until the salvaging is done. For this reason, the dtcm of directories is maintained in an unusual way. Address and name space management flags all directories with the gtms (global transparent modified switch) on. This causes page control to not notice modifications of pages within directories, and, therefore, to not set the fms (file modified switch) or the dtcm. The dtcm must be updated manually. When a directory update is done, sum$dirmod must be called. sum$dirmod finds the ASTE (activating the directory if necessary and locking the AST). The gtms switch is turned off so that pc$updates can be called. This causes the fms switch to be set for this directory and all superiors, as well as updating the dtcm. The gtms switch is then turned off. This is all done under the directory lock, so that no one else will be in a modification sequence. .brp ----------------------------------------------------------- Historical Background This edition of the Multics software materials and documentation is provided and donated to Massachusetts Institute of Technology by Group BULL including BULL HN Information Systems Inc. as a contribution to computer science knowledge. This donation is made also to give evidence of the common contributions of Massachusetts Institute of Technology, Bell Laboratories, General Electric, Honeywell Information Systems Inc., Honeywell BULL Inc., Groupe BULL and BULL HN Information Systems Inc. to the development of this operating system. Multics development was initiated by Massachusetts Institute of Technology Project MAC (1963-1970), renamed the MIT Laboratory for Computer Science and Artificial Intelligence in the mid 1970s, under the leadership of Professor Fernando Jose Corbato. Users consider that Multics provided the best software architecture for managing computer hardware properly and for executing programs. Many subsequent operating systems incorporated Multics principles. Multics was distributed in 1975 to 2000 by Group Bull in Europe , and in the U.S. by Bull HN Information Systems Inc., as successor in interest by change in name only to Honeywell Bull Inc. and Honeywell Information Systems Inc. . ----------------------------------------------------------- Permission to use, copy, modify, and distribute these programs and their documentation for any purpose and without fee is hereby granted,provided that the below copyright notice and historical background appear in all copies and that both the copyright notice and historical background and this permission notice appear in supporting documentation, and that the names of MIT, HIS, BULL or BULL HN not be used in advertising or publicity pertaining to distribution of the programs without specific prior written permission. Copyright 1972 by Massachusetts Institute of Technology and Honeywell Information Systems Inc. Copyright 2006 by BULL HN Information Systems Inc. Copyright 2006 by Bull SAS All Rights Reserved