org.apache.hadoop.fs
Class InMemoryFileSystem

java.lang.Object
  org.apache.hadoop.conf.Configured
      org.apache.hadoop.fs.FileSystem
          org.apache.hadoop.fs.FilterFileSystem
              org.apache.hadoop.fs.ChecksumFileSystem
                  org.apache.hadoop.fs.InMemoryFileSystem

All Implemented Interfaces:

Configurable

public class InMemoryFileSystem
extends ChecksumFileSystem

An implementation of the in-memory filesystem. This implementation assumes that the file lengths are known ahead of time and the total lengths of all the files is below a certain number (like 100 MB, configurable). Use the API reserveSpaceWithCheckSum(Path f, int size) (see below for a description of the API for reserving space in the FS. The uri of this filesystem starts with ramfs:// .

Author:

ddas

Field Summary

Fields inherited from class org.apache.hadoop.fs.FilterFileSystem

fs

Fields inherited from class org.apache.hadoop.fs.FileSystem

LOG

Constructor Summary

InMemoryFileSystem()

InMemoryFileSystem(URI uri, Configuration conf)

Method Summary

void	completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) Called when we're all done writing to the target.
void	copyFromLocalFile(boolean delSrc, Path src, Path dst) copy/move operations are not supported
void	copyToLocalFile(boolean delSrc, Path src, Path dst) The src file is under FS, and the dst is on the local disk.
Path[]	getFiles(PathFilter filter)
long	getFSSize()
int	getNumFiles(PathFilter filter)
float	getPercentUsed()
boolean	reserveSpaceWithCheckSum(Path f, long size) Register a file with its size.
Path	startLocalOutput(Path fsOutputFile, Path tmpLocalFile) Returns a local File that the user can write output to.

Methods inherited from class org.apache.hadoop.fs.ChecksumFileSystem

copyToLocalFile, create, delete, getApproxChkSumLength, getBytesPerSum, getChecksumFile, getChecksumFileLength, getRawFileSystem, isChecksumFile, listPaths, listPaths, lock, mkdirs, open, release, rename, reportChecksumFailure, setReplication

Methods inherited from class org.apache.hadoop.fs.FilterFileSystem

checkPath, close, exists, getBlockSize, getConf, getDefaultBlockSize, getDefaultReplication, getFileCacheHints, getLength, getName, getReplication, getUri, getWorkingDirectory, initialize, isDirectory, makeQualified, setWorkingDirectory

Methods inherited from class org.apache.hadoop.fs.FileSystem

closeAll, copyFromLocalFile, copyToLocalFile, create, create, create, create, create, create, create, createNewFile, get, get, getContentLength, getLocal, getNamed, getUsed, globPaths, globPaths, isFile, listPaths, listPaths, moveFromLocalFile, moveToLocalFile, open, parseArgs

Methods inherited from class org.apache.hadoop.conf.Configured

setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InMemoryFileSystem

public InMemoryFileSystem()

InMemoryFileSystem

public InMemoryFileSystem(URI uri,
                          Configuration conf)

Method Detail

copyFromLocalFile

public void copyFromLocalFile(boolean delSrc,
                              Path src,
                              Path dst)
                       throws IOException

copy/move operations are not supported

Overrides:

copyFromLocalFile in class ChecksumFileSystem

Throws:

IOException

copyToLocalFile

public void copyToLocalFile(boolean delSrc,
                            Path src,
                            Path dst)
                     throws IOException

Description copied from class: ChecksumFileSystem

The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name.

Overrides:

copyToLocalFile in class ChecksumFileSystem

Throws:

IOException

startLocalOutput

public Path startLocalOutput(Path fsOutputFile,
                             Path tmpLocalFile)
                      throws IOException

Description copied from class: FilterFileSystem

Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.

Overrides:

startLocalOutput in class ChecksumFileSystem

Throws:

IOException

completeLocalOutput

public void completeLocalOutput(Path fsOutputFile,
                                Path tmpLocalFile)
                         throws IOException

Description copied from class: FilterFileSystem

Called when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.

Overrides:

completeLocalOutput in class ChecksumFileSystem

Throws:

IOException

reserveSpaceWithCheckSum

public boolean reserveSpaceWithCheckSum(Path f,
                                        long size)

Register a file with its size. This will also register a checksum for the file that the user is trying to create. This is required since none of the FileSystem APIs accept the size of the file as argument. But since it is required for us to apriori know the size of the file we are going to create, the user must call this method for each file he wants to create and reserve memory for that file. We either succeed in reserving memory for both the main file and the checksum file and return true, or return false.

getFiles

public Path[] getFiles(PathFilter filter)

getNumFiles

public int getNumFiles(PathFilter filter)

getFSSize

public long getFSSize()

getPercentUsed

public float getPercentUsed()