org.apache.hadoop.mapred
Class PhasedFileSystem

java.lang.Object
  org.apache.hadoop.conf.Configured
      org.apache.hadoop.fs.FileSystem
          org.apache.hadoop.fs.FilterFileSystem
              org.apache.hadoop.mapred.PhasedFileSystem

All Implemented Interfaces:

Configurable

Deprecated. PhasedFileSystem is no longer used during speculative execution of tasks.

public class PhasedFileSystem
extends FilterFileSystem

This class acts as a proxy to the actual file system being used. It writes files to a temporary location and on commit, moves to final location. On abort or a failure in commit the temporary file is deleted PhasedFileSystem works in context of a task. A different instance of PhasedFileSystem should be used for every task. Temporary files are written in ("mapred.system.dir")// If one tasks opens a large number of files in succession then its better to commit(Path) individual files when done. Otherwise commit() can be used to commit all open files at once.

Field Summary

Fields inherited from class org.apache.hadoop.fs.FilterFileSystem

fs

Fields inherited from class org.apache.hadoop.fs.FileSystem

LOG

Constructor Summary

PhasedFileSystem(FileSystem fs, JobConf conf)
Deprecated. This Constructor is used to wrap a FileSystem object to a Phased FilsSystem.

PhasedFileSystem(FileSystem fs, String jobid, String tipid, String taskid)
Deprecated. This Constructor is used to wrap a FileSystem object to a Phased FilsSystem.

Method Summary

void	abort() Deprecated. Aborts the file creation, all uncommitted files created by this PhasedFileSystem instance are deleted.
void	abort(Path p) Deprecated. Aborts a single file.
void	commit() Deprecated. Commits files to their final locations as passed in create* methods.
void	commit(Path fPath) Deprecated. Commits a single file file to its final locations as passed in create* methods.
void	completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) Deprecated. Called when we're all done writing to the target.
void	copyFromLocalFile(boolean delSrc, Path src, Path dst) Deprecated. The src file is on the local disk.
void	copyToLocalFile(boolean delSrc, Path src, Path dst) Deprecated. The src file is under FS, and the dst is on the local disk.
FSDataOutputStream	create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) Deprecated. Opens an FSDataOutputStream at the indicated Path with write-progress reporting.
boolean	delete(Path f) Deprecated. Delete a file
String[][]	getFileCacheHints(Path f, long start, long len) Deprecated. Return a 2D array of size 1x1 or greater, containing hostnames where portions of the given file can be found.
String	getName() Deprecated.
void	lock(Path f, boolean shared) Deprecated.
void	release(Path f) Deprecated.
boolean	rename(Path src, Path dst) Deprecated. Renames Path src to Path dst.
boolean	setReplication(Path src, short replication) Deprecated. Set replication for an existing file.
Path	startLocalOutput(Path fsOutputFile, Path tmpLocalFile) Deprecated. Returns a local File that the user can write output to.

Methods inherited from class org.apache.hadoop.fs.FilterFileSystem

checkPath, close, exists, getBlockSize, getConf, getDefaultBlockSize, getDefaultReplication, getLength, getReplication, getUri, getWorkingDirectory, initialize, isDirectory, listPaths, makeQualified, mkdirs, open, setWorkingDirectory

Methods inherited from class org.apache.hadoop.fs.FileSystem

closeAll, copyFromLocalFile, copyToLocalFile, create, create, create, create, create, create, create, createNewFile, get, get, getContentLength, getLocal, getNamed, getUsed, globPaths, globPaths, isFile, listPaths, listPaths, listPaths, moveFromLocalFile, moveToLocalFile, open, parseArgs

Methods inherited from class org.apache.hadoop.conf.Configured

setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PhasedFileSystem

public PhasedFileSystem(FileSystem fs,
                        String jobid,
                        String tipid,
                        String taskid)

Deprecated.

This Constructor is used to wrap a FileSystem object to a Phased FilsSystem.

Parameters:

fs - base file system object

jobid - JobId

tipid - tipId

taskid - taskId

PhasedFileSystem

public PhasedFileSystem(FileSystem fs,
                        JobConf conf)

Deprecated.

This Constructor is used to wrap a FileSystem object to a Phased FilsSystem.

Parameters:

fs - base file system object

conf - JobConf

Method Detail

create

public FSDataOutputStream create(Path f,
                                 boolean overwrite,
                                 int bufferSize,
                                 short replication,
                                 long blockSize,
                                 Progressable progress)
                          throws IOException

Deprecated.

Description copied from class: FilterFileSystem

Opens an FSDataOutputStream at the indicated Path with write-progress reporting.

Overrides:

create in class FilterFileSystem

Parameters:

f - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

bufferSize - the size of the buffer to be used.

replication - required block replication for the file.

Throws:

IOException

commit

public void commit(Path fPath)
            throws IOException

Deprecated.

Commits a single file file to its final locations as passed in create* methods. If a file already exists in final location then temporary file is deleted.

Parameters:

fPath - path to final file.

Throws:

IOException - thrown if commit fails

commit

public void commit()
            throws IOException

Deprecated.

Commits files to their final locations as passed in create* methods. If a file already exists in final location then temporary file is deleted. This methods ignores crc files (ending with .crc). This method doesnt close the file system so it can still be used to create new files.

Throws:

IOException - if any file fails to commit

abort

public void abort(Path p)
           throws IOException

Deprecated.

Aborts a single file. The temporary created file is deleted.

Parameters:

p - the path to final file as passed in create* call

Throws:

IOException - if File delete fails

abort

public void abort()
           throws IOException

Deprecated.

Aborts the file creation, all uncommitted files created by this PhasedFileSystem instance are deleted. This does not close baseFS because handle to baseFS may still exist can be used to create new files.

Throws:

IOException

setReplication

public boolean setReplication(Path src,
                              short replication)
                       throws IOException

Deprecated.

Description copied from class: FilterFileSystem

Set replication for an existing file.

Overrides:

setReplication in class FilterFileSystem

Parameters:

src - file name

replication - new replication

Returns:

true if successful; false if file does not exist or is a directory

Throws:

IOException

rename

public boolean rename(Path src,
                      Path dst)
               throws IOException

Deprecated.

Description copied from class: FilterFileSystem

Renames Path src to Path dst. Can take place on local fs or remote DFS.

Overrides:

rename in class FilterFileSystem

Throws:

IOException

delete

public boolean delete(Path f)
               throws IOException

Deprecated.

Description copied from class: FilterFileSystem

Delete a file

Overrides:

delete in class FilterFileSystem

Throws:

IOException

lock

@Deprecated
public void lock(Path f,
                            boolean shared)
          throws IOException

Deprecated.

Description copied from class: FilterFileSystem

Obtain a lock on the given Path

Overrides:

lock in class FilterFileSystem

Throws:

IOException

release

@Deprecated
public void release(Path f)
             throws IOException

Deprecated.

Description copied from class: FilterFileSystem

Release the lock

Overrides:

release in class FilterFileSystem

Throws:

IOException

copyFromLocalFile

public void copyFromLocalFile(boolean delSrc,
                              Path src,
                              Path dst)
                       throws IOException

Deprecated.

Description copied from class: FilterFileSystem

The src file is on the local disk. Add it to FS at the given dst name. delSrc indicates if the source should be removed

Overrides:

copyFromLocalFile in class FilterFileSystem

Throws:

IOException

copyToLocalFile

public void copyToLocalFile(boolean delSrc,
                            Path src,
                            Path dst)
                     throws IOException

Deprecated.

Description copied from class: FilterFileSystem

The src file is under FS, and the dst is on the local disk. Copy it from FS control to the local dst name. delSrc indicates if the src will be removed or not.

Overrides:

copyToLocalFile in class FilterFileSystem

Throws:

IOException

startLocalOutput

public Path startLocalOutput(Path fsOutputFile,
                             Path tmpLocalFile)
                      throws IOException

Deprecated.

Description copied from class: FilterFileSystem

Returns a local File that the user can write output to. The caller provides both the eventual FS target name and the local working file. If the FS is local, we write directly into the target. If the FS is remote, we write into the tmp local area.

Overrides:

startLocalOutput in class FilterFileSystem

Throws:

IOException

completeLocalOutput

public void completeLocalOutput(Path fsOutputFile,
                                Path tmpLocalFile)
                         throws IOException

Deprecated.

Description copied from class: FilterFileSystem

Called when we're all done writing to the target. A local FS will do nothing, because we've written to exactly the right place. A remote FS will copy the contents of tmpLocalFile to the correct target at fsOutputFile.

Overrides:

completeLocalOutput in class FilterFileSystem

Throws:

IOException

getFileCacheHints

public String[][] getFileCacheHints(Path f,
                                    long start,
                                    long len)
                             throws IOException

Deprecated.

Description copied from class: FilterFileSystem

Return a 2D array of size 1x1 or greater, containing hostnames where portions of the given file can be found. For a nonexistent file or regions, null will be returned. This call is most helpful with DFS, where it returns hostnames of machines that contain the given file. The FileSystem will simply return an elt containing 'localhost'.

Overrides:

getFileCacheHints in class FilterFileSystem

Throws:

IOException

getName

public String getName()

Deprecated.

Overrides:

getName in class FilterFileSystem