org.apache.hadoop.util
Class CopyFiles.FSCopyFilesMapper

java.lang.Object
  org.apache.hadoop.mapred.MapReduceBase
      org.apache.hadoop.util.CopyFiles.CopyFilesMapper
          org.apache.hadoop.util.CopyFiles.FSCopyFilesMapper

All Implemented Interfaces:

Closeable, JobConfigurable, Mapper

Enclosing class:

CopyFiles

public static class CopyFiles.FSCopyFilesMapper
extends CopyFiles.CopyFilesMapper
implements Mapper

DFSCopyFilesMapper: The mapper for copying files from the DFS.

Author:

Milind Bhandarkar

Constructor Summary

CopyFiles.FSCopyFilesMapper()

Method Summary

void	cleanup(Configuration conf, JobConf jobConf, String srcPath, String destPath) Interface to cleanup *distcp* specific resources
void	close() Default implementation that does nothing.
void	configure(JobConf job) Mapper configuration.
void	map(WritableComparable key, Writable value, OutputCollector out, Reporter reporter) Map method.
void	setup(Configuration conf, JobConf jobConf, String[] srcPaths, String destPath, boolean ignoreReadFailures) Initialize DFSCopyFileMapper specific job-configuration.

Methods inherited from class org.apache.hadoop.util.CopyFiles.CopyFilesMapper

getMapCount, makeRelative

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CopyFiles.FSCopyFilesMapper

public CopyFiles.FSCopyFilesMapper()

Method Detail

setup

public void setup(Configuration conf,
                  JobConf jobConf,
                  String[] srcPaths,
                  String destPath,
                  boolean ignoreReadFailures)
           throws IOException

Initialize DFSCopyFileMapper specific job-configuration.

Specified by:

setup in class CopyFiles.CopyFilesMapper

Parameters:

conf - : The dfs/mapred configuration.

jobConf - : The handle to the jobConf object to be initialized.

srcPaths - : The source URIs.

destPath - : The destination URI.

ignoreReadFailures - : Ignore read failures?

Throws:

IOException

cleanup

public void cleanup(Configuration conf,
                    JobConf jobConf,
                    String srcPath,
                    String destPath)
             throws IOException

Description copied from class: CopyFiles.CopyFilesMapper

Interface to cleanup *distcp* specific resources

Specified by:

cleanup in class CopyFiles.CopyFilesMapper

Parameters:

conf - : The dfs/mapred configuration.

jobConf - : The handle to the jobConf object to be initialized.

srcPath - : The source uri.

destPath - : The destination uri.

Throws:

IOException

configure

public void configure(JobConf job)

Mapper configuration. Extracts source and destination file system, as well as top-level paths on source and destination directories. Gets the named file systems, to be used later in map.

Specified by:

configure in interface JobConfigurable

Overrides:

configure in class MapReduceBase

Parameters:

job - the configuration

map

public void map(WritableComparable key,
                Writable value,
                OutputCollector out,
                Reporter reporter)
         throws IOException

Map method. Copies one file from source file system to destination.

Specified by:

map in interface Mapper

Parameters:

key - source file name

value - not-used.

out - not-used.

reporter -

Throws:

IOException

close

public void close()

Description copied from class: MapReduceBase

Default implementation that does nothing.

Specified by:

close in interface Closeable

Overrides:

close in class MapReduceBase