org.apache.hadoop.util
Class CopyFiles

java.lang.Object
  org.apache.hadoop.util.ToolBase
      org.apache.hadoop.util.CopyFiles

All Implemented Interfaces:

Configurable, Tool

public class CopyFiles
extends ToolBase

A Map-reduce program to recursively copy directories between different file-systems.

Author:

Milind Bhandarkar

Nested Class Summary

static class	CopyFiles.CopyFilesMapper Base-class for all mappers for distcp
static class	CopyFiles.FSCopyFilesMapper DFSCopyFilesMapper: The mapper for copying files from the DFS.
static class	CopyFiles.HTTPCopyFilesMapper

Field Summary

Fields inherited from class org.apache.hadoop.util.ToolBase

conf

Constructor Summary

CopyFiles()

Method Summary

static void	copy(Configuration conf, String srcPath, String destPath, boolean srcAsList, boolean ignoreReadFailures) Driver to copy srcPath to destPath depending on required protocol.
static void	main(String[] args)
int	run(String[] args) This is the main driver for recursively copying directories across file systems.
void	setConf(Configuration conf) Set the configuration to be used by this object.
static URI	toURI(String u)

Methods inherited from class org.apache.hadoop.util.ToolBase

doMain, getConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CopyFiles

public CopyFiles()

Method Detail

setConf

public void setConf(Configuration conf)

Description copied from interface: Configurable

Set the configuration to be used by this object.

Specified by:

setConf in interface Configurable

Overrides:

setConf in class ToolBase

toURI

public static URI toURI(String u)
                 throws IOException

Throws:

IOException

copy

public static void copy(Configuration conf,
                        String srcPath,
                        String destPath,
                        boolean srcAsList,
                        boolean ignoreReadFailures)
                 throws IOException

Driver to copy srcPath to destPath depending on required protocol.

Parameters:

conf - Configuration

srcPath - Source path URL

destPath - Destination path URL

srcAsList - List of source URLs to copy.

ignoreReadFailures - True if we are to ignore read failures.

Throws:

IOException

run

public int run(String[] args)
        throws Exception

This is the main driver for recursively copying directories across file systems. It takes at least two cmdline parameters. A source URL and a destination URL. It then essentially does an "ls -lR" on the source URL, and writes the output in a round-robin manner to all the map input files. The mapper actually copies the files allotted to it. The reduce is empty.

Parameters:

args - command specific arguments

Returns:

exit code

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception