org.apache.hadoop.mapred
Class JobClient

java.lang.Object
  org.apache.hadoop.util.ToolBase
      org.apache.hadoop.mapred.JobClient

All Implemented Interfaces:

Configurable, Tool

public class JobClient
extends ToolBase

JobClient interacts with the JobTracker network interface. This object implements the job-control interface, and should be the primary method by which user programs interact with the networked job system.

Author:

Mike Cafarella

Nested Class Summary

static class

JobClient.TaskStatusFilter

Field Summary

static int	FILE_NOT_FOUND
static long	HEARTBEAT_INTERVAL
static String	MAP_OUTPUT_LENGTH The custom http header used for the map output length.
static float	MAX_INMEM_FILESIZE_FRACTION Constant denoting the max size (in terms of the fraction of the total size of the filesys) of a map output file that we will try to keep in mem.
static float	MAX_INMEM_FILESYS_USE Constant denoting when a merge of in memory files will be triggered
static int	SUCCESS

Fields inherited from class org.apache.hadoop.util.ToolBase

conf

Constructor Summary

JobClient()
Build a job client, connect to the default job tracker

JobClient(Configuration conf)

JobClient(InetSocketAddress jobTrackAddr, Configuration conf)
Build a job client, connect to the indicated job tracker.

Method Summary

void	close()
ClusterStatus	getClusterStatus()
FileSystem	getFs() Get a filesystem handle.
RunningJob	getJob(String jobid) Get an RunningJob object to track an ongoing job.
TaskReport[]	getMapTaskReports(String jobId) Get the information of the current state of the map tasks of a job.
TaskReport[]	getReduceTaskReports(String jobId) Get the information of the current state of the reduce tasks of a job.
JobClient.TaskStatusFilter	getTaskOutputFilter() Deprecated.
static JobClient.TaskStatusFilter	getTaskOutputFilter(JobConf job) Get the task output filter out of the JobConf
void	init()
JobStatus[]	jobsToComplete()
static void	main(String[] argv)
int	run(String[] argv) execute the command with the given arguments
static RunningJob	runJob(JobConf job) Utility that submits a job, then polls for progress until the job is complete.
void	setTaskOutputFilter(JobClient.TaskStatusFilter newValue) Deprecated.
static void	setTaskOutputFilter(JobConf job, JobClient.TaskStatusFilter newValue) Modify the JobConf to set the task output filter
RunningJob	submitJob(JobConf job) Submit a job to the MR system
RunningJob	submitJob(String jobFile) Submit a job to the MR system

Methods inherited from class org.apache.hadoop.util.ToolBase

doMain, getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HEARTBEAT_INTERVAL

public static final long HEARTBEAT_INTERVAL

See Also:

Constant Field Values

MAX_INMEM_FILESYS_USE

public static final float MAX_INMEM_FILESYS_USE

Constant denoting when a merge of in memory files will be triggered

See Also:

Constant Field Values

MAX_INMEM_FILESIZE_FRACTION

public static final float MAX_INMEM_FILESIZE_FRACTION

Constant denoting the max size (in terms of the fraction of the total size of the filesys) of a map output file that we will try to keep in mem. Ideally, this should be a factor of MAX_INMEM_FILESYS_USE

See Also:

Constant Field Values

SUCCESS

public static final int SUCCESS

See Also:

Constant Field Values

FILE_NOT_FOUND

public static final int FILE_NOT_FOUND

See Also:

Constant Field Values

MAP_OUTPUT_LENGTH

public static final String MAP_OUTPUT_LENGTH

The custom http header used for the map output length.

See Also:

Constant Field Values

Constructor Detail

JobClient

public JobClient()

Build a job client, connect to the default job tracker

JobClient

public JobClient(Configuration conf)
          throws IOException

Throws:

IOException

JobClient

public JobClient(InetSocketAddress jobTrackAddr,
                 Configuration conf)
          throws IOException

Build a job client, connect to the indicated job tracker.

Throws:

IOException

Method Detail

init

public void init()
          throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Throws:

IOException

getFs

public FileSystem getFs()
                 throws IOException

Get a filesystem handle. We need this to prepare jobs for submission to the MapReduce system.

Throws:

IOException

submitJob

public RunningJob submitJob(String jobFile)
                     throws FileNotFoundException,
                            InvalidJobConfException,
                            IOException

Submit a job to the MR system

Throws:

FileNotFoundException

InvalidJobConfException

IOException

submitJob

public RunningJob submitJob(JobConf job)
                     throws FileNotFoundException,
                            InvalidJobConfException,
                            IOException

Submit a job to the MR system

Throws:

FileNotFoundException

InvalidJobConfException

IOException

getJob

public RunningJob getJob(String jobid)
                  throws IOException

Get an RunningJob object to track an ongoing job. Returns null if the id does not correspond to any known job.

Throws:

IOException

getMapTaskReports

public TaskReport[] getMapTaskReports(String jobId)
                               throws IOException

Get the information of the current state of the map tasks of a job.

Parameters:

jobId - the job to query

Returns:

the list of all of the map tips

Throws:

IOException

getReduceTaskReports

public TaskReport[] getReduceTaskReports(String jobId)
                                  throws IOException

Get the information of the current state of the reduce tasks of a job.

Parameters:

jobId - the job to query

Returns:

the list of all of the map tips

Throws:

IOException

getClusterStatus

public ClusterStatus getClusterStatus()
                               throws IOException

Throws:

IOException

jobsToComplete

public JobStatus[] jobsToComplete()
                           throws IOException

Throws:

IOException

runJob

public static RunningJob runJob(JobConf job)
                         throws IOException

Utility that submits a job, then polls for progress until the job is complete.

Throws:

IOException

setTaskOutputFilter

@Deprecated
public void setTaskOutputFilter(JobClient.TaskStatusFilter newValue)

Deprecated.

Sets the output filter for tasks. only those tasks are printed whose output matches the filter.

Parameters:

newValue - task filter.

getTaskOutputFilter

public static JobClient.TaskStatusFilter getTaskOutputFilter(JobConf job)

Get the task output filter out of the JobConf

Parameters:

job - the JobConf to examine

Returns:

the filter level

setTaskOutputFilter

public static void setTaskOutputFilter(JobConf job,
                                       JobClient.TaskStatusFilter newValue)

Modify the JobConf to set the task output filter

Parameters:

job - the JobConf to modify

newValue - the value to set

getTaskOutputFilter

@Deprecated
public JobClient.TaskStatusFilter getTaskOutputFilter()

Deprecated.

Returns task output filter.

Returns:

task filter.

run

public int run(String[] argv)
        throws Exception

Description copied from interface: Tool

execute the command with the given arguments

Specified by:

run in interface Tool

Parameters:

argv - command specific arguments

Returns:

exit code

Throws:

Exception

main

public static void main(String[] argv)
                 throws Exception

Throws:

Exception