|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Object org.apache.hadoop.contrib.utils.join.JobBase org.apache.hadoop.contrib.utils.join.DataJoinReducerBase
public abstract class DataJoinReducerBase
This abstract class serves as the base class for the reducer class of a data join job. The reduce function will first group the values according to their input tags, and then compute the cross product of over the groups. For each tuple in the cross product, it calls the following method, which is expected to be implemented in a subclass. protected abstract TaggedMapOutput combine(Object[] tags, Object[] values); The above method is expected to produce one output value from an array of records of different sources. The user code can also perform filtering here. It can return null if it decides to the records do not meet certain conditions.
Field Summary | |
---|---|
protected long |
collected
|
protected JobConf |
job
|
protected long |
largestNumOfValues
|
static Text |
NUM_OF_VALUES_FIELD
|
protected long |
numOfValues
|
protected Reporter |
reporter
|
static Text |
SOURCE_TAGS_FIELD
|
Fields inherited from class org.apache.hadoop.contrib.utils.join.JobBase |
---|
LOG |
Constructor Summary | |
---|---|
DataJoinReducerBase()
|
Method Summary | |
---|---|
void |
close()
Called after the last call to any other method on this object to free and/or flush resources. |
protected void |
collect(WritableComparable key,
TaggedMapOutput aRecord,
OutputCollector output,
Reporter reporter)
The subclass can overwrite this method to perform additional filtering and/or other processing logic before a value is collected. |
protected abstract TaggedMapOutput |
combine(Object[] tags,
Object[] values)
|
void |
configure(JobConf job)
Initializes a new instance from a JobConf . |
protected ResetableIterator |
createResetableIterator()
The subclass can provide a different implementation on ResetableIterator. |
void |
map(WritableComparable arg0,
Writable arg1,
OutputCollector arg2,
Reporter arg3)
Maps a single input key/value pair into intermediate key/value pairs. |
void |
reduce(WritableComparable key,
Iterator values,
OutputCollector output,
Reporter reporter)
Combines values for a given key. |
Methods inherited from class org.apache.hadoop.contrib.utils.join.JobBase |
---|
addDoubleValue, addLongValue, getDoubleValue, getLongValue, getReport, report, setDoubleValue, setLongValue |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
protected Reporter reporter
protected long largestNumOfValues
protected long numOfValues
protected long collected
protected JobConf job
public static Text SOURCE_TAGS_FIELD
public static Text NUM_OF_VALUES_FIELD
Constructor Detail |
---|
public DataJoinReducerBase()
Method Detail |
---|
public void close() throws IOException
Closeable
IOException
public void configure(JobConf job)
JobBase
JobConf
.
configure
in interface JobConfigurable
configure
in class JobBase
job
- the configurationprotected ResetableIterator createResetableIterator()
public void reduce(WritableComparable key, Iterator values, OutputCollector output, Reporter reporter) throws IOException
Reducer
OutputCollector.collect(WritableComparable,Writable)
.
key
- the keyvalues
- the values to combineoutput
- to collect combined values
IOException
protected void collect(WritableComparable key, TaggedMapOutput aRecord, OutputCollector output, Reporter reporter) throws IOException
key
- aRecord
- output
- reporter
-
IOException
protected abstract TaggedMapOutput combine(Object[] tags, Object[] values)
tags
- a list of source tagsvalues
- a value per source
public void map(WritableComparable arg0, Writable arg1, OutputCollector arg2, Reporter arg3) throws IOException
Mapper
OutputCollector.collect(WritableComparable,Writable)
.
arg0
- the keyarg1
- the valuesarg2
- collects mapped keys and values
IOException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |