org.apache.hadoop.streaming
Class StreamBaseRecordReader

java.lang.Object
  org.apache.hadoop.streaming.StreamBaseRecordReader

All Implemented Interfaces:

RecordReader

Direct Known Subclasses:

StreamXmlRecordReader

public abstract class StreamBaseRecordReader
extends Object
implements RecordReader

Shared functionality for hadoopStreaming formats. A custom reader can be defined to be a RecordReader with the constructor below and is selected with the option bin/hadoopStreaming -inputreader ...

Author:

Michel Tourn

See Also:

StreamLineRecordReader, StreamXmlRecordReader

Field Summary

protected static org.apache.commons.logging.Log

LOG

Constructor Summary

StreamBaseRecordReader(FSDataInputStream in, FileSplit split, Reporter reporter, JobConf job, FileSystem fs)

Method Summary

void	close() Close this to future operations.
WritableComparable	createKey() Create an object of the appropriate type to be used as a key.
Writable	createValue() Create an object of the appropriate type to be used as the value.
long	getPos() Returns the current position in the input.
float	getProgress() How far has the reader gone through the input.
abstract boolean	next(Writable key, Writable value) Read a record.
abstract void	seekNextRecordBoundary() Implementation should seek forward in_ to the first byte of the next record.
void	validateInput(JobConf job) This implementation always returns true.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.apache.commons.logging.Log LOG

Constructor Detail

StreamBaseRecordReader

public StreamBaseRecordReader(FSDataInputStream in,
                              FileSplit split,
                              Reporter reporter,
                              JobConf job,
                              FileSystem fs)
                       throws IOException

Throws:

IOException

Method Detail

next

public abstract boolean next(Writable key,
                             Writable value)
                      throws IOException

Read a record. Implementation should call numRecStats at the end

Specified by:

next in interface RecordReader

Parameters:

key - the key to read data into

value - the value to read data into

Returns:

true iff a key/value was read, false if at EOF

Throws:

IOException

See Also:

Writable.readFields(DataInput)

validateInput

public void validateInput(JobConf job)
                   throws IOException

This implementation always returns true.

Throws:

IOException

getPos

public long getPos()
            throws IOException

Returns the current position in the input.

Specified by:

getPos in interface RecordReader

Throws:

IOException

close

public void close()
           throws IOException

Close this to future operations.

Specified by:

close in interface RecordReader

Throws:

IOException

getProgress

public float getProgress()
                  throws IOException

Description copied from interface: RecordReader

How far has the reader gone through the input.

Specified by:

getProgress in interface RecordReader

Returns:

progress from 0.0 to 1.0

Throws:

IOException

createKey

public WritableComparable createKey()

Description copied from interface: RecordReader

Create an object of the appropriate type to be used as a key.

Specified by:

createKey in interface RecordReader

Returns:

a new key object

createValue

public Writable createValue()

Description copied from interface: RecordReader

Create an object of the appropriate type to be used as the value.

Specified by:

createValue in interface RecordReader

Returns:

a new value object

seekNextRecordBoundary

public abstract void seekNextRecordBoundary()
                                     throws IOException

Implementation should seek forward in_ to the first byte of the next record. The initial byte offset in the stream is arbitrary.

Throws:

IOException