org.apache.hadoop.io
Class SequenceFile

java.lang.Object
  org.apache.hadoop.io.SequenceFile

public class SequenceFile
extends Object

Support for flat files of binary key/value pairs.

Nested Class Summary

static class	SequenceFile.CompressionType The type of compression.
static class	SequenceFile.Metadata The class encapsulating with the metadata of a file.
static class	SequenceFile.Reader Reads key/value pairs from a sequence-format file.
static class	SequenceFile.Sorter Sorts key/value pairs in a sequence-format file.
static interface	SequenceFile.ValueBytes The interface to 'raw' values of SequenceFiles.
static class	SequenceFile.Writer Write key/value pairs to a sequence-format file.

Field Summary

static org.apache.commons.logging.Log	LOG
static int	SYNC_INTERVAL The number of bytes between sync points.

Method Summary

static SequenceFile.Writer	createWriter(Configuration conf, FSDataOutputStream out, Class keyClass, Class valClass, SequenceFile.CompressionType compressionType, CompressionCodec codec) Construct the preferred type of 'raw' SequenceFile Writer.
static SequenceFile.Writer	createWriter(Configuration conf, FSDataOutputStream out, Class keyClass, Class valClass, SequenceFile.CompressionType compressionType, CompressionCodec codec, SequenceFile.Metadata metadata) Construct the preferred type of 'raw' SequenceFile Writer.
static SequenceFile.Writer	createWriter(FileSystem fs, Configuration conf, Path name, Class keyClass, Class valClass) Construct the preferred type of SequenceFile Writer.
static SequenceFile.Writer	createWriter(FileSystem fs, Configuration conf, Path name, Class keyClass, Class valClass, SequenceFile.CompressionType compressionType) Construct the preferred type of SequenceFile Writer.
static SequenceFile.Writer	createWriter(FileSystem fs, Configuration conf, Path name, Class keyClass, Class valClass, SequenceFile.CompressionType compressionType, CompressionCodec codec) Construct the preferred type of SequenceFile Writer.
static SequenceFile.Writer	createWriter(FileSystem fs, Configuration conf, Path name, Class keyClass, Class valClass, SequenceFile.CompressionType compressionType, CompressionCodec codec, Progressable progress) Construct the preferred type of SequenceFile Writer.
static SequenceFile.Writer	createWriter(FileSystem fs, Configuration conf, Path name, Class keyClass, Class valClass, SequenceFile.CompressionType compressionType, CompressionCodec codec, Progressable progress, SequenceFile.Metadata metadata) Construct the preferred type of SequenceFile Writer.
static SequenceFile.Writer	createWriter(FileSystem fs, Configuration conf, Path name, Class keyClass, Class valClass, SequenceFile.CompressionType compressionType, Progressable progress) Construct the preferred type of SequenceFile Writer.
static SequenceFile.CompressionType	getCompressionType(Configuration job) Get the compression type for the reduce outputs
static void	setCompressionType(Configuration job, SequenceFile.CompressionType val) Set the compression type for sequence files.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

SYNC_INTERVAL

public static final int SYNC_INTERVAL

The number of bytes between sync points.

See Also:

Constant Field Values

Method Detail

getCompressionType

public static SequenceFile.CompressionType getCompressionType(Configuration job)

Get the compression type for the reduce outputs

Parameters:

job - the job config to look in

Returns:

the kind of compression to use

setCompressionType

public static void setCompressionType(Configuration job,
                                      SequenceFile.CompressionType val)

Set the compression type for sequence files.

Parameters:

job - the configuration to modify

val - the new compression type (none, block, record)

createWriter

public static SequenceFile.Writer createWriter(FileSystem fs,
                                               Configuration conf,
                                               Path name,
                                               Class keyClass,
                                               Class valClass)
                                        throws IOException

Construct the preferred type of SequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

Returns:

Returns the handle to the constructed SequenceFile Writer.

Throws:

IOException

createWriter

public static SequenceFile.Writer createWriter(FileSystem fs,
                                               Configuration conf,
                                               Path name,
                                               Class keyClass,
                                               Class valClass,
                                               SequenceFile.CompressionType compressionType)
                                        throws IOException

Construct the preferred type of SequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

Returns:

Returns the handle to the constructed SequenceFile Writer.

Throws:

IOException

createWriter

public static SequenceFile.Writer createWriter(FileSystem fs,
                                               Configuration conf,
                                               Path name,
                                               Class keyClass,
                                               Class valClass,
                                               SequenceFile.CompressionType compressionType,
                                               Progressable progress)
                                        throws IOException

Construct the preferred type of SequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

progress - The Progressable object to track progress.

Returns:

Returns the handle to the constructed SequenceFile Writer.

Throws:

IOException

createWriter

public static SequenceFile.Writer createWriter(FileSystem fs,
                                               Configuration conf,
                                               Path name,
                                               Class keyClass,
                                               Class valClass,
                                               SequenceFile.CompressionType compressionType,
                                               CompressionCodec codec)
                                        throws IOException

Construct the preferred type of SequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

codec - The compression codec.

Returns:

Returns the handle to the constructed SequenceFile Writer.

Throws:

IOException

createWriter

public static SequenceFile.Writer createWriter(FileSystem fs,
                                               Configuration conf,
                                               Path name,
                                               Class keyClass,
                                               Class valClass,
                                               SequenceFile.CompressionType compressionType,
                                               CompressionCodec codec,
                                               Progressable progress,
                                               SequenceFile.Metadata metadata)
                                        throws IOException

Construct the preferred type of SequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

codec - The compression codec.

progress - The Progressable object to track progress.

metadata - The metadata of the file.

Returns:

Returns the handle to the constructed SequenceFile Writer.

Throws:

IOException

createWriter

public static SequenceFile.Writer createWriter(FileSystem fs,
                                               Configuration conf,
                                               Path name,
                                               Class keyClass,
                                               Class valClass,
                                               SequenceFile.CompressionType compressionType,
                                               CompressionCodec codec,
                                               Progressable progress)
                                        throws IOException

Construct the preferred type of SequenceFile Writer.

Parameters:

fs - The configured filesystem.

conf - The configuration.

name - The name of the file.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

codec - The compression codec.

progress - The Progressable object to track progress.

Returns:

Returns the handle to the constructed SequenceFile Writer.

Throws:

IOException

createWriter

public static SequenceFile.Writer createWriter(Configuration conf,
                                               FSDataOutputStream out,
                                               Class keyClass,
                                               Class valClass,
                                               SequenceFile.CompressionType compressionType,
                                               CompressionCodec codec,
                                               SequenceFile.Metadata metadata)
                                        throws IOException

Construct the preferred type of 'raw' SequenceFile Writer.

Parameters:

conf - The configuration.

out - The stream on top which the writer is to be constructed.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

codec - The compression codec.

metadata - The metadata of the file.

Returns:

Returns the handle to the constructed SequenceFile Writer.

Throws:

IOException

createWriter

public static SequenceFile.Writer createWriter(Configuration conf,
                                               FSDataOutputStream out,
                                               Class keyClass,
                                               Class valClass,
                                               SequenceFile.CompressionType compressionType,
                                               CompressionCodec codec)
                                        throws IOException

Construct the preferred type of 'raw' SequenceFile Writer.

Parameters:

conf - The configuration.

out - The stream on top which the writer is to be constructed.

keyClass - The 'key' type.

valClass - The 'value' type.

compressionType - The compression type.

codec - The compression codec.

Returns:

Returns the handle to the constructed SequenceFile Writer.

Throws:

IOException