org.apache.hadoop.dfs
Class NameNode

java.lang.Object
  org.apache.hadoop.dfs.NameNode

All Implemented Interfaces:

FSConstants

public class NameNode
extends Object
implements FSConstants

NameNode serves as both directory namespace manager and "inode table" for the Hadoop DFS. There is a single NameNode running in any DFS deployment. (Well, except when there is a second backup/failover NameNode.) The NameNode controls two critical tables: 1) filename->blocksequence (namespace) 2) block->machinelist ("inodes") The first table is stored on disk and is very precious. The second table is rebuilt every time the NameNode comes up. 'NameNode' refers to both this class as well as the 'NameNode server'. The 'FSNamesystem' class actually performs most of the filesystem management. The majority of the 'NameNode' class itself is concerned with exposing the IPC interface to the outside world, plus some configuration management. NameNode implements the ClientProtocol interface, which allows clients to ask for DFS services. ClientProtocol is not designed for direct use by authors of DFS client code. End-users should instead use the org.apache.nutch.hadoop.fs.FileSystem class. NameNode also implements the DatanodeProtocol interface, used by DataNode programs that actually store DFS data blocks. These methods are invoked repeatedly and automatically by all the DataNodes in a DFS deployment.

Author:

Mike Cafarella

Nested Class Summary

static class

DatanodeProtocol.DataNodeAction Determines actions that data node should perform when receiving a block command.

Nested classes/interfaces inherited from interface org.apache.hadoop.dfs.FSConstants

FSConstants.NodeType, FSConstants.SafeModeAction, FSConstants.StartupOption

Field Summary

static int	DISK_ERROR
static int	INVALID_BLOCK
static org.apache.commons.logging.Log	LOG
static int	NOTIFY
static org.apache.commons.logging.Log	stateChangeLog
static long	versionID
static long	versionID

Fields inherited from interface org.apache.hadoop.dfs.FSConstants

BLOCK_INVALIDATE_CHUNK, BLOCKREPORT_INTERVAL, BUFFER_SIZE, CHUNKED_ENCODING, COMPLETE_SUCCESS, HEARTBEAT_INTERVAL, LAYOUT_VERSION, LEASE_HARDLIMIT_PERIOD, LEASE_SOFTLIMIT_PERIOD, MAX_PATH_DEPTH, MAX_PATH_LENGTH, MIN_BLOCKS_FOR_WRITE, OP_ACK, OP_BLOCKRECEIVED, OP_BLOCKREPORT, OP_CLIENT_ABANDONBLOCK, OP_CLIENT_ABANDONBLOCK_ACK, OP_CLIENT_ADDBLOCK, OP_CLIENT_ADDBLOCK_ACK, OP_CLIENT_COMPLETEFILE, OP_CLIENT_COMPLETEFILE_ACK, OP_CLIENT_DATANODE_HINTS, OP_CLIENT_DATANODE_HINTS_ACK, OP_CLIENT_DATANODEREPORT, OP_CLIENT_DATANODEREPORT_ACK, OP_CLIENT_DELETE, OP_CLIENT_DELETE_ACK, OP_CLIENT_EXISTS, OP_CLIENT_EXISTS_ACK, OP_CLIENT_ISDIR, OP_CLIENT_ISDIR_ACK, OP_CLIENT_LISTING, OP_CLIENT_LISTING_ACK, OP_CLIENT_MKDIRS, OP_CLIENT_MKDIRS_ACK, OP_CLIENT_OBTAINLOCK, OP_CLIENT_OBTAINLOCK_ACK, OP_CLIENT_OPEN, OP_CLIENT_OPEN_ACK, OP_CLIENT_RAWSTATS, OP_CLIENT_RAWSTATS_ACK, OP_CLIENT_RELEASELOCK, OP_CLIENT_RELEASELOCK_ACK, OP_CLIENT_RENAMETO, OP_CLIENT_RENAMETO_ACK, OP_CLIENT_RENEW_LEASE, OP_CLIENT_RENEW_LEASE_ACK, OP_CLIENT_STARTFILE, OP_CLIENT_STARTFILE_ACK, OP_CLIENT_TRYAGAIN, OP_ERROR, OP_FAILURE, OP_HEARTBEAT, OP_INVALIDATE_BLOCKS, OP_READ_BLOCK, OP_READ_RANGE_BLOCK, OP_READSKIP_BLOCK, OP_TRANSFERBLOCKS, OP_TRANSFERDATA, OP_WRITE_BLOCK, OPERATION_FAILED, READ_TIMEOUT, RUNLENGTH_ENCODING, STILL_WAITING, WRITE_COMPLETE

Constructor Summary

NameNode(Configuration conf)
Start NameNode.

NameNode(String bindAddress, int port, Configuration conf)
Create a NameNode at the specified location and start it.

Method Summary

void	abandonBlock(org.apache.hadoop.dfs.Block b, String src) The client needs to give up on the block.
void	abandonFileInProgress(String src, String holder) A client that wants to abandon writing to the current file should call abandonFileInProgress().
org.apache.hadoop.dfs.LocatedBlock	addBlock(String src, String clientName) A client that wants to write an additional block to the indicated filename (which must currently be open for writing) should call addBlock().
void	blockReceived(org.apache.hadoop.dfs.DatanodeRegistration nodeReg, org.apache.hadoop.dfs.Block[] blocks) blockReceived() allows the DataNode to tell the NameNode about recently-received block data.
org.apache.hadoop.dfs.DatanodeCommand	blockReport(org.apache.hadoop.dfs.DatanodeRegistration nodeReg, org.apache.hadoop.dfs.Block[] blocks) blockReport() tells the NameNode about all the locally-stored blocks.
boolean	complete(String src, String clientName) The client is done writing data to the given filename, and would like to complete it.
org.apache.hadoop.dfs.LocatedBlock	create(String src, String clientName, boolean overwrite, short replication, long blockSize) Create a new file.
boolean	delete(String src) Remove the given filename from the filesystem
void	errorReport(org.apache.hadoop.dfs.DatanodeRegistration nodeReg, int errorCode, String msg) errorReport() tells the NameNode about something that has gone awry.
boolean	exists(String src) Check whether the given file exists
void	finalizeUpgrade() Finalize previous upgrade.
static void	format(Configuration conf) Format a new filesystem.
long	getBlockSize(String filename) Get the block size for the given file.
DatanodeInfo[]	getDatanodeReport() Get a full report on the system's current datanodes.
long	getEditLogSize() Returns the size of the current edit log.
File	getFsEditName() Returns the name of the edits file
File	getFsImageName() Returns the name of the fsImage file
File[]	getFsImageNameCheckpoint() Returns the name of the fsImage file uploaded by periodic checkpointing
String[][]	getHints(String src, long start, long len) getHints() returns a list of hostnames that store data for a specific file region.
org.apache.hadoop.dfs.DFSFileInfo[]	getListing(String src) Get a listing of the indicated directory
InetSocketAddress	getNameNodeAddress() Returns the address on which the NameNodes is listening to.
long	getProtocolVersion(String protocol, long clientVersion) Return protocol version corresponding to protocol interface.
long[]	getStats() Get a set of statistics about the filesystem.
boolean	isDir(String src) Check whether the given filename is a directory or not.
void	join() Wait for service to finish.
static void	main(String[] argv)
void	metaSave(String filename) Dumps namenode state into specified file
boolean	mkdirs(String src) Create a directory (or hierarchy of directories) with the given name.
boolean	obtainLock(String src, String clientName, boolean exclusive) Deprecated.
org.apache.hadoop.dfs.LocatedBlock[]	open(String src) Open an existing file, at the given name.
void	refreshNodes() Tells the namenode to reread the hosts and exclude files.
org.apache.hadoop.dfs.DatanodeRegistration	register(org.apache.hadoop.dfs.DatanodeRegistration nodeReg, String networkLocation) Register Datanode.
boolean	releaseLock(String src, String clientName) Deprecated.
boolean	rename(String src, String dst) Rename an item in the fs namespace
void	renewLease(String clientName) Client programs can cause stateful changes in the NameNode that affect other clients.
void	reportBadBlocks(org.apache.hadoop.dfs.LocatedBlock[] blocks) The client has detected an error on the specified located blocks and is reporting them to the server.
void	rollEditLog() Roll the edit log.
void	rollFsImage() Roll the image
org.apache.hadoop.dfs.DatanodeCommand	sendHeartbeat(org.apache.hadoop.dfs.DatanodeRegistration nodeReg, long capacity, long remaining, int xmitsInProgress, int xceiverCount) Data node notify the name node that it is alive Return a block-oriented command for the datanode to execute.
boolean	setReplication(String src, short replication) Set replication for an existing file.
boolean	setSafeMode(FSConstants.SafeModeAction action) Enter, leave or get safe mode.
void	stop() Stop all NameNode threads and wait for all to finish.
void	verifyRequest(org.apache.hadoop.dfs.DatanodeRegistration nodeReg) Verify request.
void	verifyVersion(int version) Verify version.
org.apache.hadoop.dfs.NamespaceInfo	versionRequest()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

stateChangeLog

public static final org.apache.commons.logging.Log stateChangeLog

versionID

public static final long versionID

See Also:

Constant Field Values

versionID

public static final long versionID

See Also:

Constant Field Values

NOTIFY

public static final int NOTIFY

See Also:

Constant Field Values

DISK_ERROR

public static final int DISK_ERROR

See Also:

Constant Field Values

INVALID_BLOCK

public static final int INVALID_BLOCK

See Also:

Constant Field Values

Constructor Detail

NameNode

public NameNode(Configuration conf)
         throws IOException

Start NameNode.

The name-node can be started with one of the following startup options:

• REGULAR - normal startup
• FORMAT - format name node
• UPGRADE - start the cluster upgrade and create a snapshot of the current file system state
• ROLLBACK - roll the cluster back to the previous state

The option is passed via configuration field: dfs.namenode.startup The conf will be modified to reflect the actual ports on which the NameNode is up and running if the user passes the port as zero in the conf.

Parameters:

conf - confirguration

Throws:

IOException

NameNode

public NameNode(String bindAddress,
                int port,
                Configuration conf)
         throws IOException

Create a NameNode at the specified location and start it. The conf will be modified to reflect the actual ports on which the NameNode is up and running if the user passes the port as zero.

Throws:

IOException

Method Detail

getProtocolVersion

public long getProtocolVersion(String protocol,
                               long clientVersion)
                        throws IOException

Description copied from interface: VersionedProtocol

Return protocol version corresponding to protocol interface.

Parameters:

protocol - The classname of the protocol interface

clientVersion - The version of the protocol that the client speaks

Returns:

the version that the server will speak

Throws:

IOException

format

public static void format(Configuration conf)
                   throws IOException

Format a new filesystem. Destroys any filesystem that may already exist at this location.

Throws:

IOException

join

public void join()

Wait for service to finish. (Normally, it runs forever.)

stop

public void stop()

Stop all NameNode threads and wait for all to finish.

open

public org.apache.hadoop.dfs.LocatedBlock[] open(String src)
                                          throws IOException

Open an existing file, at the given name. Returns block and DataNode info. DataNodes for each block are sorted by the distance to the client's address. The client will then have to contact each indicated DataNode to obtain the actual data. There is no need to call close() or any other function after calling open().

Throws:

IOException

create

public org.apache.hadoop.dfs.LocatedBlock create(String src,
                                                 String clientName,
                                                 boolean overwrite,
                                                 short replication,
                                                 long blockSize)
                                          throws IOException

Create a new file. Get back block and datanode info, which describes where the first block should be written. Successfully calling this method prevents any other client from creating a file under the given name, but the caller must invoke complete() for the file to be added to the filesystem. Blocks have a maximum size. Clients that intend to create multi-block files must also use reportWrittenBlock() and addBlock().

Throws:

IOException

setReplication

public boolean setReplication(String src,
                              short replication)
                       throws IOException

Set replication for an existing file. The NameNode sets replication to the new value and returns. The actual block replication is not expected to be performed during this method call. The blocks will be populated or removed in the background as the result of the routine block maintenance procedures.

Parameters:

src - file name

replication - new replication

Returns:

true if successful; false if file does not exist or is a directory

Throws:

IOException

addBlock

public org.apache.hadoop.dfs.LocatedBlock addBlock(String src,
                                                   String clientName)
                                            throws IOException

A client that wants to write an additional block to the indicated filename (which must currently be open for writing) should call addBlock(). addBlock() returns block and datanode info, just like the initial call to create(). A null response means the NameNode could not allocate a block, and that the caller should try again.

Throws:

IOException

abandonBlock

public void abandonBlock(org.apache.hadoop.dfs.Block b,
                         String src)
                  throws IOException

The client needs to give up on the block.

Throws:

IOException

abandonFileInProgress

public void abandonFileInProgress(String src,
                                  String holder)
                           throws IOException

A client that wants to abandon writing to the current file should call abandonFileInProgress(). After this call, any client can call create() to obtain the filename. Any blocks that have been written for the file will be garbage-collected.

Parameters:

src - The filename

holder - The datanode holding the lease

Throws:

IOException

complete

public boolean complete(String src,
                        String clientName)
                 throws IOException

The client is done writing data to the given filename, and would like to complete it. The function returns whether the file has been closed successfully. If the function returns false, the caller should try again. A call to complete() will not return true until all the file's blocks have been replicated the minimum number of times. Thus, DataNode failures may cause a client to call complete() several times before succeeding.

Throws:

IOException

reportBadBlocks

public void reportBadBlocks(org.apache.hadoop.dfs.LocatedBlock[] blocks)
                     throws IOException

The client has detected an error on the specified located blocks and is reporting them to the server. For now, the namenode will delete the blocks from the datanodes. In the future we might check the blocks are actually corrupt.

Parameters:

blocks - Array of located blocks to report

Throws:

IOException

getHints

public String[][] getHints(String src,
                           long start,
                           long len)
                    throws IOException

getHints() returns a list of hostnames that store data for a specific file region. It returns a set of hostnames for every block within the indicated region. This function is very useful when writing code that considers data-placement when performing operations. For example, the MapReduce system tries to schedule tasks on the same machines as the data-block the task processes.

Throws:

IOException

getBlockSize

public long getBlockSize(String filename)
                  throws IOException

Get the block size for the given file.

Parameters:

filename - The name of the file

Returns:

The number of bytes in each block

Throws:

IOException

rename

public boolean rename(String src,
                      String dst)
               throws IOException

Rename an item in the fs namespace

Throws:

IOException

delete

public boolean delete(String src)
               throws IOException

Remove the given filename from the filesystem

Throws:

IOException

exists

public boolean exists(String src)
               throws IOException

Check whether the given file exists

Throws:

IOException

isDir

public boolean isDir(String src)
              throws IOException

Check whether the given filename is a directory or not.

Throws:

IOException

mkdirs

public boolean mkdirs(String src)
               throws IOException

Create a directory (or hierarchy of directories) with the given name.

Throws:

IOException

obtainLock

@Deprecated
public boolean obtainLock(String src,
                                     String clientName,
                                     boolean exclusive)
                   throws IOException

Deprecated.

Throws:

IOException

releaseLock

@Deprecated
public boolean releaseLock(String src,
                                      String clientName)
                    throws IOException

Deprecated.

Throws:

IOException

renewLease

public void renewLease(String clientName)
                throws IOException

Client programs can cause stateful changes in the NameNode that affect other clients. A client may obtain a file and neither abandon nor complete it. A client might hold a series of locks that prevent other clients from proceeding. Clearly, it would be bad if a client held a bunch of locks that it never gave up. This can happen easily if the client dies unexpectedly. So, the NameNode will revoke the locks and live file-creates for clients that it thinks have died. A client tells the NameNode that it is still alive by periodically calling renewLease(). If a certain amount of time passes since the last call to renewLease(), the NameNode assumes the client has died.

Throws:

IOException

getListing

public org.apache.hadoop.dfs.DFSFileInfo[] getListing(String src)
                                               throws IOException

Get a listing of the indicated directory

Throws:

IOException

getStats

public long[] getStats()
                throws IOException

Get a set of statistics about the filesystem. Right now, only two values are returned. [0] contains the total storage capacity of the system, in bytes. [1] contains the available storage of the system, in bytes.

Throws:

IOException

getDatanodeReport

public DatanodeInfo[] getDatanodeReport()
                                 throws IOException

Get a full report on the system's current datanodes. One DatanodeInfo object is returned for each DataNode.

Throws:

IOException

setSafeMode

public boolean setSafeMode(FSConstants.SafeModeAction action)
                    throws IOException

Enter, leave or get safe mode.

Safe mode is a name node state when it

1. does not accept changes to name space (read-only), and
2. does not replicate or delete blocks.

Safe mode is entered automatically at name node startup. Safe mode can also be entered manually using setSafeMode(SafeModeAction.SAFEMODE_GET).

At startup the name node accepts data node reports collecting information about block locations. In order to leave safe mode it needs to collect a configurable percentage called threshold of blocks, which satisfy the minimal replication condition. The minimal replication condition is that each block must have at least dfs.replication.min replicas. When the threshold is reached the name node extends safe mode for a configurable amount of time to let the remaining data nodes to check in before it will start replicating missing blocks. Then the name node leaves safe mode.

If safe mode is turned on manually using setSafeMode(SafeModeAction.SAFEMODE_ENTER) then the name node stays in safe mode until it is manually turned off using setSafeMode(SafeModeAction.SAFEMODE_LEAVE). Current state of the name node can be verified using setSafeMode(SafeModeAction.SAFEMODE_GET)

Configuration parameters:

dfs.safemode.threshold.pct is the threshold parameter.
dfs.safemode.extension is the safe mode extension parameter.
dfs.replication.min is the minimal replication parameter.

Special cases:

The name node does not enter safe mode at startup if the threshold is set to 0 or if the name space is empty.
If the threshold is set to 1 then all blocks need to have at least minimal replication.
If the threshold value is greater than 1 then the name node will not be able to turn off safe mode automatically.
Safe mode can always be turned off manually.

Parameters:

action -

• 0 leave safe mode;
• 1 enter safe mode;
• 2 get safe mode state.

Returns:

• 0 if the safe mode is OFF or
• 1 if the safe mode is ON.

Throws:

IOException

refreshNodes

public void refreshNodes()
                  throws IOException

Tells the namenode to reread the hosts and exclude files.

Throws:

IOException

getEditLogSize

public long getEditLogSize()
                    throws IOException

Returns the size of the current edit log.

Returns:

The number of bytes in the current edit log.

Throws:

IOException

rollEditLog

public void rollEditLog()
                 throws IOException

Roll the edit log.

Throws:

IOException

rollFsImage

public void rollFsImage()
                 throws IOException

Roll the image

Throws:

IOException

finalizeUpgrade

public void finalizeUpgrade()
                     throws IOException

Finalize previous upgrade. Remove file system state saved during the upgrade. The upgrade will become irreversible.

Throws:

IOException

metaSave

public void metaSave(String filename)
              throws IOException

Dumps namenode state into specified file

Throws:

IOException

register

public org.apache.hadoop.dfs.DatanodeRegistration register(org.apache.hadoop.dfs.DatanodeRegistration nodeReg,
                                                           String networkLocation)
                                                    throws IOException

Register Datanode.

Returns:

updated DatanodeRegistration, which contains new storageID if the datanode did not have one and registration ID for further communication.

Throws:

IOException

See Also:

DataNode.register(), FSNamesystem.registerDatanode(DatanodeRegistration, String)

sendHeartbeat

public org.apache.hadoop.dfs.DatanodeCommand sendHeartbeat(org.apache.hadoop.dfs.DatanodeRegistration nodeReg,
                                                           long capacity,
                                                           long remaining,
                                                           int xmitsInProgress,
                                                           int xceiverCount)
                                                    throws IOException

Data node notify the name node that it is alive Return a block-oriented command for the datanode to execute. This will be either a transfer or a delete operation.

Throws:

IOException

blockReport

public org.apache.hadoop.dfs.DatanodeCommand blockReport(org.apache.hadoop.dfs.DatanodeRegistration nodeReg,
                                                         org.apache.hadoop.dfs.Block[] blocks)
                                                  throws IOException

blockReport() tells the NameNode about all the locally-stored blocks. The NameNode returns an array of Blocks that have become obsolete and should be deleted. This function is meant to upload *all* the locally-stored blocks. It's invoked upon startup and then infrequently afterwards.

Throws:

IOException

blockReceived

public void blockReceived(org.apache.hadoop.dfs.DatanodeRegistration nodeReg,
                          org.apache.hadoop.dfs.Block[] blocks)
                   throws IOException

blockReceived() allows the DataNode to tell the NameNode about recently-received block data. For example, whenever client code writes a new Block here, or another DataNode copies a Block to this DataNode, it will call blockReceived().

Throws:

IOException

errorReport

public void errorReport(org.apache.hadoop.dfs.DatanodeRegistration nodeReg,
                        int errorCode,
                        String msg)
                 throws IOException

errorReport() tells the NameNode about something that has gone awry. Useful for debugging.

Throws:

IOException

versionRequest

public org.apache.hadoop.dfs.NamespaceInfo versionRequest()
                                                   throws IOException

Throws:

IOException

verifyRequest

public void verifyRequest(org.apache.hadoop.dfs.DatanodeRegistration nodeReg)
                   throws IOException

Verify request. Verifies correctness of the datanode version, registration ID, and if the datanode does not need to be shutdown.

Parameters:

nodeReg - data node registration

Throws:

IOException

verifyVersion

public void verifyVersion(int version)
                   throws IOException

Verify version.

Parameters:

version -

Throws:

IOException

getFsImageName

public File getFsImageName()
                    throws IOException

Returns the name of the fsImage file

Throws:

IOException

getFsImageNameCheckpoint

public File[] getFsImageNameCheckpoint()
                                throws IOException

Returns the name of the fsImage file uploaded by periodic checkpointing

Throws:

IOException

getFsEditName

public File getFsEditName()
                   throws IOException

Returns the name of the edits file

Throws:

IOException

getNameNodeAddress

public InetSocketAddress getNameNodeAddress()

Returns the address on which the NameNodes is listening to.

Returns:

the address on which the NameNodes is listening to.

main

public static void main(String[] argv)
                 throws Exception

Throws:

Exception