The source code of the image decoding is here
http://lxr.mozilla.org/seamonkey/source/modules/libpr0n/

The interface that every image type implements is imgIDecoder
http://lxr.mozilla.org/seamonkey/source/modules/libpr0n/public/imgIDecoder.idl

=== about IDL ===

Note that the first character of a method name in IDL becomes
capitalized in C++.  Also note that the .h #include header files are
automatically generated from .idl files during the build.

=== Examples ===

Currently implemented decoders for various image types are in
http://lxr.mozilla.org/seamonkey/source/modules/libpr0n/decoders/
which you can use as examples.

Lines of code for various the implementation of the various image
types:

 bmp   1780
 gif   3355
 icon  5553
 jpeg  1028
 png    719
 xbm    457

So clearly start looking at [http://lxr.mozilla.org/seamonkey/source/modules/libpr0n/decoders/ xbm]
for the simplest example of how it all works.  

Note that these line counts do not reflect that the actual work, as
decoding an image is often farmed out to a library somewhere else, for
png in
http://lxr.mozilla.org/seamonkey/source/modules/libimg/png/png.h and
jpeg in http://lxr.mozilla.org/seamonkey/source/jpeg/jpeglib.h (and
containing directories)

=== implementing imgIDecoder ===

The methods you need to implement in the imgIDecoder interface are
init, close, flush, and writeFrom.

The [http://lxr.mozilla.org/seamonkey/source/modules/libpr0n/public/imgIDecoder.idl#59 cryptic @note] on the init method of imgIDecoder
 "The decode should QI \a aLoad to an imgIDecoderObserver"
means that the implementation of the Init method should do something like this:
  mObserver = do_QueryInterface(aLoad); 
(so the acronym QI == "QueryInterface") where mObserver is a data member of type
nsCOMPtr<[http://lxr.mozilla.org/seamonkey/source/modules/libpr0n/public/imgIDecoderObserver.idl imgIDecoderObserver]>
  

The guts of decoding start in the implementation of the WriteFrom
method of the imgIDecoder interface, whose first argument is a
nsIInputStream.  The WriteFrom method can call this (first) argument's
Read method directly (as jpeg does), or call its ReadSegments method
(as xbm does, for example), passing it a callback function, which is a
STATIC method or a regular function.  This callback function will
receive chunkfuls of bytes the image data arrives over the network.

The nsIInputStream interface is specified in 
http://lxr.mozilla.org/seamonkey/source/xpcom/io/nsIInputStream.idl

In order to deliver the decoded image (or partially decoded image) to
the rest of Firefox, we call various methods to data members that are
hooks.  The mObserver of type
nsCOMPtr<[http://lxr.mozilla.org/seamonkey/source/modules/libpr0n/public/imgIDecoderObserver.idl imgIDecoderObserver]>
is mentioned above, and you need to call its 
[http://lxr.mozilla.org/seamonkey/source/modules/libpr0n/public/imgIDecoderObserver.idl#93 OnDataAvailable]
 method.  You will also probably need to define and
use data members
nsCOMPtr<[http://lxr.mozilla.org/seamonkey/source/modules/libpr0n/public/imgIContainer.idl imgIContainer]> mImage and
nsCOMPtr<[http://lxr.mozilla.org/seamonkey/source/gfx/idl/gfxIImageFrame.idl gfxIImageFrame]> mFrame.  See
[http://lxr.mozilla.org/seamonkey/source/modules/libpr0n/decoders/ the decoders source] for examples.