Internals and Programmer's Guide

The document is divided into the following sections:

• Overview of the Library
• Communication Modules
• The Stream Concept
• Presentation Modules
• Programming Utilities

Overview

The control flow diagram of the Library shows the module interface to a WWW Browser. In the diagram, the common code presented in this document is to the right of the green line.

The main application module is a platform dependent module called by the operating system, and it manages the overall running of the program. In the CERN Line Mode Browser, it is HTBrowse. In the CERN HTTP Server, it is HTDaemon.

On behalf of the user, the client parses a request to the HTAccess Module that is the main entry node to the library. When the request has been fulfilled (either in terms of an load or an error message) the data is parsed back to the client through the HTML Module and the HText Module, see also Information on presentation modules.

HTAccess

The Access manager module which actually loads documents is based in HTAccess.c. This uses all the protocol modules. Given an anchor ID to jump to, it asks the anchor object for the address in order to load it.

HTFormat

The Format Manager uses the parser modules to load the document as appropriate. It can also decide on the format of a file from its name. This module defines the HTConverter type.

HTAnchor object

An anchor exists for very URL or fragment identifier. The HTAnchor Module takes care of creating anchors, managing the links between them and their attributes. This module is independent of the type of graphics object (text, line drawing etc). It stores hypertext addresses of anchors, and ensures that anchors with the same address are the same anchor. See also the Anchor Description )

HTHistory

Optional client module records and replays on request the documents which the user visits

Configuration of the Library

HTInit

The initialisation module for clients. This modules contains functions to setup a list of MIME type converters that the client is capable of handling and a set of bindings between file suffixes and MIME types. The module may be replaced in custom clients.

HTSInit

The initialisation module for servers. May be replaced for custom servers. Not part of the library, but part of the server.

HTRule

The module which reads the configuration file, and does translation of URLs according to rules. It is currently replaced by HTConfig Module in the CERN Server and no rule file is used in the Line Mode Browser in order to keep it simple.

Modules to Overwrite

HTML

The HTML parser build upon the SGML parser

HTErrorMsg

This module generates the messages on the error stack

HTAlert

See also Description of HTAlert

HTEvent

This is the internal event loop in the library for multi-threaded protocol access. See also the description of multiple threads

Communications Interface to the Network

Other topics includes:

• I/O status indication (errno etc.)
• Information on remote hosts
• Information on local host (domain name etc.)
• Information on current user (mail address)

Protocol modules

When the client parses a request to the library a HTRequest Structure is filled out and parsed to, e.g. HTLoadAnchor() in HTAccess module. HTRequest is a hierarchical data structure that contains all information needed by the client, the server and the library to fulfill the request. The default values in the structure makes it very easy for the client to do a normal request of, e.g., a HTML document.

File access

This module provides access to files on a local file system. Due to general confusion of the "file://" access scheme in the URL Specifications tries FTP access on failure, but this will be changed in a new major release, June 1994.

FTP access

Uses HTTCP for common TCP routines.

HTTP access

The HTTP module handles document search and retrieve using the HTTP protocol. See also information on the current implementation of the HTTP client. The module is now a complete state machine which is a required functionality in a multi-threaded enviromnent.

News access

The NNTP internet news protocol is handled by HTNews which builds a hypertext object.

Gopher access

The internet gopher access to menus and flat files (and links to telnet nodes, WhoIs servers, CSO Name Server etc.) is handled by HTGopher Module.

Telnet access

A dummy access which forks a session. Also rlogin, tn3270.

WAIS access

WAIS access is also implemented in a separate gateway program.

HTThread

This module provides the functionality for registrating sockets as ready for READ or WRITE (this includes the CONNECT statement that is basically a WRITE request).

HTEvent

This is the Libray's own version of the event-loop serving the HTTP client. The user can interrupt via stdin and a call-back function is used so that the user can determine whether it was an interrupt or a type-ahead.

Reading Data from a Socket

Non-reentrant Reading Functions

This module is a submodule of HTFormat and it provides the functionality of reading data from the network. When multi-threaded access gets incorporated it is essential that all requests to the network goes through one point in the library. The reason keeping these is to support the single-threaded protocol access.

Reentrant Reading Functions

This is also a submodule of the HTFormat but the difference from the other socket parsing module is that this is completely reentrant. That is, it can be used together with a multi-threaded client.

Access Authorization

HTAABrow

This module contains WWW Browser specific code, that is composing the HTTP Authorization Header, recording users information etc.

HTAAUtil

This module contains the authorization code that is common to both the servers and clients, e.g., handling information on different authentication etc.

UU Encoding and Decoding

Provides functions to encode and decode a data buffer according to the RFC 1113 printable encoding format.

Presenting Directory Listings and other Listings

HTDirBrw

This is a very configurable module to actually present the listings

HTDescript

This module handles the description field in a HTTP directory listing. For a HTML file, the default action is to peek the title of the document.

HTIcons

This module handles the set of icons used in the listings (HTTP, Gopher, FTP etc.).

Multi Linguistic Documents

HTMulti

This is the current implementation of the Multi Linguistic support in the Library.

Streams

Streams can be thought of as like files open for write. The stream-based architecture allows the software to be event-driven in the sense that when input arrives, it is put into a stream, and any necessary actions then cascade off that. Even reading from the Network can in fact be done using a stream having the read function pumping data down the input stream.

Stream might be cascaded so that one stream writes into into another stream after having performed some processing on the data. An output stream is often referred to as the "target" or "sink" stream.

Currently the following specific stream objects are implemented:

HTWriter

Writes to a socket or something opened with the UNIX file I/O open() function.

HTFWriter

Writes to an ANSI C FILE * object, as opened by fopen(), etc.

SGML

Parses the data and generates a structured stream. Each parser instance is created with reference to a particular DTD structure.

HTTee

Just writes into two streams at once. Useful for taking a copy for a cache.

Black Hole

A quite expensive way of piping data into a hole for then to be forgotten forever.

Through Line

A short circuited stream that returns the same output sink as it is called with.

Content Length Counter

This stream counts the number of bytes pumped into the stream. This is used in, e.g., POST'ing in the HTTP Protocol

Converters

HTNetToText

Converts "Net ascii" (the stuff telnet sessions are made of) info local C textual data stream.

HTMIME

Parse a MIME format message. This module also contains a Content Length Counter

HTWSRC

Parses a "WAIS source" description.

HTPlain

Takes plain ASCII text and converts into whatever -- typically styled text for display or HTML.

Structured streams

The elements and entities in the stream are referred to by numbers, rather than strings. The DTD contains the mapping between element names and numbers, so each structured stream when created is associated with the DTD which it using.

The structured stream data structures are defined in the SGML module above.

Any instance of a structured stream has a related DTD which gives the rules and element and entity names for events on the structured stream. The only DTD which is currently in the library is the HTML DTD, in the HTMLDTD module.

The SGML parser uses a DTD to output to a structured stream from a stream of SGML. A hypertext editor will output to a structured stream when writing out a document. Many protocol modules output to a structures stream when generating their data structures.

The current structured stream modules are

HTML

This module is an important one as it actually PRESENTS a hypertext object to the user. GUI writers replace this module with your own, or use it and replaced the HText module which it feeds.

HTMLGen

This structured stream regenerates a plain stream. It makes reference to the HTML DTD to regenerate the names of entities and elements. This is used by the server to pass a hypertext document on to the client, and also by the client to save HTML.

HTTeXGen

Structured streams can be used for other conversions than from SGML to HTML. As an example, this is a SGML to LaTex converter that makes it easier to construct paper versions of a HTML document.

HTMLDTD

Table describing some aspects of the HTML document type: entity and element names, element contents and allowed attributes.

HTMLPDTD

The same thing but for HTML-Plus document type.

Cascading Streams Using a Stream Stack

HTFormat

This module performs the registration of MIME types and conversion modules. Currently we only parse HTML, plain text and MIME messages, but obviously other formats can be added.

Stream Stack

As mentioned above, streams can be cascaded to perform a multistep conversion. This submodule handles a stack of streams, but or the moment only direct conversion is supported so the size of the stack is always 1.

HTGuess

If the input format is unknown at the time when putting up a stream stack, then this module scans a part of the stream and on a statistical basis determines the type of stream needed from the content-type.

The library provides code for doing this "flattening". You don't have to use these modules: if you want to work the first way, just replace the entry points in HTML with your own, to prevent the library modules from being loaded.

The HTML object mentioned above flattens structured text into styled text. The HTPlain object generates a styled text for a plain ascii document.

The style system uses a set of names styles, each of which contains paragraph and character formatting information. This is managed by the HTStyle module.

HTStyle

Style and style sheet manipulation.

Styled Text object

HText

This object is window-system dependent. In the line mode browser, the GridText module is the hypertext object, providing the generic functionality of this module. Note that this interface is an alternative to the HTML.h interface above when you are building a client: you can replace the library code at either point depending on the interface you require.

Container Modules