HDF5 documents and links Introduction to HDF5 HDF5 User Guide |
And in this document, the
HDF5 Reference Manual H5 H5A H5D H5E H5F H5G H5I H5P H5R H5S H5T H5Z Tools Datatypes |
(PDF of complete manual formatted as print volume) |
The C Interfaces:
Alphabetical Listing
Filter Behavior in HDF5:
Filters can be inserted into the HDF5 pipeline to perform functions
such as compression and conversion. As such, they are a very flexible
aspect of HDF5; for example, a user-defined filter could provide
encryption for an HDF5 dataset.
A filter can be declared as either required or optional. Required is the default status; optional status must be explicitly declared.
A required filter that fails or is not defined causes an entire output operation to fail; if it was applied when the data was written, such a filter will cause an input operation to fail.
The following table summarizes required filter behavior.
Required FILTER_X not available | FILTER_X available | |
H5Pset_<FILTER_X> |
Will fail. |
Will succeed. |
H5Dwrite with FILTER_X set |
Will fail. | Will succeed; FILTER_X will be applied to the data. |
H5Dread with FILTER_X set |
Will fail. | Will succeed. |
An optional filter can be set for an HDF5 dataset even when the filter is not available. Such a filter can then be applied to the dataset when it becomes available on the original system or when the file containing the dataset is processed on a system on which it is available.
A filter can be declared as optional through the use of
the H5Z_FLAG_OPTIONAL
flag
with H5Pset_filter
.
Consider a situation where one is creating files that will normally be used only on systems where the optional (and fictional) filter FILTER_Z is routinely available. One can create those files on system A, which lacks FILTER_Z, create chunked datasets in the files with FILTER_Z defined in the dataset creation property list, and even write data to those datasets. The dataset object header will indicate that FILTER_Z has been associated with this dataset. But since system A does not have FILTER_Z, dataset chunks will be written without it being applied.
HDF5 has a mechanism for determining whether chunks are actually written with the filters specified in the object header, so while the filter remains unavailable, system A will be able to read the data. Once the file is moved to system B, where FILTER_Z is available, HDF5 will apply FILTER_Z to any data rewritten or new data written in these datasets. Dataset chunks that have been written on system B will then be unreadable on system A; chunks that have not been re-written since being written on system A will remain readable on system A. All chunks will be readable on system B.
The following table summarizes optional filter behavior.
not available | FILTER_Z available with encode and decode | FILTER_Z available decode only | |
H5Pset_<FILTER_Z> |
Will succeed. |
Will succeed. | Will succeed. |
H5Dwrite with FILTER_Z set |
Will succeed; FILTER_Z will not be applied to the data. |
Will succeed; FILTER_Z will be applied to the data. |
Will succeed; FILTER_Z will not be applied to the data. |
H5Dread with FILTER_Z set |
Will succeed if FILTER_Z has not actually been applied to data. | Will succeed. | Will succeed. |
The above principles apply generally in the use of
HDF5 optional filters insofar as HDF5 does as much as possible
to complete an operation when an optional filter is unavailable.
(The SZIP filter is an exception to this rule; see
H5Pset_szip
for details.)
Notes:
Filters can be applied only to chunked datasets;
they cannot be used with other dataset storage methods,
such as contiguous, compact, or external datasets.
Dataset elements of variable-length and dataset region reference datatypes
are stored in separate structures in the file called heaps.
Filters cannot currently be applied to these heaps.
HDF5 documents and links Introduction to HDF5 HDF5 User Guide |
And in this document, the
HDF5 Reference Manual H5 H5A H5D H5E H5F H5G H5I H5P H5R H5S H5T H5Z Tools Datatypes |
(PDF of complete manual formatted as print volume) |