Welcome to the netCDF Operator (NCO) Homepage
If you like NCO and want to send us a little money, click here:
Current version is 2.8.1 last updated on Wednesday, 27-Aug-2003 15:02:00 EDT
What is NCO?
The netCDF
Operators, or NCO, are a suite of programs known as operators.
Each operator is a standalone, command line program which is executed at
the UNIX shell-level like, e.g., ls or mkdir.
The operators take netCDF
or HDF4 files as input, then
perform a set of operations (e.g., deriving new data, averaging,
hyperslabbing, or metadata manipulation) and produce a
netCDF file
as output.
The operators are primarily designed to aid manipulation and analysis of
gridded scientific data.
The single command style of NCO allows users to manipulate and analyze
files interactively and with simple scripts, avoiding the overhead
(and some of the power) of a higher level programming environment.
The NCO User's Guide
illustrates their use with examples from the field of climate modeling
and analysis.
The operators are as general as
netCDF
itself: there are no restrictions on the contents of the
netCDF or
HDF4 file(s) used as input.
The internal routines of NCO are completely dynamic which means that
NCO imposes no limit on the number of dimensions, sizes of variables,
files etc.
NCO is designed to be used both interactively and with large batch jobs.
There are plenty of command line (i.e., runtime) options, but the defaults
settings are often sufficient for everyday needs.
NCO works on AIX, Alpha, ARM, *BSD, IRIX, Linux, NEC-SX, OSX, Solaris,
Tru64, UNICOS, and Windows.
The following operators are thoroughly described in the
NCO User's Guide.
Note that the “averagers” are misnamed because they
perform many non-linear operations as well, e.g., total, minimum,
maximum, RMS:
- ncap netCDF Arithmetic Processor
- ncatted netCDF Attribute Editor
- ncbo netCDF Binary Operator (includes ncadd, ncsubtract, ncmultiply, ncdivide)
- ncea netCDF Ensemble Averager
- ncecat netCDF Ensemble Concatenator
- ncflint netCDF File Interpolator
- ncks netCDF Kitchen Sink
- ncra netCDF Record Averager
- ncrcat netCDF Record Concatenator
- ncrename netCDF Renamer
- ncwa netCDF Weighted Averager
How to Contribute
In March, 2000, NCO became an Open Source project hosted by
SourceForge.net in order to facilitate collaboration and code
contribution by a wide variety of people.
There is a TODO list about 200 items long!
No matter what your programming background there is a task you can
help with.
From improving this cheesy webpage, to adding documentation,
to helping design the new features and interfaces for future NCO
capability, we need your help!
The NCO project
homepage contains mailing lists, discussion forums, and
instructions for helping the NCO project.
As of June, 2003, you may use PayPal to donate money to support NCO development.
Your donation will be distributed to NCO developers as an incentive
for them to tackle TODO items on a
pay-per-feature basis.
We will update the following counters to track the influence of
monetary donations on NCO development:
Donations received between 20030624 and 20030818: US$0.00.
NCO features "incentivized" by these donations: None.
NCO News & Announcements
- New “binary” operator ncbo does file addition, subtraction, multiplication, and division
- ncap adds user-invokable type conversion functions
char(), short(), int(), float(),
double().
- ncra versions 2.4.x–2.5.5 handled multifile
averages of low precision (int, char) data incorrectly if
the data had a missing_value.
Under these conditions, this bug could result in incorrect values.
Thanks to Martin Dix for pointing this out.
User's of this feature should definitely upgrade to 2.5.6+.
- libnco_c++ reads into and writes from type long double
- Full text of past NCO announcements
Version Summary
- Most releases since 2.0.0 are not “packaged”.
This means that they are simply tagged in the CVS repository and that
tarballs are generated and stored
here.
Tarballs are only uploaded to SourceForge, and RPMs are only
generated, for those releases marked “Packaged”.
Retrieve these versions using CVS or from
this HTTP site.
- NCO 2.8.1, “Variable wildcarding”—extended regular expressions in -v var,... lists—support in all operators
- NCO 2.8.0, New “binary” operator ncbo, obsoletes ncdiff, Debian integration
- NCO 2.7.4, Autotools build support for AIX, fix UDUnits confusion with floating point hyperslab specifications
- NCO 2.7.3, Support UDUnits (and some CF) conventions, e.g., ncks -d wvl "1 picometer","1 furlong" -d time,"2001-03-19 06:00:0.0","2001-03-19 18:00:0.0"
- NCO 2.7.2, Printing (ncks -H) plays well with multi-slabbing
- NCO 2.7.1, (Packaged) Fallback to B. Sittler's getopt() for
long option support. Store fixed, packed variables correctly with ncap.
- NCO 2.7.0, Support for GNU/POSIX long options,
ncks introduces multi-slabbing (= multiple hyperslabs) capability
- NCO 2.6.6, Fix and document ncrename attribute renaming, add type conversion functions to ncap
- NCO 2.6.5, Cray, Sun build improvements
- NCO 2.6.4, Mac OS X, NEC, SGI build improvements
- NCO 2.6.3, Mac OS X builds work seamlessly, documentation fixed
- NCO 2.6.2, Major improvements in autotools support, ncap supports user-defined dimensions
- NCO 2.6.1, Minor fix to unpacking support
- NCO 2.6.0, All arithmetic operators (ncap, ncdiff, ncea, ncflint, ncra) support unpacking
- NCO 2.5.6, Handle missing_value correctly for integer variables in ncra, merge autoconf build
- NCO 2.5.5, Handle files without
existing global “History” attribute correctly
- NCO 2.5.4, Add support NEC-SX support, update Cray support, small build changes
- NCO 2.5.3, Fix native builds under MS Windows with Cygwin ncap, hint to ncwa
- NCO 2.5.2, Add math functions to ncap, hint to ncwa
- NCO 2.5.1, ncatted deletes all
attributes of att_var when att_nm not specified
- NCO 2.5.0, man(1) pages from Debian distribution
- NCO 2.4.6, Free (speech) sort routines
- NCO 2.4.5, Arithmetic speedups
- NCO 2.4.4, 64-bit AIX builds
- NCO 2.4.3, Handle non-standard “History” attributes
- NCO 2.4.2, ncap supports nesting
files with #include
- NCO 2.4.0, ncks supports writing native binary output with -b switch
- NCO 2.3.1, ncap builds on all Unices
- NCO 2.3.0, Library redesign
- NCO 2.2.4, ncap fully supports packing and unpacking variables
- NCO 2.2.3, ncap supports manually
unpacking variables stored with scale_factor and add_offset
- NCO 2.2.2, fix rare ncwa crash with weights
- NCO 2.2.1, fix ncap divide and fmodf bugs
- NCO 2.2.0, (Packaged) 20020203: ncap public debut (has division error, please upgrade)
- NCO 2.1.3, ncap implements LHS casting
- NCO 2.1.2, C++ API to netcdf, libnco_c++ open for business
- NCO 2.1.0, ncap full lexer/parser operational
- NCO 2.0.3, small fixes to 2.0.0 (e.g., ncks printing)
- NCO 2.0.2, Add packing/unpacking capabilities based on
scale_factor and add_offset convention
- NCO 2.0.1, Add user-configurable multi-threaded capability to arithmetic-intensive operators ncwa, ncea, and ncra
- NCO 2.0.0, (Packaged) 20010507: Switched all internal calls to netCDF 3.x API
- NCO 1.2.2, 20010211: Allow weights with
ncwa in min/max/ttl operations, easier building with HDF4.
ncrename can now work on specific variables rather than whole file.
ncks has -q option for cleaner printing
- NCO 1.2.1, 20000828: Added type-conversion of integers to floating point before arithmetic, fixed broken -v option
- NCO 1.2, 20000730: Added non-linear operations to ncra/ncea and ncwa: min,max,total,rms,rmssdn,sqrt. Added DODS compliance.
- NCO 1.1.48, 20000515: Potentially serious bugfix to ncdiff, recommend upgrade
- NCO 1.1.47, 20000510: Improved ncra and ncrcat to allow lazier specifications of input files
Download/CVS NCO Software
The best way to acquire NCO is with CVS.
The second best way is to download the compressed tarfile
here.
The
NCO CVS Repository
contains up-to-the-minute sources and is the easiest way to stay
synchronized with NCO features.
Retrieving NCO requires some familiarity with GNU development tools,
especially CVS and
Make.
You must first log into the NCO CVS server:
cvs -d:pserver:anonymous@cvs.nco.sourceforge.net:/cvsroot/nco login
Simply hit return when asked for the CVS password.
This logging in procedure is only required the first time you access
the CVS server.
Then you may retrieve any NCO distribution you wish.
Usually you will wish to retrieve a recent tagged (i.e., released)
version.
This command retrieves and places NCO version 2.5.3 into a
local directory named nco-2.5.3:
cvs -z3 -d:pserver:anonymous@cvs.nco.sourceforge.net:/cvsroot/nco co -r nco-2_5_3 -d nco-2.5.3 nco
This command retrieves the current ("bleeding edge") development
version of NCO into a local directory named nco:
cvs -z3 -d:pserver:anonymous@cvs.nco.sourceforge.net:/cvsroot/nco co -kk nco
Track changes to the development version using
cd nco;cvs update -kk
One difference between running a "tagged" release
(e.g., nco-2_5_3) and the development version is that the
tagged release operators will print a valid version number (e.g.,
2.5.3) when asked to do so with the -r flag
(e.g., ncks -r).
The development version simply places today's date in place of the
version.
Once the autotools builds are working more robustly, the confusion
over versions should largely disappear.
Tarballs and pre-built binaries: NCO development is too shorthanded
to upload tarballs to SourceForge (a PITA), or to provide prebuilt
binary executables of any format.
Volunteers or organizations interested in porting and providing NCO
binaries on a variety of platforms (using, e.g.,
Compile Farm or
HP Testdrive sites)
would be very welcome to join the NCO team.
Tarballs are usually up-to-date, but the binaries (e.g., RPM, .deb)
tend to lag the current distribution by many months, if not years:
- Source and documentation in tarball format:
nco.tar.gz ([an error occurred while processing this directive] compressed tar-file)
Debian GNU/Linux-compatible systems:
- nco_2.8.0-1_i386.deb ([an error occurred while processing this directive]): Pre-built executables and libraries, Intel i386-compatible, no documentation (install with apt-get install nco)
- nco-doc_2.8.0-1_all.deb ([an error occurred while processing this directive]): Pre-built documentation, all architectures (install with apt-get install nco-doc)
- nco_2.8.0-1.dsc ([an error occurred while processing this directive]): Description
- nco_2.8.0-1.tar.gz ([an error occurred while processing this directive]): Gzipped Source Tarball
- nco_2.8.0-1_i386.changes ([an error occurred while processing this directive]): Changes
Documentation and User's Guide
View the NCO User's Guide in any of the following formats.
nco.texi is the most up-to-date.
nco.html, nco.info, nco.txt, and
nco.xml are reasonably up-to-date.
Files nco.dvi, nco.ps, and nco.pdf are
out-of-date due to configuration problems with my texinfo
utilities (but you can generate them yourself from nco.texi!)
- DVI Device Independent (xdvi)
- HTML Hypertext (any browser)
- Info GNU Info (M-x Info, emacs)
- PDF Portable Document Format (ghostview, acroread, xpdf)
- Postscript Printing (ghostview)
- TeXInfo Documentation Source code (emacs)
- Text Plain text (more)
- XML Extensible Markup Language (mozilla)
Other documentation:
This
pamphlet and
slide presentation
introduce and summarize the NCO operators.
They were created by Sylvia Murphy of the NCAR
Community Climate System Model Support Network.
README/ChangeLog/TODO
Files containing useful information about the current NCO distribution:
- README Platforms and software requirements
for a successful NCO installation
- ChangeLog Up-to-date history of
NCO changes
- TODO An unordered list describing the features and
fixes we currently have in mind for NCO.
FAQ:
These questions show up almost as frequently as my mother.
But they are more predictable:
- Where can I find prebuilt NCO executables?
If you have an account at UCI, NCAR, or SDSC, try the locations listed below.
Otherwise, you are on your own.
- The executables at NCAR are out of date, where can I get the
latest, greatest version?
You must build them from source yourself. See next question.
- I thought NCAR developed NCO, so why doesn't NCAR support it?
I wrote NCO for fun—not at NCAR's request—while I was at NCAR.
The primary reason NCAR does not support NCO may be because I actively
maintained NCO there, inadvertently creating the impression that NCO
was officially maintained.
NCAR system administrators have recently (2003) begun to support NCO
like other community software packages such as lapack.
If you notice problems with the NCO installation on NCAR machines
please send a message to the relevant system administrators, e.g.,
NCAR User Support.
- Is there an easy way to keep up with new NCO releases?
Subscribe to the nco-announce mailing list (this list is for NCO-related announcements, not for questions)
- Where should I ask my questions on how to use NCO?
On the Help site.
- Where should I post suggestions/comments on NCO features and usage?
On the Discussion site.
- Where is NCO development discussed?
At the Developer
site.
Using NCO at UCI, NCAR, and SDSC
Users at UC Irvine, NCAR, and SDSC may find pre-built,
almost up-to-date NCO executables in the following locations.
These executables are unsupported and are almost
guaranteed to be from different NCO versions and thus to behave
slightly differently.
(They are built from the main trunk, not a tagged version, and
currently it is impossible to tell exactly what version they are).
Moreover they may be compiled with debugging enabled and thus may be
slower than custom built, optimized executables.
- ESS dust.ess.uci.edu (Linux 2.4.7): /home/zender/bin/LINUX
- ESS ess1.ess.uci.edu (Linux 2.2.15): /home/zender/bin/LINUXALPHA
- ESS krein.math.uci.edu (IRIX 6.5): /home/zender/zender/bin/SGIMP64
- CGD network (Solaris 2.6): /home/zender/bin/SUNMP
- CGD/SCD dataproc.ucar.edu (IRIX 6.5): /home/dataproc/zender/bin/SGIMP64
- SCD babyblue.ucar.edu (AIX 5.1.0.35): /home/babyblue/zender/bin/AIX
- SCD blackforest.ucar.edu (AIX 5.1.0.35): /home/blackforest/zender/bin/AIX
- SCD bluesky.ucar.edu (AIX 5.1.0.35): /home/bluesky/zender/bin/AIX
- SCD dave.ucar.edu (AIX 5.1.0.35): /home/dave/zender/bin/AIX
- SCD longs.ucar.edu (Linux 2.4.18): /fs/cgd/home0/zender/bin/LINUX
- SCD utefe.ucar.edu (IRIX 6.5): /home/ute/zender/bin/SGIMP64
- SDSC horizon.npaci.edu (AIX 5.0): /paci/uci/ux453797/bin/AIX
Known Problems with NCO and/or netCDF:
- Generic NCO build problems:
NCO requires recent versions of netCDF (version >= 3.5.0).
NCO requires recent versions of GNU make (version >= 3.78.1).
- Platform-dependent problems with ./configure
(Autotools) build mechanism:
The file configure.eg contains a table which describes the
success you can expect building NCO with the new ./configure
mechanism.
In summary, only Linux platforms can expect trouble-free behavior.
All platforms work continue to work fine with the
bld/Makefile method.
However, this method does not support shared libraries and other nice
features that come automagically with Autotools.
So please send in the patches required to get the ./configure
mechanism working properly for NCO, nco_c++, and DODS on your
platform.
The gory details of the failed (and successful) builds are in the
regressions archive.
- ncap build problems:
ncap requires flex to build correctly.
AT&T lex will not work because it does not understand
EOF rules that ncap_lex.l uses.
bison version 1.29 (the default version on Mac OS X) is buggy
and does not parse ncap_yacc.y correctly; previous and later
versions of bison work correctly.