This is nco.info, produced by gmakeinfo version 7.1 from nco.texi.

INFO-DIR-SECTION netCDF
START-INFO-DIR-ENTRY
* NCO::        User Guide for the netCDF Operator suite
END-INFO-DIR-ENTRY

This file documents NCO, a collection of utilities to manipulate and
analyze netCDF files.

   Copyright © 1995-2025 Charlie Zender

   This is the first edition of the ‘NCO User Guide’,
and is consistent with version 2 of ‘texinfo.tex’.

   Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  The
license is available online at <http://www.gnu.org/copyleft/fdl.html>

   The original author of this software, Charlie Zender, wants to
improve it with the help of your suggestions, improvements, bug-reports,
and patches.
Charlie Zender <surname at uci dot edu> (yes, my surname is zender)
3200 Croul Hall
Department of Earth System Science
University of California, Irvine
Irvine, CA 92697-3100


File: nco.info,  Node: Top,  Next: Foreword,  Prev: (dir),  Up: (dir)

NCO User Guide
**************

_Note to readers of the NCO User Guide in Info format_: _The NCO User
Guide in PDF format (./nco.pdf) (also on SourceForge
(http://nco.sf.net/nco.pdf)) contains the complete NCO documentation._
This Info documentation is equivalent except it refers you to the
printed (i.e., DVI, PostScript, and PDF) documentation for description
of complex mathematical expressions.

   The netCDF Operators, or NCO, are a suite of programs known as
operators.  The operators facilitate manipulation and analysis of data
stored in the self-describing netCDF format, available from
(<http://www.unidata.ucar.edu/software/netcdf>).  Each NCO operator
(e.g., ‘ncks’) takes netCDF input file(s), performs an operation (e.g.,
averaging, hyperslabbing, or renaming), and outputs a processed netCDF
file.  Although most users of netCDF data are involved in scientific
research, these data formats, and thus NCO, are generic and are equally
useful in fields from agriculture to zoology.  The NCO User Guide
illustrates NCO use with examples from the field of climate modeling and
analysis.  The NCO homepage is <http://nco.sf.net>, and the source code
is maintained at <http://github.com/nco/nco>.

   This documentation is for NCO version 5.3.6.  It was last updated 28
October 2025.  Corrections, additions, and rewrites of this
documentation are gratefully welcome.

   Enjoy,
Charlie Zender

* Menu:

* Foreword::
* Summary::
* Introduction::
* Strategies::
* Shared features::
* Reference Manual::
* Contributing::
* Quick Start::
* CMIP5 Example::
* Parallel::
* CCSM Example::
* mybibnode::
* General Index::


File: nco.info,  Node: Foreword,  Next: Summary,  Prev: Top,  Up: Top

Foreword
********

NCO is the result of software needs that arose while I worked on
projects funded by NCAR, NASA, and ARM.  Thinking they might prove
useful as tools or templates to others, it is my pleasure to provide
them freely to the scientific community.  Many users (most of whom I
have never met) have encouraged the development of NCO.  Thanks
espcially to Jan Polcher, Keith Lindsay, Arlindo da Silva, John Sheldon,
and William Weibel for stimulating suggestions and correspondence.  Your
encouragment motivated me to complete the ‘NCO User Guide’.  So if you
like NCO, send me a note!  I should mention that NCO is not connected to
or officially endorsed by Unidata, ACD, ASP, CGD, or Nike.

Charlie Zender
May 1997
Boulder, Colorado



   Major feature improvements entitle me to write another Foreword.  In
the last five years a lot of work has been done to refine NCO.  NCO is
now an open source project and appears to be much healthier for it.  The
list of illustrious institutions that do not endorse NCO continues to
grow, and now includes UCI.

Charlie Zender
October 2000
Irvine, California



   The most remarkable advances in NCO capabilities in the last few
years are due to contributions from the Open Source community.
Especially noteworthy are the contributions of Henry Butowsky and Rorik
Peterson.

Charlie Zender
January 2003
Irvine, California



   NCO was generously supported from 2004-2008 by US National Science
Foundation (NSF) grant IIS-0431203
(http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0431203).  This
support allowed me to maintain and extend core NCO code, and others to
advance NCO in new directions: Gayathri Venkitachalam helped implement
MPI; Harry Mangalam improved regression testing and benchmarking; Daniel
Wang developed the server-side capability, SWAMP; and Henry Butowsky, a
long-time contributor, developed ‘ncap2’.  This support also led NCO to
debut in professional journals and meetings.  The personal and
professional contacts made during this evolution have been immensely
rewarding.

Charlie Zender
March 2008
Grenoble, France



   The end of the NSF SEI grant in August, 2008 curtailed NCO
development.  Fortunately we could justify supporting Henry Butowsky on
other research grants until May, 2010 while he developed the key ‘ncap2’
features used in our climate research.  And recently the NASA ACCESS
program commenced funding us to support netCDF4 group functionality.
Thus NCO will grow and evade bit-rot for the foreseeable future.

   I continue to receive with gratitude the thanks of NCO users at
nearly every scientific meeting I attend.  People introduce themselves,
shake my hand and extol NCO, often effusively, while I grin in stupid
embarassment.  These exchanges lighten me like anti-gravity.  Sometimes
I daydream how many hours NCO has turned from grunt work to productive
research for researchers world-wide, or from research into early
happy-hours.  It's a cool feeling.


Charlie Zender
April, 2012
Irvine, California



   The NASA ACCESS 2011 program generously supported (Cooperative
Agreement NNX12AF48A) NCO from 2012-2014.  This allowed us to produce
the first iteration of a Group-oriented Data Analysis and Distribution
(GODAD) software ecosystem.  Shifting more geoscience data analysis to
GODAD is a long-term plan.  Then the NASA ACCESS 2013 program agreed to
support (Cooperative Agreement NNX14AH55A) NCO from 2014-2016.  This
support permits us to implement support for Swath-like Data (SLD).  Most
recently, the DOE has funded me to implement NCO re-gridding and
parallelization in support of their ACME program.  After many years of
crafting NCO as an after-hours hobby, I finally have the cushion
necessary to give it some real attention.  And I'm looking forward to
this next, and most intense yet, phase of NCO development.

Charlie Zender
June, 2015
Irvine, California

   The DOE Energy Exascale Earth System Model (E3SM) project (formerly
ACME) has generously supported NCO development for the past four years.
Supporting NCO for a mission-driven, high-performance climate model
development effort has brought unprecedented challenges and
opportunities.  After so many years of staid progress, the recent
development speed has been both exhilirating and terrifying.


Charlie Zender
May, 2019
Laguna Beach, California

   The DOE E3SM project has supported NCO development and maintenance
since 2015.  This is an eternity in the world of research funding!
Their reliable support has enabled us to add cutting-edge features
including quantization, vertical interpolation, and support for multiple
regridding weight-generators.  Recently NSF supported us to enable
user-friendly support for modern compression algorithms that can make
geoscience data analysis greener by reducing dataset size, and thereby
storage, power, and associated greenhouse gas emissions.  I am grateful
for this this agency support that inspires me to create new features
that help my amazing colleagues pursue their scientific ideas.


Charlie Zender
July, 2022
Laguna Beach, California

   The DOE E3SM project, NASA OSTFL program, and NSF CSSI program all
supported NCO for the past three years.  Our quantization and
compression features are now mature and implemented in the netCDF
C-Library where they can be used by other toolkits and languages.  These
IPCC AR7 CMIP7 model intercomparison project will allow use of these
features.  This lays the groundwork for their widespread adoption
throughout geoscientific computing, potentially saving enormous amounts
of storage and reducing its attendant power requirements and GHG
emissions.  Help from my colleagues at CF, CMOR, E3SM, NASA, NOAA, and
Unidata has been indispensable.  Unfortunately the future of US-based
support for these organizations and collaborators is dire due to the
climate denialism and science budget cutbacks of the current
administration.  Thankfully the University of California (though it too
is threatened by the federal government) still affords me the
independence to continue maintaining and improving NCO for years to
come.


Charlie Zender
September, 2025
Santa Fe, New Mexico


File: nco.info,  Node: Summary,  Next: Introduction,  Prev: Foreword,  Up: Top

Summary
*******

This manual describes NCO, which stands for netCDF Operators.  NCO is a
suite of programs known as “operators”.  Each operator is a standalone,
command line program executed at the shell-level like, e.g., ‘ls’ or
‘mkdir’.  The operators take netCDF files (including HDF5 files
constructed using the netCDF API) as input, perform an operation (e.g.,
averaging or hyperslabbing), and produce a netCDF file as output.  The
operators are primarily designed to aid manipulation and analysis of
data.  The examples in this documentation are typical applications of
the operators for processing climate model output.  This stems from
their origin, though the operators are as general as netCDF itself.


File: nco.info,  Node: Introduction,  Next: Strategies,  Prev: Summary,  Up: Top

1 Introduction
**************

* Menu:

* Availability::
* How to Use This guide::
* Compatability::
* Symbolic Links::
* Libraries::
* netCDF2/3/4 and HDF4/5 Support::
* Help Requests and Bug Reports::


File: nco.info,  Node: Availability,  Next: How to Use This guide,  Prev: Introduction,  Up: Introduction

1.1 Availability
================

The complete NCO source distribution is currently distributed as a
“compressed tarfile” from <http://sf.net/projects/nco> and from
<http://dust.ess.uci.edu/nco/nco.tar.gz>.  The compressed tarfile must
be uncompressed and untarred before building NCO.  Uncompress the file
with ‘gunzip nco.tar.gz’.  Extract the source files from the resulting
tarfile with ‘tar -xvf nco.tar’.  GNU ‘tar’ lets you perform both
operations in one step with ‘tar -xvzf nco.tar.gz’.

   The documentation for NCO is called the ‘NCO User Guide’.  The ‘User
Guide’ is available in PDF, Postscript, HTML, DVI, TeXinfo, and Info
formats.  These formats are included in the source distribution in the
files ‘nco.pdf’, ‘nco.ps’, ‘nco.html’, ‘nco.dvi’, ‘nco.texi’, and
‘nco.info*’, respectively.  All the documentation descends from a single
source file, ‘nco.texi’ (1).  Hence the documentation in every format is
very similar.  However, some of the complex mathematical expressions
needed to describe ‘ncwa’ can only be displayed in DVI, Postscript, and
PDF formats.

   A complete list of papers and publications on/about NCO is available
on the NCO homepage.  Most of these are freely available.  The primary
refereed publications are ZeM06 and Zen08.  These contain copyright
restrictions which limit their redistribution, but they are freely
available in preprint form from the NCO.

   If you want to quickly see what the latest improvements in NCO are
(without downloading the entire source distribution), visit the NCO
homepage at <http://nco.sf.net>.  The HTML version of the ‘User Guide’
is also available online through the World Wide Web at URL
<http://nco.sf.net/nco.html>.  To build and use NCO, you must have
netCDF installed.  The netCDF homepage is
<http://www.unidata.ucar.edu/software/netcdf>.

   New NCO releases are announced on the netCDF list and on the
‘nco-announce’ mailing list
<http://lists.sf.net/mailman/listinfo/nco-announce>.

   ---------- Footnotes ----------

   (1) To produce these formats, ‘nco.texi’ was simply run through the
freely available programs ‘texi2dvi’, ‘dvips’, ‘texi2html’, and
‘makeinfo’.  Due to a bug in TeX, the resulting Postscript file,
‘nco.ps’, contains the Table of Contents as the final pages.  Thus if
you print ‘nco.ps’, remember to insert the Table of Contents after the
cover sheet before you staple the manual.


File: nco.info,  Node: How to Use This guide,  Next: Compatability,  Prev: Availability,  Up: Introduction

1.2 How to Use This Guide
=========================

Detailed instructions about how to download the newest version
(http://nco.sf.net/#Source), and how to complie source code
(http://nco.sf.net/#bld), as well as a FAQ (http://nco.sf.net/#FAQ) and
descriptions of Known Problems (http://nco.sf.net/#bug) etc.  are on our
homepage (<http://nco.sf.net/>).

   There are twelve operators in the current version (5.3.6).  The
function of each is explained in *note Reference Manual: Reference
Manual.  Many of the tasks that NCO can accomplish are described during
the explanation of common NCO Features (*note Shared features::).  More
specific use examples for each operator can be seen by visiting the
operator-specific examples in the *note Reference Manual::.  These can
be found directly by prepending the operator name with the ‘xmp_’ tag,
e.g., <http://nco.sf.net/nco.html#xmp_ncks>.  Also, users can type the
operator name on the shell command line to see all the available
options, or type, e.g., ‘man ncks’ to see a help man-page.

   NCO is a command-line language.  You may either use an operator after
the prompt (e.g., ‘$’ here), like,
     $ operator [options] input [output]
   or write all commands lines into a shell script, as in the CMIP5
Example (*note CMIP5 Example::).

   If you are new to NCO, the Quick Start (*note Quick Start::) shows
simple examples about how to use NCO on different kinds of data files.
More detailed "real-world" examples are in the *note CMIP5 Example:
CMIP5 Example.  The *note Index: General Index. is presents multiple
keyword entries for the same subject.  If these resources do not help
enough, please *note Help Requests and Bug Reports::.


File: nco.info,  Node: Compatability,  Next: Symbolic Links,  Prev: How to Use This guide,  Up: Introduction

1.3 Operating systems compatible with NCO
=========================================

In its time on Earth, NCO has been successfully ported and tested on so
many 32- and 64-bit platforms that if we did not write them down here we
would forget their names: IBM AIX 4.x, 5.x, FreeBSD 4.x, GNU/Linux 2.x,
LinuxPPC, LinuxAlpha, LinuxARM, LinuxSparc64, LinuxAMD64, SGI IRIX 5.x
and 6.x, MacOS X 10.x, DEC OSF, NEC Super-UX 10.x, Sun SunOS 4.1.x,
Solaris 2.x, Cray UNICOS 8.x-10.x, and Microsoft Windows (95, 98, NT,
2000, XP, Vista, 7, 8, 10).  If you port the code to a new operating
system, please send me a note and any patches you required.

   The major prerequisite for installing NCO on a particular platform is
the successful, prior installation of the netCDF library (and, as of
2003, the UDUnits library).  Unidata has shown a commitment to
maintaining netCDF and UDUnits on all popular UNIX platforms, and is
moving towards full support for the Microsoft Windows operating system
(OS).  Given this, the only difficulty in implementing NCO on a
particular platform is standardization of various C-language API system
calls.  NCO code is tested for ANSI compliance by compiling with
C99 compilers including those from GNU (‘gcc -std=c99 -pedantic
-D_BSD_SOURCE -D_POSIX_SOURCE’ -Wall) (1), Comeau Computing (‘como
--c99’), Cray (‘cc’), HP/Compaq/DEC (‘cc’), IBM (‘xlc -c
-qlanglvl=extc99’), Intel (‘icc -std=c99’), LLVM (‘clang’), NEC (‘cc’),
PathScale (QLogic) (‘pathcc -std=c99’), PGI (‘pgcc -c9x’), SGI (‘cc
-c99’), and Sun (‘cc’).  NCO (all commands and the ‘libnco’ library) and
the C++ interface to netCDF (called ‘libnco_c++’) comply with the ISO
C++ standards as implemented by Comeau Computing (‘como’), Cray (‘CC’),
GNU (‘g++ -Wall’), HP/Compaq/DEC (‘cxx’), IBM (‘xlC’), Intel (‘icc’),
Microsoft (‘MVS’), NEC (‘c++’), PathScale (Qlogic) (‘pathCC’), PGI
(‘pgCC’), SGI (‘CC -LANG:std’), and Sun (‘CC -LANG:std’).  See
‘nco/bld/Makefile’ and ‘nco/src/nco_c++/Makefile.old’ for more details
and exact settings.

   Until recently (and not even yet), ANSI-compliant has meant
compliance with the 1989 ISO C-standard, usually called C89 (with minor
revisions made in 1994 and 1995).  C89 lacks variable-size arrays,
restricted pointers, some useful ‘printf’ formats, and many mathematical
special functions.  These are valuable features of C99, the 1999 ISO
C-standard.  NCO is C99-compliant where possible and C89-compliant where
necessary.  Certain branches in the code are required to satisfy the
native SGI and SunOS C compilers, which are strictly ANSI C89 compliant,
and cannot benefit from C99 features.  However, C99 features are fully
supported by modern AIX, GNU, Intel, NEC, Solaris, and UNICOS compilers.
NCO requires a C99-compliant compiler as of NCO version 2.9.8, released
in August, 2004.

   The most time-intensive portion of NCO execution is spent in
arithmetic operations, e.g., multiplication, averaging, subtraction.
These operations were performed in Fortran by default until August,
1999.  This was a design decision based on the relative speed of
Fortran-based object code vs. C-based object code in late 1994.
C compiler vectorization capabilities have dramatically improved since
1994.  We have accordingly replaced all Fortran subroutines with
C functions.  This greatly simplifies the task of building NCO on
nominally unsupported platforms.  As of August 1999, NCO built entirely
in C by default.  This allowed NCO to compile on any machine with an
ANSI C compiler.  In August 2004, the first C99 feature, the ‘restrict’
type qualifier, entered NCO in version 2.9.8.  C compilers can obtain
better performance with C99 restricted pointers since they inform the
compiler when it may make Fortran-like assumptions regarding pointer
contents alteration.  Subsequently, NCO requires a C99 compiler to build
correctly (2).

   In January 2009, NCO version 3.9.6 was the first to link to the GNU
Scientific Library (GSL).  GSL must be version 1.4 or later.  NCO, in
particular ‘ncap2’, uses the GSL special function library to evaluate
geoscience-relevant mathematics such as Bessel functions, Legendre
polynomials, and incomplete gamma functions (*note GSL special
functions::).

   In June 2005, NCO version 3.0.1 began to take advantage of C99
mathematical special functions.  These include the standarized gamma
function (called ‘tgamma()’ for "true gamma").  NCO automagically takes
advantage of some GNU Compiler Collection (GCC) extensions to ANSI C.

   As of July 2000 and NCO version 1.2, NCO no longer performs
arithmetic operations in Fortran.  We decided to sacrifice executable
speed for code maintainability.  Since no objective statistics were ever
performed to quantify the difference in speed between the Fortran and
C code, the performance penalty incurred by this decision is unknown.
Supporting Fortran involves maintaining two sets of routines for every
arithmetic operation.  The ‘USE_FORTRAN_ARITHMETIC’ flag is still
retained in the ‘Makefile’.  The file containing the Fortran code,
‘nco_fortran.F’, has been deprecated but a volunteer (Dr. Frankenstein?)
could resurrect it.  If you would like to volunteer to maintain
‘nco_fortran.F’ please contact me.

* Menu:

* Windows Operating System::

   ---------- Footnotes ----------

   (1) The ‘_BSD_SOURCE’ token is required on some Linux platforms where
‘gcc’ dislikes the network header files like ‘netinet/in.h’).

   (2) NCO may still build with an ANSI or ISO C89 or C94/95-compliant
compiler if the C pre-processor undefines the ‘restrict’ type qualifier,
e.g., by invoking the compiler with ‘-Drestrict=''’.


File: nco.info,  Node: Windows Operating System,  Prev: Compatability,  Up: Compatability

1.3.1 Compiling NCO for Microsoft Windows OS
--------------------------------------------

NCO has been successfully ported and tested on most Microsoft Windows
operating systems including: XP SP2/Vista/7/10.  Support is provided for
compiling either native Windows executables, using the Microsoft Visual
Studio Compiler (MVSC), or with Cygwin, the UNIX-emulating compatibility
layer with the GNU toolchain.  The switches necessary to accomplish both
are included in the standard distribution of NCO.

   With Microsoft Visual Studio compiler, one must build NCO with C++
since MVSC does not support C99.  Support for Qt, a convenient
integrated development environment, was deprecated in 2017.  As of NCO
version 4.6.9 (September, 2017) please build native Windows executables
with CMake:
     cd ~/nco/cmake
     cmake .. -DCMAKE_INSTALL_PREFIX=${HOME}
     make install
   The file ‘nco/cmake/build.bat’ shows how deal with various path
issues.

   As of NCO version 4.7.1 (December, 2017) the Conda package for NCO is
available from the ‘conda-forge’ channel on all three smithies: Linux,
MacOS, and Windows.
     # Recommended install with Conda
     conda config --add channels conda-forge # Permananently add conda-forge
     conda install nco
     # Or, specify conda-forge explicitly as a one-off:
     conda install -c conda-forge nco

   Using the freely available Cygwin (formerly gnu-win32) development
environment (1), the compilation process is very similar to installing
NCO on a UNIX system.  Set the ‘PVM_ARCH’ preprocessor token to ‘WIN32’.
Note that defining ‘WIN32’ has the side effect of disabling Internet
features of NCO (see below).  NCO should now build like it does on UNIX.

   The least portable section of the code is the use of standard UNIX
and Internet protocols (e.g., ‘ftp’, ‘rcp’, ‘scp’, ‘sftp’, ‘getuid’,
‘gethostname’, and header files ‘<arpa/nameser.h>’ and ‘<resolv.h>’).
Fortunately, these UNIX-y calls are only invoked by the single NCO
subroutine which is responsible for retrieving files stored on remote
systems (*note Remote storage::).  In order to support NCO on the
Microsoft Windows platforms, this single feature was disabled (on
Windows OS only).  This was required by Cygwin 18.x--newer versions of
Cygwin may support these protocols (let me know if this is the case).
The NCO operators should behave identically on Windows and UNIX
platforms in all other respects.

   ---------- Footnotes ----------

   (1) The Cygwin package is available from
‘http://sourceware.redhat.com/cygwin’
Currently, Cygwin 20.x comes with the GNU C/C++ compilers (‘gcc’, ‘g++’.
These GNU compilers may be used to build the netCDF distribution itself.


File: nco.info,  Node: Symbolic Links,  Next: Libraries,  Prev: Compatability,  Up: Introduction

1.4 Symbolic Links
==================

NCO relies on a common set of underlying algorithms.  To minimize
duplication of source code, multiple operators sometimes share the same
underlying source.  This is accomplished by symbolic links from a single
underlying executable program to one or more invoked executable names.
For example, ‘nces’ and ‘ncrcat’ are symbolically linked to the ‘ncra’
executable.  The ‘ncra’ executable behaves slightly differently based on
its invocation name (i.e., ‘argv[0]’), which can be ‘nces’, ‘ncra’, or
‘ncrcat’.  Logically, these are three different operators that happen to
share the same executable.

   For historical reasons, and to be more user friendly, multiple
synonyms (or pseudonyms) may refer to the same operator invoked with
different switches.  For example, ‘ncdiff’ is the same as ‘ncbo’ and
‘ncpack’ is the same as ‘ncpdq’.  We implement the symbolic links and
synonyms by the executing the following UNIX commands in the directory
where the NCO executables are installed.
     ln -s -f ncbo ncdiff    # ncbo --op_typ='-'
     ln -s -f ncra nces      # ncra --pseudonym='nces'
     ln -s -f ncra ncrcat    # ncra --pseudonym='ncrcat'
     ln -s -f ncbo ncadd     # ncbo --op_typ='+'
     ln -s -f ncbo ncsubtract # ncbo --op_typ='-'
     ln -s -f ncbo ncmultiply # ncbo --op_typ='*'
     ln -s -f ncbo ncdivide   # ncbo --op_typ='/'
     ln -s -f ncpdq ncpack    # ncpdq
     ln -s -f ncpdq ncunpack  # ncpdq --unpack
     # NB: Windows/Cygwin executable/link names have '.exe' suffix, e.g.,
     ln -s -f ncbo.exe ncdiff.exe
     ...
   The imputed command called by the link is given after the comment.
As can be seen, some these links impute the passing of a command line
argument to further modify the behavior of the underlying executable.
For example, ‘ncdivide’ is a pseudonym for ‘ncbo --op_typ='/'’.


File: nco.info,  Node: Libraries,  Next: netCDF2/3/4 and HDF4/5 Support,  Prev: Symbolic Links,  Up: Introduction

1.5 Libraries
=============

Like all executables, the NCO operators can be built using dynamic
linking.  This reduces the size of the executable and can result in
significant performance enhancements on multiuser systems.
Unfortunately, if your library search path (usually the
‘LD_LIBRARY_PATH’ environment variable) is not set correctly, or if the
system libraries have been moved, renamed, or deleted since NCO was
installed, it is possible NCO operators will fail with a message that
they cannot find a dynamically loaded (aka “shared object” or ‘.so’)
library.  This will produce a distinctive error message, such as
‘ld.so.1: /usr/local/bin/nces: fatal: libsunmath.so.1: can't open file:
errno=2’.  If you received an error message like this, ask your system
administrator to diagnose whether the library is truly missing (1), or
whether you simply need to alter your library search path.  As a final
remedy, you may re-compile and install NCO with all operators statically
linked.

   ---------- Footnotes ----------

   (1) The ‘ldd’ command, if it is available on your system, will tell
you where the executable is looking for each dynamically loaded library.
Use, e.g., ‘ldd `which nces`’.


File: nco.info,  Node: netCDF2/3/4 and HDF4/5 Support,  Next: Help Requests and Bug Reports,  Prev: Libraries,  Up: Introduction

1.6 netCDF2/3/4 and HDF4/5 Support
==================================

netCDF version 2 was released in 1993.  NCO (specifically ‘ncks’) began
soon after this in 1994.  netCDF 3.0 was released in 1996, and we were
not exactly eager to convert all code to the newer, less tested netCDF
implementation.  One netCDF3 interface call (‘nc_inq_libvers’) was added
to NCO in January, 1998, to aid in maintainance and debugging.  In
March, 2001, the final NCO conversion to netCDF3 was completed
(coincidentally on the same day netCDF 3.5 was released).  NCO
versions 2.0 and higher are built with the ‘-DNO_NETCDF_2’ flag to
ensure no netCDF2 interface calls are used.

   However, the ability to compile NCO with only netCDF2 calls is worth
maintaining because HDF version 4, aka HDF4 or simply HDF, (1)
(available from HDF (http://hdfgroup.org)) supports only the netCDF2
library calls (see
<http://hdfgroup.org/UG41r3_html/SDS_SD.fm12.html#47784>).  There are
two versions of HDF.  Currently HDF version 4.x supports the full
netCDF2 API and thus NCO version 1.2.x.  If NCO version 1.2.x (or
earlier) is built with only netCDF2 calls then all NCO operators should
work with HDF4 files as well as netCDF files (2).  The preprocessor
token ‘NETCDF2_ONLY’ exists in NCO version 1.2.x to eliminate all
netCDF3 calls.  Only versions of NCO numbered 1.2.x and earlier have
this capability.

   HDF version 5 became available in 1999, but did not support netCDF
(or, for that matter, Fortran) as of December 1999.  By early 2001, HDF5
did support Fortran90.  Thanks to an NSF-funded "harmonization"
partnership, HDF began to fully support the netCDF3 read interface
(which is employed by NCO 2.x and later).  In 2004, Unidata and THG
began a project to implement the HDF5 features necessary to support the
netCDF API. NCO version 3.0.3 added support for reading/writing
netCDF4-formatted HDF5 files in October, 2005.  See *note File Formats
and Conversion:: for more details.

   HDF support for netCDF was completed with HDF5 version version 1.8 in
2007.  The netCDF front-end that uses this HDF5 back-end was completed
and released soon after as netCDF version 4.  Download it from the
netCDF4 (http://my.unidata.ucar.edu/content/software/netcdf/netcdf-4)
website.

   NCO version 3.9.0, released in May, 2007, added support for all
netCDF4 atomic data types except ‘NC_STRING’.  Support for ‘NC_STRING’,
including ragged arrays of strings, was finally added in version 3.9.9,
released in June, 2009.  Support for additional netCDF4 features has
been incremental.  We add one netCDF4 feature at a time.  You must build
NCO with netCDF4 to obtain this support.

   NCO supports many netCDF4 features including atomic data types,
Lempel-Ziv compression (deflation), chunking, and groups.  The new
atomic data types are ‘NC_UBYTE’, ‘NC_USHORT’, ‘NC_UINT’, ‘NC_INT64’,
and ‘NC_UINT64’.  Eight-byte integer support is an especially useful
improvement from netCDF3.  All NCO operators support these types, e.g.,
‘ncks’ copies and prints them, ‘ncra’ averages them, and ‘ncap2’
processes algebraic scripts with them.  ‘ncks’ prints compression
information, if any, to screen.

   NCO version 3.9.1 (June, 2007) added support for netCDF4 Lempel-Ziv
deflation.  Lempel-Ziv deflation is a lossless compression technique.
See *note Deflation:: for more details.

   NCO version 3.9.9 (June, 2009) added support for netCDF4 chunking in
‘ncks’ and ‘ncecat’.  NCO version 4.0.4 (September, 2010) completed
support for netCDF4 chunking in the remaining operators.  See *note
Chunking:: for more details.

   NCO version 4.2.2 (October, 2012) added support for netCDF4 groups in
‘ncks’ and ‘ncecat’.  Group support for these operators was complete
(e.g., regular expressions to select groups and Group Path Editing) as
of NCO version 4.2.6 (March, 2013).  See *note Group Path Editing:: for
more details.  Group support for all other operators was finished in the
NCO version 4.3.x series completed in December, 2013.

   Support for netCDF4 in the first arithmetic operator, ‘ncbo’, was
introduced in NCO version 4.3.0 (March, 2013).  NCO version 4.3.1 (May,
2013) completed this support and introduced the first example of
automatic group broadcasting.  See *note ncbo netCDF Binary Operator::
for more details.

   netCDF4-enabled NCO handles netCDF3 files without change.  In
addition, it automagically handles netCDF4 (HDF5) files: If you feed NCO
netCDF3 files, it produces netCDF3 output.  If you feed NCO netCDF4
files, it produces netCDF4 output.  Use the handy-dandy ‘-4’ switch to
request netCDF4 output from netCDF3 input, i.e., to convert netCDF3 to
netCDF4.  See *note File Formats and Conversion:: for more details.

   When linked to a netCDF library that was built with HDF4 support (3),
NCO automatically supports reading HDF4 files and writing them as
netCDF3/netCDF4/HDF5 files.  NCO can only write through the netCDF API,
which can only write netCDF3/netCDF4/HDF5 files.  So NCO can _read_ HDF4
files, perform manipulations and calculations, and then it must _write_
the results in netCDF format.

   NCO support for HDF4 has been quite functional since December, 2013.
For best results install NCO versions 4.4.0 or later on top of netCDF
versions 4.3.1 or later.  Getting to this point has been an iterative
effort where Unidata improved netCDF library capabilities in response to
our requests.  NCO versions 4.3.6 and earlier do not explicitly support
HDF4, yet should work with HDF4 if compiled with a version of netCDF
(4.3.2 or later?)  that does not unexpectedly die when probing HDF4
files with standard netCDF calls.  NCO versions 4.3.7-4.3.9
(October-December, 2013) use a special flag to circumvent netCDF HDF4
issues.  The user must tell these versions of NCO that an input file is
HDF4 format by using the ‘--hdf4’ switch.

   When compiled with netCDF version 4.3.1 (20140116) or later, NCO
versions 4.4.0 (January, 2014) and later more gracefully handle HDF4
files.  In particular, the ‘--hdf4’ switch is obsolete.  Current
versions of NCO use netCDF to determine automatically whether the
underlying file is HDF4, and then take appropriate precautions to avoid
netCDF4 API calls that fail when applied to HDF4 files (e.g.,
‘nc_inq_var_chunking()’, ‘nc_inq_var_deflate()’).  When compiled with
netCDF version 4.3.2 (20140423) or earlier, NCO will report that
chunking and deflation properties of HDF4 files as ‘HDF4_UNKNOWN’,
because determining those properties was impossible.  When compiled with
netCDF version 4.3.3-rc2 (20140925) or later, NCO versions 4.4.6
(October, 2014) and later fully support chunking and deflation features
of HDF4 files.  Unfortunately, netCDF version 4.7.4 (20200327)
introduced a regression that breaks this functionality for all NCO
versions until we first noticed the regression a year later and
implemented a workaround to restore this functionality as of
4.9.9-alpha02 (20210327).  The ‘--hdf4’ switch is supported (for
backwards compatibility) yet redundant (i.e., does no harm) with current
versions of NCO and netCDF.

   Converting HDF4 files to netCDF: Since NCO reads HDF4 files natively,
it is now easy to convert HDF4 files to netCDF files directly, e.g.,
     ncks        fl.hdf fl.nc # Convert HDF4->netCDF4 (NCO 4.4.0+, netCDF 4.3.1+)
     ncks --hdf4 fl.hdf fl.nc # Convert HDF4->netCDF4 (NCO 4.3.7-4.3.9)
   The most efficient and accurate way to convert HDF4 data to netCDF
format is to convert to netCDF4 using NCO as above.  Many HDF4 producers
(NASA!)  love to use netCDF4 types, e.g., unsigned bytes, so this
procedure is the most typical.  Conversion of HDF4 to netCDF4 as above
suffices when the data will only be processed by NCO and other
netCDF4-aware tools.

   However, many tools are not fully netCDF4-aware, and so conversion to
netCDF3 may be desirable.  Obtaining any netCDF file from an HDF4 is
easy:
     ncks -3 fl.hdf fl.nc      # HDF4->netCDF3 (NCO 4.4.0+, netCDF 4.3.1+)
     ncks -4 fl.hdf fl.nc      # HDF4->netCDF4 (NCO 4.4.0+, netCDF 4.3.1+)
     ncks -6 fl.hdf fl.nc      # HDF4->netCDF3 64-bit  (NCO 4.4.0+, ...)
     ncks -7 -L 1 fl.hdf fl.nc # HDF4->netCDF4 classic (NCO 4.4.0+, ...)
     ncks --hdf4 -3 fl.hdf fl.nc # HDF4->netCDF3 (netCDF 4.3.0-)
     ncks --hdf4 -4 fl.hdf fl.nc # HDF4->netCDF4 (netCDF 4.3.0-)
     ncks --hdf4 -6 fl.hdf fl.nc # HDF4->netCDF3 64-bit  (netCDF 4.3.0-)
     ncks --hdf4 -7 fl.hdf fl.nc # HDF4->netCDF4 classic (netCDF 4.3.0-)
   As of NCO version 4.4.0 (January, 2014), these commands work even
when the HDF4 file contains netCDF4 atomic types (e.g., unsigned bytes,
64-bit integers) because NCO can autoconvert everything to atomic types
supported by netCDF3 (4).

   As of NCO version 4.4.4 (May, 2014) both ‘ncl_convert2nc’ and NCO
have built-in, automatic workarounds to handle element names that
contain characters that are legal in HDF though are illegal in netCDF.
For example, slashes and leading special characters are are legal in HDF
and illegal in netCDF element (i.e., group, variable, dimension, and
attribute) names.  NCO converts these forbidden characters to
underscores, and retains the original names of variables in
automatically produced attributes named ‘hdf_name’ (5).

   Finally, in February 2014, we learned that the HDF group has a
project called H4CF (described here
(http://hdfeos.org/software/h4cflib.php)) whose goal is to make HDF4
files accessible to CF tools and conventions.  Their project includes a
tool named ‘h4tonccf’ that converts HDF4 files to netCDF3 or netCDF4
files.  We are not yet sure what advantages or features ‘h4tonccf’ has
that are not in NCO, though we suspect both methods have their own
advantages.  Corrections welcome.

   As of 2012, netCDF4 is relatively stable software.  Problems with
netCDF4 and HDF libraries have mainly been fixed.  Binary NCO
distributions shipped as RPMs and as debs have used the netCDF4 library
since 2010 and 2011, respectively.

   One must often build NCO from source to obtain netCDF4 support.
Typically, one specifies the root of the netCDF4 installation directory.
Do this with the ‘NETCDF4_ROOT’ variable.  Then use your preferred NCO
build mechanism, e.g.,
     export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location
     cd ~/nco;./configure --enable-netcdf4  # Configure mechanism -or-
     cd ~/nco/bld;./make NETCDF4=Y allinone # Old Makefile mechanism

   We carefully track the netCDF4 releases, and keep the netCDF4 atomic
type support and other features working.  Our long term goal is to
utilize more of the extensive new netCDF4 feature set.  The next major
netCDF4 feature we are likely to utilize is parallel I/O. We will enable
this in the MPI netCDF operators.

   ---------- Footnotes ----------

   (1) The Hierarchical Data Format, or HDF, is another self-describing
data format similar to, but more elaborate than, netCDF. HDF comes in
two flavors, HDF4 and HDF5.  Often people use the shorthand HDF to refer
to the older format HDF4.  People almost always use HDF5 to refer to
HDF5.

   (2) One must link the NCO code to the HDF4 MFHDF library instead of
the usual netCDF library.  Apparently ‘MF’ stands for Multi-file not for
Mike Folk.  In any case, until about 2007 the MFHDF library only
supported netCDF2 calls.  Most people will never again install NCO 1.2.x
and so will never use NCO to write HDF4 files.  It is simply too much
trouble.

   (3) The procedure for doing this is documented at
<http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html>.

   (4) Prior to NCO version 4.4.0 (January, 2014), we recommended the
‘ncl_convert2nc’ tool to convert HDF to netCDF3 when both these are
true: 1.  You must have netCDF3 and 2.  the HDF file contains netCDF4
atomic types.  More recent versions of NCO handle this problem fine, and
include other advantages so we no longer recommend ‘ncl_convert2nc’
because ‘ncks’ is faster and more space-efficient.  Both automatically
convert netCDF4 types to netCDF3 types, yet ‘ncl_convert2nc’ cannot
produce full netCDF4 files.  In contrast, ‘ncks’ will happily convert
HDF straight to netCDF4 files with netCDF4 types.  Hence ‘ncks’ can and
does preserve the variable types.  Unsigned bytes stay unsigned bytes.
64-bit integers stay 64-bit integers.  Strings stay strings.  Hence,
‘ncks’ conversions often result in smaller files than ‘ncl_convert2nc’
conversions.  Another tool useful for converting netCDF3 to netCDF4
files, and whose functionality is, we think, also matched or exceeded by
‘ncks’, is the Python script ‘nc3tonc4’ by Jeff Whitaker.

   (5) Two real-world examples: NCO translates the NASA CERES dimension
‘(FOV) Footprints’ to ‘_FOV_ Footprints’, and ‘Cloud & Aerosol, Cloud
Only, Clear Sky w/Aerosol, and Clear Sky’ (yes, the dimension name
includes whitespace and special characters) to ‘Cloud & Aerosol, Cloud
Only, Clear Sky w_Aerosol, and Clear Sky’ ‘ncl_convert2nc’ makes the
element name netCDF-safe in a slightly different manner, and also stores
the original name in the ‘hdf_name’ attribute.


File: nco.info,  Node: Help Requests and Bug Reports,  Prev: netCDF2/3/4 and HDF4/5 Support,  Up: Introduction

1.7 Help Requests and Bug Reports
=================================

We generally receive three categories of mail from users: help requests,
bug reports, and feature requests.  Notes saying the equivalent of "Hey,
NCO continues to work great and it saves me more time everyday than it
took to write this note" are a distant fourth.

   There is a different protocol for each type of request.  The
preferred etiquette for all communications is via NCO Project Forums.
Do not contact project members via personal e-mail unless your request
comes with money or you have damaging information about our personal
lives.  _Please use the Forums_--they preserve a record of the questions
and answers so that others can learn from our exchange.  Also, since NCO
is both volunteer-driven and government-funded, this record helps us
provide program officers with information they need to evaluate our
project.

   Before posting to the NCO forums described below, you might first
register (https://sf.net/account/register.php) your name and email
address with SourceForge.net or else all of your postings will be
attributed to _nobody_.  Once registered you may choose to _monitor_ any
forum and to receive (or not) email when there are any postings
including responses to your questions.  We usually reply to the forum
message, not to the original poster.

   If you want us to include a new feature in NCO, please consider
implementing the feature yourself and sending us the patch.  If that is
beyond your ken, then send a note to the NCO Discussion forum
(http://sf.net/p/nco/discussion/9829).

   Read the manual before reporting a bug or posting a help request.
Sending questions whose answers are not in the manual is the best way to
motivate us to write more documentation.  We would also like to
accentuate the contrapositive of this statement.  If you think you have
found a real bug _the most helpful thing you can do is simplify the
problem to a manageable size and then report it_.  The first thing to do
is to make sure you are running the latest publicly released version of
NCO.

   Once you have read the manual, if you are still unable to get NCO to
perform a documented function, submit a help request.  Follow the same
procedure as described below for reporting bugs (after all, it might be
a bug).  That is, describe what you are trying to do, and include the
complete commands (run with ‘-D 5’), error messages, and version of NCO
(with ‘-r’).  Some commands behave differently depending on the exact
order and rank of dimensions in the pertinent variables.  In such cases
we need you to provide that metadata, e.g., the text results of ‘ncks
-m’ on your input and/or output files.  Post your help request to the
NCO Help forum (http://sf.net/p/nco/discussion/9830).

   If you think you used the right command when NCO misbehaves, then you
might have found a bug.  Incorrect numerical answers are the highest
priority.  We usually fix those within one or two days.  Core dumps and
sementation violations receive lower priority.  They are always fixed,
eventually.

   How do you simplify a problem to reveal a bug?  Cut out extraneous
variables, dimensions, and metadata from the offending files and re-run
the command until it no longer breaks.  Then back up one step and report
the problem.  Usually the file(s) will be very small, i.e., one variable
with one or two small dimensions ought to suffice.

   Run the operator with ‘-r’ and then run the command with ‘-D 5’ to
increase the verbosity of the debugging output.  It is very important
that your report contain the exact error messages and compile-time
environment.  Include a copy of your sample input file, or place one on
a publicly accessible location, of the file(s).

   You can also access the netCDF library “logging” capability.  This
might help you determine whether the problem is with NCO or with netCDF.
Invoke the operator with ‘--log_lvl=LOG-LEVEL’, where 0 < LOG-LEVEL < 5
is an integer that increases the verbosity of the library reporting.

   If you are sure you have found an NCO bug, post your report to the
NCO Project buglist (http://sf.net/p/nco/bugs).  Otherwise post all the
information to NCO Help forum (http://sf.net/p/nco/discussion/9830).

   Build failures count as bugs.  Our limited machine access means we
cannot fix all build failures.  The information we need to diagnose, and
often fix, build failures are the three files output by GNU build tools,
‘nco.config.log.${GNU_TRP}.foo’, ‘nco.configure.${GNU_TRP}.foo’, and
‘nco.make.${GNU_TRP}.foo’.  The file ‘configure.eg’ shows how to produce
these files.  Here ‘${GNU_TRP}’ is the "GNU architecture triplet", the
CHIP-VENDOR-OS string returned by ‘config.guess’.  Please send us your
improvements to the examples supplied in ‘configure.eg’.  The
regressions archive at <http://dust.ess.uci.edu/nco/rgr> contains the
build output from our standard test systems.  You may find you can solve
the build problem yourself by examining the differences between these
files and your own.


File: nco.info,  Node: Strategies,  Next: Shared features,  Prev: Introduction,  Up: Top

2 Operator Strategies
*********************

* Menu:

* Philosophy::
* Climate Model Paradigm::
* Temporary Output Files::
* Appending Variables::
* Simple Arithmetic and Interpolation::
* Statistics vs Concatenation::
* Large Numbers of Files::
* Large Datasets::
* Memory Requirements::
* Performance::


File: nco.info,  Node: Philosophy,  Next: Climate Model Paradigm,  Prev: Strategies,  Up: Strategies

2.1 Philosophy
==============

The main design goal is command line operators which perform useful,
scriptable operations on netCDF files.  Many scientists work with models
and observations which produce too much data to analyze in tabular
format.  Thus, it is often natural to reduce and massage this raw or
primary level data into summary, or second level data, e.g., temporal or
spatial averages.  These second level data may become the inputs to
graphical and statistical packages, and are often more suitable for
archival and dissemination to the scientific community.  NCO performs a
suite of operations useful in manipulating data from the primary to the
second level state.  Higher level interpretive languages (e.g., IDL,
Yorick, Matlab, NCL, Perl, Python), and lower level compiled languages
(e.g., C, Fortran) can always perform any task performed by NCO, but
often with more overhead.  NCO, on the other hand, is limited to a much
smaller set of arithmetic and metadata operations than these full blown
languages.

   Another goal has been to implement enough command line switches so
that frequently used sequences of these operators can be executed from a
shell script or batch file.  Finally, NCO was written to consume the
absolute minimum amount of system memory required to perform a given
job.  The arithmetic operators are extremely efficient; their exact
memory usage is detailed in *note Memory Requirements::.


File: nco.info,  Node: Climate Model Paradigm,  Next: Temporary Output Files,  Prev: Philosophy,  Up: Strategies

2.2 Climate Model Paradigm
==========================

NCO was developed at NCAR to aid analysis and manipulation of datasets
produced by General Circulation Models (GCMs).  GCM datasets share many
features with other gridded scientific datasets and so provide a useful
paradigm for the explication of the NCO operator set.  Examples in this
manual use a GCM paradigm because latitude, longitude, time, temperature
and other fields related to our natural environment are as easy to
visualize for the layman as the expert.


File: nco.info,  Node: Temporary Output Files,  Next: Appending Variables,  Prev: Climate Model Paradigm,  Up: Strategies

2.3 Temporary Output Files
==========================

NCO operators are designed to be reasonably fault tolerant, so that a
system failure or user-abort of the operation (e.g., with ‘C-c’) does
not cause loss of data.  The user-specified OUTPUT-FILE is only created
upon successful completion of the operation (1).  This is accomplished
by performing all operations in a temporary copy of OUTPUT-FILE.  The
name of the temporary output file is constructed by appending
‘.pid<PROCESS ID>.<OPERATOR NAME>.tmp’ to the user-specified OUTPUT-FILE
name.  When the operator completes its task with no fatal errors, the
temporary output file is moved to the user-specified OUTPUT-FILE.  This
imbues the process with fault-tolerance since fatal error (e.g., disk
space fills up) affect only the temporary output file, leaving the final
output file not created if it did not already exist.  Note the
construction of a temporary output file uses more disk space than just
overwriting existing files "in place" (because there may be two copies
of the same file on disk until the NCO operation successfully concludes
and the temporary output file overwrites the existing OUTPUT-FILE).
Also, note this feature increases the execution time of the operator by
approximately the time it takes to copy the OUTPUT-FILE (2).  Finally,
note this fault-tolerant feature allows the OUTPUT-FILE to be the same
as the INPUT-FILE without any danger of "overlap".

   Over time many "power users" have requested a way to turn-off the
fault-tolerance safety feature that automatically creates a temporary
file.  Often these users build and execute production data analysis
scripts that are repeated frequently on large datasets.  Obviating an
extra file write can then conserve significant disk space and time.  For
this purpose NCO has, since version 4.2.1 in August, 2012, made
configurable the controls over temporary file creation.  The
‘--wrt_tmp_fl’ and equivalent ‘--write_tmp_fl’ switches ensure NCO
writes output to an intermediate temporary file.  This is and has always
been the default behavior so there is currently no need to specify these
switches.  However, the default may change some day, especially since
writing to RAM disks (*note RAM disks::) may some day become the
default.  The ‘--no_tmp_fl’ switch causes NCO to write directly to the
final output file instead of to an intermediate temporary file.  "Power
users" may wish to invoke this switch to increase performance (i.e.,
reduce wallclock time) when manipulating large files.  When eschewing
temporary files, users may forsake the ability to have the same name for
both OUTPUT-FILE and INPUT-FILE since, as described above, the temporary
file prevented overlap issues.  However, if the user creates the output
file in RAM (*note RAM disks::) then it is still possible to have the
same name for both OUTPUT-FILE and INPUT-FILE.
     ncks in.nc out.nc # Default: create out.pid.tmp.nc then move to out.nc
     ncks --wrt_tmp_fl in.nc out.nc # Same as default
     ncks --no_tmp_fl in.nc out.nc # Create out.nc directly on disk
     ncks --no_tmp_fl in.nc in.nc # ERROR-prone! Overwrite in.nc with itself
     ncks --create_ram --no_tmp_fl in.nc in.nc # Create in RAM, write to disk
     ncks --open_ram --no_tmp_fl in.nc in.nc # Read into RAM, write to disk
There is no reason to expect the fourth example to work.  The behavior
of overwriting a file while reading from the same file is undefined,
much as is the shell command ‘cat foo > foo’.  Although it may "work" in
some cases, it is unreliable.  One way around this is to use
‘--create_ram’ so that the output file is not written to disk until the
input file is closed, *Note RAM disks::.  However, as of 20130328, the
behavior of the ‘--create_ram’ and ‘--open_ram’ examples has not been
thoroughly tested.

   The NCO authors have seen compelling use cases for utilizing the RAM
switches, though not (yet) for combining them with ‘--no_tmp_fl’.  NCO
implements both options because they are largely independent of
eachother.  It is up to "power users" to discover which best fit their
needs.  We welcome accounts of your experiences posted to the forums.

   Other safeguards exist to protect the user from inadvertently
overwriting data.  If the OUTPUT-FILE specified for a command is a
pre-existing file, then the operator will prompt the user whether to
overwrite (erase) the existing OUTPUT-FILE, attempt to append to it, or
abort the operation.  However, in processing large amounts of data, too
many interactive questions slows productivity.  Therefore NCO also
implements two ways to override its own safety features, the ‘-O’ and
‘-A’ switches.  Specifying ‘-O’ tells the operator to overwrite any
existing OUTPUT-FILE without prompting the user interactively.
Specifying ‘-A’ tells the operator to attempt to append to any existing
OUTPUT-FILE without prompting the user interactively.  These switches
are useful in batch environments because they suppress interactive
keyboard input.

   ---------- Footnotes ----------

   (1) The ‘ncrename’ and ‘ncatted’ operators are exceptions to this
rule.  *Note ncrename netCDF Renamer::.

   (2) The OS-specific system move command is used.  This is ‘mv’ for
UNIX, and ‘move’ for Windows.


File: nco.info,  Node: Appending Variables,  Next: Simple Arithmetic and Interpolation,  Prev: Temporary Output Files,  Up: Strategies

2.4 Appending Variables
=======================

Adding variables from one file to another is often desirable.  This is
referred to as “appending”, although some prefer the terminology
“merging” (1) or “pasting”.  Appending is often confused with what NCO
calls “concatenation”.  In NCO, concatenation refers to splicing a
variable along the record dimension.  The length along the record
dimension of the output is the sum of the lengths of the input files.
Appending, on the other hand, refers to copying a variable from one file
to another file which may or may not already contain the variable (2).
NCO can append or concatenate just one variable, or all the variables in
a file at the same time.

   In this sense, ‘ncks’ can append variables from one file to another
file.  This capability is invoked by naming two files on the command
line, INPUT-FILE and OUTPUT-FILE.  When OUTPUT-FILE already exists, the
user is prompted whether to “overwrite”, “append/replace”, or “exit”
from the command.  Selecting “overwrite” tells the operator to erase the
existing OUTPUT-FILE and replace it with the results of the operation.
Selecting “exit” causes the operator to exit--the OUTPUT-FILE will not
be touched in this case.  Selecting “append/replace” causes the operator
to attempt to place the results of the operation in the existing
OUTPUT-FILE, *Note ncks netCDF Kitchen Sink::.

   The simplest way to create the union of two files is
     ncks -A fl_1.nc fl_2.nc
   This puts the contents of ‘fl_1.nc’ into ‘fl_2.nc’.  The ‘-A’ is
optional.  On output, ‘fl_2.nc’ is the union of the input files,
regardless of whether they share dimensions and variables, or are
completely disjoint.  The append fails if the input files have
differently named record dimensions (since netCDF supports only one), or
have dimensions of the same name but different sizes.

   ---------- Footnotes ----------

   (1) The terminology “merging” is reserved for an (unwritten) operator
which replaces hyperslabs of a variable in one file with hyperslabs of
the same variable from another file

   (2) Yes, the terminology is confusing.  By all means mail me if you
think of a better nomenclature.  Should NCO use “paste” instead of
“append”?


File: nco.info,  Node: Simple Arithmetic and Interpolation,  Next: Statistics vs Concatenation,  Prev: Appending Variables,  Up: Strategies

2.5 Simple Arithmetic and Interpolation
=======================================

Users comfortable with NCO semantics may find it easier to perform some
simple mathematical operations in NCO rather than higher level
languages.  ‘ncbo’ (*note ncbo netCDF Binary Operator::) does file
addition, subtraction, multiplication, division, and broadcasting.  It
even does group broadcasting.  ‘ncflint’ (*note ncflint netCDF File
Interpolator::) does file addition, subtraction, multiplication and
interpolation.  Sequences of these commands can accomplish simple yet
powerful operations from the command line.


File: nco.info,  Node: Statistics vs Concatenation,  Next: Large Numbers of Files,  Prev: Simple Arithmetic and Interpolation,  Up: Strategies

2.6 Statistics vs Concatenation
===============================

The most frequently used operators of NCO are probably the
“statisticians” (i.e., tools that do statistics) and concatenators.
Because there are so many types of statistics like averaging (e.g.,
across files, within a file, over the record dimension, over other
dimensions, with or without weights and masks) and of concatenating
(across files, along the record dimension, along other dimensions),
there are currently no fewer than five operators which tackle these two
purposes: ‘ncra’, ‘nces’, ‘ncwa’, ‘ncrcat’, and ‘ncecat’.  These
operators do share many capabilities (1), though each has its unique
specialty.  Two of these operators, ‘ncrcat’ and ‘ncecat’, concatenate
hyperslabs across files.  The other two operators, ‘ncra’ and ‘nces’,
compute statistics across (and/or within) files (2).  First, let's
describe the concatenators, then the statistics tools.

* Menu:

* Concatenation::
* Averaging::
* Interpolating::

   ---------- Footnotes ----------

   (1) Currently ‘nces’ and ‘ncrcat’ are symbolically linked to the
‘ncra’ executable, which behaves slightly differently based on its
invocation name (i.e., ‘argv[0]’).  These three operators share the same
source code, and merely have different inner loops.

   (2) The third averaging operator, ‘ncwa’, is the most sophisticated
averager in NCO.  However, ‘ncwa’ is in a different class than ‘ncra’
and ‘nces’ because it operates on a single file per invocation (as
opposed to multiple files).  On that single file, however, ‘ncwa’
provides a richer set of averaging options--including weighting,
masking, and broadcasting.


File: nco.info,  Node: Concatenation,  Next: Averaging,  Prev: Statistics vs Concatenation,  Up: Statistics vs Concatenation

2.6.1 Concatenators ‘ncrcat’ and ‘ncecat’
-----------------------------------------

Joining together independent files along a common record dimension is
called “concatenation”.  ‘ncrcat’ is designed for concatenating record
variables, while ‘ncecat’ is designed for concatenating fixed length
variables.  Consider five files, ‘85.nc’, ‘86.nc’, ... ‘89.nc’ each
containing a year's worth of data.  Say you wish to create from them a
single file, ‘8589.nc’ containing all the data, i.e., spanning all five
years.  If the annual files make use of the same record variable, then
‘ncrcat’ will do the job nicely with, e.g., ‘ncrcat 8?.nc 8589.nc’.  The
number of records in the input files is arbitrary and can vary from file
to file.  *Note ncrcat netCDF Record Concatenator::, for a complete
description of ‘ncrcat’.

   However, suppose the annual files have no record variable, and thus
their data are all fixed length.  For example, the files may not be
conceptually sequential, but rather members of the same group, or
“ensemble”.  Members of an ensemble may have no reason to contain a
record dimension.  ‘ncecat’ will create a new record dimension (named
RECORD by default) with which to glue together the individual files into
the single ensemble file.  If ‘ncecat’ is used on files which contain an
existing record dimension, that record dimension is converted to a
fixed-length dimension of the same name and a new record dimension
(named ‘record’) is created.  Consider five realizations, ‘85a.nc’,
‘85b.nc’, ... ‘85e.nc’ of 1985 predictions from the same climate model.
Then ‘ncecat 85?.nc 85_ens.nc’ glues together the individual
realizations into the single file, ‘85_ens.nc’.  If an input variable
was dimensioned [‘lat’,‘lon’], it will have dimensions
[‘record’,‘lat’,‘lon’] in the output file.  A restriction of ‘ncecat’ is
that the hyperslabs of the processed variables must be the same from
file to file.  Normally this means all the input files are the same
size, and contain data on different realizations of the same variables.
*Note ncecat netCDF Ensemble Concatenator::, for a complete description
of ‘ncecat’.

   ‘ncpdq’ makes it possible to concatenate files along any dimension,
not just the record dimension.  First, use ‘ncpdq’ to convert the
dimension to be concatenated (i.e., extended with data from other files)
into the record dimension.  Second, use ‘ncrcat’ to concatenate these
files.  Finally, if desirable, use ‘ncpdq’ to revert to the original
dimensionality.  As a concrete example, say that files ‘x_01.nc’,
‘x_02.nc’, ... ‘x_10.nc’ contain time-evolving datasets from spatially
adjacent regions.  The time and spatial coordinates are ‘time’ and ‘x’,
respectively.  Initially the record dimension is ‘time’.  Our goal is to
create a single file that contains joins all the spatially adjacent
regions into one single time-evolving dataset.
     for idx in 01 02 03 04 05 06 07 08 09 10; do # Bourne Shell
       ncpdq -a x,time x_${idx}.nc foo_${idx}.nc  # Make x record dimension
     done
     ncrcat foo_??.nc out.nc       # Concatenate along x
     ncpdq -a time,x out.nc out.nc # Revert to time as record dimension

   Note that ‘ncrcat’ will not concatenate fixed-length variables,
whereas ‘ncecat’ concatenates both fixed-length and record variables
along a new record variable.  To conserve system memory, use ‘ncrcat’
where possible.


File: nco.info,  Node: Averaging,  Next: Interpolating,  Prev: Concatenation,  Up: Statistics vs Concatenation

2.6.2 Averagers ‘nces’, ‘ncra’, and ‘ncwa’
------------------------------------------

The differences between the averagers ‘ncra’ and ‘nces’ are analogous to
the differences between the concatenators.  ‘ncra’ is designed for
averaging record variables from at least one file, while ‘nces’ is
designed for averaging fixed length variables from multiple files.
‘ncra’ performs a simple arithmetic average over the record dimension of
all the input files, with each record having an equal weight in the
average.  ‘nces’ performs a simple arithmetic average of all the input
files, with each file having an equal weight in the average.  Note that
‘ncra’ cannot average fixed-length variables, but ‘nces’ can average
both fixed-length and record variables.  To conserve system memory, use
‘ncra’ rather than ‘nces’ where possible (e.g., if each INPUT-FILE is
one record long).  The file output from ‘nces’ will have the same
dimensions (meaning dimension names as well as sizes) as the input
hyperslabs (*note nces netCDF Ensemble Statistics::, for a complete
description of ‘nces’).  The file output from ‘ncra’ will have the same
dimensions as the input hyperslabs except for the record dimension,
which will have a size of 1 (*note ncra netCDF Record Averager::, for a
complete description of ‘ncra’).


File: nco.info,  Node: Interpolating,  Prev: Averaging,  Up: Statistics vs Concatenation

2.6.3 Interpolator ‘ncflint’
----------------------------

‘ncflint’ can interpolate data between or two files.  Since no other
operators have this ability, the description of interpolation is given
fully on the ‘ncflint’ reference page (*note ncflint netCDF File
Interpolator::).  Note that this capability also allows ‘ncflint’ to
linearly rescale any data in a netCDF file, e.g., to convert between
differing units.


File: nco.info,  Node: Large Numbers of Files,  Next: Large Datasets,  Prev: Statistics vs Concatenation,  Up: Strategies

2.7 Large Numbers of Files
==========================

Occasionally one desires to digest (i.e., concatenate or average)
hundreds or thousands of input files.  Unfortunately, data archives
(e.g., NASA EOSDIS) may not name netCDF files in a format understood by
the ‘-n LOOP’ switch (*note Specifying Input Files::) that automagically
generates arbitrary numbers of input filenames.  The ‘-n LOOP’ switch
has the virtue of being concise, and of minimizing the command line.
This helps keeps output file small since the command line is stored as
metadata in the ‘history’ attribute (*note History Attribute::).
However, the ‘-n LOOP’ switch is useless when there is no simple,
arithmetic pattern to the input filenames (e.g., ‘h00001.nc’,
‘h00002.nc’, ... ‘h90210.nc’).  Moreover, filename globbing does not
work when the input files are too numerous or their names are too
lengthy (when strung together as a single argument) to be passed by the
calling shell to the NCO operator (1).  When this occurs, the ANSI
C-standard ‘argc’-‘argv’ method of passing arguments from the calling
shell to a C-program (i.e., an NCO operator) breaks down.  There are (at
least) three alternative methods of specifying the input filenames to
NCO in environment-limited situations.

   The recommended method for sending very large numbers (hundreds or
more, typically) of input filenames to the multi-file operators is to
pass the filenames with the UNIX “standard input” feature, aka ‘stdin’:
     # Pipe large numbers of filenames to stdin
     /bin/ls | grep ${CASEID}_'......'.nc | ncecat -o foo.nc
   This method avoids all constraints on command line size imposed by
the operating system.  A drawback to this method is that the ‘history’
attribute (*note History Attribute::) does not record the name of any
input files since the names were not passed as positional arguments on
the command line.  This makes it difficult later to determine the data
provenance.  To remedy this situation, operators store the number of
input files in the ‘nco_input_file_number’ global attribute and the
input file list itself in the ‘nco_input_file_list’ global attribute
(*note File List Attributes::).  Although this does not preserve the
exact command used to generate the file, it does retains all the
information required to reconstruct the command and determine the data
provenance.

   As of NCO version 5.1.1, released in November, 2022, all operators
support specifying input files via ‘stdin’ (*note Specifying Input
Files::), and also store such input filenames in the *note File List
Attributes::).

   A second option is to use the UNIX ‘xargs’ command.  This simple
example selects as input to ‘xargs’ all the filenames in the current
directory that match a given pattern.  For illustration, consider a user
trying to average millions of files which each have a six character
filename.  If the shell buffer cannot hold the results of the
corresponding globbing operator, ‘??????.nc’, then the filename globbing
technique will fail.  Instead we express the filename pattern as an
extended regular expression, ‘......\.nc’ (*note Subsetting Files::).
We use ‘grep’ to filter the directory listing for this pattern and to
pipe the results to ‘xargs’ which, in turn, passes the matching
filenames to an NCO multi-file operator, e.g., ‘ncecat’.
     # Use xargs to transfer filenames on the command line
     /bin/ls | grep ${CASEID}_'......'.nc | xargs -x ncecat -o foo.nc
   The single quotes protect the only sensitive parts of the extended
regular expression (the ‘grep’ argument), and allow shell interpolation
(the ‘${CASEID}’ variable substitution) to proceed unhindered on the
rest of the command.  ‘xargs’ uses the UNIX pipe feature to append the
suitably filtered input file list to the end of the ‘ncecat’ command
options.  The ‘-o foo.nc’ switch ensures that the input files supplied
by ‘xargs’ are not confused with the output file name.  ‘xargs’ does,
unfortunately, have its own limit (usually about 20,000 characters) on
the size of command lines it can pass.  Give ‘xargs’ the ‘-x’ switch to
ensure it dies if it reaches this internal limit.  When this occurs, use
either the ‘stdin’ method above, or the symbolic link presented next.

   Even when its internal limits have not been reached, the ‘xargs’
technique may not be flexible enough to handle all situations.  A full
scripting language like Perl or Python can handle any level of
complexity of filtering input filenames, and any number of filenames.
The technique of last resort is to write a script that creates symbolic
links between the irregular input filenames and a set of regular,
arithmetic filenames that the ‘-n LOOP’ switch understands.  For
example, the following Perl script creates a monotonically enumerated
symbolic link to up to one million ‘.nc’ files in a directory.  If there
are 999,999 netCDF files present, the links are named ‘000001.nc’ to
‘999999.nc’:
     # Create enumerated symbolic links
     /bin/ls | grep \.nc | perl -e \
     '$idx=1;while(<STDIN>){chop;symlink $_,sprintf("%06d.nc",$idx++);}'
     ncecat -n 999999,6,1 000001.nc foo.nc
     # Remove symbolic links when finished
     /bin/rm ??????.nc
   The ‘-n LOOP’ option tells the NCO operator to automatically generate
the filnames of the symbolic links.  This circumvents any OS and shell
limits on command-line size.  The symbolic links are easily removed once
NCO is finished.  One drawback to this method is that the ‘history’
attribute (*note History Attribute::) retains the filename list of the
symbolic links, rather than the data files themselves.  This makes it
difficult to determine the data provenance at a later date.

   ---------- Footnotes ----------

   (1) The exact length which exceeds the operating system internal
limit for command line lengths varies across OSs and shells.  GNU ‘bash’
may not have any arbitrary fixed limits to the size of command line
arguments.  Many OSs cannot handle command line arguments (including
results of file globbing) exceeding 4096 characters.


File: nco.info,  Node: Large Datasets,  Next: Memory Requirements,  Prev: Large Numbers of Files,  Up: Strategies

2.8 Large Datasets
==================

“Large datasets” are those files that are comparable in size to the
amount of random access memory (RAM) in your computer.  Many users of
NCO work with files larger than 100 MB. Files this large not only push
the current edge of storage technology, they present special problems
for programs which attempt to access the entire file at once, such as
‘nces’ and ‘ncecat’.  If you work with a 300 MB files on a machine with
only 32 MB of memory then you will need large amounts of swap space
(virtual memory on disk) and NCO will work slowly, or even fail.  There
is no easy solution for this.  The best strategy is to work on a machine
with sufficient amounts of memory and swap space.  Since about 2004,
many users have begun to produce or analyze files exceeding 2 GB in
size.  These users should familiarize themselves with NCO's Large File
Support (LFS) capabilities (*note Large File Support::).  The next
section will increase your familiarity with NCO's memory requirements.
With this knowledge you may re-design your data reduction approach to
divide the problem into pieces solvable in memory-limited situations.

   If your local machine has problems working with large files, try
running NCO from a more powerful machine, such as a network server.  If
you get a memory-related core dump (e.g., ‘Error exit (core dumped)’) on
a GNU/Linux system, or the operation ends before the entire output file
is written, try increasing the process-available memory with ‘ulimit’:
     ulimit -f unlimited
   This may solve constraints on clusters where sufficient hardware
resources exist yet where system administrators felt it wise to prevent
any individual user from consuming too much of resource.  Certain
machine architectures, e.g., Cray UNICOS, have special commands which
allow one to increase the amount of interactive memory.  On Cray
systems, try to increase the available memory with the ‘ilimit’ command.

   The speed of the NCO operators also depends on file size.  When
processing large files the operators may appear to hang, or do nothing,
for large periods of time.  In order to see what the operator is
actually doing, it is useful to activate a more verbose output mode.
This is accomplished by supplying a number greater than 0 to the ‘-D
DEBUG-LEVEL’ (or ‘--debug-level’, or ‘--dbg_lvl’) switch.  When the
DEBUG-LEVEL is non-zero, the operators report their current status to
the terminal through the STDERR facility.  Using ‘-D’ does not slow the
operators down.  Choose a DEBUG-LEVEL between 1 and 3 for most
situations, e.g., ‘nces -D 2 85.nc 86.nc 8586.nc’.  A full description
of how to estimate the actual amount of memory the multi-file NCO
operators consume is given in *note Memory Requirements::.


File: nco.info,  Node: Memory Requirements,  Next: Performance,  Prev: Large Datasets,  Up: Strategies

2.9 Memory Requirements
=======================

Many people use NCO on gargantuan files which dwarf the memory available
(free RAM plus swap space) even on today's powerful machines.  These
users want NCO to consume the least memory possible so that their
scripts do not have to tediously cut files into smaller pieces that fit
into memory.  We commend these greedy users for pushing NCO to its
limits!

   This section describes the memory NCO requires during operation.  The
required memory depends on the underlying algorithms, datatypes, and
compression, if any.  The description below is the memory usage per
thread.  Users with shared memory machines may use the threaded NCO
operators (*note OpenMP Threading::).  The peak and sustained memory
usage will scale accordingly, i.e., by the number of threads.  In all
cases the memory use refers to the _uncompressed_ size of the data.  The
netCDF4 library automatically decompresses variables during reads.  The
filesize can easily belie the true size of the uncompressed data.  In
other words, the usage below can be taken at face value for netCDF3
datasets only.  Chunking will also affect memory usage on netCDF4
operations.  Memory consumption patterns of all operators are similar,
with the exception of ‘ncap2’.

* Menu:

* Single and Multi-file Operators::
* Memory for ncap2::


File: nco.info,  Node: Single and Multi-file Operators,  Next: Memory for ncap2,  Prev: Memory Requirements,  Up: Memory Requirements

2.9.1 Single and Multi-file Operators
-------------------------------------

The multi-file operators currently comprise the record operators, ‘ncra’
and ‘ncrcat’, and the ensemble operators, ‘nces’ and ‘ncecat’.  The
record operators require _much less_ memory than the ensemble operators.
This is because the record operators operate on one single record (i.e.,
time-slice) at a time, whereas the ensemble operators retrieve the
entire variable into memory.  Let MS be the peak sustained memory demand
of an operator, FT be the memory required to store the entire contents
of all the variables to be processed in an input file, FR be the memory
required to store the entire contents of a single record of each of the
variables to be processed in an input file, VR be the memory required to
store a single record of the largest record variable to be processed in
an input file, VT be the memory required to store the largest variable
to be processed in an input file, VI be the memory required to store the
largest variable which is not processed, but is copied from the initial
file to the output file.  All operators require MI = VI during the
initial copying of variables from the first input file to the output
file.  This is the _initial_ (and transient) memory demand.  The
_sustained_ memory demand is that memory required by the operators
during the processing (i.e., averaging, concatenation) phase which lasts
until all the input files have been processed.  The operators have the
following memory requirements: ‘ncrcat’ requires MS <= VR. ‘ncecat’
requires MS <= VT. ‘ncra’ requires MS = 2FR + VR. ‘nces’ requires MS =
2FT + VT. ‘ncbo’ requires MS <= 3VT (both input variables and the output
variable).  ‘ncflint’ requires MS <= 3VT (both input variables and the
output variable).  ‘ncpdq’ requires MS <= 2VT (one input variable and
the output variable).  ‘ncwa’ requires MS <= 8VT (see below).  Note that
only variables that are processed, e.g., averaged, concatenated, or
differenced, contribute to MS. Variables that do not appear in the
output file (*note Subsetting Files::) are never read and contribute
nothing to the memory requirements.

   Further note that some operators perform internal type-promotion on
some variables prior to arithmetic (*note Type Conversion::).  For
example, ‘ncra’, ‘nces’, and ‘ncwa’ all promote integer types to
double-precision floating-point prior to arithmetic, then perform the
arithmetic, then demote back to the original integer type after
arithmetic.  This preserves the on-disk storage type while obtaining the
precision advantages of double-precision floating-point arithmetic.
Since version 4.3.6 (released in September, 2013), NCO also by default
converts single-precision floating-point to double-precision prior to
arithmetic, which incurs the same RAM penalty.  Hence, the sustained
memory required for integer variables and single-precision floats are
two or four-times their on-disk, uncompressed, unpacked sizes if they
meet the rules for automatic internal promotion.  Put another way,
disabling auto-promotion of single-precision variables (with ‘--flt’)
considerably reduces the RAM footprint of arithmetic operators.

   The ‘--open_ram’ switch (and switches that invoke it like ‘--ram_all’
and ‘--diskless_all’) incurs a RAM penalty.  These switches cause each
input file to be copied to RAM upon opening.  Hence any operator
invoking these switches utilizes an additional FT of RAM (i.e., MS +=
FT). See *note RAM disks:: for further details.

   ‘ncwa’ consumes between two and eight times the memory of an
‘NC_DOUBLE’ variable in order to process it.  Peak consumption occurs
when storing simultaneously in memory one input variable, one tally
array, one input weight, one conformed/working weight, one weight tally,
one input mask, one conformed/working mask, and one output variable.
NCO's tally arrays are of type C-type ‘long’, whose size is eight-bytes
on all modern computers, the same as ‘NC_DOUBLE’ (1).  When invoked, the
weighting and masking features contribute up to three-eighths and
two-eighths of these requirements apiece.  If weights and masks are
_not_ specified (i.e., no ‘-w’ or ‘-a’ options) then ‘ncwa’ requirements
drop to MS <= 3VT (one input variable, one tally array, and the output
variable).  The output variable is the same size as the input variable
when averaging only over a degenerate dimension.  However, normally the
output variable is much smaller than the input, and is often a simple
scalar, in which case the memory requirements drop by 1VT since the
output array requires essentially no memory.

   All of this is subject to the type promotion rules mentioned above.
For example, ‘ncwa’ averaging a variable of type ‘NC_FLOAT’ requires MS
<= 16VT (rather than MS <= 8VT) since all arrays are (at least
temporarily) composed of eight-byte elements, twice the size of the
values on disk.  Without mask or weights, the requirements for
‘NC_FLOAT’ are MS <= 6VT (rather than MS <= 3VT as for ‘NC_DOUBLE’) due
to temporary internal promotion of both the input variable and the
output variable to type ‘NC_DOUBLE’.  The ‘--flt’ option that suppresses
promotion reduces this to MS <= 4VT (the tally elements do not change
size), and to MS <= 3VT when the output array is a scalar.

   The above memory requirements must be multiplied by the number of
threads THR_NBR (*note OpenMP Threading::).  If this causes problems
then reduce (with ‘-t THR_NBR’) the number of threads.

   ---------- Footnotes ----------

   (1) By contrast ‘NC_INT’ and its deprecated synonym ‘NC_LONG’ are
only four-bytes.  Perhaps this is one reason why the ‘NC_LONG’ token is
deprecated.


File: nco.info,  Node: Memory for ncap2,  Prev: Single and Multi-file Operators,  Up: Memory Requirements

2.9.2 Memory for ‘ncap2’
------------------------

‘ncap2’ has unique memory requirements due its ability to process
arbitrarily long scripts of any complexity.  All scripts acceptable to
‘ncap2’ are ultimately processed as a sequence of binary or unary
operations.  ‘ncap2’ requires MS <= 2VT under most conditions.  An
exception to this is when left hand casting (*note Left hand casting::)
is used to stretch the size of derived variables beyond the size of any
input variables.  Let VC be the memory required to store the largest
variable defined by left hand casting.  In this case, MS <= 2VC.

   ‘ncap2’ scripts are complete dynamic and may be of arbitrary length.
A script that contains many thousands of operations, may uncover a slow
memory leak even though each single operation consumes little additional
memory.  Memory leaks are usually identifiable by their memory usage
signature.  Leaks cause peak memory usage to increase monotonically with
time regardless of script complexity.  Slow leaks are very difficult to
find.  Sometimes a ‘malloc()’ (or ‘new[]’) failure is the only
noticeable clue to their existence.  If you have good reasons to believe
that a memory allocation failure is ultimately due to an NCO memory leak
(rather than inadequate RAM on your system), then we would be very
interested in receiving a detailed bug report.


File: nco.info,  Node: Performance,  Prev: Memory Requirements,  Up: Strategies

2.10 Performance
================

An overview of NCO capabilities as of about 2006 is in Zender, C. S.
(2008), "Analysis of Self-describing Gridded Geoscience Data with netCDF
Operators (NCO)", Environ.  Modell.  Softw.,
doi:10.1016/j.envsoft.2008.03.004.  This paper is also available at
<http://dust.ess.uci.edu/ppr/ppr_Zen08.pdf>.

   NCO performance and scaling for arithmetic operations is described in
Zender, C. S., and H. J. Mangalam (2007), "Scaling Properties of Common
Statistical Operators for Gridded Datasets", Int.  J. High Perform.
Comput.  Appl., 21(4), 485-498, doi:10.1177/1094342007083802.  This
paper is also available at <http://dust.ess.uci.edu/ppr/ppr_ZeM07.pdf>.

   It is helpful to be aware of the aspects of NCO design that can limit
its performance:
  1. No data buffering is performed during ‘nc_get_var’ and ‘nc_put_var’
     operations.  Hyperslabs too large to hold in core memory will
     suffer substantial performance penalties because of this.

  2. Since coordinate variables are assumed to be monotonic, the search
     for bracketing the user-specified limits should employ a quicker
     algorithm, like bisection, than the two-sided incremental search
     currently implemented.

  3. C_FORMAT, FORTRAN_FORMAT, SIGNEDNESS, SCALE_FORMAT and ADD_OFFSET
     attributes are ignored by ‘ncks’ when printing variables to screen.

  4. In the late 1990s it was discovered that some random access
     operations on large files on certain architectures (e.g., UNICOS)
     were much slower with NCO than with similar operations performed
     using languages that bypass the netCDF interface (e.g., Yorick).
     This may have been a penalty of unnecessary byte-swapping in the
     netCDF interface.  It is unclear whether such problems exist in
     present day (2007) netCDF/NCO environments, where unnecessary
     byte-swapping has been reduced or eliminated.


File: nco.info,  Node: Shared features,  Next: Reference Manual,  Prev: Strategies,  Up: Top

3 Shared Features
*****************

Many features have been implemented in more than one operator and are
described here for brevity.  The description of each feature is preceded
by a box listing the operators for which the feature is implemented.
Command line switches for a given feature are consistent across all
operators wherever possible.  If no "key switches" are listed for a
feature, then that particular feature is automatic and cannot be
controlled by the user.

* Menu:

* Internationalization::
* Metadata Optimization::
* OpenMP Threading::
* Command Line Options::
* Sanitization of Input::
* Specifying Input Files::
* Specifying Output Files::
* Remote storage::
* Retaining Retrieved Files::
* File Formats and Conversion::
* Zarr and NCZarr::
* Large File Support::
* Subsetting Files::
* Subsetting Coordinate Variables::
* Group Path Editing::
* C and Fortran Index Conventions::
* Hyperslabs::
* Stride::
* Record Appending::
* Subcycle::
* Interleave::
* Multislabs::
* Wrapped Coordinates::
* Auxiliary Coordinates::
* Grid Generation::
* Regridding::
* Climatology and Bounds Support::
* UDUnits Support::
* Rebasing Time Coordinate::
* Multiple Record Dimensions::
* Missing Values::
* Chunking::
* Quantization Algorithms::
* Compression::
* Deflation::
* MD5 digests::
* Buffer sizes::
* RAM disks::
* Unbuffered I/O::
* Packed data::
* Operation Types::
* Type Conversion::
* Batch Mode::
* Global Attribute Addition::
* Global Attribute Deletion::
* History Attribute::
* File List Attributes::
* CF Conventions::
* ARM Conventions::
* Operator Version::


File: nco.info,  Node: Internationalization,  Next: Metadata Optimization,  Prev: Shared features,  Up: Shared features

3.1 Internationalization
========================

Availability: All operators
   NCO support for “internationalization” of textual input and output
(e.g., Warning messages) is nascent.  We introduced the first foreign
language string catalogues (French and Spanish) in 2004, yet did not
activate these in distributions because the catalogues were nearly
empty.  We seek volunteers to populate our templates with translations
for their favorite languages.


File: nco.info,  Node: Metadata Optimization,  Next: OpenMP Threading,  Prev: Internationalization,  Up: Shared features

3.2 Metadata Optimization
=========================

Availability: All operators
Short options: None
Long options: ‘--hdr_pad’, ‘--header_pad’
   NCO supports padding headers to improve the speed of future metadata
operations.  Use the ‘--hdr_pad’ and ‘--header_pad’ switches to request
that HDR_PAD bytes be inserted into the metadata section of the output
file.  There is little downside to padding a header with kilobyte of
space, since subsequent manipulation of the file will annotate the
‘history’ attribute with all commands, let alone any explicit metadata
additions with ‘ncatted’.
     ncks --hdr_pad=1000  in.nc out.nc # Pad header with  1 kB space
     ncks --hdr_pad=10000 in.nc out.nc # Pad header with 10 kB space
   Future metadata expansions will not incur the netCDF3 performance
penalty of copying the entire output file unless the expansion exceeds
the amount of header padding.  This can be beneficial when it is known
that some metadata will be added at a future date.  The operators that
benefit most from judicious use of header padding are ‘ncatted’ and
‘ncrename’, since they only alter metadata.

   This optimization exploits the netCDF library ‘nc__enddef()’
function.  This function behaves differently with different storage
formats.  It will improve speed of future metadata expansion with
‘CLASSIC’ and ‘64bit’ netCDF files, though not necessarily with
‘NETCDF4’ files, i.e., those created by the netCDF interface to the HDF5
library (*note File Formats and Conversion::).  netCDF3 formats use a
simple sequential ordering that requires copying the file if the size of
new metadata exceeds the available padding.  netCDF4 files use internal
file pointers that allow flexibility at inserting and removing data
without necessitating copying the whole file.


File: nco.info,  Node: OpenMP Threading,  Next: Command Line Options,  Prev: Metadata Optimization,  Up: Shared features

3.3 OpenMP Threading
====================

Availability: ‘ncclimo’, ‘ncks’, ‘ncremap’
Short options: ‘-t’
Long options: ‘--thr_nbr’, ‘--threads’, ‘--omp_num_threads’
   NCO supports shared memory parallelism (SMP) when compiled with an
OpenMP-enabled compiler.  Threads requests and allocations occur in two
stages.  First, users may request a specific number of threads THR_NBR
with the ‘-t’ switch (or its long option equivalents, ‘--thr_nbr’,
‘--threads’, and ‘--omp_num_threads’).  If not user-specified, OpenMP
obtains THR_NBR from the ‘OMP_NUM_THREADS’ environment variable, if
present, or from the OS, if not.

   Caveat: Unfortunately, threading does not improve NCO throughput
(i.e., wallclock time) because nearly all NCO operations are I/O-bound.
This means that NCO spends negligible time doing anything compared to
reading and writing.  The only exception is regridding with ‘ncremap’
which uses ‘ncks’ under-the-hood.  As of 2017, threading works only for
regridding, thus this section is relevant only to ‘ncclimo’, ‘ncks’, and
‘ncremap’.  We have seen some and can imagine other use cases where
‘ncwa’, ‘ncpdq’, and ‘ncap2’ (with long scripts) will complete faster
due to threading.  The main benefits of threading so far have been to
isolate the serial from parallel portions of code.  This parallelism is
now exploited by OpenMP but then runs into the I/O bottleneck during
output.  The bottleneck will be ameliorated for large files by the use
of MPI-enabled calls in the netCDF4 library when the underlying
filesystem is parallel (e.g., PVFS or JFS).  Implementation of the
parallel output calls in NCO is not a goal of our current funding and
would require new volunteers or funding.

   NCO may modify THR_NBR according to its own internal settings before
it requests any threads from the system.  Certain operators contain
hard-code limits to the number of threads they request.  We base these
limits on our experience and common sense, and to reduce potentially
wasteful system usage by inexperienced users.  For example, ‘ncrcat’ is
extremely I/O-intensive so we restrict THR_NBR <= 2 for ‘ncrcat’.  This
is based on the notion that the best performance that can be expected
from an operator which does no arithmetic is to have one thread reading
and one thread writing simultaneously.  In the future (perhaps with
netCDF4), we hope to demonstrate significant threading improvements with
operators like ‘ncrcat’ by performing multiple simultaneous writes.

   Compute-intensive operators (‘ncremap’) benefit most from threading.
The greatest increases in throughput due to threading occur on large
datasets where each thread performs millions, at least, of
floating-point operations.  Otherwise, the system overhead of setting up
threads probably outweighs the speed enhancements due to SMP
parallelism.  However, we have not yet demonstrated that the SMP
parallelism scales beyond four threads for these operators.  Hence we
restrict THR_NBR <= 4 for all operators.  We encourage users to play
with these limits (edit file ‘nco_omp.c’) and send us their feedback.

   Once the initial THR_NBR has been modified for any operator-specific
limits, NCO requests the system to allocate a team of THR_NBR threads
for the body of the code.  The operating system then decides how many
threads to allocate based on this request.  Users may keep track of this
information by running the operator with DBG_LVL > 0.

   By default, threaded operators attach one global attribute,
‘nco_openmp_thread_number’, to any file they create or modify.  This
attribute contains the number of threads the operator used to process
the input files.  This information helps to verify that the answers with
threaded and non-threaded operators are equal to within machine
precision.  This information is also useful for benchmarking.


File: nco.info,  Node: Command Line Options,  Next: Sanitization of Input,  Prev: OpenMP Threading,  Up: Shared features

3.4 Command Line Options
========================

Availability: All operators
   NCO achieves flexibility by using “command line options”.  These
options are implemented in all traditional UNIX commands as single
letter “switches”, e.g., ‘ls -l’.  For many years NCO used only single
letter option names.  In late 2002, we implemented GNU/POSIX extended or
long option names for all options.  This was done in a backward
compatible way such that the full functionality of NCO is still
available through the familiar single letter options.  Many features of
NCO introduced since 2002 now require the use of long options, simply
because we have nearly run out of single letter options.  More
importantly, mnemonics for single letter options are often non-intuitive
so that long options provide a more natural way of expressing intent.

   Extended options, also called long options, are implemented using the
system-supplied ‘getopt.h’ header file, if possible.  This provides the
‘getopt_long’ function to NCO (1).

   The syntax of “short options” (single letter options) is ‘-KEY VALUE’
(dash-key-space-value).  Here, KEY is the single letter option name,
e.g., ‘-D 2’.

   The syntax of “long options” (multi-letter options) is ‘--LONG_NAME
VALUE’ (dash-dash-key-space-value), e.g., ‘--dbg_lvl 2’ or
‘--LONG_NAME=VALUE’ (dash-dash-key-equal-value), e.g., ‘--dbg_lvl=2’.
Thus the following are all valid for the ‘-D’ (short version) or
‘--dbg_lvl’ (long version) command line option.
     ncks -D 3 in.nc        # Short option, preferred form
     ncks -D3 in.nc         # Short option, alternate form
     ncks --dbg_lvl=3 in.nc # Long option, preferred form
     ncks --dbg_lvl 3 in.nc # Long option, alternate form
The third example is preferred for two reasons.  First, ‘--dbg_lvl’ is
more specific and less ambiguous than ‘-D’.  The long option format
makes scripts more self documenting and less error-prone.  Often long
options are named after the source code variable whose value they carry.
Second, the equals sign ‘=’ joins the key (i.e., LONG_NAME) to the value
in an uninterruptible text block.  Experience shows that users are less
likely to mis-parse commands when restricted to this form.

* Menu:

* Truncating Long Options::
* Multi-arguments::

   ---------- Footnotes ----------

   (1) If a ‘getopt_long’ function cannot be found on the system, NCO
will use the ‘getopt_long’ from the ‘my_getopt’ package by Benjamin
Sittler <bsittler@iname.com>.  This is BSD-licensed software available
from <http://www.geocities.com/ResearchTriangle/Node/9405/#my_getopt>.


File: nco.info,  Node: Truncating Long Options,  Next: Multi-arguments,  Prev: Command Line Options,  Up: Command Line Options

3.4.1 Truncating Long Options
-----------------------------

GNU implements a superset of the POSIX standard.  Their superset accepts
any unambiguous truncation of a valid option:
     ncks -D 3 in.nc        # Short option
     ncks --dbg_lvl=3 in.nc # Long option, full form
     ncks --dbg=3 in.nc     # Long option, OK unambiguous truncation
     ncks --db=3 in.nc      # Long option, OK unambiguous truncation
     ncks --d=3 in.nc       # Long option, ERROR ambiguous truncation
The first four examples are equivalent and will work as expected.  The
final example will exit with an error since ‘ncks’ cannot disambiguate
whether ‘--d’ is intended as a truncation of ‘--dbg_lvl’, of
‘--dimension’, or of some other long option.

   NCO provides many long options for common switches.  For example, the
debugging level may be set in all operators with any of the switches
‘-D’, ‘--debug-level’, or ‘--dbg_lvl’.  This flexibility allows users to
choose their favorite mnemonic.  For some, it will be ‘--debug’ (an
unambiguous truncation of ‘--debug-level’, and other will prefer
‘--dbg’.  Interactive users usually prefer the minimal amount of typing,
i.e., ‘-D’.  We recommend that re-usable scripts employ long options to
facilitate self-documentation and maintainability.

   This manual generally uses the short option syntax in examples.  This
is for historical reasons and to conserve space in printed output.
Users are expected to pick the unambiguous truncation of each option
name that most suits their taste.


File: nco.info,  Node: Multi-arguments,  Prev: Truncating Long Options,  Up: Command Line Options

3.4.2 Multi-arguments
---------------------

As of NCO version 4.6.2 (November, 2016), NCO accepts multiple key-value
pair options for a single feature to be joined together into a single
extended argument called a “multi-argument”, sometimes abbreviated MTA.
Only four NCO features accept multiple key-value pairs that can be
aggregated into multi-arguments.  These features are: Global Attribute
Addition options indicated via ‘--gaa’ (*note Global Attribute
Addition::); Image Manipulation indicated via ‘--trr’(1),
Precision-Preserving Compression options are indicated via ‘--ppc’
(*note Precision-Preserving Compression::); and Regridding options are
indicated via ‘--rgr’ (*note Regridding::).  Arguments to these four
indicator options take the form of key-value pairs, e.g., ‘--rgr
KEY=VAL’.  As of version 5.2.5 (May 2024), NCO changed to preferring
‘--qnt’ over ‘--ppc’ for quantization algorithms.  The two are simply
synonyms, and backward compatibility is maintained.

   These features have so many options that making each key its own
command line option would pollute the namespace of NCO's global options.
Yet supplying multiple options to each indicator option one-at-a-time
can result in command lines overpopulated with indicator switches (e.g.,
‘--rgr’):
     ncks --rgr grd_ttl='Title' --rgr grid=grd.nc --rgr latlon=129,256 \
          --rgr lat_typ=fv --rgr lon_typ=grn_ctr ...

   Multi-arguments combine all the indicator options into one option
that receives a single argument that comprises all the original
arguments glued together by a delimiter, which is, by default, ‘#’.
Thus the multi-argument version of the above example is
     ncks --rgr grd_ttl='Title'#grid=grd.nc#latlon=129,256#lat_typ=fv#lon_typ=grn_ctr
   Note the aggregation of all KEY=VAL pairs into a single argument.
NCO simply splits this argument at each delimiter, and processes the
sub-arguments as if they had been passed with their own indicator
option.  Multi-arguments produce the same results, and may be mixed
with, traditional indicator options supplied one-by-one.

   As mentioned previously, the multi-argument delimiter string is, by
default, the hash-sign ‘#’.  When any KEY=VAL pair contains the default
delimiter, the user must specify a custom delimiter string so that
options are parsed correctly.  The options to change the multi-argument
delimiter string are ‘--mta_dlm=DELIM_STRING’ or
‘--dlm_mta=DELIM_STRING’, where DELIM_STRING can be any single or
multi-character string that (1) is not contained in any KEY or VAL
string; and (2) will not confuse the shell.  For example, to use
multi-arguments to pass a string that includes the hash symbol (the
default delimiter is ‘#’), one must also change the delimiter so
something besides hash, e.g., a colon ‘:’:
     ncks --dlm=":" --gaa foo=bar:foo2=bar2:foo3,foo4="hash # is in value" 
     ncks --dlm=":" --gaa foo=bar:foo2=bar2:foo3,foo4="Thu Sep 15 13\:03\:18 PDT 2016"
     ncks --dlm="csz" --gaa foo=barcszfoo2=bar2cszfoo3,foo4="Long text"
   In the second example, the colons that are escaped with the backslash
become literal characters.  Many characters have special shell meanings
and so must be escaped by a single or double backslash or enclosed in
single quotes to prevent interpolation.  These special characters
include ‘:’, ‘$’, ‘%’, ‘*’, ‘@’, and ‘&’.  If VAL is a long text string
that could contain the default delimiter, then delimit with a unique
multi-character string such as ‘csz’ in the third example.

   As of NCO version 4.6.7 (May, 2017), multi-argument flags no longer
need be specified as key-value pairs.  By definition a flag sets a
boolean value to either True or False.  Previously MTA flags had to
employ key-value pair syntax, e.g., ‘--rgr infer=Y’ or ‘--rgr
no_cll_msr=anything’ in order to parse correctly.  Now the MTA parser
accepts flags in the more intuitive syntax where they are listed by
name, i.e., the flag name alone indicates the flag to set, e.g., ‘--rgr
infer’ or ‘--rgr no_cll_msr’ are valid.  A consequence of this is that
flags in multi-argument strings appear as straightforward flag names,
e.g., ‘--rgr infer#no_cll_msr#latlon=129,256’.  It is also valid to
prefix flags in multi-arument strings with single or double-dashes to
make the flags more visible, e.g., ‘--rgr
latlon=129,256#--infer#-no_cll_msr’.

   ---------- Footnotes ----------

   (1) NCO supports decoding ENVI images in support of the DOE Terraref
project.  These options are indicated via the ‘ncks’ ‘--trr’ switch, and
are otherwise undocumented.  Please contact us if more support and
documentation of handling of ENVI BIL, BSQ, and BIP images would be
helpful


File: nco.info,  Node: Sanitization of Input,  Next: Specifying Input Files,  Prev: Command Line Options,  Up: Shared features

3.5 Sanitization of Input
=========================

Availability: All operators
   NCO is often installed in system directories (although not with
Conda), and on some production machines it may have escalated
privileges.  Since NCO manipulates files by using ‘system()’ calls
(e.g., to move and copy them with ‘mv’ and ‘cp’) it makes sense to audit
it for vulnerabilities and protect it from malicious users trying to
exploit security gaps.  Securing NCO against malicious attacks is
multi-faceted, and involves careful memory management and auditing of
user-input.  In versions 4.7.3-4.7.6 (March-September, 2018), NCO
implements a whitelist of characters allowed in user-specified
filenames.  This whitelist proved unpopular mainly because it proscribed
certain character combinations that could appear in automatically
generated files, and was therefore turned-off in 4.7.7 and following
versions.  The whitelist is described here for posterity and for
possible improvement and re-introduction: The purpose of the whitelist
was to prevent malicious users from injecting filename strings that
could be used for attacks.  The whitelist allowed only these characters:
     abcdefghijklmnopqrstuvwxyz
     ABCDEFGHIJKLMNOPQRSTUVWXYZ
     1234567890_-.@ :%/
   The backslash character ‘\’ was also whitelisted for Windows only.
This whitelist allows filenames to be URLs, include username prefixes,
and standard non-alphabetic characters.  The implied blacklist included
these characters
     ;|<>[](),&*?'"
   This blacklist rules-out strings that may contain dangerous commands
and injection attacks.  If you would like any of these characters
whitelisted, please contact us and include a compelling real-world
use-case.

   The DAP protocol supports accessing files with so-called "constraint
expressions".  NCO allows access to a wider set of whitelisted
characters for files whose names indicate the DAP protocol.  This is
defined as any filename beginning with the string ‘http://’, ‘https://’,
or ‘dap4://’.  The whitelist for these files is expanded to include
these characters:
     #=:[];|{}/<>

   The whitelist method is straightforward, and does not interfere with
NCO's globbing feature.  The whitelist applies only to filenames because
they are handled by shell commands passed to the ‘system()’ function.
However, the whitelist method is applicable to other user-input such as
variable lists, hyperslab arguments, etc.  Hence, the whitelist could be
applied to other user-input in the future.


File: nco.info,  Node: Specifying Input Files,  Next: Specifying Output Files,  Prev: Sanitization of Input,  Up: Shared features

3.6 Specifying Input Files
==========================

Availability (‘-n’): ‘nces’, ‘ncecat’, ‘ncra’, ‘ncrcat’
Availability (‘-p’): All operators
Availability (‘stdin’): All operators
Short options: ‘-n’, ‘-p’
Long options: ‘--nintap’, ‘--pth’, ‘--path’
   It is important that users be able to specify multiple input files
without typing every filename in full, often a tedious task even by
graduate student standards.  There are four different ways of specifying
input files to NCO: explicitly typing each, using UNIX shell wildcards,
and using the NCO ‘-n’ and ‘-p’ switches (or their long option
equivalents, ‘--nintap’ or ‘--pth’ and ‘--path’, respectively).
Techniques to augment these methods to specify arbitrary numbers (e.g.,
thousands) and patterns of filenames are discussed separately (*note
Large Numbers of Files::).

   To illustrate these methods, consider the simple problem of using
‘ncra’ to average five input files, ‘85.nc’, ‘86.nc’, ... ‘89.nc’, and
store the results in ‘8589.nc’.  Here are the four methods in order.
They produce identical answers.
     ncra 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
     ncra 8[56789].nc 8589.nc
     ncra 8?.nc 8589.nc
     ncra -p INPUT-PATH 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
     ncra -n 5,2,1 85.nc 8589.nc
   The first method (explicitly specifying all filenames) works by brute
force.  The second method relies on the operating system shell to “glob”
(expand) the “regular expression” ‘8[56789].nc’.  The shell then passes
the valid filenames (those which match the regular expansion) to ‘ncra’.
In this case ‘ncra’ never knows that a regular expression was used,
because the shell intercepts and expands and matches the regular
expression before ‘ncra’ is actually invoked.  The third method is uses
globbing with a different regular expression that is less safe (it will
also match unwanted files such as ‘81.nc’ and ‘8Z.nc’ if present).  The
fourth method uses the ‘-p INPUT-PATH’ argument to specify the directory
where all the input files reside.  NCO prepends INPUT-PATH (e.g.,
‘/data/username/model’) to all INPUT-FILES (though not to OUTPUT-FILE).
Thus, using ‘-p’, the path to any number of input files need only be
specified once.  Note INPUT-PATH need not end with ‘/’; the ‘/’ is
automatically generated if necessary.

   The last method passes (with ‘-n’) syntax concisely describing the
entire set of filenames (1).  This option is only available with the
“multi-file operators”: ‘ncra’, ‘ncrcat’, ‘nces’, and ‘ncecat’.  By
definition, multi-file operators are able to process an arbitrary number
of INPUT-FILES.  This option is very useful for abbreviating lists of
filenames representable as
ALPHANUMERIC_PREFIX+NUMERIC_SUFFIX+‘.’+FILETYPE where
ALPHANUMERIC_PREFIX is a string of arbitrary length and composition,
NUMERIC_SUFFIX is a fixed width field of digits, and FILETYPE is a
standard filetype indicator.  For example, in the file ‘ccm3_h0001.nc’,
we have ALPHANUMERIC_PREFIX = ‘ccm3_h’, NUMERIC_SUFFIX = ‘0001’, and
FILETYPE = ‘nc’.

   NCO decodes lists of such filenames encoded using the ‘-n’ syntax.
The simpler (three-argument) ‘-n’ usage takes the form ‘-n
FILE_NUMBER,DIGIT_NUMBER,NUMERIC_INCREMENT’ where FILE_NUMBER is the
number of files, DIGIT_NUMBER is the fixed number of numeric digits
comprising the NUMERIC_SUFFIX, and NUMERIC_INCREMENT is the constant,
integer-valued difference between the NUMERIC_SUFFIX of any two
consecutive files.  The value of ALPHANUMERIC_PREFIX is taken from the
input file, which serves as a template for decoding the filenames.  In
the example above, the encoding ‘-n 5,2,1’ along with the input file
name ‘85.nc’ tells NCO to construct five (5) filenames identical to the
template ‘85.nc’ except that the final two (2) digits are a numeric
suffix to be incremented by one (1) for each successive file.  Currently
FILETYPE may be either be empty, ‘nc’, ‘h5’, ‘cdf’, ‘hdf’, ‘hd5’, or
‘he5’.  If present, these FILETYPE suffixes (and the preceding ‘.’) are
ignored by NCO as it uses the ‘-n’ arguments to locate, evaluate, and
compute the NUMERIC_SUFFIX component of filenames.

   Recently the ‘-n’ option has been extended to allow convenient
specification of filenames with "circular" characteristics.  This means
it is now possible for NCO to automatically generate filenames which
increment regularly until a specified maximum value, and then wrap back
to begin again at a specified minimum value.  The corresponding ‘-n’
usage becomes more complex, taking one or two additional arguments for a
total of four or five, respectively: ‘-n
FILE_NUMBER,DIGIT_NUMBER,NUMERIC_INCREMENT[,NUMERIC_MAX[,NUMERIC_MIN]]’
where NUMERIC_MAX, if present, is the maximum integer-value of
NUMERIC_SUFFIX and NUMERIC_MIN, if present, is the minimum integer-value
of NUMERIC_SUFFIX.  Consider, for example, the problem of specifying
non-consecutive input files where the filename suffixes end with the
month index.  In climate modeling it is common to create summertime and
wintertime averages which contain the averages of the months
June-July-August, and December-January-February, respectively:
     ncra -n 3,2,1 85_06.nc 85_0608.nc
     ncra -n 3,2,1,12 85_12.nc 85_1202.nc
     ncra -n 3,2,1,12,1 85_12.nc 85_1202.nc
   The first example shows that three arguments to the ‘-n’ option
suffice to specify consecutive months (‘06, 07, 08’) which do not "wrap"
back to a minimum value.  The second example shows how to use the
optional fourth and fifth elements of the ‘-n’ option to specify a wrap
value.  The fourth argument to ‘-n’, when present, specifies the maximum
integer value of NUMERIC_SUFFIX.  In the example the maximum value
is 12, which will be formatted as ‘12’ in the filename string.  The
fifth argument to ‘-n’, when present, specifies the minimum integer
value of NUMERIC_SUFFIX.  The default minimum filename suffix is 1,
which is formatted as ‘01’ in this case.  Thus the second and third
examples have the same effect, that is, they automatically generate, in
order, the filenames ‘85_12.nc’, ‘85_01.nc’, and ‘85_02.nc’ as input to
NCO.

   As of NCO version 4.5.2 (September, 2015), NCO supports an optional
sixth argument to ‘-n’, the month-indicator.  The month-indicator
affirms to NCO that the right-most digits being manipulated in the
generated filenames correspond to month numbers (with January formatted
as ‘01’ and December as ‘12’).  Moreover, it assumes digits to the left
of the month are the year.  The full (six-argument) ‘-n’ usage takes the
form ‘-n
FILE_NUMBER,DIGIT_NUMBER,MONTH_INCREMENT,MAX_MONTH,MIN_MONTH,‘yyyymm’’.
The ‘yyyymm’ string is a clunky way (can you think of a clearer way?)
to tell NCO to enumerate files in year-month mode.  When present,
‘yyyymm’ string causes NCO to automatically generate a filename series
whose right-most two digits increment from MIN_MONTH by MONTH_INCREMENT
up to MAX_MONTH and then the leftmost digits (i.e., the year) increment
by one, and the whole process is repeated until the FILE_NUMBER
filenames are generated.
     ncrcat -n 3,6,1,12,1         198512.nc 198512_198502.nc
     ncrcat -n 3,6,1,12,1,yyyymm  198512.nc 198512_198602.nc
     ncrcat -n 3,6,1,12,12,yyyymm 198512.nc 198512_198712.nc
   The first command above concatenates three files (‘198512.nc’,
‘198501.nc’, ‘198502.nc’) into the output file.  The second command
above concatenates three files (‘198512.nc’, ‘198601.nc’, ‘198602.nc’).
The ‘yyyymm’-indicator causes the left-most digits to increment each
time the right-most two digits reach their maximum and then wrap.  The
first command does not have the indicator so it is always 1985.  The
third command concatenates three files (‘198512.nc’, ‘198612.nc’,
‘198712.nc’).

   As of NCO version 5.1.1, released in November, 2022, all operators
support specifying input files via ‘stdin’.  This capability was
implemented with NCZarr in mind, though it can also be used with
traditional POSIX files.  The ‘ncap2’, ‘ncks’, ‘ncrename’, and ‘ncatted’
operators accept one or two filenames as positional arguments.  If the
input file for these operators is provided via ‘stdin’, then the output
file, if any, must be specified with ‘-o out.nc’ so the operators know
whether to check ‘stdin’.  Multi-file operators (‘ncra’, ‘ncea’,
‘ncrcat’, ‘ncecat’) will continue to identify the last positional
argument as the output file unless the ‘-o out.nc’ form is used.  The
best best practice is to use ‘-o out.nc’ to specify output filenames
when ‘stdin’ is used for input filenames:
     echo in.nc | ncks            
     echo in.nc | ncks -o out.nc
     echo "in1.nc in2.nc" | ncbo -o out.nc
     echo "in1.nc in2.nc" | ncflint -o out.nc
   For the provenance reasons dicussed above (*note Large Numbers of
Files::), all filenames input via ‘stdin’ are stored as global
attributes in the *note File List Attributes::).

   ---------- Footnotes ----------

   (1) The ‘-n’ option is a backward-compatible superset of the ‘NINTAP’
option from the NCAR CCM Processor.  The CCM Processor was
custom-written Fortran code maintained for many years by Lawrence Buja
at NCAR, and phased-out in the late 1990s.  NCO copied some ideas, like
‘NINTAP’-functionality, from CCM Processor capabilities.


File: nco.info,  Node: Specifying Output Files,  Next: Remote storage,  Prev: Specifying Input Files,  Up: Shared features

3.7 Specifying Output Files
===========================

Availability: All operators
Short options: ‘-o’
Long options: ‘--fl_out’, ‘--output’
   NCO commands produce no more than one output file, FL_OUT.
Traditionally, users specify FL_OUT as the final argument to the
operator, following all input file names.  This is the “positional
argument” method of specifying input and ouput file names.  The
positional argument method works well in most applications.  NCO also
supports specifying FL_OUT using the command line switch argument
method, ‘-o FL_OUT’.

   Specifying FL_OUT with a switch, rather than as a positional
argument, allows FL_OUT to precede input files in the argument list.
This is particularly useful with multi-file operators for three reasons.
Multi-file operators may be invoked with hundreds (or more) filenames.
Visual or automatic location of FL_OUT in such a list is difficult when
the only syntactic distinction between input and output files is their
position.  Second, specification of a long list of input files may be
difficult (*note Large Numbers of Files::).  Making the input file list
the final argument to an operator facilitates using ‘xargs’ for this
purpose.  Some alternatives to ‘xargs’ are heinous and undesirable.
Finally, many users are more comfortable specifying output files with
‘-o FL_OUT’ near the beginning of an argument list.  Compilers and
linkers are usually invoked this way.

   Users should specify FL_OUT using either (not both) method.  If
FL_OUT is specified twice (once with the switch and once as the last
positional argument), then the positional argument takes precedence.


File: nco.info,  Node: Remote storage,  Next: Retaining Retrieved Files,  Prev: Specifying Output Files,  Up: Shared features

3.8 Accessing Remote Files
==========================

Availability: All operators
Short options: ‘-p’, ‘-l’
Long options: ‘--pth’, ‘--path’, ‘--lcl’, ‘--local’
   All NCO operators can retrieve files from remote sites as well as
from the local file system.  A remote site can be an anonymous FTP
server, a machine on which the user has ‘rcp’, ‘scp’, or ‘sftp’
privileges, NCAR's Mass Storage System (MSS), or an OPeNDAP server.
Examples of each are given below, following a brief description of the
particular access protocol.

   To access a file via an anonymous FTP server, simply supply the
remote file's URL.  Anonymous FTP usually requires no further
credentials, e.g., no ‘.netrc’ file is necessary.  FTP is an
intrinsically insecure protocol because it transfers passwords in plain
text format.  Users should access sites using anonymous FTP, or better
yet, secure FTP (SFTP, see below) when possible.  Some FTP servers
require a login/password combination for a valid user account.  NCO
allows transactions that require additional credentials so long as the
required information is stored in the ‘.netrc’ file.  Usually this
information is the remote machine name, login, and password, in plain
text, separated by those very keywords, e.g.,
     machine dust.ess.uci.edu login zender password bushlied
   Eschew using valuable passwords for FTP transactions, since ‘.netrc’
passwords are potentially exposed to eavesdropping software (1).

   SFTP, i.e., secure FTP, uses SSH-based security protocols that solve
the security issues associated with plain FTP.  NCO supports SFTP
protocol access to files specified with a homebrew syntax of the form
     sftp://machine.domain.tld:/path/to/filename
   Note the second colon following the top-level-domain, ‘tld’.  This
syntax is a hybrid between an FTP URL and standard remote file syntax.

   To access a file using ‘rcp’ or ‘scp’, specify the Internet address
of the remote file.  Of course in this case you must have ‘rcp’ or ‘scp’
privileges which allow transparent (no password entry required) access
to the remote machine.  This means that ‘~/.rhosts’ or
‘~/ssh/authorized_keys’ must be set accordingly on both local and remote
machines.

   To access a file on a High Performance Storage System (HPSS) (such as
that at NCAR, ECMWF, LANL, DKRZ, LLNL) specify the full HPSS pathname of
the remote file and use the ‘--hpss’ flag.  Then NCO will attempt to
detect whether the local machine has direct (synchronous) HPSS access.
If so, NCO attempts to use the Hierarchical Storage Interface (HSI)
command ‘hsi get’ (2).

   The following examples show how one might analyze files stored on
remote systems.
     ncks -l . ftp://dust.ess.uci.edu/pub/zender/nco/in.nc
     ncks -l . sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco/in.nc
     ncks -l . dust.ess.uci.edu:/home/zender/nco/data/in.nc
     ncks -l . /ZENDER/nco/in.nc # NCAR (broken old MSS path)
     ncks -l . --hpss /home/zender/nco/in.nc # NCAR HPSS
     ncks -l . http://thredds-test.ucar.edu/thredds/dodsC/testdods/in.nc
The first example works verbatim if your system is connected to the
Internet and is not behind a firewall.  The second example works if you
have ‘sftp’ access to the machine ‘dust.ess.uci.edu’.  The third example
works if you have ‘rcp’ or ‘scp’ access to the machine
‘dust.ess.uci.edu’.  The fourth and fifth examples work on NCAR
computers with local access to the HPSS ‘hsi get’ command (3).  The
sixth command works if your local version of NCO is OPeNDAP-enabled
(this is fully described in *note OPeNDAP::), or if the remote file is
accessible via ‘wget’.  The above commands can be rewritten using the
‘-p INPUT-PATH’ option as follows:
     ncks -p ftp://dust.ess.uci.edu/pub/zender/nco -l . in.nc
     ncks -p sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco -l . in.nc
     ncks -p dust.ess.uci.edu:/home/zender/nco -l . in.nc
     ncks -p /ZENDER/nco -l . in.nc
     ncks -p /home/zender/nco -l . --hpss in.nc # HPSS
     ncks -p http://thredds-test.ucar.edu/thredds/dodsC/testdods \
          -l . in.nc
Using ‘-p’ is recommended because it clearly separates the INPUT-PATH
from the filename itself, sometimes called the “stub”.  When INPUT-PATH
is not explicitly specified using ‘-p’, NCO internally generates an
INPUT-PATH from the first input filename.  The automatically generated
INPUT-PATH is constructed by stripping the input filename of everything
following the final ‘/’ character (i.e., removing the stub).  The ‘-l
OUTPUT-PATH’ option tells NCO where to store the remotely retrieved
file.  It has no effect on locally-retrieved files, or on the output
file.  Often the path to a remotely retrieved file is quite different
than the path on the local machine where you would like to store the
file.  If ‘-l’ is not specified then NCO internally generates an
OUTPUT-PATH by simply setting OUTPUT-PATH equal to INPUT-PATH stripped
of any machine names.  If ‘-l’ is not specified and the remote file
resides on a detected HPSS system, then the leading character of
INPUT-PATH, ‘/’, is also stripped from OUTPUT-PATH.  Specifying
OUTPUT-PATH as ‘-l ./’ tells NCO to store the remotely retrieved file
and the output file in the current directory.  Note that ‘-l .’ is
equivalent to ‘-l ./’ though the latter is syntactically more clear.

* Menu:

* OPeNDAP::

   ---------- Footnotes ----------

   (1) NCO does not implement command line options to specify FTP logins
and passwords because copying those data into the ‘history’ global
attribute in the output file (done by default) poses an unacceptable
security risk.

   (2) The ‘hsi’ command must be in the user's path in one of the
following directories: ‘/usr/local/bin’, ‘/opt/hpss/bin’, or
‘/ncar/opt/hpss/hsi’.  Tell us if the HPSS installation at your site
places the ‘hsi’ command in a different location, and we will add that
location to the list of acceptable paths to search for ‘hsi’.

   (3) NCO supported the old NCAR Mass Storage System (MSS) until
version 4.0.7 in April, 2011.  NCO supported MSS-retrievals via a
variety of mechanisms including the ‘msread’, ‘msrcp’, and ‘nrnet’
commands invoked either automatically or with sentinels like ‘ncks -p
mss:/ZENDER/nco -l . in.nc’.  Once the MSS was decommissioned in March,
2011, support for these retrieval mechanisms was replaced by support for
HPSS.


File: nco.info,  Node: OPeNDAP,  Prev: Remote storage,  Up: Remote storage

3.8.1 OPeNDAP
-------------

The Distributed Oceanographic Data System (DODS) provides useful
replacements for common data interface libraries like netCDF. The DODS
versions of these libraries implement network transparent access to data
via a client-server data access protocol that uses the HTTP protocol for
communication.  Although DODS-technology originated with oceanography
data, it applyies to virtually all scientific data.  In recognition of
this, the data access protocol underlying DODS (which is what NCO cares
about) has been renamed the Open-source Project for a Network Data
Access Protocol, OPeNDAP.  We use the terms DODS and OPeNDAP
interchangeably, and often write OPeNDAP/DODS for now.  In the future we
will deprecate DODS in favor of DAP or OPeNDAP, as appropriate (1).

   NCO may be DAP-enabled by linking NCO to the OPeNDAP libraries.  This
is described in the OPeNDAP documentation and automagically implemented
in NCO build mechanisms (2).  The ‘./configure’ mechanism automatically
enables NCO as OPeNDAP clients if it can find the required OPeNDAP
libraries.  Since about 2010 the netCDF library can be configured (with
‘--enable-dap’) to build DAP directly into the netCDF library, which NCO
automatically links to, so DAP need not be installed as a third-party
library.  It has been so many years since NCO has needed to support
linking to DAP installed outside of the netCDF library that is is
unclear whether this configuration (3).  still works.  The ‘$DODS_ROOT’
environment variable may be used to override the default OPeNDAP library
location at NCO compile-time.  Building NCO with ‘bld/Makefile’ and the
command ‘make DODS=Y’ adds the (non-intuitive) commands to link to the
OPeNDAP libraries installed in the ‘$DODS_ROOT’ directory.  The file
‘doc/opendap.sh’ contains a generic script intended to help users
install OPeNDAP before building NCO.  The documentation at the OPeNDAP
Homepage (http://www.opendap.org) is voluminous.  Check there and on the
DODS mail lists
(http://www.unidata.ucar.edu/software/dods/home/mailLists/).  to learn
more about the extensive capabilities of OPeNDAP (4).

   Once NCO is DAP-enabled the operators are OPeNDAP clients.  All
OPeNDAP clients have network transparent access to any files controlled
by a OPeNDAP server.  Simply specify the input file path(s) in URL
notation and all NCO operations may be performed on remote files made
accessible by a OPeNDAP server.  This command tests the basic
functionality of OPeNDAP-enabled NCO clients:
     % ncks -O -o ~/foo.nc -C -H -v one -l /tmp \
       -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in.nc
     % ncks -H -v one ~/foo.nc
     one = 1
   The ‘one = 1’ outputs confirm (first) that ‘ncks’ correctly retrieved
data via the OPeNDAP protocol and (second) that ‘ncks’ created a valid
local copy of the subsetted remote file.  With minor changes to the
above command, netCDF4 can be used as both the input and output file
format:
     % ncks -4 -O -o ~/foo.nc -C -H -v one -l /tmp \
       -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in_4.nc
     % ncks -H -v one ~/foo.nc
     one = 1
   And, of course, OPeNDAP-enabled NCO clients continue to support
orthogonal features such as UDUnits (*note UDUnits Support::):
     % ncks -u -C -H -v wvl -d wvl,'0.4 micron','0.7 micron' \
       -p http://thredds-test.ucar.edu/thredds/dodsC/testdods in_4.nc
     % wvl[0]=5e-07 meter

   The next command is a more advanced example which demonstrates the
real power of OPeNDAP-enabled NCO clients.  The ‘ncwa’ client requests
an equatorial hyperslab from remotely stored NCEP reanalyses data of the
year 1969.  The NOAA OPeNDAP server (hopefully!)  serves these data.
The local ‘ncwa’ client then computes and stores (locally) the regional
mean surface pressure (in Pa).
     ncwa -O -C -a lat,lon,time -d lon,-10.,10. -d lat,-10.,10. \
     http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.dailyavgs/surface/pres.sfc.1969.nc ~/foo.nc
All with one command!  The data in this particular input file also
happen to be packed (*note Methods and functions::), although this
complication is transparent to the user since NCO automatically unpacks
data before attempting arithmetic.

   NCO obtains remote files from the OPeNDAP server (e.g.,
‘www.cdc.noaa.gov’) rather than the local machine.  Input files are
first copied to the local machine, then processed.  The OPeNDAP server
performs data access, hyperslabbing, and transfer to the local machine.
This allows the I/O to appear to NCO as if the input files were local.
The local machine performs all arithmetic operations.  Only the
hyperslabbed output data are transferred over the network (to the local
machine) for the number-crunching to begin.  The advantages of this are
obvious if you are examining small parts of large files stored at remote
locations.

   Natually there are many versions of OPeNDAP servers supplying data
and bugs in the server can appear to be bugs in NCO.  However, with very
few exceptions (5) an NCO command that works on a local file must work
across an OPeNDAP connection or else there is a bug in the server.  This
is because NCO does nothing special to handle files served by OPeNDAP,
the whole process is (supposed to be) completely transparent to the
client NCO software.  Therefore it is often useful to try NCO commands
on various OPeNDAP servers in order to isolate whether a problem may be
due to a bug in the OPeNDAP server on a particular machine.  For this
purpose, one might try variations of the following commands that access
files on public OPeNDAP servers:
     # Strided access to HDF5 file
     ncks -v Time -d Time,0,10,2 http://eosdap.hdfgroup.uiuc.edu:8080/opendap/data/NASAFILES/hdf5/BUV-Nimbus04_L3zm_v01-00-2012m0203t144121.h5
     # Strided access to netCDF3 file
     ncks -O -D 1 -d time,1 -d lev,0 -d lat,0,100,10 -d lon,0,100,10 -v u_velocity http://nomads.ncep.noaa.gov:9090/dods/rtofs/rtofs_global20140303/rtofs_glo_2ds_forecast_daily_prog ~/foo.nc
These servers were operational at the time of writing, March 2014.
Unfortunately, administrators often move or rename path directories.
Recommendations for additional public OPeNDAP servers on which to test
NCO are welcome.

   ---------- Footnotes ----------

   (1) DODS is being deprecated because it is ambiguous, referring both
to a protocol and to a collection of (oceanography) data.  It is
superceded by two terms.  DAP is the discipline-neutral Data Access
Protocol at the heart of DODS.  The National Virtual Ocean Data System
(NVODS) refers to the collection of oceanography data and oceanographic
extensions to DAP.  In other words, NVODS is implemented with OPeNDAP.
OPeNDAP is _also_ the open source project which maintains, develops, and
promulgates the DAP standard.  OPeNDAP and DAP really are
interchangeable.  Got it yet?

   (2) Automagic support for DODS version 3.2.x was deprecated in
December, 2003 after NCO version 2.8.4.  NCO support for OPeNDAP
versions 3.4.x commenced in December, 2003, with NCO version 2.8.5.  NCO
support for OPeNDAP versions 3.5.x commenced in June, 2005, with NCO
version 3.0.1.  NCO support for OPeNDAP versions 3.6.x commenced in
June, 2006, with NCO version 3.1.3.  NCO support for OPeNDAP versions
3.7.x commenced in January, 2007, with NCO version 3.1.9.

   (3) The minimal set of libraries required to build NCO as OPeNDAP
clients, where OPeNDAP is supplied as a separate library apart from
‘libnetcdf.a’, are, in link order, ‘libnc-dap.a’, ‘libdap.a’, and
‘libxml2’ and ‘libcurl.a’.

   (4) We are most familiar with the OPeNDAP ability to enable
network-transparent data access.  OPeNDAP has many other features,
including sophisticated hyperslabbing and server-side processing via
“constraint expressions”.  If you know more about this, please consider
writing a section on "OPeNDAP Capabilities of Interest to NCO Users" for
incorporation in the ‘NCO User Guide’.

   (5) For example, DAP servers do not like variables with periods (".")
in their names even though this is perfectly legal with netCDF. Such
names may cause the DAP service to fail because DAP interprets the
period as structure delimiter in an HTTP query string.


File: nco.info,  Node: Retaining Retrieved Files,  Next: File Formats and Conversion,  Prev: Remote storage,  Up: Shared features

3.9 Retaining Retrieved Files
=============================

Availability: All operators
Short options: ‘-R’
Long options: ‘--rtn’, ‘--retain’
   In order to conserve local file system space, files retrieved from
remote locations are automatically deleted from the local file system
once they have been processed.  Many NCO operators were constructed to
work with numerous large (e.g., 200 MB) files.  Retrieval of multiple
files from remote locations is done serially.  Each file is retrieved,
processed, then deleted before the cycle repeats.  In cases where it is
useful to keep the remotely-retrieved files on the local file system
after processing, the automatic removal feature may be disabled by
specifying ‘-R’ on the command line.

   Invoking ‘-R’ disables the default printing behavior of ‘ncks’.  This
allows ‘ncks’ to retrieve remote files without automatically trying to
print them.  See *note ncks netCDF Kitchen Sink::, for more details.

   Note that the remote retrieval features of NCO can always be used to
retrieve _any_ file, including non-netCDF files, via ‘SSH’, anonymous
FTP, or ‘msrcp’.  Often this method is quicker than using a browser, or
running an FTP session from a shell window yourself.  For example, say
you want to obtain a JPEG file from a weather server.
     ncks -R -p ftp://weather.edu/pub/pix/jpeg -l . storm.jpg
In this example, ‘ncks’ automatically performs an anonymous FTP login to
the remote machine and retrieves the specified file.  When ‘ncks’
attempts to read the local copy of ‘storm.jpg’ as a netCDF file, it
fails and exits, leaving ‘storm.jpg’ in the current directory.

   If your NCO is DAP-enabled (*note OPeNDAP::), then you may use NCO to
retrieve any files (including netCDF, HDF, etc.)  served by an OPeNDAP
server to your local machine.  For example,
     ncks -R -l . -p \
     http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.dailyavgs/surface \
       pres.sfc.1969.nc
   It may occasionally be useful to use NCO to transfer files when your
other preferred methods are not available locally.


File: nco.info,  Node: File Formats and Conversion,  Next: Zarr and NCZarr,  Prev: Retaining Retrieved Files,  Up: Shared features

3.10 File Formats and Conversion
================================

Availability: ‘ncap2’, ‘nces’, ‘ncecat’, ‘ncflint’, ‘ncks’, ‘ncpdq’,
‘ncra’, ‘ncrcat’, ‘ncwa’
Short options: ‘-3’, ‘-4’, ‘-5’, ‘-6’, ‘-7’
Long options: ‘--3’, ‘--4’, ‘--5’, ‘--6’, ‘--64bit_offset’, ‘--7’,
‘--fl_fmt’, ‘--netcdf4’
   All NCO operators support (read and write) all three (or four,
depending on how one counts) file formats supported by netCDF4.  The
default output file format for all operators is the input file format.
The operators listed under "Availability" above allow the user to
specify the output file format independent of the input file format.
These operators allow the user to convert between the various file
formats.  (The operators ‘ncatted’ and ‘ncrename’ do not support these
switches so they always write the output netCDF file in the same format
as the input netCDF file.)

* Menu:

* File Formats::
* Determining File Format::
* File Conversion::
* Autoconversion::


File: nco.info,  Node: File Formats,  Next: Determining File Format,  Prev: File Formats and Conversion,  Up: File Formats and Conversion

3.10.1 File Formats
-------------------

netCDF supports five types of files: ‘CLASSIC’, ‘64BIT_OFFSET’,
‘64BIT_DATA’, ‘NETCDF4’, and ‘NETCDF4_CLASSIC’.  The ‘CLASSIC’ (aka
‘CDF1’) format is the traditional 32-bit offset written by netCDF2 and
netCDF3.  As of 2005, nearly all netCDF datasets were in ‘CLASSIC’
format.  The ‘64BIT_OFFSET’ (originally called plain old ‘64BIT’) (aka
‘CDF2’) format was added in Fall, 2004.  As of 2010, many netCDF
datasets were in ‘64BIT_OFFSET’ format.  As of 2013, an increasing
number of netCDF datasets were in ‘NETCDF4_CLASSIC’ format.  The
‘64BIT_DATA’ (aka ‘CDF5’ or ‘PNETCDF’) format was added to netCDF in
January, 2016 and immediately supported by NCO.  Support for *note Zarr
and NCZarr:: backend storage formats was added to netCDF in 2021 and
supported by NCO in 2022.

   The ‘NETCDF4’ format uses HDF5 as the file storage layer.  The files
are (usually) created, accessed, and manipulated using the traditional
netCDF3 API (with numerous extensions).  The ‘NETCDF4_CLASSIC’ format
refers to netCDF4 files created with the ‘NC_CLASSIC_MODEL’ mask.  Such
files use HDF5 as the back-end storage format (unlike netCDF3), though
they incorporate only netCDF3 features.  Hence ‘NETCDF4_CLASSIC’ files
are entirely readable by applications that use only the netCDF3 API
(though the applications must be linked with the netCDF4 library).  NCO
must be built with netCDF4 to write files in the new ‘NETCDF4’ and
‘NETCDF4_CLASSIC’ formats, and to read files in these formats.  Datasets
in the default ‘CLASSIC’ or the newer ‘64BIT_OFFSET’ formats have
maximum backwards-compatibility with older applications.  NCO has deep
support for ‘NETCDF4’ formats.  If backwards compatibility is important,
and your datasets are too large for netCDF3, use ‘NETCDF4_CLASSIC’
instead of ‘CLASSIC’ format files.  NCO support for the ‘NETCDF4’ format
is complete and many high-performance disk/RAM efficient workflows
utilize this format.

   As mentioned above, all operators write use the input file format for
output files unless told otherwise.  Toggling the short option ‘-6’ or
the long option ‘--6’ or ‘--64bit_offset’ (or their KEY-VALUE equivalent
‘--fl_fmt=64bit_offset’) produces the netCDF3 64-bit offset format named
‘64BIT_OFFSET’.  NCO must be built with netCDF 3.6 or higher to produce
a ‘64BIT_OFFSET’ file.  As of NCO version 4.6.9 (September, 2017),
toggling the short option ‘-5’ or the long options ‘--5’,
‘--64bit_data’, ‘--cdf5’, or ‘--pnetcdf’ (or their KEY-VALUE equivalent
‘--fl_fmt=64bit_data’) produces the netCDF3 64-bit data format named
‘64BIT_DATA’.  This format is widely used by MPI-enabled modeling codes
because of its long association with PnetCDF. NCO must be built with
netCDF 4.4 or higher to produce a ‘64BIT_DATA’ file.

   Using the ‘-4’ switch (or its long option equivalents ‘--4’ or
‘--netcdf4’), or setting its KEY-VALUE equivalent ‘--fl_fmt=netcdf4’
produces a ‘NETCDF4’ file (i.e., with all supported HDF5 features).
Using the ‘-7’ switch (or its long option equivalent ‘--7’ (1), or
setting its KEY-VALUE equivalent ‘--fl_fmt=netcdf4_classic’ produces a
‘NETCDF4_CLASSIC’ file (i.e., with all supported HDF5 features like
compression and chunking but without groups or new atomic types).
Operators given the ‘-3’ (or ‘--3’) switch without arguments will
(attempt to) produce netCDF3 ‘CLASSIC’ output, even from netCDF4 input
files.

   Note that ‘NETCDF4’ and ‘NETCDF4_CLASSIC’ are the same binary format.
The latter simply causes a writing application to fail if it attempts to
write a ‘NETCDF4’ file that cannot be completely read by the netCDF3
library.  Conversely, ‘NETCDF4_CLASSIC’ indicates to a reading
application that all of the file contents are readable with the netCDF3
library.  NCO has supported reading/writing basic ‘NETCDF4’ and
‘NETCDF4_CLASSIC’ files since October, 2005.

   ---------- Footnotes ----------

   (1) The reason (and mnemonic) for ‘-7’ is that ‘NETCDF4_CLASSIC’
files include great features of both netCDF3 (compatibility) and netCDF4
(compression, chunking) and, well, 3+4=7.


File: nco.info,  Node: Determining File Format,  Next: File Conversion,  Prev: File Formats,  Up: File Formats and Conversion

3.10.2 Determining File Format
------------------------------

Input files often end with the generic ‘.nc’ suffix that leaves (perhaps
by intention) the internal file format ambiguous.  There are at least
three ways to discover the internal format of a netCDF-supported file.
These methods determine whether it is a classic (32-bit offset) or newer
64-bit offset netCDF3 format, or is a netCDF4 format.  Each method
returns the information using slightly different terminology that
becomes easier to understand with practice.

   First, examine the first line of global metadata output by ‘ncks -M’:
     % ncks -M foo_3.nc
     Summary of foo_3.nc: filetype = NC_FORMAT_CLASSIC, 0 groups ...
     % ncks -M foo_6.nc
     Summary of foo_6.nc: filetype = NC_FORMAT_64BIT_OFFSET, 0 groups ...
     % ncks -M foo_5.nc
     Summary of foo_5.nc: filetype = NC_FORMAT_CDF5, 0 groups ...
     % ncks -M foo_7.nc
     Summary of foo_7.nc: filetype = NC_FORMAT_NETCDF4_CLASSIC, 0 groups ...
     % ncks -M foo_4.nc
     Summary of foo_4.nc: filetype = NC_FORMAT_NETCDF4, 0 groups ...
   This method requires a netCDF4-enabled NCO version 3.9.0+ (i.e., from
2007 or later).  As of NCO version 4.4.0 (January, 2014), ‘ncks’ will
also print the extended or underlying format of the input file.  The
extended filetype will be one of the six underlying formats that are
accessible through the netCDF API.  These formats are ‘NC_FORMATX_NC3’
(classic and 64-bit versions of netCDF3 formats), ‘NC_FORMATX_NC_HDF5’
(classic and extended versions of netCDF4, and "pure" HDF5 format),
‘NC_FORMATX_NC_HDF4’ (HDF4 format), ‘NC_FORMATX_PNETCDF’ (PnetCDF
format), ‘NC_FORMATX_DAP2’ (accessed via DAP2 protocol), and
‘NC_FORMATX_DAP4’ (accessed via DAP4 protocol).  For example,
     % ncks -D 2 -M hdf.hdf
     Summary of hdf.hdf: filetype = NC_FORMAT_NETCDF4 (representation of \
       extended/underlying filetype NC_FORMAT_HDF4), 0 groups ...
     % ncks -D 2 -M http://thredds-test.ucar.edu/thredds/dodsC/testdods/in.nc
     Summary of http://thredds-test.ucar.edu/thredds/dodsC/testdods/in.nc: \
       filetype = NC_FORMAT_CLASSIC (representation of extended/underlying \
       filetype NC_FORMATX_DAP2), 0 groups
     % ncks -D 2 -M foo_4.nc
     Summary of foo_4.nc: filetype = NC_FORMAT_NETCDF4 (representation of \
       extended/underlying filetype NC_FORMAT_HDF5), 0 groups
   The extended filetype determines some of the capabilities that netCDF
has to alter the file.

   Second, query the file with ‘ncdump -k’:
     % ncdump -k foo_3.nc
     classic
     % ncdump -k foo_6.nc
     64-bit offset
     % ncdump -k foo_5.nc
     cdf5
     % ncdump -k foo_7.nc
     netCDF-4 classic model
     % ncdump -k foo_4.nc
     netCDF-4
   This method requires a netCDF4-enabled netCDF 3.6.2+ (i.e., from 2007
or later).

   The third option uses the POSIX-standard ‘od’ (octal dump) command:
     % od -An -c -N4 foo_3.nc
        C   D   F 001
     % od -An -c -N4 foo_6.nc
        C   D   F 002
     % od -An -c -N4 foo_5.nc
        C   D   F 005
     % od -An -c -N4 foo_7.nc
      211   H   D   F
     % od -An -c -N4 foo_4.nc
      211   H   D   F
   This option works without NCO and ‘ncdump’.  Values of ‘C D F 001’
and ‘C D F 002’ indicate 32-bit (classic) and 64-bit netCDF3 formats,
respectively, while values of ‘211 H D F’ indicate either of the newer
netCDF4 file formats.


File: nco.info,  Node: File Conversion,  Next: Autoconversion,  Prev: Determining File Format,  Up: File Formats and Conversion

3.10.3 File Conversion
----------------------

Let us demonstrate converting a file from any netCDF-supported input
format into any netCDF output format (subject to limits of the output
format).  Here the input file ‘in.nc’ may be in any of these formats:
netCDF3 (classic, 64bit_offset, 64bit_data), netCDF4 (classic and
extended), HDF4, HDF5, HDF-EOS (version 2 or 5), and DAP. The switch
determines the output format written in the comment: (1)
     ncks --fl_fmt=classic in.nc foo_3.nc # netCDF3 classic
     ncks --fl_fmt=64bit_offset in.nc foo_6.nc # netCDF3 64bit-offset
     ncks --fl_fmt=64bit_data in.nc foo_5.nc # netCDF3 64bit-data
     ncks --fl_fmt=cdf5 in.nc foo_5.nc # netCDF3 64bit-data
     ncks --fl_fmt=netcdf4_classic in.nc foo_7.nc # netCDF4 classic
     ncks --fl_fmt=netcdf4 in.nc foo_4.nc # netCDF4
     ncks -3 in.nc foo_3.nc # netCDF3 classic
     ncks --3 in.nc foo_3.nc # netCDF3 classic
     ncks -6 in.nc foo_6.nc # netCDF3 64bit-offset
     ncks --64 in.nc foo_6.nc # netCDF3 64bit-offset
     ncks -5 in.nc foo_5.nc # netCDF3 64bit-data
     ncks --5 in.nc foo_5.nc # netCDF3 64bit-data
     ncks -4 in.nc foo_4.nc # netCDF4
     ncks --4 in.nc foo_4.nc # netCDF4
     ncks -7 in.nc foo_7.nc # netCDF4 classic
     ncks --7 in.nc foo_7.nc # netCDF4 classic
   Of course since most operators support these switches, the
"conversions" can be done at the output stage of arithmetic or metadata
processing rather than requiring a separate step.  Producing (netCDF3)
‘CLASSIC’ or ‘64BIT_OFFSET’ or ‘64BIT_DATA’ files from ‘NETCDF4_CLASSIC’
files always works.

   ---------- Footnotes ----------

   (1) The switches ‘-5’, ‘--5’, and ‘pnetcdf’ are reserved for PnetCDF
files, i.e., ‘NC_FORMAT_CDF5’.  Such files are similar to netCDF3
classic files, yet also support 64-bit offsets and the additional
netCDF4 atomic types.


File: nco.info,  Node: Autoconversion,  Prev: File Conversion,  Up: File Formats and Conversion

3.10.4 Autoconversion
---------------------

Because of the dearth of support for netCDF4 amongst tools and user
communities (including the CF conventions), it is often useful to
convert netCDF4 to netCDF3 for certain applications.  Until NCO version
4.4.0 (January, 2014), producing netCDF3 files from netCDF4 files only
worked if the input files contained no netCDF4-specific features (e.g.,
atomic types, multiple record dimensions, or groups).  As of NCO version
4.4.0, ‘ncks’ supports “autoconversion” of many netCDF4 features to
their closest netCDF3-compatible representations.  Since converting
netCDF4 to netCDF3 results in loss of features, "automatic
down-conversion" may be a more precise description of what we term
autoconversion.

   NCO employs three algorithms to downconvert netCDF4 to netCDF3:
  1. Autoconversion of atomic types: Autoconversion automatically
     promotes ‘NC_UBYTE’ to ‘NC_SHORT’, and ‘NC_USHORT’ to ‘NC_INT’.  It
     automatically demotes the three types ‘NC_UINT’, ‘NC_UINT64’, and
     ‘NC_INT64’ to ‘NC_INT’.  And it converts ‘NC_STRING’ to ‘NC_CHAR’.
     All numeric conversions work for attributes and variables of any
     rank.  Two numeric types (‘NC_UBYTE’ and ‘NC_USHORT’) are
     _promoted_ to types with greater range (and greater storage).  This
     extra range is often not used so promotion perhaps conveys the
     wrong impression.  However, promotion never truncates values or
     loses data (this perhaps justifies the extra storage).  Three
     numeric types (‘NC_UINT’, ‘NC_UINT64’ and ‘NC_INT64’) are
     _demoted_.  Since the input range is larger than the output range,
     demotion can result in numeric truncation and thus loss of data.
     In such cases, it would possible to convert the data to
     floating-point values instead.  If this feature interests you,
     please be the squeaky wheel and let us know.

     String conversions (from ‘NC_STRING’ to ‘NC_CHAR’) work for all
     attributes, but not for all variables.  This is because attributes
     are at most one-dimensional and may be of any size whereas
     variables require gridded dimensions that usually do not fit the
     ragged sizes of text strings.  Hence scalar ‘NC_STRING’ attributes
     are correctly converted to and stored as ‘NC_CHAR’ attributes in
     the netCDF3 output file, but not all ‘NC_STRING’ variables are
     correctly converted.

     As of version 5.3.1 (January, 2025), NCO has two distinct levels of
     support for autoconversion of ‘NC_STRING’ variables.  ‘ncremap’
     fully supports autoconversion of scalar ‘NC_STRING’ variables in
     both horizontal regridder and vertical interpolation mode.  This
     means that ‘ncremap’ will produce netCDF3 output files from netCDF4
     input files (if requested) and that any ‘NC_STRING’ variables in
     the input files will be translated to ‘NC_CHAR’ arrays in the
     netCDF3 output files.  There is no loss of information since the
     entire contents of the string is preserved in the character array.
     The autoconversion stores the character array with a new dimension
     named ‘sng_lng_X’ where X is the smallest power of ten between 10
     and 10000 that is capable of containing the string contents
     (including the terminating ‘NUL’ character).  This type of variable
     autoconversion fails if the input string length exceeds 10000 (this
     is a safety measure, and will be expanded if people lobby me).

     All operators besides ‘ncremap’ currently retain only the first
     character of the input ‘NC_STRING’ array.  This loses information.
     We are working to implement the same feature that ‘ncremap’ enjoys.
     Both types of autoconversion fail if the input string variable is
     multidimensional, i.e., an array of type ‘NC_STRING’.  A feature to
     solve that case would be tractable, though we have not yet received
     any requests for it.

  2. Convert multiple record dimensions to fixed-size dimensions.  Many
     netCDF4 and HDF5 datasets have multiple unlimited dimensions.
     Since a netCDF3 file may have at most one unlimited dimension, all
     but possibly one unlimited dimension from the input file must be
     converted to fixed-length dimensions prior to storing netCDF4 input
     as netCDF3 output.  By invoking ‘--fix_rec_dmn all’ the user
     ensures the output file will adhere to netCDF3 conventions and the
     user need not know the names of the specific record dimensions to
     fix.  See *note ncks netCDF Kitchen Sink:: for a description of the
     ‘--fix_rec_dmn’ option.

  3. Flattening (removal) of groups.  Many netCDF4 and HDF5 datasets
     have group hierarchies.  Since a netCDF3 file may not have any
     groups, groups in the input file must be removed.  This is also
     called "flattening" the hierarchical file.  See *note Group Path
     Editing:: for a description of the GPE option ‘-G :’ to flatten
     files.

   Putting the three algorithms together, one sees that the recipe to
convert netCDF4 to netCDF4 becomes increasingly complex as the netCDF4
features in the input file become more elaborate:
     # Convert file with netCDF4 atomic types
     ncks -3 in.nc4 out.nc3
     # Convert file with multiple record dimensions + netCDF4 atomic types
     ncks -3 --fix_rec_dmn=all in.nc4 out.nc3
     # Convert file with groups, multiple record dimensions + netCDF4 atomic types
     ncks -3 -G : --fix_rec_dmn=all in.nc4 out.nc3
   Future versions of NCO may automatically invoke the record dimension
fixation and group flattening when converting to netCDF3 (rather than
requiring it be specified manually).  If this feature would interest
you, please let us know.


File: nco.info,  Node: Zarr and NCZarr,  Next: Large File Support,  Prev: File Formats and Conversion,  Up: Shared features

3.11 Zarr and NCZarr
====================

Availability: All Operators
   As of version 5.1.1 (November 2022), all NCO operators support NCZarr
I/O. This support is currently limited to the ‘file://’ scheme.  Support
for the S3 scheme is next.  All NCO commands should work as expected
independent of the back-end storage format of the I/O. Operators can
ingest and output ‘POSIX’ or Zarr backend file formats:
     in_ncz="file://${HOME}/in_zarr4#mode=nczarr,file"
     in_psx="${HOME}/in_zarr4.nc"
     out_ncz="file://${HOME}/foo#mode=nczarr,file"
     out_psx="${HOME}/foo.nc"
     
     ncks ${in_ncz} # Print contents of Zarr file
     ncks -O -v var ${in_psx} ${out_psx} # POSIX input to POSIX output
     ncks -O -v var ${in_psx} ${out_ncz} # POSIX input to Zarr output
     ncks -O -v var ${in_ncz} ${out_psx} # Zarr input to  POSIX output
     ncks -O -v var ${in_ncz} ${out_ncz} # Zarr input to Zarr output
     ncks -O --cmp='gbr|shf|zst' ${in_psx} ${out_ncz} # Quantize/Compress
     ncks -O --cmp='gbr|shf|zst' ${in_ncz} ${out_ncz} # Quantize/Compress
   Note that NCZarr only supports the netCDF4 (not netCDF3) data model.
This is because NCZarr needs to know chunking and compression
information by default (it is not optional).  Hence if the input format
is netCDF3, then the user must explicitly specify a netCDF4 format for
the output NCZarr storage:
     in_psx="${HOME}/in_zarr3.nc" # As above, but a netCDF3 input file
     
     ncks -O -4 ${in_psx} ${out_ncz} # netCDF3 POSIX input to Zarr output
     ncks -O -7 ${in_psx} ${out_ncz} # netCDF3 POSIX input to Zarr output
   Furthermore, the current NCZarr library (version 4.9.2, March 2023)
does not yet support record dimensions (this is a significant and high
priority netCDF library limitation, not an NCO limitation).  All
dimensions must be fixed, not record To workaround this limitation,
first fix any record dimensions with, e.g.,
     ncks -O --fix_rec_dmn=all ${in_psx} ${in_psx}

   Commands with Zarr I/O behave mostly as expected.  NCO treats Zarr
and POSIX files identically once they are opened via the netCDF API.
The main difference between Zarr and POSIX, from the viewpoint of NCO,
is in handling the filenames.  By default NCO performs operations in
temporary files that it moves to a final destination once the rest of
the command succeeds.  Supporting Zarr in NCO requires applying the
correct procedures to create, copy, move/rename, and delete files and
directories correctly depending on the backend format.

   Many NCO users rely on POSIX filename globbing for multi-file
operations, e.g., ‘ncra in*.nc out.nc’.  Globbing returns matches in
POSIX format (e.g., ‘in1.nc in2.nc in3.nc’) which lacks the ‘scheme://’
indicator and the ‘#mode=...’ fragment that the netCDF API needs to open
a Zarr store.  There is no perfect solution to this.

   A partial solution is available by judiciously using NCO's new
‘stdin’ capabilities for all operators (*note Specifying Input Files::).
The procedure uses the ‘ls’ command (instead of globbing) to identify
the desired Zarr stores, and pipes the (POSIX-style) results of that
through the newly supplied NCO filter-script ‘ncz2psx’ that will prepend
the desired scheme and append the desired fragment to the matched Zarr
stores, and pipe those results onward to an NCO operator:
     nces in*.nc out.nc      # POSIX input files via globbing
     ls in*.nc | nces out.nc # POSIX input files via stdin
     ls -d in* | ncz2psx | nces out.nc # Zarr input via stdin
     ls -d in* | ncz2psx --scheme=file --mode=nczarr,file | nces out.nc


File: nco.info,  Node: Large File Support,  Next: Subsetting Files,  Prev: Zarr and NCZarr,  Up: Shared features

3.12 Large File Support
=======================

Availability: All operators
Short options: none
Long options: none
   NCO has Large File Support (LFS), meaning that NCO can write files
larger than 2 GB on some 32-bit operating systems with netCDF libraries
earlier than version 3.6.  If desired, LFS support must be configured
when both netCDF and NCO are installed.  netCDF versions 3.6 and higher
support 64-bit file addresses as part of the netCDF standard.  We
recommend that users ignore LFS support which is difficult to configure
and is implemented in NCO only to support netCDF versions prior to 3.6.
This obviates the need for configuring explicit LFS support in
applications (such as NCO) that now support 64-bit files directly
through the netCDF interface.  See *note File Formats and Conversion::
for instructions on accessing the different file formats, including
64-bit files, supported by the modern netCDF interface.

   If you are still interested in explicit LFS support for netCDF
versions prior to 3.6, know that LFS support depends on a complex,
interlocking set of operating system (1) and netCDF support issues.  The
netCDF LFS FAQ
(http://my.unidata.ucar.edu/content/software/netcdf/faq-lfs.html)
describes the various file size limitations imposed by different
versions of the netCDF standard.  NCO and netCDF automatically attempt
to configure LFS at build time.

   ---------- Footnotes ----------

   (1) Linux and AIX do support LFS.


File: nco.info,  Node: Subsetting Files,  Next: Subsetting Coordinate Variables,  Prev: Large File Support,  Up: Shared features

3.13 Subsetting Files
=====================

Options ‘-g GRP’
Availability: ‘ncbo’, ‘nces’, ‘ncecat’, ‘ncflint’, ‘ncks’, ‘ncpdq’,
‘ncra’, ‘ncrcat’, ‘ncwa’
Short options: ‘-g’
Long options: ‘--grp’ and ‘--group’
Options ‘-v VAR’ and ‘-x’
Availability: (‘ncap2’), ‘ncbo’, ‘nces’, ‘ncecat’, ‘ncflint’, ‘ncks’,
‘ncpdq’, ‘ncra’, ‘ncrcat’, ‘ncwa’
Short options: ‘-v’, ‘-x’
Long options: ‘--variable’, ‘--var_lst’, ‘--exclude’ or ‘--xcl’
Options ‘--unn’
Availability: ‘ncbo’, ‘nces’, ‘ncecat’, ‘ncflint’, ‘ncks’, ‘ncpdq’,
‘ncra’, ‘ncrcat’, ‘ncwa’
Short options:
Long options: ‘--unn’ and ‘--union’
Options ‘--grp_xtr_var_xcl’
Availability: ‘ncks’
Short options:
Long options: ‘--gxvx’ and ‘--grp_xtr_var_xcl’
   Subsetting variables refers to explicitly specifying variables and
groups to be included or excluded from operator actions.  Subsetting is
controlled by the ‘-v VAR[,...]’ and ‘-x’ options for directly
specifying variables.  Specifying groups, whether in addition to or
instead of variables, is quite similar and is controlled by the ‘-g
GRP[,...]’ and ‘-x’ options.  A list of variables or groups to extract
is specified following the ‘-v’ and ‘-g’ options, e.g., ‘-v
time,lat,lon’ or ‘-g grp1,grp2’.  Both options may be specified
simultaneously and NCO will extract the intersection of the lists, i.e.,
only variables of the specified names found in groups of the specified
names.  The ‘--unn’ option causes NCO to extract the union, rather than
the intersection, of the specified groups and variables.  Not using the
‘-v’ or ‘-g’ option is equivalent to specifying all variables or groupp,
respectively.

   The ‘-x’ option causes the list of variables specified with ‘-v’ to
be _excluded_ rather than _extracted_.  Thus ‘-x’ saves typing when you
only want to extract fewer than half of the variables in a file.
     ncks -x -v v1,v2 in.nc out.nc # Extract all variables except v1, v2
     ncks -C -x -v lat,lon in.nc out.nc # Extract all except lat, lon
   The first example above shows the typical use of ‘-x’ to subset all
variables except a few into the output.  Note that ‘v1’ and ‘v2’ will be
retained in the output if they are coordinate-like variables (*note
Subsetting Coordinate Variables::) associated with any extracted
variable.  If one wishes to exclude coordinate-like variables despite
their being referenced by extracted variables, one must use the ‘-C’ (or
synonym ‘--xcl_ass_var’) option as shown in the second example.

   Variables or groups explicitly specified for extraction with ‘-v
VAR[,...]’ or ‘-g GRP[,...]’ _must_ be present in the input file or an
error will result.  Variables explicitly specified for _exclusion_ with
‘-x -v VAR[,...]’ need not be present in the input file.  To accord with
the sophistication of the underlying hierarchy, group subsetting is
controlled by a few powerful yet subtle syntactical distinctions.  When
learning this syntax it is helpful to keep in mind the similarity
between group hierarchies and directory structures.

   As of NCO 4.4.4 (June, 2014), ‘ncks’ (alone) supports an option to
include specified groups yet exclude specified variables.  The
‘--grp_xtr_var_xcl’ switch (with long option equivalent ‘--gxvx’)
extracts all contents of groups given as arguments to ‘-g GRP[,...]’,
except for variables given as arguments to ‘-v VAR[,...]’.  Use this
when one or a few variables in hierarchical files are not to be
extracted, and all other variables are.  This is useful when coercing
netCDF4 files into netCDF3 files such as with converting, flattening, or
dismembering files (see *note Flattening Groups::).
     ncks --grp_xtr_var_xcl -g g1 -v v1 # Extract all of group g1 except v1

   Two properties of subsetting, recursion and anchoring, are best
illustrated by reminding the user of their UNIX equivalents.  The UNIX
command ‘mv src dst’ moves ‘src’ _and all its subdirectories_ (and all
their subdirectories etc.)  to ‘dst’.  In other words ‘mv’ is, by
default, _recursive_.  In contrast, the UNIX command ‘cp src dst’ moves
‘src’, and only ‘src’, to ‘dst’, If ‘src’ is a directory, not a file,
then that command fails.  One must explicitly request to copy
directories recursively, i.e., with ‘cp -r src dst’.  In NCO recursive
extraction (and copying) of groups is the default (like with ‘mv’, not
with ‘cp’).  Recursion is turned off by appending a trailing slash to
the path.

   These UNIX commands also illustrate a property we call _anchoring_.
The command ‘mv src dst’ moves (recursively) the source directory ‘src’
to the destination directory ‘dst’.  If ‘src’ begins with the slash
character then the specified path is relative to the root directory,
otherwise the path is relative to the current working directory.  In
other words, an initial slash character anchors the subsequent path to
the root directory.  In NCO an initial slash anchors the path at the
root group.  Paths that begin and end with slash characters (e.g., ‘//’,
‘/g1/’, and ‘/g1/g2/’) are both anchored and non-recursive.

   Consider the following commands, all of which may be assumed to end
with ‘in.nc out.nc’:
     ncks -g  g1  # Extract, recursively, all groups with a g1 component
     ncks -g  g1/ # Extract, non-recursively, all groups terminating in g1
     ncks -g /g1  # Extract, recursively, root group g1
     ncks -g /g1/ # Extract, non-recursively root group g1
     ncks -g //   # Extract, non-recursively the root group
   The first command is probably the most useful and common.  It would
extract these groups, if present, and all their direct ancestors and
children: ‘/g1’, ‘/g2/g1’, and ‘/g3/g1/g2’.  In other words, the
simplest form of ‘-g grp’ grabs all groups that (and their direct
ancestors and children, recursively) that have ‘grp’ as a complete
component of their path.  A simple string match is insufficient, GRP
must be a complete component (i.e., group name) in the path.  The option
‘-g g1’ would not extract these groups because ‘g1’ is not a complete
component of the path: ‘/g12’, ‘/fg1’, and ‘/g1g1’.  The second command
above shows how a terminating slash character ‘/’ cancels the recursive
copying of groups.  An argument to ‘-g’ which terminates with a slash
character extracts the group and its direct ancestors, but none of its
children.  The third command above shows how an initial slash character
‘/’ anchors the argument to the root group.  The third command would not
extract the group ‘/g2/g1’ because the ‘g1’ group is not at the root
level, but it would extract, any group ‘/g1’ at the root level and all
its children, recursively.  The fourth command is the non-recursive
version of the third command.  The fifth command is a special case of
the fourth command.

   As mentioned above, both ‘-v’ and ‘-g’ options may be specified
simultaneously and NCO will, by default, extract the intersection of the
lists, i.e., the specified variables found in the specified groups (1).
The ‘--unn’ option causes NCO to extract the union, rather than the
intersection, of the specified groups and variables.  Consider the
following commands (which may be assumed to end with ‘in.nc out.nc’):
     # Intersection-mode subsetting (default)
     ncks -g  g1  -v v1 # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1
     ncks -g /g1  -v v1 # Yes: /g1/v1, /g1/g2/v1. No: /v1, /g2/v1, /g2/g1/v1
     ncks -g  g1/ -v v1 # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1, /g1/g2/v1
     ncks -v  g1/v1     # Yes: /g1/v1, /g2/g1/v1. No: /v1, /g2/v1, /g1/g2/v1
     ncks -g /g1/ -v v1 # Yes: /g1/v1. No: /g2/g1/v1, /v1, /g2/v1 ...
     ncks -v /g1/v1     # Yes: /g1/v1. No: /g2/g1/v1, /v1, /g2/v1 ...

     # Union-mode subsetting (invoke with --unn or --union)
     ncks -g  g1  -v v1 --unn # All variables in  g1 or progeny, or named v1
     ncks -g /g1  -v v1 --unn # All variables in /g1 or progeny, or named v1
     ncks -g  g1/ -v v1 --unn # All variables in  g1 or named v1
     ncks -g /g1/ -v v1 --unn # All variables in /g1 or named v1
   The first command (‘-g g1 -v v1’) extracts the variable ‘v1’ from any
group named ‘g1’ or descendent ‘g1’.  The second command extracts ‘v1’
from any root group named ‘g1’ and any descendent groups as well.  The
third and fourth commands are equivalent ways of extracting ‘v1’ only
from the root group named ‘g1’ (not its descendents).  The fifth and
sixth commands are equivalent ways of extracting the variable ‘v1’ only
from the root group named ‘g1’.  Subsetting in union-mode (with ‘--unn’)
causes all variables to be extracted which meet either one or both of
the specifications of the variable and group specifications.  Union-mode
subsetting is simply the logical "OR" of intersection-mode subsetting.
As discussed below, the group and variable specifications may be comma
separated lists of regular expressions for added control over
subsetting.

   Remember, if averaging or concatenating large files stresses your
systems memory or disk resources, then the easiest solution is often to
subset (with ‘-g’ and/or ‘-v’) to retain only the most important
variables (*note Memory Requirements::).
     ncks          in.nc out.nc # Extract all groups and variables
     ncks -v scl   # Extract variable scl from all groups
     ncks -g g1    # Extract group g1 and descendents
     ncks -x -g g1 # Extract all groups except g1 and descendents
     ncks -g g2,g3 -v scl # Extract scl from groups g2 and g3
   Overwriting and appending work as expected:
     # Replace scl in group g2 in out.nc with scl from group g2 from in.nc
     ncks -A -g g2 -v scl in.nc out.nc

   Due to its special capabilities, ‘ncap2’ interprets the ‘-v’ switch
differently (*note ncap2 netCDF Arithmetic Processor::).  For ‘ncap2’,
the ‘-v’ switch takes no arguments and indicates that _only_
user-defined variables should be output.  ‘ncap2’ neither accepts nor
understands the -X and -G switches.

   Regular expressions the syntax that NCO use pattern-match object
names in netCDF file against user requests.  The user can select all
variables beginning with the string ‘DST’ from an input file by
supplying the regular expression ‘^DST’ to the ‘-v’ switch, i.e., ‘-v
'^DST'’.  The meta-characters used to express pattern matching
operations are ‘^$+?.*[]{}|’.  If the regular expression pattern matches
_any_ part of a variable name then that variable is selected.  This
capability is also called “wildcarding”, and is very useful for
sub-setting large data files.

   Extended regular expressions are defined by the POSIX ‘grep -E’ (aka
‘egrep’) command.  As of NCO 2.8.1 (August, 2003), variable name
arguments to the ‘-v’ switch may contain “extended regular expressions”.
As of NCO 3.9.6 (January, 2009), variable names arguments to ‘ncatted’
may contain “extended regular expressions”.  As of NCO 4.2.4 (November,
2012), group name arguments to the ‘-g’ switch may contain “extended
regular expressions”.

   Because of its wide availability, NCO uses the POSIX regular
expression library ‘regex’.  Regular expressions of arbitary complexity
may be used.  Since netCDF variable names are relatively simple
constructs, only a few varieties of variable wildcards are likely to be
useful.  For convenience, we define the most useful pattern matching
operators here:
‘^’
     Matches the beginning of a string
‘$’
     Matches the end of a string
‘.’
     Matches any single character
The most useful repetition and combination operators are
‘?’
     The preceding regular expression is optional and matched at most
     once
‘*’
     The preceding regular expression will be matched zero or more times
‘+’
     The preceding regular expression will be matched one or more times
‘|’
     The preceding regular expression will be joined to the following
     regular expression.  The resulting regular expression matches any
     string matching either subexpression.

   To illustrate the use of these operators in extracting variables and
groups, consider file ‘in_grp.nc’ with groups ‘g0’-‘g9’, and subgroups
‘s0’-‘s9’, in each of those groups, and file ‘in.nc’ with variables ‘Q’,
‘Q01’-‘Q99’, ‘Q100’, ‘QAA’-‘QZZ’, ‘Q_H2O’, ‘X_H2O’, ‘Q_CO2’, ‘X_CO2’.
     ncks -v '.+' in.nc               # All variables (default)
     ncks -v 'Q.?' in.nc              # Variables that contain Q
     ncks -v '^Q.?' in.nc             # Variables that start with Q
     ncks -v '^Q+.?.' in.nc           # Q, Q0--Q9, Q01--Q99, QAA--QZZ, etc.
     ncks -v '^Q..' in.nc             # Q01--Q99, QAA--QZZ, etc.
     ncks -v '^Q[0-9][0-9]' in.nc     # Q01--Q99, Q100
     ncks -v '^Q[[:digit:]]{2}' in.nc # Q01--Q99
     ncks -v 'H2O$' in.nc             # Q_H2O, X_H2O 
     ncks -v 'H2O$|CO2$' in.nc        # Q_H2O, X_H2O, Q_CO2, X_CO2 
     ncks -v '^Q[0-9][0-9]$' in.nc    # Q01--Q99
     ncks -v '^Q[0-6][0-9]|7[0-3]' in.nc # Q01--Q73, Q100
     ncks -v '(Q[0-6][0-9]|7[0-3])$' in.nc # Q01--Q73
     ncks -v '^[a-z]_[a-z]{3}$' in.nc # Q_H2O, X_H2O, Q_CO2, X_CO2
     ncks -g 'g.' in_grp.nc           # 10 Groups g0-g9
     ncks -g 's.' in_grp.nc       # 100 sub-groups g0/s0, g0/s1, ... g9/s9
     ncks -g 'g.' -v 'v.' in_grp.nc   # All variables 'v.' in groups 'g.'
   Beware--two of the most frequently used repetition pattern matching
operators, ‘*’ and ‘?’, are also valid pattern matching operators for
filename expansion (globbing) at the shell-level.  Confusingly, their
meanings in extended regular expressions and in shell-level filename
expansion are significantly different.  In an extended regular
expression, ‘*’ matches zero or more occurences of the preceding regular
expression.  Thus ‘Q*’ selects all variables, and ‘Q+.*’ selects all
variables containing ‘Q’ (the ‘+’ ensures the preceding item matches at
least once).  To match zero or one occurence of the preceding regular
expression, use ‘?’.  Documentation for the UNIX ‘egrep’ command details
the extended regular expressions which NCO supports.

   One must be careful to protect any special characters in the regular
expression specification from being interpreted (globbed) by the shell.
This is accomplish by enclosing special characters within single or
double quotes
     ncra -v Q?? in.nc out.nc   # Error: Shell attempts to glob wildcards
     ncra -v '^Q+..' in.nc out.nc # Correct: NCO interprets wildcards
     ncra -v '^Q+..' in*.nc out.nc # Correct: NCO interprets, Shell globs
   The final example shows that commands may use a combination of
variable wildcarding and shell filename expansion (globbing).  For
globbing, ‘*’ and ‘?’ _have nothing to do_ with the preceding regular
expression!  In shell-level filename expansion, ‘*’ matches any string,
including the null string and ‘?’ matches any single character.
Documentation for ‘bash’ and ‘csh’ describe the rules of filename
expansion (globbing).

   ---------- Footnotes ----------

   (1) Intersection-mode can also be explicitly invoked with the ‘--nsx’
or ‘--intersection’ switches.  These switches are supplied for clarity
and consistency and do absolutely nothing since intersection-mode is the
default.


File: nco.info,  Node: Subsetting Coordinate Variables,  Next: Group Path Editing,  Prev: Subsetting Files,  Up: Shared features

3.14 Subsetting Coordinate Variables
====================================

Availability: ‘ncap2’, ‘ncbo’, ‘nces’, ‘ncecat’, ‘ncflint’, ‘ncks’,
‘ncpdq’, ‘ncra’, ‘ncrcat’, ‘ncwa’
Short options: ‘-C’, ‘-c’
Long options: ‘--no_coords’, ‘--no_crd’, ‘--xcl_ass_var’, ‘--crd’,
‘--coords’, ‘--xtr_ass_var’
   By default, coordinates variables associated with any variable
appearing in the INPUT-FILE will be placed in the OUTPUT-FILE, even if
they are not explicitly specified, e.g., with the ‘-v’ switch.  Thus
variables with a latitude coordinate ‘lat’ always carry the values of
‘lat’ with them into the OUTPUT-FILE.  This automatic inclusion feature
can be disabled with ‘-C’, which causes NCO to exclude (or, more
precisely, not to automatically include) coordinates and associated
variables from the extraction list.  However, using ‘-C’ does not
preclude the user from including some coordinates in the output files
simply by explicitly selecting the coordinates and associated variables
with the -V option.  The ‘-c’ option, on the other hand, is a shorthand
way of automatically specifying that _all_ coordinate and associated
variables in INPUT-FILES should appear in OUTPUT-FILE.  The user can
thereby select all coordinate variables without even knowing their
names.

   The meaning of "coordinates" in these two options has expanded since
about 2009 from simple one dimensional coordinates (per the NUG)
definition) to any and all associated variables.  This includes
multi-dimensional coordinates as well as a menagerie of associated
variables defined by the CF metadata conventions: As of NCO version
4.4.5 (July, 2014) both ‘-c’ and ‘-C’ honor the CF ‘ancillary_variables’
convention described in *note CF Conventions::.  As of NCO version 4.0.8
(April, 2011) both ‘-c’ and ‘-C’ honor the CF ‘bounds’ convention
described in *note CF Conventions::.  As of NCO version 4.6.4 (January,
2017) both ‘-c’ and ‘-C’ honor the CF ‘cell_measures’ convention
described in *note CF Conventions::.  As of NCO version 4.4.9 (May,
2015) both ‘-c’ and ‘-C’ honor the CF ‘climatology’ convention described
in *note CF Conventions::.  As of NCO version 3.9.6 (January, 2009) both
‘-c’ and ‘-C’ honor the CF ‘coordinates’ convention described in *note
CF Conventions::.  As of NCO version 4.6.4 (January, 2017) both ‘-c’ and
‘-C’ honor the CF ‘formula_terms’ convention described in *note CF
Conventions::.  As of NCO version 4.6.0 (May, 2016) both ‘-c’ and ‘-C’
honor the CF ‘grid_mapping’ convention described in *note CF
Conventions::.

   The expanded categories of variables controlled by ‘-c’ and ‘-C’
justified adding a more descriptive switch.  As of NCO version 4.8.0
(May, 2019) the switch ‘--xcl_ass_var’, which stands for "exclude
associated variables", is synonymous with ‘-C’ and ‘--xtr_ass_var’,
which stands for "extract associated variables", is synonymous with
‘-c’.


File: nco.info,  Node: Group Path Editing,  Next: C and Fortran Index Conventions,  Prev: Subsetting Coordinate Variables,  Up: Shared features

3.15 Group Path Editing
=======================

Options ‘-G GPE_DSC’
Availability: ‘ncbo’, ‘ncecat’, ‘nces’, ‘ncflint’, ‘ncks’, ‘ncpdq’,
‘ncra’, ‘ncrcat’, ‘ncwa’
Short options: ‘-G’
Long options: ‘--gpe’

   “Group Path Editing”, or GPE, allows the user to restructure (i.e.,
add, remove, and rename groups) in the output file relative to the input
file based on the instructions they provide.  As of NCO 4.2.3 (November,
2012), all operators that accept netCDF4 files with groups accept the
‘-G’ switch, or its long-option equivalent ‘--gpe’.  To master GPE one
must understand the meaning of the required GPE_DSC structure/argument
that specifies the transformation of input-to-output group paths.

   Each GPE_DSC contains up to three elements (two are optional) in the
following order:
GPE_DSC = GRP_PTH:LVL_NBR or GRP_PTH@LVL_NBR

GRP_PTH
     Group Path.  This (optional) component specifies the output group
     path that should be appended after any editing (i.e., deletion or
     truncation) of the input path is performed.
LVL_NBR
     The number of levels to delete (from the head) or truncate (from
     the tail) of the input path.
If both components of the argument are present, then a single character,
either the colon or at-sign (‘:’ or ‘@’), must separate them.  If only
GRP_PTH is specifed, the separator character may be omitted, e.g., ‘-G
g1’.  If only LVL_NBR is specifed, the separator character is still
required to indicate it is a LVL_NBR arugment and not a GRP_PTH, e.g.,
‘-G :-1’ or ‘-G @1’.

   If the at-sign separator character ‘@’ is used instead of the colon
separator character ‘:’, then the following LVL_NBR arugment must be
positive and it will be assumed to refer to Truncation-Mode.  Hence, ‘-G
:-1’ is the same as ‘-G @1’.  This is simply a way of making the LVL_NBR
argument positive-definite.

* Menu:

* Flattening Groups::
* Moving Groups::
* Dismembering Files::
* Checking CF-compliance::


File: nco.info,  Node: Flattening Groups,  Next: Moving Groups,  Prev: Group Path Editing,  Up: Group Path Editing

3.15.1 Deletion, Truncation, and Flattening of Groups
-----------------------------------------------------

GPE has three editing modes: Delete, Truncate, and Flatten.  Select one
of GPE's three editing modes by supplying a LVL_NBR that is positive,
negative, or zero for Delete-, Truncate- and Flatten-mode, respectively.

   In Delete-mode, LVL_NBR is a positive integer which specifies the
maximum number of group path components (i.e., groups) that GPE will try
to delete from the head of GRP_PTH.  For example LVL_NBR = 3 changes the
input path ‘/g1/g2/g3/g4/g5’ to the output path ‘/g4/g5’.  Input paths
with LVL_NBR or fewer components (groups) are completely erased and the
output path commences from the root level.

   In other words, GPE is tolerant of specifying too many group
components to delete.  It deletes as many as possible, without
complaint, and then begins to flatten the file (which fails if namespace
conflicts arise).

   In Truncate-mode, LVL_NBR is a negative integer which specifies the
maximum number of group path components (i.e., groups) that GPE will try
to truncate from the tail of GRP_PTH.  For example LVL_NBR = -3 changes
the input path ‘/g1/g2/g3/g4/g5’ to the output path ‘/g1/g2’.  Input
paths with LVL_NBR or fewer components (groups) are completely erased
and the output path commences from the root level.

   In Flatten-mode, indicated by the separator character alone or with
LVL_NBR = 0, GPE removes the entire group path from the input file and
constructs the output path beginning at the root level.  For example ‘-G
:0’ and ‘-G :’ are identical and change the input path ‘/g1/g2/g3/g4/g5’
to the output path ‘/’ whereas ‘-G g1:0’ and ‘-G g1:’ are identical and
result in the output path ‘/g1’ for all variables.

   Subsequent to the alteration of the input path by the specified
editing mode, if any, GPE prepends (in Delete Mode) or Appends (in
Truncate-mode) any specifed GRP_PTH to the output path.  For example ‘-G
g2’ changes the input paths ‘/’ and ‘/g1’ to ‘/g2’ and ‘/g1/g2’,
respectively.  Likewise, ‘-G g2/g3’ changes the input paths ‘/’ and
‘/g1’ to ‘/g2/g3’ and ‘/g1/g2/g3’, respectively.  When GRP_PTH and
LVL_NBR are both specified, the editing actions are taken in sequence so
that, e.g., ‘-G g1/g2:2’ changes the input paths ‘/’ and ‘/h1/h2/h3/h4’
to ‘/g1/g2’ and ‘/g1/g2/h3/h4’, respectively.  Likewise, ‘-G g1/g2:-2’
changes the input paths ‘/’ and ‘/h1/h2/h3/h4’ to ‘/g1/g2’ and
‘/h1/h2/g1/g2’, respectively.

   Combining GPE with subsetting (*note Subsetting Files::) yields
powerful control over the extracted (or excluded) variables and groups
and their placement in the output file as shown by the following
commands.  All commands below may be assumed to end with ‘in.nc out.nc’.
     # Prepending paths without editing:
     ncks                   # /g?/v? -> /g?/v?
     ncks             -v v1 # /g?/v1 -> /g?/v1
     ncks       -g g1       # /g1/v? -> /g1/v?
     ncks -G o1             # /g?/v? -> /o1/g?/v?
     ncks -G o1 -g g1       # /g1/v? -> /o1/g1/v?
     ncks       -g g1 -v v1 # /g1/v1 -> /g1/v1
     ncks -G o1       -v v1 # /g?/v1 -> /o1/g?/v1
     ncks -G o1 -g g1 -v v1 # /g1/v1 -> /o1/g1/v1
     ncks -G g1 -g /  -v v1 # /v1    -> /g1/v1
     ncks -G g1/g2    -v v1 # /g?/v1 -> /g1/g2/g?/v1
     # Delete-mode: Delete from and Prepend to path head
     # Syntax: -G [ppn]:lvl_nbr = # of levels to delete
     ncks -G :1    -g g1    -v v1 # /g1/v1    -> /v1
     ncks -G :1    -g g1/g1 -v v1 # /g1/g1/v1 -> /g1/v1
     ncks -G :2    -g g1/g1 -v v1 # /g1/g1/v1 -> /v1
     ncks -G :2    -g g1    -v v1 # /g1/v1    -> /v1
     ncks -G g2:1  -g g1    -v v1 # /g1/v1    -> /g2/v1
     ncks -G g2:2  -g g1/g1 -v v1 # /g1/g1/v1 -> /g2/v1
     ncks -G g2:1  -g /     -v v1 # /v1       -> /g2/v1
     ncks -G g2:1           -v v1 # /v1       -> /g2/v1
     ncks -G g2:1  -g g1/g1 -v v1 # /g1/g1/v1 -> /g2/g1/v1
     # Flatten-mode: Remove all input path components
     # Syntax: -G [apn]: colon without numerical argument
     ncks -G :            -v v1 # /g?/v1    -> /v1
     ncks -G :   -g g1    -v v1 # /g1/v1    -> /v1
     ncks -G :   -g g1/g1 -v v1 # /g1/g1/v1 -> /v1
     ncks -G g2:          -v v1 # /g?/v1    -> /g2/v1
     ncks -G g2:                # /g?/v?    -> /g2/v?
     ncks -G g2: -g g1/g1 -v v1 # /g1/g1/v1 -> /g2/v1
     # Truncate-mode: Truncate from and Append to path tail
     # Syntax: -G [apn]:-lvl_nbr = # of levels to truncate
     # NB: -G [apn]:-lvl_nbr is equivalent to -G [apn]@lvl_nbr
     ncks -G :-1   -g g1    -v v1 # /g1/v1    -> /v1
     ncks -G :-1   -g g1/g2 -v v1 # /g1/g2/v1 -> /g1/v1
     ncks -G :-2   -g g1/g2 -v v1 # /g1/g2/v1 -> /v1
     ncks -G :-2   -g g1    -v v1 # /g1/v1    -> /v1
     ncks -G g2:-1          -v v1 # /g?/v1    -> /g2/v1
     ncks -G g2:-1 -g g1    -v v1 # /g1/v1    -> /g2/v1
     ncks -G g1:-1 -g g1/g2 -v v1 # /g1/g2/v1 -> /g1/g1/v1


File: nco.info,  Node: Moving Groups,  Next: Dismembering Files,  Prev: Flattening Groups,  Up: Group Path Editing

3.15.2 Moving Groups
--------------------

Until fall 2013 (netCDF version 4.3.1-pre1), netCDF contained no library
function for renaming groups, and therefore ‘ncrename’ cannot rename
groups.  However, NCO built on earlier versions of netCDF than 4.3.1 can
use a GPE-based workaround mechanism to "rename" groups.  The GPE
mechanism actually _moves_ (i.e., copies to a new location) groups, a
more arduous procedure than simply renaming them.  GPE applies to all
selected groups, so, in the general case, one must move only the desired
group to a new file, and then merge that new file with the original to
obtain a file where the desired group has been "renamed" and all else is
unchanged.  Here is how to "rename" group ‘/g4’ to group ‘/f4’ with GPE
instead of ‘ncrename’
     ncks -O -G f4:1 -g g4 ~/nco/data/in_grp.nc ~/tmp.nc # Move /g4 to /f4
     ncks -O -x -g g4 ~/nco/data/in_grp.nc ~/out.nc # Excise /g4
     ncks -A ~/tmp.nc ~/out.nc # Add /f4 to new file
   If the original group ‘g4’ is not excised from ‘out.nc’ (step two
above), then the final output file would contain both ‘g4’ and a copy
named ‘f4’.  Thus GPE can be used to both "rename" and copy groups.  The
recommended way to rename groups when when netCDF version 4.3.1 is
availale is to use ‘ncrename’ (*note ncrename netCDF Renamer::).

   One may wish to flatten hierarchical group files for many reasons.
These include 1.  To obtain flat netCDF3 files for use with tools that
do not work with netCDF4 files, 2.  To split-apart hierarchies to
re-assemble into different hierarchies, and 3.  To provide a subset of a
hierarchical file with the simplest possible storage structure.
     ncks -O -G : -g cesm -3 ~/nco/data/cmip5.nc ~/cesm.nc # Extract /cesm to /
   The ‘-3’ switch (1) specifies the output dataset should be in netCDF3
format, the ‘-G :’ option flattens all extracted groups, and the ‘-g
cesm’ option extracts only the ‘cesm’ group and leaves all other groups
(e.g., ‘ecmwf’, ‘giss’).

   ---------- Footnotes ----------

   (1) Note that the ‘-3’ switch should appear _after_ the ‘-G’ and ‘-g’
switches.  This is due to an artifact of the GPE implementation which we
wish to remove in the future.


File: nco.info,  Node: Dismembering Files,  Next: Checking CF-compliance,  Prev: Moving Groups,  Up: Group Path Editing

3.15.3 Dismembering Files
-------------------------

Let us show how to completely disaggregate (or, more memorably)
_dismember_ a hierarchical dataset.  For now we take this to mean: store
each group as a standalone flat dataset in netCDF3 format.  This can be
accomplished by looping the previous example over all groups.  This
script ‘ncdismember’ dismembers the input file FL_IN specified in the
first argument and places the resulting files in the directory DRC_OUT
specified by the second argument:
     cat > ~/ncdismember << 'EOF'
     #!/bin/sh
     
     # Purpose: Dismember netCDF4/HDF5 hierarchical files. CF-check them.
     # Place each input file group in separate netCDF3 output file
     # Described in NCO User Guide at http://nco.sf.net/nco.html#dismember
     # Requirements: NCO 4.3.x+, UNIX shell utilities awk, grep, sed
     # Optional: Decker CFchecker https://bitbucket.org/mde_/cfchecker
     
     # Usage:
     # ncdismember <fl_in> <drc_out> [cf_chk] [cf_vrs] [opt]
     # where fl_in is input file/URL to dismember, drc_out is output directory
     # CF-compliance check is performed when optional third argument is not '0'
     # Default checker is Decker's cfchecker installed locally
     # Specify cf_chk=nerc for smallified uploads to NERC checker
     # Optional fourth argument cf_vrs is CF version to check
     # Optional fifth argument opt passes straight-through to ncks
     # Arguments must not use shell expansion/globbing
     # NB: ncdismember does not clean-up output directory, so user must
     # chmod a+x ~/sh/ncdismember
     # Examples:
     # ncdismember ~/nco/data/mdl_1.nc /data/zender/tmp
     # ncdismember http://dust.ess.uci.edu/nco/mdl_1.nc /tmp
     # ncdismember http://thredds-test.ucar.edu/thredds/dodsC/testdods/foo.nc /tmp
     # ncdismember ~/nco/data/mdl_1.nc /data/zender/nco/tmp cf
     # ncdismember ~/nco/data/mdl_1.nc /data/zender/nco/tmp nerc
     # ncdismember ~/nco/data/mdl_1.nc /data/zender/nco/tmp cf 1.3
     # ncdismember ~/nco/data/mdl_1.nc /data/zender/nco/tmp cf 1.5 --fix_rec_dmn=all
     
     # Command-line argument defaults
     fl_in="${HOME}/nco/data/mdl_1.nc" # [sng] Input file to dismember/check
     drc_out="${DATA}/nco/tmp" # [sng] Output directory
     cf_chk='0' # [flg] Perform CF-compliance check? Which checker?
     cf_vrs='1.5' # [sng] Compliance-check this CF version (e.g., '1.5')
     opt='' # [flg] Additional ncks options (e.g., '--fix_rec_dmn=all')
     # Use single quotes to pass multiple arguments to opt=${5}
     # Otherwise arguments would be seen as ${5}, ${6}, ${7} ...
     
     # Command-line argument option parsing
     if [ -n "${1}" ]; then fl_in=${1}; fi
     if [ -n "${2}" ]; then drc_out=${2}; fi
     if [ -n "${3}" ]; then cf_chk=${3}; fi
     if [ -n "${4}" ]; then cf_vrs=${4}; fi
     if [ -n "${5}" ]; then opt=${5}; fi
     
     # Prepare output directory
     echo "NCO dismembering file ${fl_in}"
     fl_stb=$(basename ${fl_in})
     drc_out=${drc_out}/${fl_stb}
     mkdir -p ${drc_out}
     cd ${drc_out}
     chk_dck='n'
     chk_nrc='n'
     if [ ${cf_chk} = 'nerc' ]; then
         chk_nrc='y'
     fi # chk_nrc
     if [ ${cf_chk} != '0' ] && [ ${cf_chk} != 'nerc' ]; then
         chk_dck='y'
         hash cfchecker 2>/dev/null || { echo >&2 "Local cfchecker command not found, will smallify and upload to NERC checker instead"; chk_nrc='y'; chk_dck='n'; }
     fi # !cf_chk
     # Obtain group list
     grp_lst=`ncks -m ${fl_in} | grep '// group' | awk '{$1=$2=$3="";sub(/^  */,"",$0);print}'`
     IFS=$'\n' # Change Internal-Field-Separator from <Space><Tab><Newline> to <Newline>
     for grp_in in ${grp_lst} ; do
         # Replace slashes by dots for output group filenames
         grp_out=`echo ${grp_in} | sed 's/\///' | sed 's/\//./g'`
         if [ "${grp_out}" = '' ]; then grp_out='root' ; fi
         # Tell older NCO/netCDF if HDF4 with --hdf4 switch (signified by .hdf/.HDF suffix)
         hdf4=`echo ${fl_in} | awk '{if(match(tolower($1),".hdf$")) hdf4="--hdf4"; print hdf4}'`
         # Flatten to netCDF3, anchor, no history, no temporary file, padding, HDF4 flag, options
         cmd="ncks -O -3 -G : -g ${grp_in}/ -h --no_tmp_fl --hdr_pad=40 ${hdf4} ${opt} ${fl_in} ${drc_out}/${grp_out}.nc"
         # Use eval in case ${opt} contains multiple arguments separated by whitespace
         eval ${cmd}
         if [ ${chk_dck} = 'y' ]; then
            # Decker checker needs Conventions <= 1.6
            no_bck_sls=`echo ${drc_out}/${grp_out} | sed 's/\\\ / /g'`
            ncatted -h -a Conventions,global,o,c,CF-${cf_vrs} ${no_bck_sls}.nc
         else # !chk_dck
            echo ${drc_out}/${grp_out}.nc
         fi # !chk_dck
     done
     if [ ${chk_dck} = 'y' ]; then
         echo 'Decker CFchecker reports CF-compliance of each group in flat netCDF3 format'
         cfchecker -c ${cf_vrs} *.nc
     fi
     if [ ${chk_nrc} = 'y' ]; then
         # Smallification and NERC upload from qdcf script by Phil Rasch (PJR)
         echo 'Using remote CFchecker http://puma.nerc.ac.uk/cgi-bin/cf-checker.pl'
         cf_lcn='http://puma.nerc.ac.uk/cgi-bin/cf-checker.pl'
         for fl in ${drc_out}/*.nc ; do
     	fl_sml=${fl}
     	cf_out=${fl%.nc}.html
     	dmns=`ncdump -h ${fl_in} | sed -n -e '/dimensions/,/variables/p' | grep = | sed -e 's/=.*//'`
     	hyp_sml=''
     	for dmn in ${dmns}; do
     	    dmn_lc=`echo ${dmn} | tr "[:upper:]" "[:lower:]"`
     	    if [ ${dmn_lc} = 'lat' ] || [ ${dmn_lc} = 'latitude' ] || [ ${dmn_lc} = 'lon' ] || [ ${dmn_lc} = 'longitude' ] || [ ${dmn_lc} = 'time' ]; then
     		hyp_sml=`echo ${hyp_sml}" -d ${dmn},0"`
     	    fi # !dmn_lc
     	done
     	# Create small version of input file by sampling only first element of lat, lon, time
     	ncks -O ${hyp_sml} ${fl} ${fl_sml}
     	# Send small file to NERC checker
     	curl --form cfversion=1.6 --form upload=@${fl_sml} --form press="Check%20file" ${cf_lcn} -o ${cf_out}
     	# Strip most HTML to improve readability
     	cat ${cf_out} | sed -e "s/<[^>]*>//g" -e "/DOCTYPE/,/\]\]/d" -e "s/CF-Convention//g" -e "s/Output of//g" -e "s/Compliance Checker//g" -e "s/Check another//g" -e "s/CF-Checker follows//g" -e "s/Received//g" -e "s/for NetCDF//g" -e "s/NetCDF format//g" -e "s/against CF version 1//g" -e "s/\.\.\.//g"
     	echo "Full NERC compliance-check log for ${fl} in ${cf_out}"
         done
     fi # !nerc
     EOF
     chmod 755 ~/ncdismember # Make command executable
     /bin/mv -f ~/ncdismember ~/sh # Store in location on $PATH, e.g., /usr/local/bin
     
     zender@roulee:~$ ncdismember ~/nco/data/mdl_1.nc ${DATA}/nco/tmp
     NCO dismembering file /home/zender/nco/data/mdl_1.nc
     /data/zender/nco/tmp/mdl_1.nc/cesm.cesm_01.nc
     /data/zender/nco/tmp/mdl_1.nc/cesm.cesm_02.nc
     /data/zender/nco/tmp/mdl_1.nc/cesm.nc
     /data/zender/nco/tmp/mdl_1.nc/ecmwf.ecmwf_01.nc
     /data/zender/nco/tmp/mdl_1.nc/ecmwf.ecmwf_02.nc
     /data/zender/nco/tmp/mdl_1.nc/ecmwf.nc
     /data/zender/nco/tmp/mdl_1.nc/root.nc
   A (potentially more portable) binary executable could be written to
dismember all groups with a single invocation, yet dismembering without
loss of information is possible now with this simple script on all
platforms with UNIXy utilities.  Note that all dimensions inherited by
groups in the input file are correctly placed by ‘ncdismember’ into the
flat files.  Moreover, each output file preserves the group metadata of
all ancestor groups, including the global metadata from the input file.
As written, the script could fail on groups that contain advanced
netCDF4 features because the user requests (with the ‘-3’ switch) that
output be netCDF3 classic format.  However, ‘ncks’ detects many format
incompatibilities in advance and works around them.  For example, ‘ncks’
autoconverts netCDF4-only atomic-types (such as ‘NC_STRING’ and
‘NC_UBYTE’) to corresponding netCDF3 atomic types (‘NC_CHAR’ and
‘NC_SHORT’) when the output format is netCDF3.


File: nco.info,  Node: Checking CF-compliance,  Prev: Dismembering Files,  Up: Group Path Editing

3.15.4 Checking CF-compliance
-----------------------------

One application of dismembering is to check the CF-compliance of each
group in a file.  When invoked with the optional third argumnt ‘cf’,
‘ncdismember’ passes each file it generates to freely available
compliance checkers, such as ‘cfchecker’ (1).
     zender@roulee:~$ ncdismember ~/nco/data/mdl_1.nc /data/zender/nco/tmp cf
     NCO dismembering file /home/zender/nco/data/mdl_1.nc
     CFchecker reports CF-compliance of each group in flat netCDF3 format
     WARNING: Using the default (non-CF) Udunits database
     cesm.cesm_01.nc: 
     INFO: INIT:     running CFchecker version 1.5.15
     INFO: INIT:     checking compliance with convention CF-1.5
     INFO: INIT:     using standard name table version: 25, last modified: 2013-07-05T05:40:30Z
     INFO: INIT:     using area type table version: 2, date: 10 July 2013
     INFO: 2.4:      no axis information found in dimension variables, not checking dimension order
     WARNING: 3:     variable "tas1" contains neither long_name nor standard_name attribute
     WARNING: 3:     variable "tas2" contains neither long_name nor standard_name attribute
     INFO: 3.1:      variable "tas1" does not contain units attribute
     INFO: 3.1:      variable "tas2" does not contain units attribute
     --------------------------------------------------
     cesm.cesm_02.nc: 
     ...
   By default the CF version checked is determined automatically by
‘cfchecker’.  The user can override this default by supplying a
supported CF version, e.g., ‘1.3’, as an optional fourth argument to
‘ncdismember’.  Current valid CF options are ‘1.0’, ‘1.1’, ‘1.2’, ‘1.3’,
‘1.4’, and ‘1.5’.

   Our development and testing of ‘ncdismember’ is funded by our
involvement in NASA's Dataset Interoperability Working Group (DIWG
(https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group)),
though our interest extends beyond NASA datasets.  Taken together, NCO's
features (autoconversion to netCDF3 atomic types, fixing multiple record
dimensions, autosensing HDF4 input, scoping rules for CF conventions)
make ‘ncdismember’ reliable and friendly for both dismembering
hierarchical files and for CF-compliance checks.  Most HDF4 and HDF5
datasets can be checked for CF-compliance with a one-line command.
Example compliance checks of common NASA datasets are at
<http://dust.ess.uci.edu/diwg>.  Our long-term goal is to enrich the
hierarchical data model with the expressivity and syntactic power of CF
conventions.

   NASA asked the DIWG to prepare a one-page summary of the procedure
necessary to check HDF files for CF-compliance:
     cat > ~/ncdismember.txt << 'EOF'
         Preparing an RPM-based OS to Test HDF & netCDF Files for CF-Compliance
     
     By Charlie Zender, UCI & NASA Dataset Interoperability Working Group (DIWG)
     
     Installation Summary:
     1. HDF4 [with internal netCDF support _disabled_]
     2. HDF5
     3. netCDF [with external HDF4 support _enabled_]
     4. NCO
     5. numpy
     6. netcdf4-python
     7. python-lxml
     8. CFunits-python
     9. CFChecker
     10. ncdismember
     
     All 10 packages can use default installs _except_ HDF4 and netCDF.
     Following instructions for Fedora Core 20 (FC20), an RPM-based Linux OS
     Feedback and changes for other Linux-based OS's welcome to zender at uci.edu
     ${H4DIR}, ${H5DIR}, ${NETCDFDIR}, ${NCODIR}, may all be different
     For simplicity CZ sets them all to /usr/local
     
     # 1. HDF4. Build in non-default manner. Turn-off its own netCDF support.
     # Per http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html
     # HDF4 support not necessary though it makes ncdismember more comprehensive
     wget -c http://www.hdfgroup.org/ftp/HDF/HDF_Current/src/hdf-4.2.9.tar.gz
     tar xvzf hdf-4.2.9.tar.gz
     cd hdf-4.2.9
     ./configure --enable-shared --disable-netcdf --disable-fortran --prefix=${H4DIR}
     make && make check && make install
     
     # 2. HDF5. Build normally. RPM may work too. Please let me know if so.
     # HDF5 is a necessary pre-requisite for netCDF4
     wget -c ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4/hdf5-1.8.11.tar.gz
     tar xvzf hdf5-1.8.11.tar.gz
     cd hdf5-1.8.11
     ./configure --enable-shared --prefix=${H5DIR}
     make && make check && make install
     
     # 3. netCDF version 4.3.1 or later. Build in non-default manner with HDF4.
     # Per http://www.unidata.ucar.edu/software/netcdf/docs/build_hdf4.html
     # Earlier versions of netCDF may fail checking some HDF4 files
     wget -c ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.3.2.tar.gz
     tar xvzf netcdf-4.3.2.tar.gz
     cd netcdf-4.3.2
     CPPFLAGS="-I${H5DIR}/include -I${H4DIR}/include" \
     LDFLAGS="-L${H5DIR}/lib -L${H4DIR}/lib" \
     ./configure --enable-hdf4 --enable-hdf4-file-tests
     make && make check && make install
     
     # 4. NCO version 4.4.0 or later. Some RPMs available. Or install by hand.
     # Later versions of NCO have much better support for ncdismember
     wget http://nco.sourceforge.net/src/nco-4.4.4.tar.gz .
     tar xvzf nco-4.4.4.tar.gz
     cd nco-4.4.4
     ./configure --prefix=${NCODIR}
     make && make install
     
     # 5. numpy
     sudo yum install numpy -y
     
     # 6. netcdf4-python
     sudo yum install netcdf4-python -y
     
     # 7. python-lxml
     sudo yum install python-lxml -y
     
     # 8. CFunits-python. No RPM available. Must install by hand.
     # http://code.google.com/p/cfunits-python/
     wget http://cfunits-python.googlecode.com/files/cfunits-0.9.6.tar.gz .
     tar xvzf cfunits-0.9.6.tar.gz
     cd cfunits-0.9.6
     sudo python setup.py install
     
     # 9. CFChecker. No RPM available. Must install by hand.
     # https://bitbucket.org/mde_/cfchecker
     wget https://bitbucket.org/mde_/cfchecker/downloads/CFchecker-1.5.15.tar.bz2 . 
     tar xvjf CFchecker-1.5.15.tar.bz2 
     cd CFchecker
     sudo python setup.py install
     
     # 10. ncdismember. Copy script from http://nco.sf.net/nco.html#ncdismember
     # Store dismembered files somewhere, e.g., ${DATA}/nco/tmp/hdf
     mkdir -p ${DATA}/nco/tmp/hdf
     # Many datasets work with a simpler command...
     ncdismember ~/nco/data/in.nc ${DATA}/nco/tmp/hdf cf 1.5
     ncdismember ~/nco/data/mdl_1.nc ${DATA}/nco/tmp/hdf cf 1.5
     ncdismember ${DATA}/hdf/AMSR_E_L2_Rain_V10_200905312326_A.hdf \
                 ${DATA}/nco/tmp/hdf cf 1.5
     ncdismember ${DATA}/hdf/BUV-Nimbus04_L3zm_v01-00-2012m0203t144121.h5 \
                 ${DATA}/nco/tmp/hdf cf 1.5
     ncdismember ${DATA}/hdf/HIRDLS-Aura_L3ZAD_v06-00-00-c02_2005d022-2008d077.he5 ${DATA}/nco/tmp/hdf cf 1.5
     # Some datasets, typically .h5, require the --fix_rec_dmn=all argument
     ncdismember_${DATA}/hdf/GATMO_npp_d20100906_t1935191_e1935505_b00012_c20110707155932065809_noaa_ops.h5 ${DATA}/nco/tmp/hdf cf 1.5 --fix_rec_dmn=all
     ncdismember ${DATA}/hdf/mabel_l2_20130927t201800_008_1.h5 \
                 ${DATA}/nco/tmp/hdf cf 1.5 --fix_rec_dmn=all
     EOF
   A PDF version of these instructions is available here
(http://dust.ess.uci.edu/diwg/ncdismember.pdf).

   ---------- Footnotes ----------

   (1) CFchecker is developed by Michael Decker and Martin Schultz at
Forschungszentrum Jülich and distributed at
<https://bitbucket.org/mde_/cfchecker>.


File: nco.info,  Node: C and Fortran Index Conventions,  Next: Hyperslabs,  Prev: Group Path Editing,  Up: Shared features

3.16 C and Fortran Index conventions
====================================

Availability: ‘ncbo’, ‘nces’, ‘ncecat’, ‘ncflint’, ‘ncks’, ‘ncpdq’,
‘ncra’, ‘ncrcat’, ‘ncwa’
Short options: ‘-F’
Long options: ‘--fortran’
   The ‘-F’ switch changes NCO to read and write with the Fortran index
convention.  By default, NCO uses C-style (0-based) indices for all I/O.
In C, indices count from 0 (rather than 1), and dimensions are ordered
from slowest (inner-most) to fastest (outer-most) varying.  In Fortran,
indices count from 1 (rather than 0), and dimensions are ordered from
fastest (inner-most) to slowest (outer-most) varying.  Hence C and
Fortran data storage conventions represent mathematical transposes of
eachother.  Note that record variables contain the record dimension as
the most slowly varying dimension.  See *note ncpdq netCDF Permute
Dimensions Quickly:: for techniques to re-order (including transpose)
dimensions and to reverse data storage order.

   Consider a file ‘85.nc’ containing 12 months of data in the record
dimension ‘time’.  The following hyperslab operations produce identical
results, a June-July-August average of the data:
     ncra -d time,5,7 85.nc 85_JJA.nc
     ncra -F -d time,6,8 85.nc 85_JJA.nc

   Printing variable THREE_DMN_VAR in file ‘in.nc’ first with the
C indexing convention, then with Fortran indexing convention results in
the following output formats:
     % ncks --trd -v three_dmn_var in.nc
     lat[0]=-90 lev[0]=1000 lon[0]=-180 three_dmn_var[0]=0
     ...
     % ncks --trd -F -v three_dmn_var in.nc
     lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0
     ...


File: nco.info,  Node: Hyperslabs,  Next: Stride,  Prev: C and Fortran Index Conventions,  Up: Shared features

3.17 Hyperslabs
===============

Availability: ‘ncbo’, ‘nces’, ‘ncecat’, ‘ncflint’, ‘ncks’, ‘ncpdq’,
‘ncra’, ‘ncrcat’, ‘ncwa’
Short options: ‘-d DIM,[MIN][,[MAX][,[STRIDE]]]’
Long options: ‘--dimension DIM,[MIN][,[MAX][,[STRIDE]]]’,
‘--dmn DIM,[MIN][,[MAX][,[STRIDE]]]’
   A “hyperslab” is a subset of a variable's data.  The coordinates of a
hyperslab are specified with the ‘-d DIM,[MIN][,[MAX][,[STRIDE]]]’ short
option (or with the same arguments to the ‘--dimension’ or ‘--dmn’ long
options).  At least one hyperslab argument (MIN, MAX, or STRIDE) must be
present.  The bounds of the hyperslab to be extracted are specified by
the associated MIN and MAX values.  A half-open range is specified by
omitting either the MIN or MAX parameter.  The separating comma must be
present to indicate the omission of one of these arguments.  The
unspecified limit is interpreted as the maximum or minimum value in the
unspecified direction.  A cross-section at a specific coordinate is
extracted by specifying only the MIN limit and omitting a trailing
comma.  Dimensions not mentioned are passed with no reduction in range.
The dimensionality of variables is not reduced (in the case of a
cross-section, the size of the constant dimension will be one).
     # First and second longitudes
     ncks -F -d lon,1,2 in.nc out.nc
     # Second and third longitudes
     ncks -d lon,1,2 in.nc out.nc

   As of version 4.2.1 (August, 2012), NCO allows one to extract the
last N elements of a hyperslab.  Negative integers as MIN or MAX
elements of a hyperslab specification indicate offsets from the end
(Python also uses this convention).  Consistent with this convention,
the value ‘-1’ (negative one) indicates the last element of a dimension,
and negative zero is algebraically equivalent to zero and so indicates
the first element of a dimension.  Previously, for example, ‘-d
time,-2,-1’ caused a domain error.  Now it means select the penultimate
and last timesteps, independent of the size of the ‘time’ dimension.
Select only the first and last timesteps, respectively, with ‘-d time,0’
and ‘-d time,-1’.  Negative integers work for MIN and MAX indices,
though not for STRIDE.
     # Second through penultimate longitudes
     ncks -d lon,1,-2 in.nc out.nc
     # Second through last longitude
     ncks -d lon,1,-1 in.nc out.nc
     # Second-to-last to last longitude
     ncks -d lon,-3,-1 in.nc out.nc
     # Second-to-last to last longitude
     ncks -d lon,-3, in.nc out.nc
The ‘-F’ argument, if any, applies the Fortran index convention only to
indices specified as positive integers:
     # First through penultimate longitudes
     ncks -F -d lon,1,-2 in.nc out.nc (-F affects only start index)
     # First through last longitude
     ncks -F -d lon,1,-1 in.nc out.nc
     # Second-to-last to penultimate longitude (-F has no effect)
     ncks -F -d lon,-3,-1 in.nc out.nc
     # Second-to-last to last longitude (-F has no effect)
     ncks -F -d lon,-3, in.nc out.nc

   Coordinate values should be specified using real notation with a
decimal point required in the value, whereas dimension indices are
specified using integer notation without a decimal point.  This
convention serves only to differentiate coordinate values from dimension
indices.  It is independent of the type of any netCDF coordinate
variables.  In other words, even if coordinates are defined as integers,
specify them with decimal points to have the command interpret them as
values, rather than indices.  For a given dimension, the specified
limits must both be coordinate values (with decimal points) or dimension
indices (no decimal points).

   If values of a coordinate-variable are used to specify a range or
cross-section, then the coordinate variable must be monotonic (values
either increasing or decreasing).  In this case, command-line values
need not exactly match coordinate values for the specified dimension.
Ranges are determined by seeking the first coordinate value to occur in
the closed range [MIN,MAX] and including all subsequent values until one
falls outside the range.  The coordinate value for a cross-section is
the coordinate-variable value closest to the specified value and must
lie within the range or coordinate-variable values.  The STRIDE
argument, if any, must be a dimension index, not a coordinate value.
*Note Stride::, for more information on the STRIDE option.
     # All longitude values between 1 and 2 degrees
     ncks -d lon,1.0,2.0 in.nc out.nc
     # All longitude values between 1 and 2 degrees
     ncks -F -d lon,1.0,2.0 in.nc out.nc
     # Every other longitude value between 0 and 90 degrees
     ncks -F -d lon,0.0,90.0,2 in.nc out.nc
   As shown, we recommend using a full floating-point suffix of ‘.0’
instead of simply ‘.’ in order to make obvious the selection of
hyperslab elements based on coordinate value rather than index.

   User-specified coordinate limits are promoted to double-precision
values while searching for the indices which bracket the range.  Thus,
hyperslabs on coordinates of type ‘NC_CHAR’ are computed numerically
rather than lexically, so the results are unpredictable.

   The relative magnitude of MIN and MAX indicate to the operator
whether to expect a “wrapped coordinate” (*note Wrapped Coordinates::),
such as longitude.  If MIN > MAX, the NCO expects the coordinate to be
wrapped, and a warning message will be printed.  When this occurs, NCO
selects all values outside the domain [MAX < MIN], i.e., all the values
exclusive of the values which would have been selected if MIN and MAX
were swapped.  If this seems confusing, test your command on just the
coordinate variables with ‘ncks’, and then examine the output to ensure
NCO selected the hyperslab you expected (coordinate wrapping is
currently only supported by ‘ncks’).

   Because of the way wrapped coordinates are interpreted, it is very
important to make sure you always specify hyperslabs in the
monotonically increasing sense, i.e., MIN < MAX (even if the underlying
coordinate variable is monotonically decreasing).  The only exception to
this is when you are indeed specifying a wrapped coordinate.  The
distinction is crucial to understand because the points selected by,
e.g., ‘-d longitude,50.,340.’, are exactly the complement of the points
selected by ‘-d longitude,340.,50.’.

   Not specifying any hyperslab option is equivalent to specifying full
ranges of all dimensions.  This option may be specified more than once
in a single command (each hyperslabbed dimension requires its own ‘-d’
option).


File: nco.info,  Node: Stride,  Next: Record Appending,  Prev: Hyperslabs,  Up: Shared features

3.18 Stride
===========

Availability: ‘ncbo’, ‘nces’, ‘ncecat’, ‘ncflint’, ‘ncks’, ‘ncpdq’,
‘ncra’, ‘ncrcat’, ‘ncwa’
Short options: ‘-d DIM,[MIN][,[MAX][,[STRIDE]]]’
Long options: ‘--dimension DIM,[MIN][,[MAX][,[STRIDE]]]’,
‘--dmn DIM,[MIN][,[MAX][,[STRIDE]]]’
   All data operators support specifying a “stride” for any and all
dimensions at the same time.  The STRIDE is the spacing between
consecutive points in a hyperslab.  A STRIDE of 1 picks all the elements
of the hyperslab, and a STRIDE of 2 skips every other element, etc..
‘ncks’ multislabs support strides, and are more powerful than the
regular hyperslabs supported by the other operators (*note
Multislabs::).  Using the STRIDE option for the record dimension with
‘ncra’ and ‘ncrcat’ makes it possible, for instance, to average or
concatenate regular intervals across multi-file input data sets.

   The STRIDE is specified as the optional fourth argument to the ‘-d’
hyperslab specification: ‘-d DIM,[MIN][,[MAX][,[STRIDE]]]’.  Specify
STRIDE as an integer (i.e., no decimal point) following the third comma
in the ‘-d’ argument.  There is no default value for STRIDE.  Thus using
‘-d time,,,2’ is valid but ‘-d time,,,2.0’ and ‘-d time,,,’ are not.
When STRIDE is specified but MIN is not, there is an ambiguity as to
whether the extracted hyperslab should begin with (using C-style,
0-based indexes) element 0 or element ‘stride-1’.  NCO must resolve this
ambiguity and it chooses element 0 as the first element of the hyperslab
when MIN is not specified.  Thus ‘-d time,,,STRIDE’ is syntactically
equivalent to ‘-d time,0,,STRIDE’.  This means, for example, that
specifying the operation ‘-d time,,,2’ on the array ‘1,2,3,4,5’ selects
the hyperslab ‘1,3,5’.  To obtain the hyperslab ‘2,4’ instead, simply
explicitly specify the starting index as 1, i.e., ‘-d time,1,,2’.

   For example, consider a file ‘8501_8912.nc’ which contains 60
consecutive months of data.  Say you wish to obtain just the March data
from this file.  Using 0-based subscripts (*note C and Fortran Index
Conventions::) these data are stored in records 2, 14, ... 50 so the
desired STRIDE is 12.  Without the STRIDE option, the procedure is very
awkward.  One could use ‘ncks’ five times and then use ‘ncrcat’ to
concatenate the resulting files together:
     for idx in 02 14 26 38 50; do # Bourne Shell
       ncks -d time,${idx} 8501_8912.nc foo.${idx}
     done
     foreach idx (02 14 26 38 50) # C Shell
       ncks -d time,${idx} 8501_8912.nc foo.${idx}
     end
     ncrcat foo.?? 8589_03.nc
     rm foo.??
   With the STRIDE option, ‘ncks’ performs this hyperslab extraction in
one operation:
     ncks -d time,2,,12 8501_8912.nc 8589_03.nc
   *Note ncks netCDF Kitchen Sink::, for more information on ‘ncks’.

   Applying the STRIDE option to the record dimension in ‘ncra’ and
‘ncrcat’ makes it possible, for instance, to average or concatenate
regular intervals across multi-file input data sets.
     ncra -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8589_03.nc
     ncrcat -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8503_8903.nc


File: nco.info,  Node: Record Appending,  Next: Subcycle,  Prev: Stride,  Up: Shared features

3.19 Record Appending
=====================

Availability: ‘ncra’, ‘ncrcat’
Short options: None
Long options: ‘--rec_apn’, ‘--record_append’
   As of version 4.2.6 (March, 2013), NCO allows both Multi-File,
Multi-Record operators (‘ncra’ and ‘ncrcat’) to append their output
directly to the end of an existing file.  This feature may be used to
augment a target file, rather than construct it from scratch.  This
helps, for example, when a timeseries is concatenated from input data
that becomes available in stages rather than all at once.  In such cases
this switch significantly speeds writing.

   Consider the use case where one wishes to preserve the contents of
‘fl_1.nc’, and add to them new records contained in ‘fl_2.nc’.
Previously the output had to be placed in a third file, ‘fl_3.nc’ (which
could also safely be named ‘fl_2.nc’), via
     ncrcat -O fl_1.nc fl_2.nc fl_3.nc
   Under the hood this operation copies all information in ‘fl_1.nc’ and
‘fl_2.nc’ not once but twice.  The first copy is performed through the
netCDF interface, as all data from ‘fl_1.nc’ and ‘fl_2.nc’ are extracted
and placed in the output file.  The second copy occurs (usually much)
more quickly as the (by default) temporary output file is copied
(sometimes a quick re-link suffices) to the final output file (*note
Temporary Output Files::).  All this copying is expensive for large
files.

   The ‘--record_append’ switch appends all records in ‘fl_2.nc’ to the
end (after the last record) of ‘fl_1.nc’:
     ncrcat --rec_apn fl_2.nc fl_1.nc
   The ordering of the filename arguments may seem non-intuitive.  If
the record variable represents time in these files, then the values in
‘fl_1.nc’ precede those in ‘fl_2.nc’, so why do the files appear in the
reverse order on the command line?  ‘fl_1.nc’ is the last file named
because it is the pre-existing output file to which we will append all
the other input files listed (in this case only ‘fl_2.nc’).  The
contents of ‘fl_1.nc’ are completely preserved, and only values in
‘fl_2.nc’ (and any other input files) are copied.  This switch avoids
the necessity of copying all of ‘fl_1.nc’ through the netCDF interface
to a new output file.  The ‘--rec_apn’ switch automatically puts NCO
into append mode (*note Appending Variables::), so specifying ‘-A’ is
redundant, and simultaneously specifying overwrite mode with ‘-O’ causes
an error.  By default, NCO works in an intermediate temporary file.
Power users may combine ‘--rec_apn’ with the ‘--no_tmp_fl’ switch (*note
Temporary Output Files::):
     ncrcat --rec_apn --no_tmp_fl fl_2.nc fl_1.nc
   This avoids creating an intermediate file, and copies only the
minimal amount of data (i.e., all of ‘fl_2.nc’).  Hence, it is fast.  We
recommend users try to understand the safety trade-offs involved.

   One side-effect of ‘--rec_apn’ to be aware of is how attributes are
handled.  When appending files, NCO typically overwrites attributes for
existing variables in the destination file with the corresponding
attributes from the same variable in the source file.  The exception to
this rule is when ‘--rec_apn’ is invoked.  As of version 4.7.9 (January,
2019), NCO leaves unchanged the attributes for existing variables in the
destination file.  This is primarily to ensure that calendar attributes
(e.g., ‘units’, ‘calendar’) of the record coordinate, if any, are
maintained, so that the data appended to them can be re-based to the
existing units.  Otherwise rebasing would fail or require rewriting the
entire file which is counter to the purpose of ‘--rec_apn’.


File: nco.info,  Node: Subcycle,  Next: Interleave,  Prev: Record Appending,  Up: Shared features

3.20 Subcycle
=============

Availability: ‘ncra’, ‘ncrcat’
Short options: ‘-d DIM,[MIN][,[MAX][,[STRIDE][,[SUBCYCLE]]]]’
Long options: ‘--mro’ ‘--dimension
DIM,[MIN][,[MAX][,[STRIDE][,[SUBCYCLE]]]]’
‘--dmn DIM,[MIN][,[MAX][,[STRIDE][,[SUBCYCLE]]]]’
   As of version 4.2.1 (August, 2012), NCO allows both Multi-File,
Multi-Record operators, ‘ncra’ and ‘ncrcat’, to extract and operate on
multiple groups of records.  These groups may be connected to physical
_sub-cycles_ of a periodic nature, e.g., months of a year, or hours of a
day.  Or they may be thought of as groups of a specifed duration.  We
call this the “subcycle feature”, sometimes abbreviated SSC (1).

   The subcycle feature allows processing of groups of records separated
by regular intervals of records.  It is perhaps best illustrated by an
extended example that describes how to solve the same problem both with
and without the SSC feature.

   Creating seasonal cycles is a common task in climate data processing.
Suppose a 150-year climate simulation produces 150 output files, each
comprising 12 records, each record a monthly mean: ‘1850.nc’, ‘1851.nc’,
...  ‘1999.nc’.  Our goal is to create a single file that contains the
climatological summertime (June, July, and August, aka JJA) mean.
Traditionally, we would first compute the climatological monthly mean
for each month of summer.  Each of these is a 150-year mean, i.e.,
     # Step 1: Create climatological monthly files clm06.nc..clm08.nc
     for mth in {6..8}; do
       mm=`printf "%02d" $mth`
       ncra -O -F -d time,${mm},,12 -n 150,4,1 1850.nc clm${mm}.nc
     done
     # Step 2: Average climatological monthly files into summertime mean
     ncra -O clm06 clm07.nc clm08.nc clm_JJA.nc
So far, nothing is unusual and this task can be performed by any NCO
version.  The SSC feature makes obsolete the need for the shell loop
used in Step 1 above.

   The new SSC option aggregates more than one input record at a time
before performing arithmetic operations, and, with an additional switch,
allows archival of those results in multiple-record output (MRO) files.
This reduces the task of producing the climatological summertime mean to
_one_ step:
     # Step 1: Compute climatological summertime mean
     ncra -O -F -d time,6,,12,3 -n 150,4,1 1850.nc clm_JJA.nc
The SSC option instructs ‘ncra’ (or ‘ncrcat’) to process files in groups
of three records.  To better understand the meaning of each argument to
the ‘-d’ hyperslab option, read it this way: "for the time dimension
start with the sixth record, continue without end, repeat the process
every twelfth record, and define a sub-cycle as three consecutive
records".

   A separate option, ‘--mro’, instructs ‘ncra’ to output its results
from each sub-group, and to produce a “Multi-Record Output” (MRO) file
rather than a “Single-Record Output” (SRO) file.  Unless
Multi-Record-Output is indicated (either with ‘--mro’ or implicitly, as
with interleave-mode), ‘ncra’ collects together all sub-groups, operates
on their ensemble, and produces a single output record.  Adding ‘--mro’
to the above example causes ‘ncra’ to archive all (150) annual
summertime means to one file:
     # Step 1: Archive all 150 summertime means in one file
     ncra --mro -O -F -d time,6,,12,3 -n 150,4,1 1850.nc 1850_2009_JJA.nc
     # ...or all (150) annual means...
     ncra --mro -O -d time,,,12,12 -n 150,4,1 1850.nc 1850_2009.nc
These operations generate and require no intermediate files.  This
contrasts to previous NCO methods, which require generating, averaging,
then catenating 150 files.  The ‘--mro’ option only works on ‘ncra’ and
has no effect on (or rather, is redundant for) ‘ncrcat’, since ‘ncrcat’
always outputs all selected records.

   Another example of the power of this sub-cycle feature is in
coarsening the resolution of high frequency timeseries.  Say you have
3-hourly data stored in a series of consecutive input files named
‘in1.nc, in2.nc, inN.nc’.  You can coarsen this to a single file of
daily-mean data with:
     ncra -O --mro -d time,${srt_idx},,8,8 in*.nc out.nc
Set SRT_IDX=0 if the first timestep is identical to the rest, or
SRT_IDX=1 if the first timestep is an instantaneous output (e.g.,
restart value) associated with no time interval, e.g.,
‘time_bnds[0,0]=time_bnds[0,1]’).  The first eight in the temporal
hyperslab argument is the stride between first elements of a group
(i.e., every 8*3=24 hours), and the second eight is the number of
records to average within the group (i.e., average all 8*3=24 hours).
The ‘--mro’ (multi-record output) switch tells ‘ncra’ to create a single
output record from each input group.

   ---------- Footnotes ----------

   (1) When originally released in 2012 this was called the “duration
feature”, and was abbreviated DRN.


File: nco.info,  Node: Interleave,  Next: Multislabs,  Prev: Subcycle,  Up: Shared features

3.21 Interleave
===============

Availability: ‘ncra’, ‘ncrcat’
Short options: ‘-d
DIM,[MIN][,[MAX][,[STRIDE][,[SUBCYCLE][,[INTERLEAVE]]]]]’
Long options: ‘--mro’ ‘--dimension
DIM,[MIN][,[MAX][,[STRIDE][,[SUBCYCLE][,[INTERLEAVE]]]]]’
‘--dmn DIM,[MIN][,[MAX][,[STRIDE][,[SUBCYCLE][,[INTERLEAVE]]]]]’

   Caveat lector: Unforunately NCO versions from 4.9.4-5.1.8 (September,
2020 through October, 2023) contained a bug that affected the subcycle
and interleave (SSC and ILV) hyperslab features.  The bug was triggered
by invoking the SSC feature without explicitly providing an ILV
parameter.  The software failed to initialize an internal flag that
indicated whether ILV had been invoked.  The resulting behavior was
compiler-dependent.  Most compilers set the flag to false (as it should
have been), but occasionally it was set to true).  The bug expressed
itself by extracting only a single timestep, rather than the number of
timesteps indicated by the SSC parameter.  This behavior was fixed in
NCO version 5.1.9 (November, 2023).

   As of version 4.9.4 (September, 2020), NCO allows both Multi-File,
Multi-Record operators (‘ncra’ and ‘ncrcat’) to extract, interleave, and
operate on multiple groups of records.  Interleaving (or
de-interleaving, depending on one's perspective) means altering the
order of records in a group to be processed.  Specifically, the
interleaving feature (sometimes abbreviated ILV) causes the operator to
treat as sequential records those that are separated by multiples of the
specified INTERLEAVE parameter within a group or sub-cycle of records.

   The interleave feature sequences records with respect to their
position relative to the beginning of each sub-cycle.  Records an
integer multiple of INTERLEAVE from the sub-cycle start are first
extracted (‘ncrcat’) or reduced (‘ncra’), then records offset from these
by one, two, et cetera up to INTERLEAVE-1.  In this manner interleaving
extracts an inner (intra-sub-cycle) loop that preserves high-frequency
signals relative to the longer stride between sub-cycles.  Thus
interleaving allows deconvolution of periodic phenomena within a
time-series.

   Processing simple arithmetic sequences is a helpful way to understand
what interleaving does.  Here are some examples to reify the abstract.
Let ‘in1.nc’ contain the record-array [1..10], ‘in2.nc’ contain
[11..20], and ‘in12.nc’ contain [1..20].
     ncra   -O -d time,,,,10,5 ~/in1.nc ~/foo.nc # 3.5, 4.5, 5.5, 6.5, 7.5
     ncrcat -O -d time,0,4,,6,2 ~/in1.nc ~/foo.nc # 1, 3, 5, 2, 4, 6 (+WARNING)
     ncrcat -O -d time,2,,10,4,2 ~/in12.nc ~/foo.nc # 3, 5, 4, 6, 13, 15, 14, 16
     ncra   -O -d time,2,,10,4,2 ~/in12.nc ~/foo.nc # 4, 5, 14, 15
     ncra   -O -d time,,,,10,2 ~/in1.nc ~/in2.nc ~/foo.nc # 5, 6, 15, 16
     ncra   -O -d time,,,,10,2 ~/in12.nc ~/foo.nc # 5, 6, 15, 16

   Interleaving is perhaps best illustrated by an extended example that
describes how to solve the same problem both with and without the ILV
feature.  Consider as an example an interannual timeseries archived at a
high-enough temporal frequency to resolve the diurnal cycle with TPD
timesteps-per-day.  Many climate models and re-analyses are archived at
hourly, tri-hourly, or six-hourly resolution yielding TPD = 24, 8, or 6,
respectively.  Our goal is to extract a monthly mean diurnal cycle from
this timeseries.

   Suppose a 150-year climate simulation produces 150 output files, each
comprising 365 days of hourly data, or 8760 records, each record an
hourly mean: ‘1850.nc’, ‘1851.nc’, ...  ‘1999.nc’.  Our goal is to
create a single file that contains the climatological monthly mean
diurnal cycle for, say, March, which contains 31 days or 744 hourly
records that commence on the 60th day of the 356-day year, with record
index 1416.  Traditionally, we might first compute the climatological
monthly mean for hour of the day, then combine those into a full diurnal
cycle:
     # Step 1: Create climatological hourly files hr00.nc..hr23.nc 
     for hr in {0..23}; do
       hh=`printf "%02d" $hr`
       let srt=${hr}+1416
       # Alternatively, use UDUnits by setting srt=1850-03-01T00:00:01 
       ncra -O -d time,${srt},,8760 -n 150,4,1 1850.nc hr${hh}.nc
     done
     # Step 2: Concatenate climatological hourly files into diurnal cycle
     ncrcata -O hr??.nc clm_drn.nc
So far, nothing is unusual and this task can be performed by any NCO
version.  The ILV feature obsoletes the need for the shell loop used in
Step 1 above.

   The new ILV option aggregates more than one input record at a time
before performing arithmetic operations, and, with an additional switch,
allows archival of those results in multiple-record output (MRO) files.
This reduces the task of producing the climatological summertime mean to
_one_ step:
     # Step 1: Archive all 150 March-mean diurnal cycles in one file
     ncra -O -d time,1850-03-01T00:00:01,,8760,744,24 -n 150,4,1 1850.nc clm_drn.nc
The ILV option instructs ‘ncra’ (or ‘ncrcat’) to process files in groups
of 31 days (744 hourly records) interleaved with a 24-record cycle.  The
end result will have 150 sets of 24-timesteps representing the diurnal
cycle of March in every year.  A given timestep is the mean of the same
hour of the day for every day in March of that year.


File: nco.info,  Node: Multislabs,  Next: Wrapped Coordinates,  Prev: Interleave,  Up: Shared features

3.22 Multislabs
===============

Availability: ‘ncbo’, ‘nces’, ‘ncecat’, ‘ncflint’, ‘ncks’, ‘ncpdq’,
‘ncra’, ‘ncrcat’
Short options: ‘-d DIM,[MIN][,[MAX][,[STRIDE]]]’
Long options: ‘--dimension DIM,[MIN][,[MAX][,[STRIDE]]]’,
‘--dmn DIM,[MIN][,[MAX][,[STRIDE]]]’
‘--msa_usr_rdr’, ‘--msa_user_order’
   A multislab is a union of one or more hyperslabs.  One defines
multislabs by chaining together hyperslab commands, i.e., ‘-d’ options
(*note Hyperslabs::).  Support for specifying a “multi-hyperslab” or
“multislab” for any variable was first added to ‘ncks’ in late 2002.
The other operators received these capabilities in April 2008.
Multi-slabbing is often referred to by the acronym MSA, which stands for
"Multi-Slabbing Algorithm".  As explained below, the user may
additionally request that the multislabs be returned in the
user-specified order, rather than the on-disk storage order.  Although
MSA user-ordering has been available in all operators since 2008, most
users were unaware of it since the documentation (below, and in the man
pages) was not written until July 2013.

   Multislabs overcome many restraints that limit simple hyperslabs.
A single ‘-d’ option can only specify a contiguous and/or a regularly
spaced multi-dimensional data array.  Multislabs are constructed from
multiple ‘-d’ options and may therefore have non-regularly spaced
arrays.  For example, suppose it is desired to operate on all longitudes
from 10.0 to 20.0 and from 80.0 to 90.0 degrees.  The combined range of
longitudes is not selectable in a single hyperslab specfication of the
form ‘-d DIMENSION,MIN,MAX’ or ‘-d DIMENSION,MIN,MAX,STRIDE’ because its
elements are irregularly spaced in coordinate space (and presumably in
index space too).  The multislab specification for obtaining these
values is simply the union of the hyperslabs specifications that
comprise the multislab, i.e.,
     ncks -d lon,10.,20. -d lon,80.,90. in.nc out.nc
     ncks -d lon,10.,15. -d lon,15.,20. -d lon,80.,90. in.nc out.nc
Any number of hyperslabs specifications may be chained together to
specify the multislab.  MSA creates an output dimension equal in size to
the sum of the sizes of the multislabs.  This can be used to extend and
or pad coordinate grids.

   Users may specify redundant ranges of indices in a multislab, e.g.,
     ncks -d lon,0,4 -d lon,2,9,2 in.nc out.nc
This command retrieves the first five longitudes, and then every other
longitude value up to the tenth.  Elements 0, 2, and 4 are specified by
both hyperslab arguments (hence this is redundant) but will count only
once if an arithmetic operation is being performed.  This example uses
index-based (not coordinate-based) multislabs because the STRIDE option
only supports index-based hyper-slabbing.  *Note Stride::, for more
information on the STRIDE option.

   Multislabs are more efficient than the alternative of sequentially
performing hyperslab operations and concatenating the results.  This is
because NCO employs a novel multislab algorithm to minimize the number
of I/O operations when retrieving irregularly spaced data from disk.
The NCO multislab algorithm retrieves each element from disk once and
only once.  Thus users may take some shortcuts in specifying multislabs
and the algorithm will obtain the intended values.  Specifying redundant
ranges is not encouraged, but may be useful on occasion and will not
result in unintended consequences.

   Suppose the Q variable contains three dimensional arrays of distinct
chemical constituents in no particular order.  We are interested in the
NOy species in a certain geographic range.  Say that NO, NO2, and N2O5
are elements 0, 1, and 5 of the SPECIES dimension of Q.  The multislab
specification might look something like
     ncks -d species,0,1 -d species,5 -d lon,0,4 -d lon,2,9,2 in.nc out.nc
Multislabs are powerful because they may be specified for every
dimension at the same time.  Thus multislabs obsolete the need to
execute multiple ‘ncks’ commands to gather the desired range of data.

   The MSA user-order switch ‘--msa_usr_rdr’ (or ‘--msa_user_order’,
both of which shorten to ‘--msa’) requests that the multislabs be output
in the user-specified order from the command-line, rather than in the
input-file on-disk storage order.  This allows the user to perform
complex data re-ordering in one operation that would otherwise require
cumbersome steps of hyperslabbing, concatenating, and permuting.
Consider the example of converting datasets stored with the longitude
coordinate ‘Lon’ ranging from [−180,180) to datasets that follow the
[0,360) convention.
     % ncks -H -v Lon in.nc
     Lon[0]=-180
     Lon[1]=-90
     Lon[2]=0
     Lon[3]=90
What is needed is a simple way to rotate longitudes.  Although simple in
theory, this task requires both mathematics to change the numerical
value of the longitude coordinate, data hyperslabbing to split the input
on-disk arrays at Greenwich, and data re-ordering within to stitch the
western hemisphere onto the eastern hemisphere at the date-line.  The
‘--msa’ user-order switch overrides the default that data are output in
the same order in which they are stored on-disk in the input file, and
instead stores them in the same order as the multi-slabs are given to
the command line.  This default is intuitive and is not important in
most uses.  However, the MSA user-order switch allows users to meet
their output order needs by specifying multi-slabs in a certain order.
Compare the results of default ordering to user-ordering for longitude:
     % ncks -O -H       -v Lon -d Lon,0.,180. -d Lon,-180.,-1.0 in.nc
     Lon[0]=-180
     Lon[1]=-90
     Lon[2]=0
     Lon[3]=90
     % ncks -O -H --msa -v Lon -d Lon,0.,180. -d Lon,-180.,-1.0 in.nc
     Lon[0]=0
     Lon[1]=90
     Lon[2]=-180
     Lon[3]=-90
The two multi-slabs are the same but they can be presented to screen, or
to an output file, in either order.  The second example shows how to
place the western hemisphere after the eastern hemisphere, although they
are stored in the opposite order in the input file.

   With this background, one sees that the following commands suffice to
rotate the input file by 180 degrees longitude:
     % ncks -O -v LatLon --msa -d Lon,0.,180. -d Lon,-180.,-1.0 in.nc out.nc
     % ncap2 -O -s 'where(Lon < 0) Lon=Lon+360' out.nc out.nc
     % ncks --trd -C -H -v LatLon ~/nco/data/in.nc
     Lat[0]=-45 Lon[0]=-180 LatLon[0]=0
     Lat[0]=-45 Lon[1]=-90 LatLon[1]=1
     Lat[0]=-45 Lon[2]=0 LatLon[2]=2
     Lat[0]=-45 Lon[3]=90 LatLon[3]=3
     Lat[1]=45 Lon[0]=-180 LatLon[4]=4
     Lat[1]=45 Lon[1]=-90 LatLon[5]=5
     Lat[1]=45 Lon[2]=0 LatLon[6]=6
     Lat[1]=45 Lon[3]=90 LatLon[7]=7
     % ncks --trd -C -H -v LatLon ~/out.nc
     Lat[0]=-45 Lon[0]=0 LatLon[0]=2
     Lat[0]=-45 Lon[1]=90 LatLon[1]=3
     Lat[0]=-45 Lon[2]=180 LatLon[2]=0
     Lat[0]=-45 Lon[3]=270 LatLon[3]=1
     Lat[1]=45 Lon[0]=0 LatLon[4]=6
     Lat[1]=45 Lon[1]=90 LatLon[5]=7
     Lat[1]=45 Lon[2]=180 LatLon[6]=4
     Lat[1]=45 Lon[3]=270 LatLon[7]=5
The analogous commands to rotate all fields in a global dataset by
180 degrees in the other direction, i.e., from [0,360) to [−180,180),
are:
     ncks -O --msa -d lon,181.,360. -d lon,0.,180.0 in.nc out.nc
     ncap2 -O -s 'where(lon > 180) lon=lon-360' out.nc out.nc

   There are other workable, valid methods to rotate data, yet none are
simpler nor more efficient than utilizing MSA user-ordering.  Some final
comments on applying this algorithm: Be careful to specify hemispheres
that do not overlap, e.g., by inadvertently specifying coordinate ranges
that both include Greenwich or the date-line.  Some users will find
using index-based rather than coordinate-based hyperslabs makes this
clearer.


File: nco.info,  Node: Wrapped Coordinates,  Next: Auxiliary Coordinates,  Prev: Multislabs,  Up: Shared features

3.23 Wrapped Coordinates
========================

Availability: ‘ncks’
Short options: ‘-d DIM,[MIN][,[MAX][,[STRIDE]]]’
Long options: ‘--dimension DIM,[MIN][,[MAX][,[STRIDE]]]’,
‘--dmn DIM,[MIN][,[MAX][,[STRIDE]]]’
   A “wrapped coordinate” is a coordinate whose values increase or
decrease monotonically (nothing unusual so far), but which represents a
dimension that ends where it begins (i.e., wraps around on itself).
Longitude (i.e., degrees on a circle) is a familiar example of a wrapped
coordinate.  Longitude increases to the East of Greenwich, England,
where it is defined to be zero.  Halfway around the globe, the longitude
is 180 degrees East (or West).  Continuing eastward, longitude increases
to 360 degrees East at Greenwich.  The longitude values of most
geophysical data are either in the range [0,360), or [−180,180).  In
either case, the Westernmost and Easternmost longitudes are numerically
separated by 360 degrees, but represent contiguous regions on the globe.
For example, the Saharan desert stretches from roughly 340 to 50 degrees
East.  Extracting the hyperslab of data representing the Sahara from a
global dataset presents special problems when the global dataset is
stored consecutively in longitude from 0 to 360 degrees.  This is
because the data for the Sahara will not be contiguous in the INPUT-FILE
but is expected by the user to be contiguous in the OUTPUT-FILE.  In
this case, ‘ncks’ must invoke special software routines to assemble the
desired output hyperslab from multiple reads of the INPUT-FILE.

   Assume the domain of the monotonically increasing longitude
coordinate ‘lon’ is 0 < LON < 360.  ‘ncks’ will extract a hyperslab
which crosses the Greenwich meridian simply by specifying the
westernmost longitude as MIN and the easternmost longitude as MAX.  The
following commands extract a hyperslab containing the Saharan desert:
     ncks -d lon,340.,50. in.nc out.nc
     ncks -d lon,340.,50. -d lat,10.,35. in.nc out.nc
The first example selects data in the same longitude range as the
Sahara.  The second example further constrains the data to having the
same latitude as the Sahara.  The coordinate ‘lon’ in the OUTPUT-FILE,
‘out.nc’, will no longer be monotonic!  The values of ‘lon’ will be,
e.g., ‘340, 350, 0, 10, 20, 30, 40, 50’.  This can have serious
implications should you run ‘out.nc’ through another operation which
expects the ‘lon’ coordinate to be monotonically increasing.
Fortunately, the chances of this happening are slim, since ‘lon’ has
already been hyperslabbed, there should be no reason to hyperslab ‘lon’
again.  Should you need to hyperslab ‘lon’ again, be sure to give
dimensional indices as the hyperslab arguments, rather than coordinate
values (*note Hyperslabs::).


File: nco.info,  Node: Auxiliary Coordinates,  Next: Grid Generation,  Prev: Wrapped Coordinates,  Up: Shared features

3.24 Auxiliary Coordinates
==========================

Availability: ‘ncbo’, ‘nces’, ‘ncecat’, ‘ncflint’, ‘ncks’, ‘ncpdq’,
‘ncra’, ‘ncrcat’
Short options: ‘-X LON_MIN,LON_MAX,LAT_MIN,LAT_MAX’
Long options: ‘--auxiliary LON_MIN,LON_MAX,LAT_MIN,LAT_MAX’
   Utilize auxiliary coordinates specified in values of the coordinate
variable's ‘standard_name’ attributes, if any, when interpreting
hyperslab and multi-slab options.  Also ‘--auxiliary’.  This switch
supports hyperslabbing cell-based grids (aka unstructured grids) over
coordinate ranges.  When these grids are stored as 1D-arrays of cell
data, this feature is helpful at hyperslabbing and/or performing
arithmetic on selected geographic regions.  This feature cannot be used
to select regions of 2D grids (instead use the ‘ncap2’ ‘where’ statement
for such grids *note Where statement::).  This feature works on datasets
that associate coordinate variables to grid-mappings using the
CF-convention (*note CF Conventions::) ‘coordinates’ and ‘standard_name’
attributes described here
(http://cfconventions.org/cf-conventions/cf-conventions.html#coordinate-system).
Currently, NCO understands auxiliary coordinate variables pointed to by
the ‘standard_name’ attributes for LATITUDE and LONGITUDE.  Cells that
contain a value within the user-specified West-East-South-North (aka
WESN) bounding box [LON_MIN,LON_MAX,LAT_MIN,LAT_MAX] are included in the
output hyperslab.

   The sides of the WESN) bounding box must be specified in degrees (not
radians).  The specified coordinates must be within the valid data
range.  This includes boxes that wrap the origin of the longitude
coordinate.  For example, if the longitude coordinate is stored in
[0,360], then a bounding box that straddles the Greenwich meridian in
Africa would be specified as, e.g., [350,10,-20,20], not as
[350,370,-20,20].

   A cell-based or unstructured grid collapses the horizontal spatial
information (latitude and longitude) and stores it along a
one-dimensional coordinate that has a one-to-one mapping to both
latitude and longitude coordinates.  Rectangular (in longitude and
latitude) horizontal hyperslabs cannot be selected using the typical
procedure (*note Hyperslabs::) of separately specifying ‘-d’ arguments
for longitude and latitude.  Instead, when the ‘-X’ is used, NCO learns
the names of the latitude and longitude coordinates by searching the
‘standard_name’ attribute of all variables until it finds the two
variables whose ‘standard_name’'s are "latitude" and "longitude",
respectively.  This ‘standard_name’ attribute for latitude and longitude
coordinates follows the CF-convention (*note CF Conventions::).

   Putting it all together, consider a variable GDS_3DVAR output from
simulations on a cell-based geodesic grid.  Although the variable
contains three dimensions of data (time, latitude, and longitude), it is
stored in the netCDF file with only two dimensions, ‘time’ and
‘gds_crd’.
     % ncks -m -C -v gds_3dvar ~/nco/data/in.nc
     gds_3dvar: type NC_FLOAT, 2 dimensions, 4 attributes, chunked? no, \
      compressed? no, packed? no, ID = 41
     gds_3dvar RAM size is 10*8*sizeof(NC_FLOAT) = 80*4 = 320 bytes
     gds_3dvar dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 20 \
      (CRD)(REC)
     gds_3dvar dimension 1: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
     gds_3dvar attribute 0: long_name, size = 17 NC_CHAR, value = \
      Geodesic variable
     gds_3dvar attribute 1: units, size = 5 NC_CHAR, value = meter
     gds_3dvar attribute 2: coordinates, size = 15 NC_CHAR, value = \
      lat_gds lon_gds
     gds_3dvar attribute 3: purpose, size = 64 NC_CHAR, value = \
      Test auxiliary coordinates like those that define geodesic grids
   The ‘coordinates’ attribute lists the names of the latitude and
longitude coordinates, ‘lat_gds’ and ‘lon_gds’, respectively.  The
‘coordinates’ attribute is recommended though optional.  With it, the
user can immediately identify which variables contain the latitude and
longitude coordinates.  Without a ‘coordinates’ attribute it would be
unclear at first glance whether a variable resides on a cell-based grid.
In this example, ‘time’ is a normal record dimension and ‘gds_crd’ is
the cell-based dimension.

   The cell-based grid file must contain two variables whose
‘standard_name’ attributes are "latitude", and "longitude":
     % ncks -m -C -v lat_gds,lon_gds ~/nco/data/in.nc
     lat_gds: type NC_DOUBLE, 1 dimensions, 4 attributes, \
      chunked? no, compressed? no, packed? no, ID = 37
     lat_gds RAM size is 8*sizeof(NC_DOUBLE) = 8*8 = 64 bytes
     lat_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
     lat_gds attribute 0: long_name, size = 8 NC_CHAR, value = Latitude
     lat_gds attribute 1: standard_name, size = 8 NC_CHAR, value = latitude
     lat_gds attribute 2: units, size = 6 NC_CHAR, value = degree
     lat_gds attribute 3: purpose, size = 62 NC_CHAR, value = \
      1-D latitude coordinate referred to by geodesic grid variables

     lon_gds: type NC_DOUBLE, 1 dimensions, 4 attributes, \
      chunked? no, compressed? no, packed? no, ID = 38
     lon_gds RAM size is 8*sizeof(NC_DOUBLE) = 8*8 = 64 bytes
     lon_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
     lon_gds attribute 0: long_name, size = 9 NC_CHAR, value = Longitude
     lon_gds attribute 1: standard_name, size = 9 NC_CHAR, value = longitude
     lon_gds attribute 2: units, size = 6 NC_CHAR, value = degree
     lon_gds attribute 3: purpose, size = 63 NC_CHAR, value = \
      1-D longitude coordinate referred to by geodesic grid variables
   In this example ‘lat_gds’ and ‘lon_gds’ represent the latitude or
longitude, respectively, of cell-based variables.  These coordinates
(must) have the same single dimension (‘gds_crd’, in this case) as the
cell-based variables.  And the coordinates must be
one-dimensional--multidimensional coordinates will not work.

   This infrastructure allows NCO to identify, interpret, and process
(i.e., hyperslab) the variables on cell-based grids as easily as it
works with regular grids.  To time-average all the values between zero
and 180 degrees longitude and between plus and minus 30 degress
latitude, we use
     ncra -O -X 0.,180.,-30.,30. -v gds_3dvar in.nc out.nc
   NCO accepts multiple ‘-X’ arguments for cell-based grid multi-slabs,
just as it accepts multiple ‘-d’ arguments for multi-slabs of regular
coordinates.
     ncra -O -X 0.,180.,-30.,30. -X 270.,315.,45.,90. in.nc out.nc
   The arguments to ‘-X’ are always interpreted as floating-point
numbers, i.e., as coordinate values rather than dimension indices so
that these two commands produce identical results
     ncra -X 0.,180.,-30.,30. in.nc out.nc
     ncra -X 0,180,-30,30 in.nc out.nc
   By contrast, arguments to ‘-d’ require decimal places to be
recognized as coordinates not indices (*note Hyperslabs::).  We
recommend always using decimal points with ‘-X’ arguments to avoid
confusion.


File: nco.info,  Node: Grid Generation,  Next: Regridding,  Prev: Auxiliary Coordinates,  Up: Shared features

3.25 Grid Generation
====================

Availability: ‘ncks’
Short options: None
Long options: ‘--rgr KEY=VAL’ (multiple invocations allowed)

   As of NCO version 4.5.2 (August, 2015), ‘ncks’ generates accurate and
complete SCRIP-format gridfiles for select grid types, including
uniform, capped and Gaussian rectangular, latitude/longitude grids,
global or regional.  The grids are stored in an external GRID-FILE.

   All options pertinent to the grid geometry and metadata are passed to
NCO via key-value pairs prefixed by the ‘--rgr’ option, or its synonym,
‘--regridding’.  The option ‘--rgr’ (and its long option equivalents
such as ‘--regridding’) indicates the argument syntax will be KEY=VAL.
As such, ‘--rgr’ and its synonyms are indicator options that accept
arguments supplied one-by-one like ‘--rgr KEY1=VAL1 --rgr KEY2=VAL2’, or
aggregated together in multi-argument format like ‘--rgr
KEY1=VAL1#KEY2=VAL2’ (*note Multi-arguments::).

   The text strings that describe the grid and name the file are
important aids to convey the grid geometry to other users.  These
arguments, and their corresponding keys, are the grid title (GRD_TTL),
and grid filename (GRID), respectively.  The numbers of latitudes
(LAT_NBR) and longitudes (LON_NBR) are independent, and together
determine the grid storage size.  These four options should be
considered mandatory, although NCO provides defaults for any arguments
omitted.

   The remaining arguments depend on the whether the grid is global or
regional.  For global grids, one should specify only two more arguments,
the latitude (LAT_TYP) and longitude (LON_TYP) grid-types.  These types
are chosen as described below from a small selection of options that
together define the most common rectangular global grids.  For regional
grids, one must specify the bounding box, i.e., the edges of the
rectangular grid on the North (LAT_NRT), South (LAT_STH), East
(LAT_EST), and West (LAT_NRT) sides.  Specifying a bounding box for
global grids is redundant and will cause an error to ensure the user
intends a global grid.  NCO assumes that regional grids are uniform,
though it will attempt to produce regional grids of other types if the
user specifies other latitude (LAT_TYP) and longitude (LON_TYP)
grid-types, e.g., Gaussian or Cap.  Edges of a regional bounding box may
be specified individually, or in the single-argument forms.

   The full description of grid-generation arguments, and their
corresponding keys, is:
“Grid Title: GRD_TTL”
     It is surprisingly difficult to discern the geometric configuration
     of a grid from the coordinates of a SCRIP-format gridfile.  A
     human-readable grid description should be placed in GRD_TTL.
     Examples include "CAM-FV scalar grid 129x256" and "T42 Gaussian
     grid".

“Grid File: SCRIP_GRID”
     The grid-generation API was bolted-on to NCO and contains some
     temporary kludges.  For example, the output grid filename is
     distinct from the output filename of the host ‘ncks’ command.
     Specify the output gridfile name SCRIP_GRID with keywords ‘grid’ or
     ‘scrip’, e.g., ‘--rgr grid=SCRIP_GRID’ or ‘--rgr
     scrip=‘t42_SCRIP.20150901.nc’’.  It is conventional to include a
     datestamp in the gridfile name.  This helps users identify
     up-to-date and out-of-date grids.  Any valid netCDF file may be
     named as the source (e.g., ‘in.nc’).  It will not be altered.  The
     destination file (e.g., ‘foo.nc’) will be overwritten.  Its
     contents are immaterial.

“Grid Types: LAT_TYP, LON_TYP”
     The keys that hold the longitude and latitude gridtypes (which are,
     by the way, independent of eachother) are LON_TYP and LAT_TYP.  The
     LAT_TYP options for global grids are ‘uni’ for Uniform, ‘cap’ (or
     ‘fv’) for Cap(1), and ‘gss’ for Gaussian.

     These values are all case-independent, so ‘Gss’ and ‘gss’ both
     work.  As of version 4.7.7 (September, 2018), NCO generates
     perfectly symmetric interface latitudes for Gaussian grids.
     Previously the interface latitude generation mechanism could
     accumulate small rounding errors (~1.0e-14).  Now symmetry
     properties are used to ensure perfect symmetry.  All other Gaussian
     grids we have seen compute interfaces as the arithmetic mean of the
     adjacent Gaussian latitudes, which is patently wrong.  To our
     knowledge NCO is the only map software that generates accurate
     interface latitudes for a Gaussian grid.  We use a Newton-Raphson
     iteration technique to identify the interface latitudes that
     enclose the area indicated by the Gaussian weight.

     As its name suggests, the latitudes in a Uniform-latitude grid are
     uniformly spaced (2).  The Uniform-latitude grid may have any
     number of latitudes.  NCO can only generate longitude grids (below)
     that are uniformly spaced, so the Uniform-latitude grids we
     describe are also uniform in the 2D sense.  Uniform grids are
     intuitive, easy to visualize, and simple to program.  Hence their
     popularity in data exchange, visualization, and archives.
     Moreover, regional grids (unless they include the poles), are free
     of polar singularities, and thus are well-suited to storage on
     Uniform grids.  Theoretically, a Uniform-latitude grid could have
     non-uniform longitudes, but NCO currently does not implement
     non-uniform longitude grids.

     Their mathematical properties (convergence and excessive resolution
     at the poles, which can appear as singularities) make Uniform grids
     fraught for use in global models.  One purpose Uniform grids serve
     in modeling is as "offset" or "staggered" grids, meaning grids
     whose centers are the interfaces of another grid.  The
     Finite-Volume (FV) method is often used to represent and solve the
     equations of motion in climate-related fields.  Many FV solutions
     (including the popular Lin-Rood method as used in the CESM CAM-FV
     atmospheric model) evaluate scalar (i.e., non-vector) fields (e.g.,
     temperature, water vapor) at gridcell centers of what is therefore
     called the scalar grid.  FV methods (like Lin-Rood) that employ an
     Arakawa C-grid or D-grid formulation define velocities on the edges
     of the scalar grid.  This CAM-FV velocity grid is therefore
     "staggered" or "offset" from the CAM-FV scalar grid by one-half
     gridcell.  The CAM-FV scalar latitude grid has gridpoints (the
     "caps") centered on each pole to avoid singularities.  The offset
     of a Cap-grid is a Uniform-grid, so the Uniform grid is often
     called an FV-"offset" or "staggered" grid.  Hence an NCO Uniform
     grid is equivalent to an NCL "Fixed Offset" grid.  For example, a
     128x256 Uniform grid is the offset or staggered version of a
     129x256 Cap grid (aka FV-grid).

     Referring the saucer-like cap-points at the poles, NCO uses the
     term "Cap grid" to describe the latitude portion of the FV-scalar
     grid as used by the CAM-FV Lin-Rood dynamics formulation.  NCO
     accepts the shorthand FV, and the more descriptive "Yarmulke", as
     synonyms for Cap.  A Cap-latitude grid differs from a
     Uniform-latitude grid in many ways:

     Most importantly, Cap grids are 2D-representations of numerical
     grids with cap-midpoints instead of zonal-teeth convergence at the
     poles.  The rectangular 2D-representation of each cap contains
     gridcells shaped like sharp teeth that converge at the poles
     similar to the Uniform grid, but the Cap gridcells are meant to be
     aggregated into a single cell centered at the pole in a dynamical
     transport algorithm.  In other words, the polar teeth are a
     convenient way to encode a non-rectangular grid in memory into a
     rectangular array on disk.  Hence Cap grids have the unusual
     property that the poles are labeled as being both the centers and
     the outer interfaces of all polar gridcells.  Second, Cap grids are
     uniform in angle except at the poles, where the latitudes span half
     the meridional range of the rest of the gridcells.  Even though in
     the host dynamical model the Cap grid polar points are melded into
     caps uniform (in angle) with the rest of the grid, the disk
     representation on disk is not uniform.  Nevertheless, some call the
     Cap grid a uniform-angle grid because the information contained at
     the poles is aggregated in memory to span twice the range of a
     single polar gridcell (which has half the normal width).  NCL uses
     the term "Fixed grid" for a Cap grid.  The "Fixed" terminology
     seems broken.

     Finally, Gaussian grids are the Cartesian representation of global
     spectral transform models.  Gaussian grids typically have an even
     number of latitudes and so do not have points at the poles.  All
     three latitude grid-type supported by NCO (Uniform, Cap, and
     Gaussian) are Regular grids in that they are monotonic.

     The LON_TYP options for global grids are ‘grn_ctr’ and ‘180_ctr’
     for the first gridcell centered at Greenwich or 180 degrees,
     respecitvely.  And ‘grn_wst’ and ‘180_wst’ for Greenwich or
     180 degress lying on the western edge of the first gridcell.  Many
     global models use the ‘grn_ctr’ longitude grid as their "scalar
     grid" (where, e.g., temperature, humidity, and other scalars are
     defined).  The "staggered" or "offset" grid (where often the
     dynamics variables are defined) then must have the ‘grn_wst’
     longitude convention.  That way the centers of the scalar grid are
     the vertices of the offset grid, and visa versa.

“Grid Resolution: LAT_NBR, LON_NBR”
     The number of gridcells in the horizontal spatial dimensions are
     LAT_NBR and LON_NBR, respectively.  There are no restrictions on
     LON_NBR for any gridtype.  Latitude grids do place some
     restrictions on LAT_NBR (see above).  As of NCO version 4.5.3,
     released in October, 2015, the ‘--rgr latlon=LAT_NBR,LON_NBR’
     switch may be used to simultaneously specify both latitude and
     longitude, e.g., ‘--rgr latlon=180,360’.

“Latitude Direction: LAT_DRC”
     The LAT_DRC option is specifies whether latitudes monotonically
     increase or decrease in rectangular grids.  The two possible values
     are ‘s2n’ for grids that begin with the most southerly latitude and
     end with the most northerly, and ‘n2s’ for grids that begin with
     the most northerly latitude and end with the most southerly.  By
     default NCO creates grids whose latitudes run south-to-north.
     Hence this option is only necessary to create a grid whose
     latitudes run north-to-south.

“Grid Edges: LON_WST, LON_EST, LAT_STH, LAT_NRT”
     The outer edges of a regional rectangular grid are specified by the
     North (LAT_NRT), South (LAT_STH), East (LAT_EST), and West
     (LAT_NRT) sides.  Latitudes and longigudes must be specified in
     degrees (not radians).  Latitude edges must be between -90 and 90.
     Longitude edges may be positive or negative and separated by no
     more than 360 degrees.  The edges may be specified individually
     with four arguments, consecutively separated by the multi-argument
     delimiter (‘#’ by default), or together in a short list to the
     pre-ordered options ‘wesn’ or ‘snwe’.  These three specifications
     are equivalent:
          ncks ... --rgr lat_sth=30.0 --rgr lat_nrt=70.0 --rgr lon_wst=-120.0 --rgr lon_est=-90.0 ...
          ncks ... --rgr lat_sth=30.0#lat_nrt=70.0#lon_wst=-120.0#lon_est=-90.0 ...
          ncks ... --rgr snwe=30.0,70.0,-120.0,-90.0 ...
   The first example above supplies the bounding box with four KEY=VAL
pairs.  The second example above supplies the bounding box with a single
option in multi-argument format (*note Multi-arguments::).  The third
example uses a convenience switch introduced to reduce typing.

   Generating common grids:
     # Access to grid-generation was through ncks, not ncremap, until version 4.7.6
     # 180x360 (1x1 degree) Equi-Angular grid, first longitude centered at Greenwich
     # This is NOT the CMIP6 1x1 grid
     ncks --rgr ttl='Equi-Angular grid 180x360'#latlon=180,360#lat_typ=uni#lon_typ=grn_ctr \
          --rgr scrip=${DATA}/grids/180x360_SCRIP.20150901.nc \
          ~zender/nco/data/in.nc ~/foo.nc
     
     # Version 4.7.6+ (August, 2018), supports the preferred, more concise, ncremap syntax:
     # This is NOT the CMIP6 1x1 grid
     ncremap -G ttl='Equi-Angular grid 180x360'#latlon=180,360#lat_typ=uni#lon_typ=grn_ctr \
             -g ${DATA}/grids/180x360_SCRIP.20180901.nc
     
     # 180x360 (1x1 degree) Equi-Angular grid, first longitude west edge at Greenwich
     # This IS the CMIP6 1x1 grid
     ncremap -G ttl='Equi-Angular grid 180x360'#latlon=180,360#lat_typ=uni#lon_typ=grn_wst \
             -g ${DATA}/grids/180x360wst_SCRIP.20180301.nc
     
     # 129x256 CAM-FV grid, first longitude centered at Greenwich
     ncremap -G ttl='CAM-FV scalar grid 129x256'#latlon=129,256#lat_typ=fv#lon_typ=grn_ctr \
             -g ${DATA}/grids/129x256_SCRIP.20150901.nc
     
     # 192x288 CAM-FV grid, first longitude centered at Greenwich
     ncremap -G ttl='CAM-FV scalar grid 192x288'#latlon=192,288#lat_typ=fv#lon_typ=grn_ctr \
             -g ${DATA}/grids/192x288_SCRIP.20160301.nc
     
     # 361x576 NASA MERRA2 FV grid, first longitude centered at DateLine
     ncremap -G ttl='NASA MERRA2 Cap grid 361x576'#latlon=361,576#lat_typ=cap#lon_typ=180_ctr \
             -g ${DATA}/grids/merra2_361x576.20201001.nc
     
     # 1441x2880 CAM-FV grid, first longitude centered at Greenwich
     ncremap -G ttl='CAM-FV scalar grid 1441x2880'#latlon=1441,2880#lat_typ=fv#lon_typ=grn_ctr \
             -g ${DATA}/grids/1441x2880_SCRIP.20170901.nc
     
     # 1440x2880 ELM/MOSART grid, first longitude west edge at DateLine
     ncremap -7 -L 1 \
             -G ttl='ELM/MOSART 1440x2880 one-eighth degree uniform grid (r0125)'#latlon=1440,2880#lat_typ=uni#lon_typ=180_wst \
             -g ${DATA}/grids/r0125_1440x2880.20210401.nc
     
     # 91x180 CAM-FV grid, first longitude centered at Greenwich (2 degree grid)
     ncremap -G ttl='CAM-FV scalar grid 91x180'#latlon=91,180#lat_typ=fv#lon_typ=grn_ctr \
             -g ${DATA}/grids/91x180_SCRIP.20170401.nc
     
     # 25x48 CAM-FV grid, first longitude centered at Greenwich (7.5 degree grid)
     ncremap -G ttl='CAM-FV scalar grid 25x48'#latlon=25,48#lat_typ=fv#lon_typ=grn_ctr \
             -g ${DATA}/grids/25x48_SCRIP.20170401.nc
     
     # 128x256 Equi-Angular grid, Greenwich west edge of first longitude
     # CAM-FV offset grid for 129x256 CAM-FV scalar grid above
     ncremap -G ttl='Equi-Angular grid 128x256'#latlon=128,256#lat_typ=uni#lon_typ=grn_wst \
             -g ${DATA}/grids/128x256_SCRIP.20150901.nc
     
     # T42 Gaussian grid, first longitude centered at Greenwich
     ncremap -G ttl='T42 Gaussian grid'#latlon=64,128#lat_typ=gss#lon_typ=grn_ctr \
             -g ${DATA}/grids/t42_SCRIP.20180901.nc
     
     # T62 Gaussian grid, first longitude centered at Greenwich, NCEP2 T62 Gaussian grid 
     ncremap -G ttl='NCEP2 T62 Gaussian grid'#latlon=94,192#lat_typ=gss#lon_typ=grn_ctr#lat_drc=n2s \
             -g ${DATA}/grids/ncep2_t62_SCRIP.20191001.nc
     
     # F256 Full Gaussian grid, first longitude centered at Greenwich
     ncremap -7 -L 1 \
             -G ttl='ECMWF IFS F256 Full Gaussian grid 512x1024'#latlon=512,1024#lat_typ=gss#lon_typ=grn_ctr#lat_drc=n2s \
             -g ${DATA}/grids/f256_scrip.20201001.nc
     
     # 513x1024 FV grid, first longitude centered at Greenwich
     ncremap -7 -L 1 \
             -G ttl='FV scalar grid 513x1024'#latlon=513,1024#lat_typ=fv#lon_typ=grn_ctr \
             -g ${DATA}/grids/513x1024_SCRIP.20201001.nc
     
     # 1025x2048 FV grid, first longitude centered at Greenwich
     ncremap -7 -L 1 \
             -G ttl='FV scalar grid 1025x2048'#latlon=1025,2048#lat_typ=fv#lon_typ=grn_ctr \
             -g ${DATA}/grids/1025x2048_SCRIP.20201001.nc
     
     # F640 Full Gaussian grid, first longitude centered at Greenwich
     ncremap -7 -L 1 \
          -G ttl='ECMWF IFS F640 Full Gaussian grid 1280x2560'#latlon=1280,2560#lat_typ=gss#lon_typ=grn_ctr#lat_drc=n2s \
          -g ${DATA}/grids/f640_scrip.20190601.nc
     
     # NASA Climate Modeling Grid (CMG) 3600x7200 (0.05x0.05 degree) Equi-Angular grid
     # Date-line west edge of first longitude, east edge of last longitude
     # Write to compressed netCDF4-classic file to reduce filesize ~140x from 2.2 GB to 16 MB
     ncremap -7 -L 1 \
          -G ttl='Equi-Angular grid 3600x7200 (NASA CMG)'#latlon=3600,7200#lat_typ=uni#lon_typ=180_wst \
          -g ${DATA}/grids/3600x7200_SCRIP.20160301.nc
     
     # DOE E3SM/ACME High Resolution Topography (1 x 1 km grid) for Elevation Classes
     # Write to compressed netCDF4-classic file to reduce filesize from ~85 GB to 607 MB
     ncremap -7 -L 1 \
          -G ttl='Global latxlon = 18000x36000 ~1 x 1 km'#latlon=18000,36000#lat_typ=uni#lon_typ=grn_ctr \
          -g ${DATA}/grids/grd_18000x36000_SCRIP.nc
     
     # 1x1 degree Equi-Angular Regional grid over Greenland, centered longitudes
     ncremap -G ttl='Equi-Angular Greenland 1x1 degree grid'#latlon=30,90#snwe=55.0,85.0,-90.0,0.0#lat_typ=uni#lon_typ=grn_ctr \
             -g ${HOME}/greenland_1x1.nc
     
     # 721x1440 ECMWF ERA5 resolution in north-to-south order (ERA5/CAMS default order)
     ncremap -7 --dfl_lvl=1 -G ttl='Cap/FV ECMWF ERA5 grid 0.25x0.25 degree, dimensions 721x1440, cell centers on Poles/Equator (in north-to-south order) and Prime Meridian/Date Line'#latlon=721,1440#lat_drc=n2s#lat_typ=cap#lon_typ=grn_ctr \
              -g ${DATA}/grids/era5_n2s_721x1440.nc
     
     # 721x1440 ECMWF ERA5 resolution in south-to-north order (E3SM/ELM-offline forcing order)
     ncremap -7 --dfl_lvl=1 -G ttl='Cap/FV ECMWF ERA5 grid 0.25x0.25 degree, dimensions 721x1440, cell centers on Poles/Equator (in south-to-north order) and Prime Meridian/Date Line'#latlon=721,1440#lat_drc=s2n#lat_typ=cap#lon_typ=grn_ctr \
             -g ${DATA}/grids/era5_s2n_721x1440.nc
     
     # 360x720 CRUNCEP (E3SM/ELM-offline forcing grid) (NB: CRUNCEP starts at Greenwich, is not r05)
     ncremap -7 --dfl_lvl=1 -G ttl='CRUNCEP Equi-Angular 0.5x0.5 degree uniform grid, dimensions 360x720, cell edges on Poles/Equator and Prime Meridian/Date Line'#latlon=360,720#lat_typ=uni#lon_typ=Grn_wst \
             -g ${DATA}/grids/cruncep_360x720.nc
     
     # 360x720 ELM/MOSART grid, first longitude west edge at DateLine (NB: starts at Dateline, "r" stands for "river" grid)
     ncremap -7 --dfl_lvl=1 -G ttl='Equi-Angular 0.5x0.5 degree uniform grid (r05), dimensions 360x720, cell edges on Poles/Equator and Date Line/Prime Meridian'#latlon=360,720#lat_typ=uni#lon_typ=180_wst \
             -g ${DATA}/grids/r05_360x720.nc
     
     # 720x1440 ELM/MOSART grid, first longitude west edge at DateLine (NB: starts at Dateline, "r" stands for "river" grid)
     ncremap -7 --dfl_lvl=1 -G ttl='Equi-Angular 0.25x0.25 degree uniform grid (r025), dimensions 720x1440, cell edges on Poles/Equator and Date Line/Prime Meridian'#latlon=720,1440#lat_typ=uni#lon_typ=180_wst \
             -g ${DATA}/grids/r025_720x1440.nc
     
     # 105x401 Greenland ERA5
     ncremap -G ttl='Equi-Angular Greenland 0.25x0.25 degree ERA5 north-to-south grid'#latlon=105,401#snwe=58.875,85.125,-87.125,13.125#lat_typ=uni#lat_drc=n2s#lon_typ=grn_ctr \
             -g ${DATA}/grids/greenland_0.25x0.25_era5.nc
     
     # Greenland r025 with SNWE = 59,84,-73,-11 (in round numbers) with RACMO ice mask
     ncremap -G ttl='Equi-Angular Greenland 0.25x0.25 degree r025 south-to-north grid'#latlon=100,250#snwe=58.875,83.875,-73.25,-10.75#lat_typ=uni#lat_drc=s2n#lon_typ=grn_ctr \
             -g ${DATA}/grids/greenland_r025_100x250.nc
     
     # NASA Climate Modeling Grid (CMG) 3600x7200 (0.05x0.05 degree, 3'x3') Equi-Angular grid
     # With land mask derived mainly from GLOBE 30" topography and anywhere Gardner 30" land ice data is valid
     # Date-line west edge of first longitude, east edge of last longitude
     # Write to compressed netCDF4-classic file to reduce filesize ~140x from 2.2 GB to 16 MB
     ncremap -7 -L 1 \
          -G ttl='Equi-Angular grid 3-minute=0.05 degree resolution = 3600x7200, NASA CMG boundaries, with land mask derived mainly from GLOBE 30" topography and anywhere Gardner 30" land ice data is valid'#latlon=3600,7200#lat_typ=uni#lon_typ=180_wst \
          -g ${DATA}/grids/r005_3600x7200_globe_gardner_landmask.20210501.nc

   Often researchers face the problem not of generating a known,
idealized grid but of understanding an unknown, possibly irregular or
curvilinear grid underlying a dataset produced elsewhere.  NCO will
“infer” the grid of a datafile by examining its coordinates (and
boundaries, if available), reformat that information as necessary to
diagnose gridcell areas, and output the results in SCRIP format.  As of
NCO version 4.5.3, released in October, 2015, the ‘--rgr infer’ flag
activates the machinery to infer the grid rather than construct the grid
from other user-specified switches.  To infer the grid properties, NCO
interrogates INPUT-FILE for horizontal coordinate information, such as
the presence of dimension names rooted in latitude/longitude-naming
traditions and conventions.  Once NCO identifies the likely horizontal
dimensions it looks for horizontal coordinates and bounds.  If bounds
are not found, NCO assumes the underlying grid comprises quadrilateral
cells whose edges are midway between cell centers, for both rectilinear
and curvilinear grids.
     # Infer AIRS swath grid from input, write it to grd_scrip.nc
     ncks --rgr infer --rgr scrip=${DATA}/sld/rgr/grd_scrip.nc \
          ${DATA}/sld/raw/AIRS.2014.10.01.202.L2.TSurfStd.Regrid010.1DLatLon.nc ~/foo.nc
   When inferring grids, the grid file (‘grd_scrip.nc’) is written in
SCRIP format, the input file (‘AIRS...nc’) is read, and the output file
(‘foo.nc’) is overwritten (its contents are immaterial).

   As of NCO version 4.6.6, released in April, 2017, inferred 2D
rectangular grids may also be written in UGRID-format (defined here
(http://ugrid-conventions.github.io/ugrid-conventions)).  Request a
UGRID mesh with the option ‘--rgr ugrid=FL_UGRID’.  Currently both UGRID
and SCRIP grids must be requested in order to produce the UGRID output,
e.g.,
     ncks --rgr infer --rgr ugrid=${HOME}/grd_ugrid.nc \
          --rgr scrip=${HOME}/grd_scrip.nc ~/skl_180x360.nc ~/foo.nc

   The SCRIP gridfile and UGRID meshfile metadata produced for the
equiangular 1-by-1 degree global grid are:
     zender@aerosol:~$ ncks -m ~/grd_scrip.nc 
     netcdf grd_scrip {
       dimensions:
         grid_corners = 4 ;
         grid_rank = 2 ;
         grid_size = 64800 ;
     
       variables:
         double grid_area(grid_size) ;
           grid_area:units = "steradian" ;
     
         double grid_center_lat(grid_size) ;
           grid_center_lat:units = "degrees" ;
     
         double grid_center_lon(grid_size) ;
           grid_center_lon:units = "degrees" ;
     
         double grid_corner_lat(grid_size,grid_corners) ;
           grid_corner_lat:units = "degrees" ;
     
         double grid_corner_lon(grid_size,grid_corners) ;
           grid_corner_lon:units = "degrees" ;
     
         int grid_dims(grid_rank) ;
     
         int grid_imask(grid_size) ;
     } // group /
     
     zender@aerosol:~$ ncks -m ~/grd_ugrid.nc 
     netcdf grd_ugrid {
       dimensions:
         maxNodesPerFace = 4 ;
         nEdges = 129240 ;
         nFaces = 64800 ;
         nNodes = 64442 ;
         two = 2 ;
     
       variables:
         int mesh ;
           mesh:cf_role = "mesh_topology" ;
           mesh:standard_name = "mesh_topology" ;
           mesh:long_name = "Topology data" ;
           mesh:topology_dimension = 2 ;
           mesh:node_coordinates = "mesh_node_x mesh_node_y" ;
           mesh:face_node_connectivity = "mesh_face_nodes" ;
           mesh:face_coordinates = "mesh_face_x mesh_face_y" ;
           mesh:face_dimension = "nFaces" ;
           mesh:edge_node_connectivity = "mesh_edge_nodes" ;
           mesh:edge_coordinates = "mesh_edge_x mesh_edge_y" ;
           mesh:edge_dimension = "nEdges" ;
     
         int mesh_edge_nodes(nEdges,two) ;
           mesh_edge_nodes:cf_role = "edge_node_connectivity" ;
           mesh_edge_nodes:long_name = "Maps every edge to the two nodes that it connects" ;
           mesh_edge_nodes:start_index = 0 ;
     
         double mesh_edge_x(nEdges) ;
           mesh_edge_x:standard_name = "longitude" ;
           mesh_edge_x:long_name = "Characteristic longitude of 2D mesh face" ;
           mesh_edge_x:units = "degrees_east" ;
     
         double mesh_edge_y(nEdges) ;
           mesh_edge_y:standard_name = "latitude" ;
           mesh_edge_y:long_name = "Characteristic latitude of 2D mesh face" ;
           mesh_edge_y:units = "degrees_north" ;
     
         int mesh_face_nodes(nFaces,maxNodesPerFace) ;
           mesh_face_nodes:cf_role = "face_node_connectivity" ;
           mesh_face_nodes:long_name = "Maps every face to its corner nodes" ;
           mesh_face_nodes:start_index = 0 ;
           mesh_face_nodes:_FillValue = -2147483648 ;
     
         double mesh_face_x(nFaces) ;
           mesh_face_x:standard_name = "longitude" ;
           mesh_face_x:long_name = "Characteristic longitude of 2D mesh edge" ;
           mesh_face_x:units = "degrees_east" ;
     
         double mesh_face_y(nFaces) ;
           mesh_face_y:standard_name = "latitude" ;
           mesh_face_y:long_name = "Characteristic latitude of 2D mesh edge" ;
           mesh_face_y:units = "degrees_north" ;
     
         double mesh_node_x(nNodes) ;
           mesh_node_x:standard_name = "longitude" ;
           mesh_node_x:long_name = "Longitude of mesh nodes" ;
           mesh_node_x:units = "degrees_east" ;
     
         double mesh_node_y(nNodes) ;
           mesh_node_y:standard_name = "latitude" ;
           mesh_node_y:long_name = "Latitude of mesh nodes" ;
           mesh_node_y:units = "degrees_north" ;
     } // group /

   Another task that arises in regridding is characterizing new grids.
In such cases it can be helpful to have a "skeleton" version of a
dataset on the grid, so that grid center and interfaces locations can be
assessed, continental outlines can be examined, or the skeleton can be
manually populated with data rather than relying on a model.  SCRIP
files can be difficult to visualize and manipulate, so NCO will provide,
if requested, a so-called skeleton file on the user-specified grid.  As
of NCO version 4.5.3, released in October, 2015, the ‘--rgr skl=FL_SKL’
switch outputs the skeleton file to FL_SKL.  The skeleton file may then
be examined in a dataset viewer, populated with data, and generally
serve as a template for what to expect from datasets of the same
geometry.
     # Generate T42 Gaussian grid file t42_SCRIP.nc and skeleton file t42_skl.nc
     ncks --rgr skl=${DATA}/grids/t42_skl.nc --rgr scrip=${DATA}/grids/t42_SCRIP.nc \
          --rgr latlon=64,128#lat_typ=gss#lon_typ=Grn_ctr \
          ~zender/nco/data/in.nc ~/foo.nc
   When generating skeleton files, both the grid file (‘t42_SCRIP.nc’)
and the skeleton file (‘t42_skl.nc’) are written, the input file
(‘in.nc’) is ignored, and the output file (‘foo.nc’) is overwritten (its
contents are immaterial).

   ---------- Footnotes ----------

   (1) The term FV confusing because it is correct to call any Finite
Volume grid (including arbitrary polygons) an FV grid.  However, an FV
grid has also been used for many years to described the particular type
of rectangular grid with caps at the poles used to discretize global
model grids for use with the Lin-Rood dynamical core.  To reduce
confusion, we use "Cap grid" to refer to the latter and reserv FV as a
straightforward acronym for Finite Volume.

   (2) A Uniform grid in latitude could be called "equi-angular" in
latitude, but NCO reserves the term Equi-angular or "eqa" for grids that
have the same uniform spacing in both latitude and longitude, e.g.,
1°x1° or 2°x2°.  NCO reserves the term Regular to refer to grids that
are monotonic and rectangular grids.  Confusingly, the angular spacing
in a Regular grid need not be uniform, it could be irregular, such as in
a Gaussian grid.  The term Regular is not too useful in grid-generation,
because so many other parameters (spacing, centering) are necessary to
disambiguate it.


File: nco.info,  Node: Regridding,  Next: Climatology and Bounds Support,  Prev: Grid Generation,  Up: Shared features

3.26 Regridding
===============

Availability: ‘ncclimo’, ‘ncks’, ‘ncremap’
Short options: None
Long options: ‘--map MAP-FILE’ or ‘--rgr_map MAP-FILE’
‘--rgr KEY=VAL’ (multiple invocations allowed)
‘--rnr=RNR_THR’ or ‘--rgr_rnr=RNR_THR’ or ‘--renormalize=RNR_THR’ or
‘--renormalization_threshold=RNR_THR’

   NCO includes extensive regridding features in ‘ncclimo’ (as of
version 4.6.0 in May, 2016), ‘ncremap’ (as of version 4.5.4 in November,
2015) and ‘ncks’ (since version 4.5.0 in June, 2015).  Regridding can
involve many choices, options, inputs, and outputs.  The appropriate
operator for this workflow is the ‘ncremap’ script which automatically
handles many details of regridding and passes the required commands to
‘ncks’ and external programs.  Occasionally users need access to
lower-level remapping functionality present in ‘ncks’ and not exposed to
direct manipulation through ‘ncremap’ or ‘ncclimo’.  This section
describes the lower-level functionality and switches as implemented in
‘ncks’.  Knowing what these features are will help ‘ncremap’ and
‘ncclimo’ users understand the full potential of these operators.

   ‘ncks’ supports horizontal regridding of datasets where the grids and
weights are all stored in an external MAP-FILE.  Use the ‘--map’ or
‘--rgr_map’ options to specify the MAP-FILE, and NCO will regrid the
INPUT-FILE to a new (or possibly the same, aka, an identity mapping)
horizontal grid in the OUTPUT-FILE, using the input and output grids and
mapping weights specified in the ESMF- or SCRIP-format MAP-FILE.
Currently NCO understands the mapfile formats pioneered by SCRIP
(<http://oceans11.lanl.gov/svn/SCRIP/trunk/SCRIP>) and later extended by
ESMF (<http://www.earthsystemcog.org/projects/regridweightgen>), and
adopted (along with Exodus) by TempestRemap
(<https://github.com/ClimateGlobalChange/tempestremap.git>).  Those
references document quirks in their respectively weight-generation
algorithms as to map formats, grid specification, and weight generation.
NCO itself produces map-files in the format recommended by CMIP6 and
described here
(https://docs.google.com/document/d/1BfVVsKAk9MAsOYstwFSWI2ZBt5mrO_Nmcu7rLGDuL08).
This format differs from ESMF map-file format chiefly in that its
metadata are slightly more evolved, self-descriptive, and standardized.

   Originally NCO supported only weight-application, which is what most
people mean by "regridding".  As of version 4.9.0, released in December,
2019, NCO also supports weight-generation by its own conservative
algorithm.  Thus NCO can now apply weights generated by ESMF, NCO,
SCRIP, and TempestRemap.  NCO reads-in pre-stored weights from the
MAP-FILE and applies them to (almost) every variable, thereby creating a
regridded OUTPUT-FILE.  Specify regridding with a standard ‘ncks’
command and options along with the additional specification of a
MAP-FILE:
     # Regrid entire file, same output format as input:
     ncks --map=map.nc in.nc out.nc
     # Entire file, netCDF4 output:
     ncks -4 --map=map.nc in.nc out.nc
     # Deflated netCDF4 output
     ncks -4 -L 1 --map=map.nc in.nc out.nc
     # Selected variables
     ncks -v FS.?,T --map=map.nc in.nc out.nc
     # Threading
     ncks -t 8 --map=map.nc in.nc out.nc
     # Deflated netCDF4 output, threading, selected variables:
     ncks -4 -L 1 -t 8 -v FS.?,T --map=map.nc in.nc out.nc
   OpenMP threading works well with regridding large datasets.
Threading improves throughput of regridding 1-10 GB files by factors of
2-5.  Options specific to regridding are described below.

   NCO supports 1D⇒1D, 1D⇒2D, 2D⇒1D, and 2D⇒2D regridding for any
unstructured 1D-grid and any rectangular 2D-grid.  This has been tested
by converting among and between Gaussian, equiangular, FV, unstructured
cubed-sphere grids, and regionally refined grids.  Support for irregular
2D- and regional grids (e.g., swath-like data) is planned.

Renormalization
---------------

Conservative regridding is, for first-order accurate algorithms, a
straightforward procedure of identifying gridcell overlap and
apportioning values correctly from source to destination.  The presence
of missing values forces a decision on how to handle destination
gridcells where some but not all source cells are valid.  NCO allows the
user to choose between two distinct weight-application algorithms:
"conservative" and "renormalized".  The "conservative" algorithm uses
all valid data from the input grid on the output grid once and only
once.  Destination cells receive the weighted valid values of the source
cells.  This is conservative because the global integrals of the source
and destination fields are equal.  Another name for the "conservative"
weight-application method is therefore "integral-preserving".  The
"renormalized" algorithm divides the destination value by the sum of the
valid weights.  This produces values equal to the mean of the valid
input values, but extended to the entire destination gridcell.  Thus
renormalization is equivalent to extrapolating valid data to missing
regions.  Another name for the "renormalized" weight-application method
is therefore "mean-preserving".  Input and output integrals are unequal
and renormalized regridding is not conservative.  Both algorithms
produce identical answers when no missing data maps to the destination
gridcell.

   The renormalized algorithm is useful because it solves some problems,
like producing physically unrealistic temperature values, at the expense
of incurring others, like non-conservation.  Many land and ocean
modelers eschew unrealistic gridpoint values, and conservative
weight-application often produces "weird" values along coastlines or
missing data gaps where state variables are regridded to/from small
fractions of a gridcell.  Renormalization ensures the output values are
physically consistent, although the integral of their value times area
is not preserved.

   By default, NCO implements the "conservative" algorithm because it
has useful properties, is simpler to understand, and requires no
additional parameters.  To employ the "renormalized" algorithm instead,
use the ‘--rnr’, ‘--rgr_rnr’, ‘--rnr_thr’, or ‘--renormalize’ options to
supply RNR_THR, the threshold weight for valid destination values.
Valid values must cover at least the fraction RNR_THR of the destination
gridcell to meet the threshold for a non-missing destination value.
When RNR_THR is exceeded, the mean valid value is renormalized by the
valid area and placed in the destination gridcell.  If the valid area
covers less than RNR_THR, then the destination gridcell is assigned the
missing value.  Valid values of RNR_THR range from zero to one.  Keep in
mind though, that this threshold is potentially a divisor, and values of
zero or very near to zero can lead to floating-point underflow and
divide-by-zero errors.  For convenience NCO permits users to specify a
RNR_THR = 0.0 threshold weight.  This indicates that any valid data
should be represented and renormalized on the output grid.  Also,
renormalization can be explicitly prevented or turned-off by setting
RNR_THR to either of the values ‘off’ or ‘none’:
     ncks           --map=map.nc in.nc out.nc # Conservative (global integral-preserving)
     ncks --rnr=off --map=map.nc in.nc out.nc # Conservative (global integral-preserving)
     ncks --rnr=0.1 --map=map.nc in.nc out.nc # Renormalized (local mean-preserving with threshold)
     ncks --rnr=0.0 --map=map.nc in.nc out.nc # Renormalized (local mean-preserving)
   The first and second examples use the default conservative algorithm.
The third example specifies that valid values must cover at least 10% of
the destination gridcell to meet the threshold for a non-missing
destination value.  With valid destination areas of, say 25% or 50%, the
renormalized algorithm would produce destination values greater than the
conservative algorithm by factors of four or two, respectively.

   In practice, it may make sense to use the default "conservative"
algorithm when performing conservative regridding, and the
"renormalized" algorithm when performing other regridding such as
bilinear interpolation or nearest-neighbor.  Another consideration is
whether the fields being regridded are fluxes or state variables.  For
example, temperature (unlike heat) and concentrations (amount per unit
volume) are not physically conserved quantities under areal-regridding
so it often makes sense to interpolate them in a non-conservative
fashion, to preserve their fine-scale structure.  Few researchers can
digest the unphysical values of temperature that the "conservative"
option will produce in regions rife with missing values.  A
counter-example is fluxes, which should be physically conserved under
areal-regridding.  One should consider both the type of field and its
conservation properties when choosing a regridding strategy.

Regridder Options Table
-----------------------

NCO automatically annotates the output with relevant metadata such as
coordinate bounds, axes, and vertices (à la CF).  These annotations
include
“Horizontal Dimension Names: LAT_DMN, LON_DMN”
     The name of the horizontal spatial dimensions assumed to represent
     latitude and longitude in 2D rectangular input files are LAT_DMN_NM
     and LON_DMN_NM, which default to ‘lat’ and ‘lon’, respectively.
     Variables that contain a LAT_DMN_NM-dimension and a
     LON_DMN_NM-dimension on a 2D-rectangular input grid will be
     regridded, and variables regridded to a 2D-rectangular output grid
     will all contain the LAT_DMN_NM- and LON_DMN_NM-dimensions.  To
     treat different dimensions as latitude and longitude, use the
     options ‘--rgr lat_dmn_nm=LAT_DMN_NM’ and ‘--rgr
     lon_dmn_nm=LON_DMN_NM’.  These options applied only to inferring
     and generating grids until NCO version 4.7.9 (February, 2019).
     Since then, these options also determine the dimension names in
     regridded output files.
“Horizontal Coordinate Names: LAT, LON”
     The name of the horizontal spatial coordinates that represent
     latitude and longitude in input files are LAT_NM and LON_NM, and
     default to ‘lat’ and ‘lon’, respectively.  Variables that contain a
     LAT_DMN_NM-dimension and a LON_DMN_NM-dimension on a 2D input grid
     will be regridded, and output regridded variables will all contain
     the LAT_NM- and LON_NM-variables.  Unless the LAT_DMN_NM- and
     LON_DMN_NM-dimensions are explicitly configured otherwise, they
     will share the same name as the LAT_NM- and LON_NM-variables.  Thus
     variables regridded to a 2D-rectangular output grid usually have
     LAT_NM- and LON_NM as coordinate variables.  Variables regridded to
     a 1D-unstructured output grid will have LAT_NM and LON_NM as
     auxiliary coordinate variables.  Variables regridded to a
     2D-curvilinear output grid will have LAT_NM and LON_NM as
     multi-dimensional auxiliary coordinate variables.  To treat
     different variables as latitude and longitude, use the options
     ‘--rgr lat_nm=LAT_NM’ and ‘--rgr lon_nm=LON_NM’.  Before NCO
     version 4.7.9 (February, 2019), LAT_NM and LON_NM specified both
     the variable names _and_, where applicable (i.e., on 2D-grids), the
     dimensions of the horizontal coordinates in output files.  Now the
     horizontal variable and dimension names in output files may be
     separately specified.
“Unstructured Dimension Name: COL”
     The name of the horizontal spatial dimension assumed to delineate
     an unstructured grid is COL_NM, which defaults to ‘ncol’ (number of
     columns), the name CAM employs.  Other common names for the columns
     in an unstructured grid include ‘lndgrid’ (used by CLM), and
     ‘nCells’ (used by MPAS-O).  Variables that contain the
     COL_NM-dimension on an unstructured input grid will be regridded,
     and regridded variables written to an unstructured output grid will
     all contain the COL_NM-dimension.  To treat a different dimension
     as unstructured, use the option ‘--rgr col_nm=COL_NM’.  Note: Often
     there is no coordinate variable for the COL_NM-dimension, i.e.,
     there is no variable named COL_NM, although such a coordinate could
     contain useful information about the unstructured grid.
“Structured Grid Standard Names and Units”
     Longitude and latitude coordinates (both regular and auxiliary,
     i.e., for unstructured grids) receive CF ‘standard_name’ values of
     ‘latitude’ and ‘longitude’, CF ‘axes’ attributes with values ‘X’
     and ‘Y’, and ‘units’ attributes with values ‘degrees_east’ and
     ‘degrees_north’, respectively.
“Unstructured Grid Auxiliary Coordinates”
     Unstructured grid auxiliary coordinates for longitude and latitude
     receive CF ‘coordinates’ attributes with values ‘lon’ and ‘lat’,
     respectively.
“Structured Grid Bounds Variables: BND, LAT_BND, LON_BND”
     Structured grids with 1D-coordinates use the dimension BND_NM
     (which defaults to ‘nbnd’) with the spatial bounds variables in
     LAT_BND_NM and LON_BND_NM which default to ‘lon_bnds’ and
     ‘lat_bnds’, respectively.  By default spatial bounds for such
     structured grids parallel the oft-used temporal bounds dimension
     (‘nbnd=2’) and variable (‘time_bnds’).  Bounds are attached to the
     horizontal spatial dimensions via their ‘bounds’ attributes.
     Change the spatial bounds dimension with the option ‘--rgr
     bnd_nm=BND_NM’.  Rename the spatial bounds variables with the
     options ‘--rgr lat_bnd_nm=LAT_BND_NM’ and ‘--rgr
     lon_bnd_nm=LON_BND_NM’.
“Unstructured Grid Bounds Variables: BND, LAT_BND, LON_BND”
     Unstructured grids with 1D-coordinates use the dimension BND_NM
     (which defaults to ‘nv’, number of vertices) for the spatial bounds
     variables LAT_BND_NM and LON_BND_NM which default to ‘lat_vertices’
     and ‘lon_vertices’, respectively.  It may be impossible to re-use
     the temporal bounds dimension (often ‘nbnd’) for unstructure grids,
     because the gridcells are not rectangles, and thus require
     specification of all vertices for each gridpoint, rather than only
     two parallel interfaces per dimension.  These bounds are attached
     to the horizontal spatial dimensions via their ‘bounds’ attributes.
     Change the spatial bounds dimension with the option ‘--rgr
     bnd_nm=BND_NM’.  Rename the spatial bounds variables with the
     options ‘--rgr lat_bnd_nm=LAT_BND_NM’ and ‘--rgr
     lon_bnd_nm=LON_BND_NM’.  The temporal bounds dimension in
     unstructured grid output remains as in the INPUT-FILE, usually
     ‘nbnd’.
“Vertical Dimension Names: LEV_DMN, ILEV_DMN”
     The name of the dimension(s) associated with the vertical
     coordinate(s) in multi-level input files are LEV_DMN_NM and
     ILEV_DMN_NM, which default to ‘lev’ and ‘ilev’, respectively.
     Variables that contain a LEV_DMN_NM-dimension or an
     ILEV_DMN_NM-dimension will be vertically interpolated to the
     specified (with ‘vrt_out=VRT_FL’) vertical output grid, and will
     all contain the LEV_DMN_NM- and, for hybrid sigma-pressure
     interface variables, ILEV_DMN_NM-dimensions.  To treat different
     dimensions as the midlayer and interface level dimensions, use the
     options ‘--rgr lev_dmn_nm=LEV_DMN_NM’ and ‘--rgr
     ilev_dmn_nm=ILEV_DMN_NM’ options.  Pure-pressure grids should use
     the ‘--rgr lev_dmn_nm=LEV_DMN_NM’ option (to reduce option
     proliferation, there is no PLEV_DMN_NM option).  These options were
     introduced in NCO version 4.9.0 (December, 2019).  These options
     also determine the vertical dimension names in vertically
     interpolated output files.
“Vertical Coordinate Names: LEV, ILEV, PLEV”
     The name of the vertical coordinate variables that represent
     midpoint levels and interface levels in hybrid sigma-pressure input
     files are LEV_NM and ILEV_NM, and default to ‘lev’ and ‘ilev’,
     respectively.  While the vertical coordinate in pure-pressure
     vertical grid files (i.e., the template-file to which data will be
     interpolated) must be named ‘plev’, the vertical coordinate in
     pure-pressure _data_ files (i.e., the files to be interpolated) may
     be changed with the ‘--rgr plev_nm=PLEV_NM’ option.  The name of
     the vertical coordinate variable that represents pressure levels in
     pure-pressure grid input data files is PLEV_NM, and it defaults to
     ‘plev’.  To reduce proliferation of command-line options and
     internal code complexity, the variable and dimension options for
     pure-pressure vertical coordinate output names re-use the "lev"
     options, i.e., ‘--rgr lev_nm_out=LEV_NM_OUT’ option.  Variables
     that contain a LEV_DMN_NM-dimension or a ILEV_DMN_NM-dimension on
     hybrid sigma-pressure input grid, or a PLEV_DMN_NM-dimension on a
     pure pressure grid, will be regridded, and output in vertically
     interpolated files on a hybrid sigma-pressure grid will all contain
     the LEV_NM- and ILEV_NM-variables, and output on a pure-pressure
     grid will contain the LEV_NM coordinate.  Unless the LEV_DMN_NM and
     ILEV_DMN_NM dimensions are explicitly configured otherwise, they
     will share the same name as the LEV_NM/PLEV_NM and
     ILEV_NM-variables, respectively.  Thus variables regridded to a
     hybrid sigma-pressure output grid usually have LEV_NM- and ILEV_NM
     as coordinate variables.  Variables regridded to a pure-pressure
     output grid will only have a single vertical coordinate variable,
     LEV_NM, which will be an associated coordinate variable if
     LEV_DMN_NM differs from LEV_NM.  To treat different variables as
     level and interface-level coordinates, use the options ‘--rgr
     lev_nm=LEV_NM’ and ‘--rgr ilev_nm=ILEV_NM’.  Before NCO version
     4.9.0 (December, 2019), LEV_NM and ILEV_NM specified both the
     variable names _and_, where applicable (i.e., on 2D-grids), the
     dimensions of the vertical coordinates in output files.  Now the
     vertical variable and dimension names in output files may be
     separately specified.
“Surface Pressure Names: PS, PS”
     The name of the surface pressure field necessary to reconstruct the
     layer pressures in the hybrid sigma-pressure coordinate system is
     PS_NM which defaults to ‘PS’.  As of NCO version 5.1.2, released in
     November, 2022, one may change this with the ‘--rgr ps_nm=PS_NM’
     option.  There are, in fact, three similar options, one each for
     the surface pressure variable in the input data file, the vertical
     grid file, and in the output (interpolated file).  The full option
     key names are ‘ps_nm’ (equivalent to ‘ps_nm_in’), ‘ps_nm_tpl’, and
     ‘ps_nm_out’, respectively.
“Gridcell Area: AREA”
     The variable AREA_NM (which defaults to ‘area’) is, by default,
     (re-)created in the OUTPUT_FILE to hold the gridcell area in
     steradians.  To store the area in a different variable, use the
     option ‘--rgr area=AREA_NM’.  The AREA_NM variable receives a
     ‘standard_name’ attribute of ‘cell_area’, a ‘units’ attribute of
     ‘steradian’ (the SI unit of solid angle), and a ‘cell_methods’
     attribute with value ‘lat, lon: sum’, which indicates that AREA_NM
     is “extensive”, meaning that its value depends on the gridcell
     boundaries.  Since AREA_NM is a property of the grid, it is read
     directly from the MAP-FILE rather than regridded itself.  To omit
     the area variable from the output file, set the NO_AREA_OUT flag.
     The ‘--no_cll_msr’ switch to ‘ncremap’ and ‘ncclimo’ does this
     automatically.
“Gridcell Fraction: FRC”
     The variable FRC_NM (which defaults to ‘frac_b’) is automatically
     copied to the OUTPUT_FILE to hold the valid fraction of each
     gridcell when certain conditions are met.  First, the regridding
     method must be conservative.  Second, at least one value of FRC_NM
     must be non-unity.  These conditions ensure that whenever
     fractional gridcells affect the regridding, they are also placed in
     the output file.  To store the fraction in a different variable,
     use the option ‘--rgr frc_nm=FRC_NM’.  The FRC_NM variable receives
     a ‘cell_methods’ attribute with value ‘lat, lon: sum’, which
     indicates that FRC_NM is “extensive”, meaning that its value
     depends on the gridcell boundaries.  Since FRC_NM is a property of
     the grid, it is read directly from the MAP-FILE rather than
     regridded itself.
“Gridcell Mask: MASK”
     The variable MSK_NM (which defaults to ‘mask_b’) can, if present,
     be copied from the MAP-FILE to hold the gridcell mask on the
     destination grid in OUTPUT-FILE.  To name the mask differently in
     the output file, use the option ‘--rgr msk_nm=MSK_NM’.  Since
     MSK_NM is a property of the grid, it is read directly from the
     MAP-FILE rather than regridded itself.  To include the mask
     variable in the output file, set the MSK_OUT flag.  To omit the
     mask variable from the output file, set the NO_MSK_OUT flag.  In
     grid inferral and map-generation modes, this option tells the
     regridder to generate an integer mask map from the variable MSK_NM.
     The mask will be one (i.e., points at that location will contribute
     to regridding weights) where MSK_NM has valid values.  The mask
     will be zero (i.e., points at that location will not contribute to
     regridding weights) where MSK_NM has a missing value.  This feature
     is useful when creating weights between masked grids, e.g.,
     ocean-only points or land-only points.
“Latitude weights: LAT_WGT”
     Rectangular 2D-grids use the variable LAT_WGT_NM, which defaults to
     ‘gw’ (originally for "Gaussian weight"), to store the 1D-weight
     appropriate for area-weighting the latitude grid.  To store the
     latitude weight in a different variable, use the option ‘--rgr
     lat_wgt=LAT_WGT_NM’.  The LAT_WGT_NM variable will not appear in
     1D-grid output.  Weighting statistics by latitude (i.e., by
     LAT_WGT_NM will produce the same answers (up-to round-off error) as
     weighting by area (i.e., by AREA_NM) in grids that have both
     variables.  The former requires less memory because LAT_WGT_NM is
     1D), whereas the latter is more general because AREA_NM works on
     _any_ grid.
“Provenance Attributes”
     The MAP-FILE and INPUT-FILE names are stored in the OUTPUT-FILE
     global attributes ‘mapping_file’ and ‘source_file’, respectively.
“Staggered Grid Coordinates and Weights”
     Owing to its heritage as an early CCM analysis tool, NCO tries to
     create output interoperable with other CESM analysis tools.  Like
     many models, CAM computes and archives thermodynamic state
     variables on gridcell centers, and computes dynamics variables
     (zonal and meridional winds U and V, respectively) on gridcell
     edges (interfaces).  The dual-grid, sometimes called the "staggered
     grid", formed by connecting edge centers is thus the natural
     location for storing output dynamics variables.  Most dynamical
     cores of CAM archives horizontal winds at gridcell centers under
     the names ‘U’, and ‘V’.  For CAM-FV, these are interpolated from
     the computed interface winds archived as ‘US’, and ‘VS’ (which are
     on the staggered grid coordinate system).  Some analysis packages,
     such as the AMWG diagnostics, require access to these dual-grid
     coordinates with the names ‘slat’ and ‘slon’ (for "staggered"
     latitude and longitude).  Until NCO version 4.9.8 (released March,
     2021), the NCO regridder output these coordinates, along with the
     latitude weights (called ‘w_stag’), by default when the input was
     on a cap (aka FV) grid so that the result could be processed by
     AMWG diagnostics.  Setting the NO_STAGGER flag turns-off archiving
     the staggered grid (i.e., ‘slat’, ‘slon’, and ‘w_stag’).  Do this
     with the ‘--no_stg_grd’ flag in ‘ncremap’.  ‘ncclimo’ always sets
     this ‘--no_stagger’ flag.  As of NCO version 4.9.8 (released March,
     2021), the default ‘ncremap’ and ‘ncclimo’ behavior is to omit the
     staggered grid.  The new flag ‘--stg_grd’ turns-on outputting the
     staggered grid, and thus recovers the previous default behavior.

   One may supply muliple ‘--rgr KEY=VALUE’ options to simultaneously
customize multiple grid-field names.  The following examples may all be
assumed to end with the standard options ‘--map=map.nc in.nc out.nc’.
     ncks --rgr lat_nm=latitude --rgr lon_nm=longitude
     ncks --rgr col_nm=column --rgr lat_wgt=lat_wgt
     ncks --rgr bnd_nm=bounds --rgr lat_bnd_nm=lat_bounds --rgr lon_bnd_nm=lon_bounds
     ncks --rgr bnd_nm=vertices --rgr lat_bnd_nm=lat_vrt --rgr lon_bnd_nm=lon_vrt
   The first command causes the regridder to associate the latitude and
longitude dimensions with the dimension names ‘latitude’ and ‘longitude’
(instead of the defaults, ‘lat’ and ‘lon’).  The second command causes
the regridder to associate the independent columns in an unstructured
grid with the dimension name ‘column’ (instead of the default, ‘ncol’)
and the variable containing latitude weights to be named ‘lat_wgt’
(instead of the default, ‘gw’).  The third command associates the
latitude and longitude bounds with the dimension ‘bounds’ (instead of
the default, ‘nbnd’) and the variables ‘lat_bounds’ and ‘lon_bounds’
(instead of the defaults, ‘lat_bnds’ and ‘lon_bnds’, respectively).  The
fourth command associates the latitude and longitude bounds with the
dimension ‘vertices’ (instead of the default, ‘nv’) and the variables
‘lat_vrt’ and ‘lon_vrt’ (instead of the defaults, ‘lat_vertices’ and
‘lon_vertices’, respectively).

   When used with an identity remapping files, regridding can
signficantly enhance the metadata and therefore the dataset usability.
Consider these selected metadata (those unchanged are not shown for
brevity) associated with the variable ‘FSNT’ from typical unstructured
grid (CAM-SE cubed-sphere) output before and after an identity
regridding:
     # Raw model output before regridding
     netcdf ne30_FSNT {
       dimensions:
         nbnd = 2 ;
         ncol = 48602 ;
         time = UNLIMITED ; // (1 currently)
     
       variables:
         float FSNT(time,ncol) ;
           FSNT:long_name = "Net solar flux at top of model" ;
     
         double time(time) ;
           time:long_name = "time" ;
           time:bounds = "time_bnds" ;
     
         double time_bnds(time,nbnd) ;
           time_bnds:long_name = "time interval endpoints" ;
     } // group /
     
     # Same model output after identity regridding
     netcdf dogfood {
       dimensions:
         nbnd = 2 ;
         ncol = 48602 ;
         nv = 5 ;
         time = 1 ;
     
       variables:
         float FSNT(time,ncol) ;
           FSNT:long_name = "Net solar flux at top of model" ;
           FSNT:coordinates = "lat lon" ;
     
         double lat(ncol) ;
           lat:long_name = "latitude" ;
           lat:standard_name = "latitude" ;
           lat:units = "degrees_north" ;
           lat:axis = "Y" ;
           lat:bounds = "lat_vertices" ;
           lat:coordinates = "lat lon" ;
     
         double lat_vertices(ncol,nv) ;
           lat_vertices:long_name = "gridcell latitude vertices" ;
     
         double lon(ncol) ;
           lon:long_name = "longitude" ;
           lon:standard_name = "longitude" ;
           lon:units = "degrees_east" ;
           lon:axis = "X" ;
           lon:bounds = "lon_vertices" ;
           lon:coordinates = "lat lon" ;
     
         double lon_vertices(ncol,nv) ;
           lon_vertices:long_name = "gridcell longitude vertices" ;
     
         double time(time) ;
           time:long_name = "time" ;
           time:bounds = "time_bnds" ;
     
         double time_bnds(time,nbnd) ;
           time_bnds:long_name = "time interval endpoints" ;
     } // group /
   The raw model output lacks the CF ‘coordinates’ and ‘bounds’
attributes that the regridder adds.  The metadata turns ‘lat’ and ‘lon’
into auxiliary coordinate variables (*note Auxiliary Coordinates::)
which can then be hyperslabbed (with ‘-X’) using latitude/longitude
coordinates bounding the region of interest:
     % ncks -u -H -X 314.6,315.3,-35.6,-35.1 -v FSNT dogfood.nc
     time[0]=31 ncol[0] FSNT[0]=344.575 W/m2
     
     ncol[0] lat[0]=-35.2643896828 degrees_north
     
     ncol[0] nv[0] lat_vertices[0]=-35.5977213708 
     ncol[0] nv[1] lat_vertices[1]=-35.5977213708 
     ncol[0] nv[2] lat_vertices[2]=-35.0972113817 
     ncol[0] nv[3] lat_vertices[3]=-35.0972113817 
     ncol[0] nv[4] lat_vertices[4]=-35.0972113817 
     
     ncol[0] lon[0]=315 degrees_east
     
     ncol[0] nv[0] lon_vertices[0]=315 
     ncol[0] nv[1] lon_vertices[1]=315 
     ncol[0] nv[2] lon_vertices[2]=315.352825437 
     ncol[0] nv[3] lon_vertices[3]=314.647174563 
     ncol[0] nv[4] lon_vertices[4]=314.647174563 
     
     time[0]=31 days since 1979-01-01 00:00:00
     
     time[0]=31 nbnd[0] time_bnds[0]=0 
     time[0]=31 nbnd[1] time_bnds[1]=31 
   Thus auxiliary coordinate variables help to structure unstructured
grids.  The expanded metadata annotations from an identity regridding
may obviate the need to place unstructured data on a rectangular grid.
For example, statistics for regions that can be expressed as unions of
rectangular regions can now be performed on the native (unstructured)
grid.

   Here are some quick examples of regridding from common models.  All
examples require ‘in.nc out.nc’ at the end.
     # Identity re-map E3SM/ACME CAM-SE Cubed-Sphere output (to improve metadata)
     ncks --map=${DATA}/maps/map_ne30np4_to_ne30np4_aave.20150603.nc
     # Convert E3SM/ACME CAM-SE Cubed Sphere output to rectangular lat/lon
     ncks --map=${DATA}/maps/map_ne30np4_to_fv129x256_aave.150418.nc
     # Convert CAM3 T42 output to Cubed-Sphere grid
     ncks --map=${DATA}/maps/map_ne30np4_to_t42_aave.20150601.nc

