This is nco.info, produced by gmakeinfo version 7.1 from nco.texi.

INFO-DIR-SECTION netCDF
START-INFO-DIR-ENTRY
* NCO::        User Guide for the netCDF Operator suite
END-INFO-DIR-ENTRY

This file documents NCO, a collection of utilities to manipulate and
analyze netCDF files.

   Copyright © 1995-2025 Charlie Zender

   This is the first edition of the ‘NCO User Guide’,
and is consistent with version 2 of ‘texinfo.tex’.

   Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  The
license is available online at <http://www.gnu.org/copyleft/fdl.html>

   The original author of this software, Charlie Zender, wants to
improve it with the help of your suggestions, improvements, bug-reports,
and patches.
Charlie Zender <surname at uci dot edu> (yes, my surname is zender)
3200 Croul Hall
Department of Earth System Science
University of California, Irvine
Irvine, CA 92697-3100


File: nco.info,  Node: GSL interpolation,  Next: GSL least-squares fitting,  Prev: GSL special functions,  Up: ncap2 netCDF Arithmetic Processor

4.1.23 GSL interpolation
------------------------

As of version 3.9.9 (released July, 2009), NCO has wrappers to the GSL
interpolation functions.

Given a set of data points (x1,y1)...(xn, yn) the GSL functions computes
a continuous interpolating function Y(x) such that Y(xi) = yi.  The
interpolation is piecewise smooth, and its behavior at the end-points is
determined by the type of interpolation used.  For more information
consult the GSL manual.

Interpolation with ‘ncap2’ is a two stage process.  In the first stage,
a RAM variable is created from the chosen interpolating function and the
data set.  This RAM variable holds in memory a GSL interpolation object.
In the second stage, points along the interpolating function are
calculated.  If you have a very large data set or are interpolating many
sets then consider deleting the RAM variable when it is redundant.  Use
the command ‘ram_delete(var_nm)’.

A simple example

     x_in[$lon]={1.0,2.0,3.0,4.0};
     y_in[$lon]={1.1,1.2,1.5,1.8};
     
     // Ram variable is declared and defined here 
     gsl_interp_cspline(&ram_sp,x_in,y_in);
     
     x_out[$lon_grd]={1.1,2.0,3.0,3.1,3.99};
     
     y_out=gsl_spline_eval(ram_sp,x_out);
     y2=gsl_spline_eval(ram_sp,1.3);
     y3=gsl_spline_eval(ram_sp,0.0);
     ram_delete(ram_sp);
     
     print(y_out); // 1.10472, 1.2, 1.4, 1.42658, 1.69680002 
     print(y2);    // 1.12454 
     print(y3);    // '_' 

Note in the above example y3 is set to 'missing value' because 0.0 isn't
within the input X range.

   *GSL Interpolation Types*
All the interpolation functions have been implemented.  These are:
gsl_interp_linear()
gsl_interp_polynomial()
gsl_interp_cspline()
gsl_interp_cspline_periodic()
gsl_interp_akima()
gsl_interp_akima_periodic()




   * Evaluation of Interpolating Types *
*Implemented*
gsl_spline_eval()
*Not implemented*
gsl_spline_deriv()
gsl_spline_deriv2()
gsl_spline_integ()


File: nco.info,  Node: GSL least-squares fitting,  Next: GSL statistics,  Prev: GSL interpolation,  Up: ncap2 netCDF Arithmetic Processor

4.1.24 GSL least-squares fitting
--------------------------------

Least Squares fitting is a method of calculating a straight line through
a set of experimental data points in the XY plane.  Data may be weighted
or unweighted.  For more information please refer to the GSL manual.

These GSL functions fall into three categories:
*A)* Fitting data to Y=c0+c1*X
*B)* Fitting data (through the origin) Y=c1*X
*C)* Multi-parameter fitting (not yet implemented)

   *Section A*
‘status=*gsl_fit_linear*
(data_x,stride_x,data_y,stride_y,n,&co,&c1,&cov00,&cov01,&cov11,&sumsq)’

*Input variables*: data_x, stride_x, data_y, stride_y, n
From the above variables an X and Y vector both of length 'n' are
derived.  If data_x or data_y is less than type double then it is
converted to type ‘double’.  It is up to you to do bounds checking on
the input data.  For example if stride_x=3 and n=8 then the size of
data_x must be at least 24

*Output variables*: c0, c1, cov00, cov01, cov11,sumsq
The '&' prefix indicates that these are call-by-reference variables.  If
any of the output variables don't exist prior to the call then they are
created on the fly as scalar variables of type ‘double’.  If they
already exist then their existing value is overwritten.  If the function
call is successful then ‘status=0’.

   ‘status=
*gsl_fit_wlinear*(data_x,stride_x,data_w,stride_w,data_y,stride_y,n,&co,&c1,&cov00,&cov01,&cov11,&chisq)
’

Similar to the above call except it creates an additional weighting
vector from the variables data_w, stride_w, n

   ‘ data_y_out=*gsl_fit_linear_est*(data_x,c0,c1,cov00,cov01,cov11) ’

This function calculates y values along the line Y=c0+c1*X


   *Section B*
‘status=*gsl_fit_mul*(data_x,stride_x,data_y,stride_y,n,&c1,&cov11,&sumsq)’

*Input variables*: data_x, stride_x, data_y, stride_y, n
From the above variables an X and Y vector both of length 'n' are
derived.  If data_x or data_y is less than type ‘double’ then it is
converted to type ‘double’.

*Output variables*: c1,cov11,sumsq

   ‘status=
*gsl_fit_wmul*(data_x,stride_x,data_w,stride_w,data_y,stride_y,n,&c1,&cov11,&sumsq)’

Similar to the above call except it creates an additional weighting
vector from the variables data_w, stride_w, n

   ‘data_y_out=*gsl_fit_mul_est*(data_x,c0,c1,cov11)’

This function calculates y values along the line Y=c1*X

The below example shows *gsl_fit_linear()* in action
     defdim("d1",10);
     xin[d1]={1,2,3,4,5,6,7,8,9,10.0};
     yin[d1]={3.1,6.2,9.1,12.2,15.1,18.2,21.3,24.0,27.0,30.0};
     gsl_fit_linear(xin,1,yin,1,$d1.size,&c0,&c1,&cov00,&cov01,&cov11,&sumsq);
     print(c0);  // 0.2
     print(c1);  // 2.98545454545
     
     defdim("e1",4);
     xout[e1]={1.0,3.0,4.0,11};
     yout[e1]=0.0;
     
     yout=gsl_fit_linear_est(xout,c0,c1,cov00,cov01,cov11,sumsq);
     
     print(yout); // 3.18545454545, 9.15636363636, 12.1418181818, 33.04

The following code does linear regression of sst(time,lat,lon) for each
time-step
     // Declare variables
     c0[$lat, $lon]=0.; // Intercept
     c1[$lat, $lon]=0.; // Slope
     sdv[$lat, $lon]=0.; // Standard deviation
     covxy[$lat, $lon]=0.; // Covariance
     for (i=0;i<$lat.size;i++) // Loop over lat
     {
       for (j=0;j<$lon.size;j++) // Loop over lon
       {
           // Linear regression function
           gsl_fit_linear(time,1,sst(:, i, j),1,$time.size,&tc0,&tc1,&cov00,&cov01,&cov11,&sumsq); 
           c0(i,j)=tc0; // Output results
           c1(i,j)=tc1; // Output results
           // Covariance function
           covxy(i,j)=gsl_stats_covariance(time,1,$time.size,double(sst(:,i,j)),1,$time.size); 
           // Standard deviation function
           sdv(i,j)=gsl_stats_sd(sst(:,i,j),1,$time.size); 
       }
      }
     // slope (c1) missing values are set to '0', change to -999. (variable c0 intercept value)
     where(c0 == -999) c1=-999;


File: nco.info,  Node: GSL statistics,  Next: GSL random number generation,  Prev: GSL least-squares fitting,  Up: ncap2 netCDF Arithmetic Processor

4.1.25 GSL statistics
---------------------

Wrappers for most of the GSL Statistical functions have been
implemented.  The GSL function names include a type specifier (except
for type double functions).  To obtain the equivalent NCO name simply
remove the type specifier; then depending on the data type the
appropriate GSL function is called.  The weighed statistical functions
e.g., ‘ gsl_stats_wvariance()’ are only defined in GSL for
floating-point types; so your data must of type ‘float’ or ‘double’
otherwise ncap2 will emit an error message.  To view the implemented
functions use the shell command ‘ncap2 -f|grep _stats’

GSL Functions
     short gsl_stats_max (short data[], size_t stride, size_t n);
     double gsl_stats_int_mean (int data[], size_t stride, size_t n);
     double gsl_stats_short_sd_with_fixed_mean (short data[], size_t stride, size_t n, double mean);
     double gsl_stats_wmean (double w[], size_t wstride, double data[], size_t stride, size_t n);
     double gsl_stats_quantile_from_sorted_data (double sorted_data[], size_t stride, size_t n, double f) ;

Equivalent ncap2 wrapper functions
     short gsl_stats_max (var_data, data_stride, n);
     double gsl_stats_mean (var_data, data_stride, n);
     double gsl_stats_sd_with_fixed_mean (var_data, data_stride, n, var_mean);
     double gsl_stats_wmean (var_weight, weight_stride, var_data, data_stride, n, var_mean);
     double gsl_stats_quantile_from_sorted_data (var_sorted_data, data_stride, n, var_f) ;

GSL has no notion of missing values or dimensionality beyond one.  If
your data has missing values which you want ignored in the calculations
then use the ‘ncap2’ built in aggregate functions(*note Methods and
functions::).  The GSL functions operate on a vector of values created
from the var_data/stride/n arguments.  The ncap wrappers check that
there is no bounding error with regard to the size of the data and the
final value in the vector.
     a1[time]={1,2,3,4,5,6,7,8,9,10};
     
     a1_avg=gsl_stats_mean(a1,1,10);
     print(a1_avg); // 5.5
     
     a1_var=gsl_stats_variance(a1,4,3);
     print(a1_var); // 16.0
     
     // bounding error, vector attempts to access element a1(10)
     a1_sd=gsl_stats_sd(a1,5,3); 

For functions with the signature *func_nm(var_data,data_stride,n)*, one
may omit the second or third arguments.  The default value for STRIDE is
‘1’.  The default value for N is ‘1+(data.size()-1)/stride’.

     // Following statements are equvalent
     n2=gsl_stats_max(a1,1,10)
     n2=gsl_stats_max(a1,1);
     n2=gsl_stats_max(a1);

     // Following statements are equvalent
     n3=gsl_stats_median_from_sorted_data(a1,2,5);
     n3=gsl_stats_median_from_sorted_data(a1,2);

     // Following statements are NOT equvalent
     n4=gsl_stats_kurtosis(a1,3,2);
     n4=gsl_stats_kurtosis(a1,3); //default n=4

   The following example illustrates some of the weighted functions.
The data are randomly generated.  In this case the value of the weight
for each datum is either 0.0 or 1.0
     defdim("r1",2000);
     data[r1]=1.0;

     // Fill with random numbers [0.0,10.0)
     data=10.0*gsl_rng_uniform(data);

     // Create a weighting variable
     weight=(data>4.0);

     wmean=gsl_stats_wmean(weight,1,data,1,$r1.size);
     print(wmean);

     wsd=gsl_stats_wsd(weight,1,data,1,$r1.size);
     print(wsd);

     // number of values in data that are greater than 4
     weight_size=weight.total();
     print(weight_size);

     // print min/max of data
     dmin=data.gsl_stats_min();
     dmax=data.gsl_stats_max();
     print(dmin);print(dmax);


File: nco.info,  Node: GSL random number generation,  Next: Examples ncap2,  Prev: GSL statistics,  Up: ncap2 netCDF Arithmetic Processor

4.1.26 GSL random number generation
-----------------------------------

The GSL library has a large number of random number generators.  In
addition there are a large set of functions for turning uniform random
numbers into discrete or continuous probabilty distributions.  The
random number generator algorithms vary in terms of quality numbers
output, speed of execution and maximum number output.  For more
information see the GSL documentation.  The algorithm and seed are set
via environment variables, these are picked up by the ‘ncap2’ code.

*Setup*
The number algorithm is set by the environment variable ‘GSL_RNG_TYPE’.
If this variable isn't set then the default rng algorithm is
gsl_rng_19937.  The seed is set with the environment variable
‘GSL_RNG_SEED’.  The following wrapper functions in ncap2 provide
information about the chosen algorithm.

‘gsl_rng_min()’
     the minimum value returned by the rng algorithm.
‘gsl_rng_max()’
     the maximum value returned by the rng algorithm.

*Uniformly Distributed Random Numbers*
‘gsl_rng_get(var_in)’
     This function returns var_in with integers from the chosen rng
     algorithm.  The min and max values depend uoon the chosen rng
     algorthm.
‘gsl_rng_uniform_int(var_in)’
     This function returns var_in with random integers from 0 to n-1.
     The value n must be less than or equal to the maximum value of the
     chosen rng algorithm.
‘gsl_rng_uniform(var_in)’
     This function returns var_in with double-precision numbers in the
     range [0.0,1).  The range includes 0.0 and excludes 1.0.
‘gsl_rng_uniform_pos(var_in)’
     This function returns var_in with double-precision numbers in the
     range (0.0,1), excluding both 0.0 and 1.0.

Below are examples of ‘gsl_rng_get()’ and ‘gsl_rng_uniform_int()’ in
action.

     export GSL_RNG_TYPE=ranlux
     export GSL_RNG_SEED=10
     ncap2 -v -O -s 'a1[time]=0;a2=gsl_rng_get(a1);' in.nc foo.nc
     // 10 random numbers from the range 0 - 16777215
     // a2=9056646, 12776696, 1011656, 13354708, 5139066, 1388751, 11163902, 7730127, 15531355, 10387694 ;

     ncap2 -v -O -s 'a1[time]=21;a2=gsl_rng_uniform_int(a1).sort();' in.nc foo.nc
     // 10 random numbers from the range 0 - 20
     a2 = 1, 1, 6, 9, 11, 13, 13, 15, 16, 19 ;


The following example produces an ‘ncap2’ runtime error.  This is
because the chose rng algorithm has a maximum value greater than
‘NC_MAX_INT=2147483647’; the wrapper functions to ‘gsl_rng_get()’ and
‘gsl_rng_uniform_int()’ return variable of type ‘NC_INT’.  Please be
aware of this when using random number distribution functions functions
from the GSL library which return ‘unsigned int’.  Examples of these are
‘gsl_ran_geometric()’ and ‘gsl_ran_pascal()’.

     export GSL_RNG_TYPE=mt19937
     ncap2 -v -O -s 'a1[time]=0;a2=gsl_rng_get(a1);' in.nc foo.nc

To find the maximum value of the chosen rng algorithm use the following
code snippet.
     ncap2 -v -O -s 'rng_max=gsl_rng_max();print(rng_max)' in.nc foo.nc

*Random Number Distributions*
The GSL library has a rich set of random number disribution functions.
The library also provides cumulative distribution functions and inverse
cumulative distribution functions sometimes referred to a quantile
functions.  To see whats available on your build use the shell command
‘ncap2 -f|grep -e _ran -e _cdf’.

The following examples all return variables of type ‘NC_INT’
     defdim("out",15);
     a1[$out]=0.5;
     a2=gsl_ran_binomial(a1,30).sort();
     //a2 = 10, 11, 12, 12, 13, 14, 14, 15, 15, 16, 16, 16, 16, 17, 22 ;
     a3=gsl_ran_geometric(a2).sort();
     //a2 = 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 4, 5 ;
     a4=gsl_ran_pascal(a2,50);
     //a5 = 37, 40, 40, 42, 43, 45, 46, 49, 52, 58, 60, 62, 62, 65, 67 ;

The following all return variables of type ‘NC_DOUBLE’;
     defdim("b1",1000);
     b1[$b1]=0.8;
     b2=gsl_ran_exponential(b1);
     b2_avg=b2.avg();
     print(b2_avg);
     // b2_avg = 0.756047976787

     b3=gsl_ran_gaussian(b1);
     b3_avg=b3.avg();
     b3_rms=b3.rms();
     print(b3_avg);
     // b3_avg = -0.00903446534258;
     print(b3_rms);
     // b3_rms = 0.81162979889;

     b4[$b1]=10.0;
     b5[$b1]=20.0;
     b6=gsl_ran_flat(b4,b5);
     b6_avg=b6.avg();
     print(b6_avg);
     // b6_avg=15.0588129413


File: nco.info,  Node: Examples ncap2,  Next: Intrinsic mathematical methods,  Prev: GSL random number generation,  Up: ncap2 netCDF Arithmetic Processor

4.1.27 Examples ncap2
---------------------

See the ‘ncap.in’ and ‘ncap2.in’ scripts released with NCO for more
complete demonstrations of ‘ncap2’ functionality (script available
on-line at <http://nco.sf.net/ncap2.in>).

   Define new attribute NEW for existing variable ONE as twice the
existing attribute DOUBLE_ATT of variable ATT_VAR:
     ncap2 -s 'one@new=2*att_var@double_att' in.nc out.nc

   Average variables of mixed types (result is of type ‘double’):
     ncap2 -s 'average=(var_float+var_double+var_int)/3' in.nc out.nc

   Multiple commands may be given to ‘ncap2’ in three ways.  First, the
commands may be placed in a script which is executed, e.g., ‘tst.nco’.
Second, the commands may be individually specified with multiple ‘-s’
arguments to the same ‘ncap2’ invocation.  Third, the commands may be
chained into a single ‘-s’ argument to ‘ncap2’.  Assuming the file
‘tst.nco’ contains the commands ‘a=3;b=4;c=sqrt(a^2+b^2);’, then the
following ‘ncap2’ invocations produce identical results:
     ncap2 -v -S tst.nco in.nc out.nc
     ncap2 -v -s 'a=3' -s 'b=4' -s 'c=sqrt(a^2+b^2)' in.nc out.nc
     ncap2 -v -s 'a=3;b=4;c=sqrt(a^2+b^2)' in.nc out.nc
   The second and third examples show that ‘ncap2’ does not require that
a trailing semi-colon ‘;’ be placed at the end of a ‘-s’ argument,
although a trailing semi-colon ‘;’ is always allowed.  However,
semi-colons are required to separate individual assignment statements
chained together as a single ‘-s’ argument.

   ‘ncap2’ may be used to "grow" dimensions, i.e., to increase dimension
sizes without altering existing data.  Say ‘in.nc’ has ‘ORO(lat,lon)’
and the user wishes a new file with ‘new_ORO(new_lat,new_lon)’ that
contains zeros in the undefined portions of the new grid.
     defdim("new_lat",$lat.size+1); // Define new dimension sizes
     defdim("new_lon",$lon.size+1);
     new_ORO[$new_lat,$new_lon]=0.0f; // Initialize to zero
     new_ORO(0:$lat.size-1,0:$lon.size-1)=ORO; // Fill valid data
   The commands to define new coordinate variables ‘new_lat’ and
‘new_lon’ in the output file follow a similar pattern.  One would might
store these commands in a script ‘grow.nco’ and then execute the script
with
     ncap2 -v -S grow.nco in.nc out.nc

   Imagine you wish to create a binary flag based on the value of an
array.  The flag should have value 1.0 where the array exceeds 1.0, and
value 0.0 elsewhere.  This example creates the binary flag ‘ORO_flg’ in
‘out.nc’ from the continuous array named ‘ORO’ in ‘in.nc’.
     ncap2 -s 'ORO_flg=(ORO > 1.0)' in.nc out.nc
   Suppose your task is to change all values of ‘ORO’ which equal 2.0 to
the new value 3.0:
     ncap2 -s 'ORO_msk=(ORO==2.0);ORO=ORO_msk*3.0+!ORO_msk*ORO' in.nc out.nc
   This creates and uses ‘ORO_msk’ to mask the subsequent arithmetic
operation.  Values of ‘ORO’ are only changed where ‘ORO_msk’ is true,
i.e., where ‘ORO’ equals 2.0
Using the ‘where’ statement the above code simplifies to :
     ncap2 -s 'where(ORO == 2.0) ORO=3.0;' in.nc foo.nc

   This example uses ‘ncap2’ to compute the covariance of two variables.
Let the variables U and V be the horizontal wind components.  The
“covariance” of U and V is defined as the time mean product of the
deviations of U and V from their respective time means.  Symbolically,
the covariance [U'V'] = [UV]-[U][V] where [X] denotes the time-average
of X and X' denotes the deviation from the time-mean.  The covariance
tells us how much of the correlation of two signals arises from the
signal fluctuations versus the mean signals.  Sometimes this is called
the “eddy covariance”.  We will store the covariance in the variable
‘uprmvprm’.
     ncwa -O -a time -v u,v in.nc foo.nc # Compute time mean of u,v
     ncrename -O -v u,uavg -v v,vavg foo.nc # Rename to avoid conflict
     ncks -A -v uavg,vavg foo.nc in.nc # Place time means with originals
     ncap2 -O -s 'uprmvprm=u*v-uavg*vavg' in.nc in.nc # Covariance
     ncra -O -v uprmvprm in.nc foo.nc # Time-mean covariance
   The mathematically inclined will note that the same covariance would
be obtained by replacing the step involving ‘ncap2’ with
     ncap2 -O -s 'uprmvprm=(u-uavg)*(v-vavg)' foo.nc foo.nc # Covariance

   As of NCO version 3.1.8 (December, 2006), ‘ncap2’ can compute
averages, and thus covariances, by itself:
     ncap2 -s 'uavg=u.avg($time);vavg=v.avg($time);uprmvprm=u*v-uavg*vavg' \
           -s 'uprmvrpmavg=uprmvprm.avg($time)' in.nc foo.nc
   We have not seen a simpler method to script and execute powerful
arithmetic than ‘ncap2’.

   ‘ncap2’ utilizes many meta-characters (e.g., ‘$’, ‘?’, ‘;’, ‘()’,
‘[]’) that can confuse the command-line shell if not quoted properly.
The issues are the same as those which arise in utilizing extended
regular expressions to subset variables (*note Subsetting Files::).  The
example above will fail with no quotes and with double quotes.  This is
because shell globbing tries to “interpolate” the value of ‘$time’ from
the shell environment unless it is quoted:
     ncap2 -s 'uavg=u.avg($time)'  in.nc foo.nc # Correct (recommended)
     ncap2 -s  uavg=u.avg('$time') in.nc foo.nc # Correct (and dangerous)
     ncap2 -s  uavg=u.avg($time)   in.nc foo.nc # Wrong ($time = '')
     ncap2 -s "uavg=u.avg($time)"  in.nc foo.nc # Wrong ($time = '')
   Without the single quotes, the shell replaces ‘$time’ with an empty
string.  The command ‘ncap2’ receives from the shell is ‘uavg=u.avg()’.
This causes ‘ncap2’ to average over all dimensions rather than just the
TIME dimension, and unintended consequence.

   We recommend using single quotes to protect ‘ncap2’ command-line
scripts from the shell, even when such protection is not strictly
necessary.  Expert users may violate this rule to exploit the ability to
use shell variables in ‘ncap2’ command-line scripts (*note CCSM
Example::).  In such cases it may be necessary to use the shell
backslash character ‘\’ to protect the ‘ncap2’ meta-character.

   A dimension of size one is said to be _degenerate_.  Whether a
degenerate record dimension is desirable or not depends on the
application.  Often a degenerate TIME dimension is useful, e.g., for
concatenating, though it may cause problems with arithmetic.  Such is
the case in the above example, where the first step employs ‘ncwa’
rather than ‘ncra’ for the time-averaging.  Of course the numerical
results are the same with both operators.  The difference is that,
unless ‘-b’ is specified, ‘ncwa’ writes no TIME dimension to the output
file, while ‘ncra’ defaults to keeping TIME as a degenerate (size 1)
dimension.  Appending ‘u’ and ‘v’ to the output file would cause ‘ncks’
to try to expand the degenerate time axis of ‘uavg’ and ‘vavg’ to the
size of the non-degenerate TIME dimension in the input file.  Thus the
append (‘ncks -A’) command would be undefined (and should fail) in this
case.  Equally important is the ‘-C’ argument (*note Subsetting
Coordinate Variables::) to ‘ncwa’ to prevent any scalar TIME variable
from being written to the output file.  Knowing when to use ‘ncwa -a
time’ rather than the default ‘ncra’ for time-averaging takes, well,
time.


File: nco.info,  Node: Intrinsic mathematical methods,  Next: Operator precedence and associativity,  Prev: Examples ncap2,  Up: ncap2 netCDF Arithmetic Processor

4.1.28 Intrinsic mathematical methods
-------------------------------------

‘ncap2’ supports the standard mathematical functions supplied with most
operating systems.  Standard calculator notation is used for addition
‘+’, subtraction ‘-’, multiplication ‘*’, division ‘/’, exponentiation
‘^’, and modulus ‘%’.  The available elementary mathematical functions
are:
‘abs(x)’
     “Absolute value” Absolute value of X.  Example: abs(-1) = 1
‘acos(x)’
     “Arc-cosine” Arc-cosine of X where X is specified in radians.
     Example: acos(1.0) = 0.0
‘acosh(x)’
     “Hyperbolic arc-cosine” Hyperbolic arc-cosine of X where X is
     specified in radians.  Example: acosh(1.0) = 0.0
‘asin(x)’
     “Arc-sine” Arc-sine of X where X is specified in radians.  Example:
     asin(1.0) = 1.57079632679489661922
‘asinh(x)’
     “Hyperbolic arc-sine” Hyperbolic arc-sine of X where X is specified
     in radians.  Example: asinh(1.0) = 0.88137358702
‘atan(x)’
     “Arc-tangent” Arc-tangent of X where X is specified in radians
     between -pi/2 and pi/2.  Example: atan(1.0) =
     0.78539816339744830961

‘atan2(y,x)’
     “Arc-tangent2” Arc-tangent of Y/X :Example atan2(1,3) = 0.321689857

‘atanh(x)’
     “Hyperbolic arc-tangent” Hyperbolic arc-tangent of X where X is
     specified in radians between -pi/2 and pi/2.  Example:
     atanh(3.14159265358979323844) = 1.0
‘ceil(x)’
     “Ceil” Ceiling of X.  Smallest integral value not less than
     argument.  Example: ceil(0.1) = 1.0
‘cos(x)’
     “Cosine” Cosine of X where X is specified in radians.  Example:
     cos(0.0) = 1.0
‘cosh(x)’
     “Hyperbolic cosine” Hyperbolic cosine of X where X is specified in
     radians.  Example: cosh(0.0) = 1.0
‘erf(x)’
     “Error function” Error function of X where X is specified between
     -1 and 1.  Example: erf(1.0) = 0.842701
‘erfc(x)’
     “Complementary error function” Complementary error function of X
     where X is specified between -1 and 1.  Example: erfc(1.0) =
     0.15729920705
‘exp(x)’
     “Exponential” Exponential of X, e^x.  Example: exp(1.0) =
     2.71828182845904523536
‘floor(x)’
     “Floor” Floor of X.  Largest integral value not greater than
     argument.  Example: floor(1.9) = 1
‘gamma(x)’
     “Gamma function” Gamma function of X, Gamma(x).  The well-known and
     loved continuous factorial function.  Example: gamma(0.5) =
     sqrt(pi)
‘gamma_inc_P(x)’
     “Incomplete Gamma function” Incomplete Gamma function of parameter
     A and variable X, gamma_inc_P(a,x).  One of the four incomplete
     gamma functions.  Example: gamma_inc_P(1,1) = 1-1/e
‘ln(x)’
     “Natural Logarithm” Natural logarithm of X, ln(x).  Example:
     ln(2.71828182845904523536) = 1.0
‘log(x)’
     “Natural Logarithm” Exact synonym for ‘ln(x)’.
‘log10(x)’
     “Base 10 Logarithm” Base 10 logarithm of X, log10(x).  Example:
     log(10.0) = 1.0
‘nearbyint(x)’
     “Round inexactly” Nearest integer to X is returned in
     floating-point format.  No exceptions are raised for “inexact
     conversions”.  Example: nearbyint(0.1) = 0.0
‘pow(x,y)’
     “Power” Value of X is raised to the power of Y.  Exceptions are
     raised for “domain errors”.  Due to type-limitations in the
     C language ‘pow’ function, integer arguments are promoted (*note
     Type Conversion::) to type ‘NC_FLOAT’ before evaluation.  Example:
     pow(2,3) = 8
‘rint(x)’
     “Round exactly” Nearest integer to X is returned in floating-point
     format.  Exceptions are raised for “inexact conversions”.  Example:
     rint(0.1) = 0
‘round(x)’
     “Round” Nearest integer to X is returned in floating-point format.
     Round halfway cases away from zero, regardless of current IEEE
     rounding direction.  Example: round(0.5) = 1.0
‘sin(x)’
     “Sine” Sine of X where X is specified in radians.  Example:
     sin(1.57079632679489661922) = 1.0
‘sinh(x)’
     “Hyperbolic sine” Hyperbolic sine of X where X is specified in
     radians.  Example: sinh(1.0) = 1.1752
‘sqrt(x)’
     “Square Root” Square Root of X, sqrt(x).  Example: sqrt(4.0) = 2.0
‘tan(x)’
     “Tangent” Tangent of X where X is specified in radians.  Example:
     tan(0.78539816339744830961) = 1.0
‘tanh(x)’
     “Hyperbolic tangent” Hyperbolic tangent of X where X is specified
     in radians.  Example: tanh(1.0) = 0.761594155956
‘trunc(x)’
     “Truncate” Nearest integer to X is returned in floating-point
     format.  Round halfway cases toward zero, regardless of current
     IEEE rounding direction.  Example: trunc(0.5) = 0.0
The complete list of mathematical functions supported is
platform-specific.  Functions mandated by ANSI C are _guaranteed_ to be
present and are indicated with an asterisk (1).  and are indicated with
an asterisk.  Use the ‘-f’ (or ‘fnc_tbl’ or ‘prn_fnc_tbl’) switch to
print a complete list of functions supported on your platform.  (2)

   ---------- Footnotes ----------

   (1) ANSI C compilers are guaranteed to support double-precision
versions of these functions.  These functions normally operate on netCDF
variables of type ‘NC_DOUBLE’ without having to perform intrinsic
conversions.  For example, ANSI compilers provide ‘sin’ for the sine of
C-type ‘double’ variables.  The ANSI standard does not require, but many
compilers provide, an extended set of mathematical functions that apply
to single (‘float’) and quadruple (‘long double’) precision variables.
Using these functions (e.g., ‘sinf’ for ‘float’, ‘sinl’ for ‘long
double’), when available, is (presumably) more efficient than casting
variables to type ‘double’, performing the operation, and then
re-casting.  NCO uses the faster intrinsic functions when they are
available, and uses the casting method when they are not.

   (2) Linux supports more of these intrinsic functions than other OSs.


File: nco.info,  Node: Operator precedence and associativity,  Next: ID Quoting,  Prev: Intrinsic mathematical methods,  Up: ncap2 netCDF Arithmetic Processor

4.1.29 Operator precedence and associativity
--------------------------------------------

This page lists the ‘ncap2’ operators in order of precedence (highest to
lowest).  Their associativity indicates in what order operators of equal
precedence in an expression are applied.

Operator      Description                                   Associativity
---------------------------------------------------------------------------
‘++ --’       Postfix Increment/Decrement                   Right to
                                                            Left
‘()’          Parentheses (function call)
‘.’           Method call
‘++ --’       Prefix Increment/Decrement                    Right to
                                                            Left
‘+ -’         Unary Plus/Minus
‘!’           Logical Not
‘^’           Power of Operator                             Right to
                                                            Left
‘* / %’       Multiply/Divide/Modulus                       Left To
                                                            Right
‘+ -’         Addition/Subtraction                          Left To
                                                            Right
‘>> <<’       Fortran style array clipping                  Left to
                                                            Right
‘< <=’        Less than/Less than or equal to               Left to
                                                            Right
‘> >=’        Greater than/Greater than or equal to
‘== !=’       Equal to/Not equal to                         Left to
                                                            Right
‘&&’          Logical AND                                   Left to
                                                            Right
‘||’          Logical OR                                    Left to
                                                            Right
‘?:’          Ternary Operator                              Right to
                                                            Left
‘=’           Assignment                                    Right to
                                                            Left
‘+= -=’       Addition/subtraction assignment
‘*= /=’       Multiplication/division assignment


File: nco.info,  Node: ID Quoting,  Next: make_bounds() function,  Prev: Operator precedence and associativity,  Up: ncap2 netCDF Arithmetic Processor

4.1.30 ID Quoting
-----------------

In this section a name refers to a variable, attribute, or dimension
name.  The allowed characters in a valid netCDF name vary from release
to release.  (See end section).  To use metacharacters in a name, or to
use a method name as a variable name, the name must be quoted wherever
it occurs.

The default NCO name is specified by the regular expressions:

     DGT:     ('0'..'9');
     LPH:     ( 'a'..'z' | 'A'..'Z' | '_' );
     name:    (LPH)(LPH|DGT)+

The first character of a valid name must be alphabetic or the
underscore.  Subsequent characters must be alphanumeric or underscore,
e.g., a1, _23, hell_is_666.

The valid characters in a quoted name are specified by the regular
expressions:
     LPHDGT:  ( 'a'..'z' | 'A'..'Z' | '_' | '0'..'9');
     name:    (LPHDGT|'-'|'+'|'.'|'('|')'|':' )+  ;

Quote a variable:
'avg' , '10_+10','set_miss' '+-90field' , '-test'=10.0d

Quote an attribute:
'three@10', 'set_mss@+10', '666@hell', 't1@+units'="kelvin"

Quote a dimension:
'$10', '$t1-', '$-odd', c1['$10','$t1-']=23.0d


   The following comments are from the netCDF library definitions and
detail the naming conventions for each release.  netcdf-3.5.1
netcdf-3.6.0-p1
netcdf-3.6.1
netcdf-3.6.2
     /*
      * ( [a-zA-Z]|[0-9]|'_'|'-'|'+'|'.'|'|':'|'@'|'('|')' )+
      * Verify that name string is valid CDL syntax, i.e., all characters are
      * alphanumeric, '-', '_', '+', or '.'.
      * Also permit ':', '@', '(', or ')' in names for chemists currently making 
      * use of these characters, but don't document until ncgen and ncdump can 
      * also handle these characters in names.
      */

netcdf-3.6.3
netcdf-4.0 Final 2008/08/28
     /*
      * Verify that a name string is valid syntax.  The allowed name
      * syntax (in RE form) is:
      *
      * ([a-zA-Z_]|{UTF8})([^\x00-\x1F\x7F/]|{UTF8})*
      *
      * where UTF8 represents a multibyte UTF-8 encoding.  Also, no
      * trailing spaces are permitted in names.  This definition
      * must be consistent with the one in ncgen.l.  We do not allow '/'
      * because HDF5 does not permit slashes in names as slash is used as a
      * group separator.  If UTF-8 is supported, then a multi-byte UTF-8
      * character can occur anywhere within an identifier.  We later
      * normalize UTF-8 strings to NFC to facilitate matching and queries.
      */ 


File: nco.info,  Node: make_bounds() function,  Next: solar_zenith_angle function,  Prev: ID Quoting,  Up: ncap2 netCDF Arithmetic Processor

4.1.31 make_bounds() function
-----------------------------

The ‘ncap2’ custom function 'make_bounds()' takes any monotonic 1D
coordinate variable with regular or irregular (e.g., Gaussian) spacing
and creates a bounds variable.

   _<bounds_var_out>=make_bounds( <coordinate_var_in>, <dim in>,
<string>)_

1ST ARGUMENT
     The name of the input coordinate variable.
2ND ARGUMENT
     The second dimension of the output variable, referenced as a
     dimension (i.e., the name preceded by a dollarsign) not as a string
     name.  The size of this dimension should always be 2.  If the
     dimension does not yet exist create it first using ‘defdim()’.
3RD ARGUMENT
     This optional string argument will be placed in the "bounds"
     attribute that will be created in the input coordinate variable.
     Normally this is the name of the bounds variable:
   Typical usage:
     defdim("nv",2);
     longitude_bounds=make_bounds(longitude,$nv,"longitude_bounds");

   Another common CF convention:
     defdim("nv",2);
     climatology_bounds=make_bounds(time,$nv,"climatology_bounds");


File: nco.info,  Node: solar_zenith_angle function,  Prev: make_bounds() function,  Up: ncap2 netCDF Arithmetic Processor

4.1.32 solar_zenith_angle function
----------------------------------

_<zenith_out>=solar_zenith_angle( <time_in>, <latitude in>)_

   This function takes two arguments, mean local solar time and
latitude.  Calculation and output is done with type ‘NC_DOUBLE’.  The
calendar attribute for <time_in> in is NOT read and is assumed to be
Gregorian (this is the calendar that UDUnits uses).  As part of the
calculation <time_in> is converted to days since start of year.  For
some input units e.g., seconds, this function may produce gobbledygook.
The output <zenith_out> is in ‘degrees’.  For more details of the
algorithm used please examine the function ‘solar_geometry()’ in
‘fmc_all_cls.cc’.  Note that this routine does not account for the
equation of time, and so can be in error by the angular equivalent of up
to about fifteen minutes time depending on the day of year.

     my_time[time]={10.50, 11.0, 11.50, 12.0, 12.5, 13.0, 13.5, 14.0, 14.50, 15.00};
     my_time@units="hours since 2017-06-21";

     // Assume we are at Equator
     latitude=0.0;

     // 32.05428, 27.61159, 24.55934, 23.45467, 24.55947, 27.61184, 32.05458, 37.39353, 43.29914, 49.55782 ;
     zenith=solar_zenith_angle(my_time,latitude);


File: nco.info,  Node: ncatted netCDF Attribute Editor,  Next: ncbo netCDF Binary Operator,  Prev: ncap2 netCDF Arithmetic Processor,  Up: Reference Manual

4.2 ‘ncatted’ netCDF Attribute Editor
=====================================

SYNTAX
     ncatted [-a ATT_DSC] [-a ...] [-D DBG]
     [-H] [-h] [--hdr_pad NBR] [--hpss]
     [-l PATH] [-O] [-o OUTPUT-FILE] [-p PATH]
     [-R] [-r] [--ram_all] [-t] INPUT-FILE [[OUTPUT-FILE]]

DESCRIPTION

   ‘ncatted’ edits attributes in a netCDF file.  If you are editing
attributes then you are spending too much time in the world of metadata,
and ‘ncatted’ was written to get you back out as quickly and painlessly
as possible.  ‘ncatted’ can “append”, “create”, “delete”, “modify”, and
“overwrite” attributes (all explained below).  ‘ncatted’ allows each
editing operation to be applied to every variable in a file.  This saves
time when changing attribute conventions throughout a file.  ‘ncatted’
is for _writing_ attributes.  To _read_ attribute values in plain text,
use ‘ncks -m -M’, or define something like ‘ncattget’ as a shell command
(*note Filters for ncks::).

   Because repeated use of ‘ncatted’ can considerably increase the size
of the ‘history’ global attribute (*note History Attribute::), the ‘-h’
switch is provided to override automatically appending the command to
the ‘history’ global attribute in the OUTPUT-FILE.

   According to the ‘netCDF User Guide’, altering metadata in netCDF
files does not incur the penalty of recopying the entire file when the
new metadata occupies less space than the old metadata.  Thus ‘ncatted’
may run much faster (at least on netCDF3 files) if judicious use of
header padding (*note Metadata Optimization::) was made when producing
the INPUT-FILE.  Similarly, using the ‘--hdr_pad’ option with ‘ncatted’
helps ensure that future metadata changes to OUTPUT-FILE occur as
swiftly as possible.

   When ‘ncatted’ is used to change the ‘_FillValue’ attribute, it
changes the associated missing data self-consistently.  If the internal
floating-point representation of a missing value, e.g., 1.0e36, differs
between two machines then netCDF files produced on those machines will
have incompatible missing values.  This allows ‘ncatted’ to change the
missing values in files from different machines to a single value so
that the files may then be concatenated, e.g., by ‘ncrcat’, without
losing information.  *Note Missing Values::, for more information.

   To master ‘ncatted’ one must understand the meaning of the structure
that describes the attribute modification, ATT_DSC specified by the
required option ‘-a’ or ‘--attribute’.  This option is repeatable and
may be used multiple time in a single ‘ncatted’ invocation to increase
the efficiency of altering multiple attributes.  Each ATT_DSC contains
five elements.  This makes using ‘ncatted’ somewhat complicated, though
powerful.  The ATT_DSC fields are in the following order:

   ATT_DSC = ATT_NM, VAR_NM, MODE, ATT_TYPE, ATT_VAL

ATT_NM
     Attribute name.  Example: ‘units’ As of NCO 4.5.1 (July, 2015),
     ‘ncatted’ accepts regular expressions (*note Subsetting Files::)
     for attribute names (it has "always" accepted regular expressions
     for variable names).  Regular expressions will select all matching
     attribute names.
VAR_NM
     Variable name.  Example: ‘pressure’, ‘'^H2O'’.  Regular expressions
     (*note Subsetting Files::) are accepted and will select all
     matching variable (and/or group) names.  The names ‘global’ and
     ‘group’ have special meaning.
MODE
     Edit mode abbreviation.  Example: ‘a’.  See below for complete
     listing of valid values of MODE.
ATT_TYPE
     Attribute type abbreviation.  Example: ‘c’.  See below for complete
     listing of valid values of ATT_TYPE.
ATT_VAL
     Attribute value.  Example: ‘pascal’.
There should be no empty space between these five consecutive arguments.
The description of these arguments follows in their order of appearance.

   The value of ATT_NM is the name of the attribute to edit.  The
meaning of this should be clear to all ‘ncatted’ users.  Both ATT_NM and
VAR_NM may be specified as regular expressions.  If ATT_NM is omitted
(i.e., left blank) and “Delete” mode is selected, then all attributes
associated with the specified variable will be deleted.

   The value of VAR_NM is the name of the variable containing the
attribute (named ATT_NM) that you want to edit.  There are three very
important and useful exceptions to this rule.  The value of VAR_NM can
also be used to direct ‘ncatted’ to edit global attributes, or to repeat
the editing operation for every group or variable in a file.  A value of
VAR_NM of ‘global’ indicates that ATT_NM refers to a global (i.e.,
root-level) attribute, rather than to a particular variable's attribute.
This is the method ‘ncatted’ supports for editing global attributes.
A value of VAR_NM of ‘group’ indicates that ATT_NM refers to all groups,
rather than to a particular variable's or group's attribute.  The
operation will proceed to edit group metadata for every group.  Finally,
if VAR_NM is left blank, then ‘ncatted’ attempts to perform the editing
operation on every variable in the file.  This option may be convenient
to use if you decide to change the conventions you use for describing
the data.  As of NCO 4.6.0 (May, 2016), ‘ncatted’ accepts the ‘-t’ (or
long-option equivalent ‘--typ_mch’ or ‘--type_match’) option.  This
causes ‘ncatted’ to perform the editing operation only on variables that
are the same type as the specified attribute.

   The value of MODE is a single character abbreviation (‘a’, ‘c’, ‘d’,
‘m’, ‘n’, ‘o’, or ‘p’) standing for one of seven editing modes:
‘a’
     “Append”.  Append value ATT_VAL to current VAR_NM attribute ATT_NM
     value ATT_VAL, if any.  If VAR_NM does not already have an existing
     attribute ATT_NM, it is created with the value ATT_VAL.
‘c’
     “Create”.  Create variable VAR_NM attribute ATT_NM with ATT_VAL if
     ATT_NM does not yet exist.  If VAR_NM already has an attribute
     ATT_NM, there is no effect, so the existing attribute is preserved
     without change.
‘d’
     “Delete”.  Delete current VAR_NM attribute ATT_NM.  If VAR_NM does
     not have an attribute ATT_NM, there is no effect.  If ATT_NM is
     omitted (left blank), then all attributes associated with the
     specified variable are automatically deleted.  When “Delete” mode
     is selected, the ATT_TYPE and ATT_VAL arguments are superfluous and
     may be left blank.
‘m’
     “Modify”.  Change value of current VAR_NM attribute ATT_NM to value
     ATT_VAL.  If VAR_NM does not have an attribute ATT_NM, there is no
     effect.
‘n’
     “Nappend”.  Append value ATT_VAL to VAR_NM attribute ATT_NM value
     ATT_VAL if ATT_NM already exists.  If VAR_NM does not have an
     attribute ATT_NM, there is no effect.  In other words, if ATT_NM
     already exists, Nappend behaves like Append otherwise it does
     nothing.  The mnenomic is "non-create append".  Nappend mode was
     added to ‘ncatted’ in version 4.6.0 (May, 2016).
‘o’
     “Overwrite”.  Write attribute ATT_NM with value ATT_VAL to variable
     VAR_NM, overwriting existing attribute ATT_NM, if any.  This is the
     default mode.
‘p’
     “Prepend”.  Prepend value ATT_VAL to VAR_NM attribute ATT_NM value
     ATT_VAL if ATT_NM already exists.  If VAR_NM does not have an
     attribute ATT_NM, there is no effect.  Prepend mode was added to
     ‘ncatted’ in version 5.0.5 (January, 2022).

   The value of ATT_TYPE is a single character abbreviation (‘f’, ‘d’,
‘l’, ‘i’, ‘s’, ‘c’, ‘b’, ‘u’) or a short string standing for one of the
twelve primitive netCDF data types:
‘f’
     “Float”.  Value(s) specified in ATT_VAL will be stored as netCDF
     intrinsic type ‘NC_FLOAT’.
‘d’
     “Double”.  Value(s) specified in ATT_VAL will be stored as netCDF
     intrinsic type ‘NC_DOUBLE’.
‘i, l’
     “Integer” or (its now deprecated synonym) “Long”.  Value(s)
     specified in ATT_VAL will be stored as netCDF intrinsic type
     ‘NC_INT’.
‘s’
     “Short”.  Value(s) specified in ATT_VAL will be stored as netCDF
     intrinsic type ‘NC_SHORT’.
‘c’
     “Char”.  Value(s) specified in ATT_VAL will be stored as netCDF
     intrinsic type ‘NC_CHAR’.
‘b’
     “Byte”.  Value(s) specified in ATT_VAL will be stored as netCDF
     intrinsic type ‘NC_BYTE’.
‘ub’
     “Unsigned Byte”.  Value(s) specified in ATT_VAL will be stored as
     netCDF intrinsic type ‘NC_UBYTE’.
‘us’
     “Unsigned Short”.  Value(s) specified in ATT_VAL will be stored as
     netCDF intrinsic type ‘NC_USHORT’.
‘u, ui, ul’
     “Unsigned Int”.  Value(s) specified in ATT_VAL will be stored as
     netCDF intrinsic type ‘NC_UINT’.
‘ll, int64’
     “Int64”.  Value(s) specified in ATT_VAL will be stored as netCDF
     intrinsic type ‘NC_INT64’.
‘ull, uint64’
     “Uint64”.  Value(s) specified in ATT_VAL will be stored as netCDF
     intrinsic type ‘NC_UINT64’.
‘sng, string’
     “String”.  Value(s) specified in ATT_VAL will be stored as netCDF
     intrinsic type ‘NC_STRING’.  Note that ‘ncatted’ handles type
     ‘NC_STRING’ attributes correctly beginning with version 4.3.3
     released in July, 2013.  Earlier versions fail when asked to handle
     ‘NC_STRING’ attributes.
In “Delete” mode the specification of ATT_TYPE is optional (and is
ignored if supplied).

   The value of ATT_VAL is what you want to change attribute ATT_NM to
contain.  The specification of ATT_VAL is optional in “Delete” (and is
ignored) mode.  Attribute values for all types besides ‘NC_CHAR’ must
have an attribute length of at least one.  Thus ATT_VAL may be a single
value or one-dimensional array of elements of type ‘att_type’.  If the
ATT_VAL is not set or is set to empty space, and the ATT_TYPE is
‘NC_CHAR’, e.g., ‘-a units,T,o,c,""’ or ‘-a units,T,o,c,’, then the
corresponding attribute is set to have zero length.  When specifying an
array of values, it is safest to enclose ATT_VAL in single or double
quotes, e.g., ‘-a levels,T,o,s,"1,2,3,4"’ or ‘-a
levels,T,o,s,'1,2,3,4'’.  The quotes are strictly unnecessary around
ATT_VAL except when ATT_VAL contains characters which would confuse the
calling shell, such as spaces, commas, and wildcard characters.

   NCO processing of ‘NC_CHAR’ attributes is a bit like Perl in that it
attempts to do what you want by default (but this sometimes causes
unexpected results if you want unusual data storage).  If the ATT_TYPE
is ‘NC_CHAR’ then the argument is interpreted as a string and it may
contain C-language escape sequences, e.g., ‘\n’, which NCO will
interpret before writing anything to disk.  NCO translates valid escape
sequences and stores the appropriate ASCII code instead.  Since two byte
escape sequences, e.g., ‘\n’, represent one-byte ASCII codes, e.g.,
ASCII 10 (decimal), the stored string attribute is one byte shorter than
the input string length for each embedded escape sequence.  The most
frequently used C-language escape sequences are ‘\n’ (for linefeed) and
‘\t’ (for horizontal tab).  These sequences in particular allow
convenient editing of formatted text attributes.  The other valid ASCII
codes are ‘\a’, ‘\b’, ‘\f’, ‘\r’, ‘\v’, and ‘\\’.  *Note ncks netCDF
Kitchen Sink::, for more examples of string formatting (with the ‘ncks’
‘-s’ option) with special characters.

   Analogous to ‘printf’, other special characters are also allowed by
‘ncatted’ if they are "protected" by a backslash.  The characters ‘"’,
‘'’, ‘?’, and ‘\’ may be input to the shell as ‘\"’, ‘\'’, ‘\?’, and
‘\\’.  NCO simply strips away the leading backslash from these
characters before editing the attribute.  No other characters require
protection by a backslash.  Backslashes which precede any other
character (e.g., ‘3’, ‘m’, ‘$’, ‘|’, ‘&’, ‘@’, ‘%’, ‘{’, and ‘}’) will
not be filtered and will be included in the attribute.

   Note that the NUL character ‘\0’ which terminates C language strings
is assumed and need not be explicitly specified.  If ‘\0’ is input, it
is translated to the NUL character.  However, this will make the
subsequent portion of the string, if any, invisible to C standard
library string functions.  And that may cause unintended consequences.
Because of these context-sensitive rules, one must use ‘ncatted’ with
care in order to store data, rather than text strings, in an attribute
of type ‘NC_CHAR’.

   Note that ‘ncatted’ interprets character attributes (i.e., attributes
of type ‘NC_CHAR’) as strings.  EXAMPLES

   Append the string ‘Data version 2.0.\n’ to the global attribute
‘history’:
     ncatted -a history,global,a,c,'Data version 2.0\n' in.nc
   Note the use of embedded C language ‘printf()’-style escape
sequences.

   Change the value of the ‘long_name’ attribute for variable ‘T’ from
whatever it currently is to "temperature":
     ncatted -a long_name,T,o,c,temperature in.nc

   Many model and observational datasets use missing values that are not
annotated in the standard manner.  For example, at the time (2015-2018)
of this writing, the MPAS ocean and ice models use
-9.99999979021476795361e+33 as the missing value, yet do not store a
‘_FillValue’ attribute with any variables.  To prevent arithmetic from
treating these values as normal, designate this value as the
‘_FillValue’ attribute:
     ncatted    -a _FillValue,,o,d,-9.99999979021476795361e+33 in.nc
     ncatted -t -a _FillValue,,o,d,-9.99999979021476795361e+33 in.nc
     ncatted -t -a _FillValue,,o,d,-9.99999979021476795361e+33 \
                -a _FillValue,,o,f,1.0e36 -a _FillValue,,o,i,-999 in.nc
   The first example adds the attribute to all variables.  The ‘-t’
switch causes the second example to add the attribute only to double
precision variables.  This is often more useful, and can be used to
provide distinct missing value attributes to each numeric type, as in
the third example.

   NCO arithmetic operators may not work as expected on IEEE NaN (short
for Not-a-Number) and NaN-like numbers such as positive infinity and
negative infinity (1).  One way to work-around this problem is to change
IEEE NaNs to normal missing values.  As of NCO 4.1.0 (March, 2012),
‘ncatted’ works with NaNs (though none of NCO's arithmetic operators
do).  This limited support enables users to change NaN to a normal
number before performing arithmetic or propagating a NaN-tainted
dataset.  First set the missing value (i.e., the value of the
‘_FillValue’ attribute) for the variable(s) in question to the IEEE NaN
value.
     ncatted -a _FillValue,,o,f,NaN in.nc
   Then change the missing value from the IEEE NaN value to a normal
IEEE number, like 1.0e36 (or to whatever the original missing value
was).
     ncatted -a _FillValue,,m,f,1.0e36 in.nc
   Some NASA MODIS datasets provide a real-world example.
     ncatted -O -a _FillValue,,m,d,1.0e36 -a missing_value,,m,d,1.0e36 \
             MODIS_L2N_20140304T1120.nc MODIS_L2N_20140304T1120_noNaN.nc

   Delete all existing ‘units’ attributes:
     ncatted -a units,,d,, in.nc
The value of VAR_NM was left blank in order to select all variables in
the file.  The values of ATT_TYPE and ATT_VAL were left blank because
they are superfluous in “Delete” mode.

   Delete all attributes associated with the ‘tpt’ variable, and delete
all global attributes
     ncatted -a ,tpt,d,, -a ,global,d,, in.nc
The value of ATT_NM was left blank in order to select all attributes
associated with the variable.  To delete all global attributes, simply
replace ‘tpt’ with ‘global’ in the above.

   Modify all existing ‘units’ attributes to ‘meter second-1’:
     ncatted -a units,,m,c,'meter second-1' in.nc

   Add a ‘units’ attribute of ‘kilogram kilogram-1’ to all variables
whose first three characters are ‘H2O’:
     ncatted -a units,'^H2O',c,c,'kilogram kilogram-1' in.nc

   Remove the ‘_FillValue’ attribute from ‘lat’ and ‘lon’ variables.
     ncatted -O -a _FillValue,'[lat]|[lon]',d,, in.nc

   Overwrite the ‘quanta’ attribute of variable ‘energy’ to an array of
four integers.
     ncatted -a quanta,energy,o,s,'010,101,111,121' in.nc

   As of NCO 3.9.6 (January, 2009), ‘ncatted’ accepts “extended regular
expressions” as arguments for variable names, and, since NCO 4.5.1
(July, 2015), for attribute names.
     ncatted -a isotope,'^H2O*',c,s,'18' in.nc
     ncatted -a '.?_iso19115$','^H2O*',d,, in.nc
   The first example creates ‘isotope’ attributes for all variables
whose names contain ‘H2O’.  The second deletes all attributes whose
names end in ‘_iso19115’ from all variables whose names contain ‘H2O’.
See *note Subsetting Files:: for more details on using regular
expressions.

   As of NCO 4.3.8 (November, 2013), ‘ncatted’ accepts full and partial
group paths in names of attributes, variables, dimensions, and groups.
     # Overwrite units attribute of specific 'lon' variable
     ncatted -O -a units,/g1/lon,o,c,'degrees_west' in_grp.nc
     # Overwrite units attribute of all 'lon' variables
     ncatted -O -a units,lon,o,c,'degrees_west' in_grp.nc
     # Delete units attribute of all 'lon' variables
     ncatted -O -a units,lon,d,, in_grp.nc
     # Overwrite units attribute with new type for specific 'lon' variable
     ncatted -O -a units,/g1/lon,o,sng,'degrees_west' in_grp.nc
     # Add new_att attribute to all variables
     ncatted -O -a new_att,,c,sng,'new variable attribute' in_grp.nc
     # Add new_grp_att group attribute to all groups
     ncatted -O -a new_grp_att,group,c,sng,'new group attribute' in_grp.nc
     # Add new_grp_att group attribute to single group
     ncatted -O -a g1_grp_att,g1,c,sng,'new group attribute' in_grp.nc
     # Add new_glb_att global attribute to root group
     ncatted -O -a new_glb_att,global,c,sng,'new global attribute' in_grp.nc
   Note that regular expressions work well in conjuction with group path
support.  In other words, the variable name (including group path
component) and the attribute names may both be extended regular
expressions.

   Demonstrate input of C-language escape sequences (e.g., ‘\n’) and
other special characters (e.g., ‘\"’)
     ncatted -h -a special,global,o,c,
     '\nDouble quote: \"\nTwo consecutive double quotes: \"\"\n
     Single quote: Beyond my shell abilities!\nBackslash: \\\n
     Two consecutive backslashes: \\\\\nQuestion mark: \?\n' in.nc
   Note that the entire attribute is protected from the shell by single
quotes.  These outer single quotes are necessary for interactive use,
but may be omitted in batch scripts.

   Although ‘ncatted’ accepts multiple ‘-a ATT_DST’ options
simultaneously, modifying lengthy commands can become unwieldy.  To
preserve simplicity in storing/modifying multiple attribute edits,
consider storing the options separately in a text file and assembling
them at run-time to generate and submit the correct command.  One such
method uses the ‘xargs’ command to intermediate between an on-disk list
attributes to change and the ‘ncatted’ command.  For example, use an
intermediate file named ‘options.txt’ to store one option per line
thusly
     cat > opt.txt << EOF
     -a institution,global,m,c,\"Super Cool University\"
     -a source,global,c,c,\"My Awesome Radar\"
     -a contributors,global,c,c,\"Enrico Fermi, Galileo Galilei, Leonardo Da Vinci\"
     ...
     EOF
   The backslashes preserve the whitespace in the individual attributes
for correct parsing by the shell.  Simply substituting the expansion of
this file through ‘xargs’ directly on the command line fails to work
(why?).  However, a simple workaround is to use ‘xargs’ to construct the
command string, and execute that string with ‘eval’:
     opt=$(cat opt.txt | xargs)
     cmd="ncatted -O ${opt} in.nc out.nc"
     eval $cmd
   This procedure can by modified to employ more complex option
pre-processing using other tools such as Awk, Perl, or Python.

   ---------- Footnotes ----------

   (1) NaN is a special floating point value (not a string).  Arithmetic
comparisons to NaN and NaN-like numbers always return False, contrary to
the behavior of all other numbers.  This behavior is difficult to
intuit, yet IEEE 754 mandates it.  To correctly handle NaNs during
arithmetic, code must use special math library macros (e.g.,
‘isnormal()’) to determine whether any operand requires special
treatment.  If so, additional logic must be added to correctly perform
the arithmetic.  This is in addition to the normal handling incurred to
correctly handle missing values.  Handling field and missing values
(either or both of which may be NaN) in binary operators thus incurs
four-to-eight extra code paths.  Each code path slows down arithmetic
relative to normal numbers.  This makes supporting NaN arithmetic costly
and inefficient.  Hence NCO supports NaN only to the extent necessary to
replace it with a normal number.  Although using NaN for the missing
value (or any value) in datasets is legal in netCDF, we discourage it.
We recommend avoiding NaN entirely.


File: nco.info,  Node: ncbo netCDF Binary Operator,  Next: ncchecker netCDF Compliance Checker,  Prev: ncatted netCDF Attribute Editor,  Up: Reference Manual

4.3 ‘ncbo’ netCDF Binary Operator
=================================

SYNTAX
     ncbo [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c] [--cmp CMP_SNG]
     [--cnk_byt SZ_BYT] [--cnk_csh SZ_BYT] [--cnk_dmn NM,SZ_LMN]
     [--cnk_map MAP] [--cnk_min SZ_BYT] [--cnk_plc PLC] [--cnk_scl SZ_LMN]
     [-D DBG] [-d DIM,[MIN][,[MAX][,[STRIDE]]] [-F] [--fl_fmt FL_FMT]
     [-G GPE_DSC] [-g GRP[,...]] [--gaa ...] [--gad ATT[,...]]
     [-H] [-h] [--hdr_pad NBR] [--hpss]
     [-L DFL_LVL] [-l PATH] [--no_cll_msr] [--no_frm_trm] [--no_tmp_fl]
     [-O] [-o FILE_3] [-p PATH] [--qnt ...] [--qnt_alg ALG_NM]
     [-R] [-r] [--ram_all] [-t THR_NBR] [--unn] [-v VAR[,...]]
     [-X ...] [-x] [-y OP_TYP]
     FILE_1 FILE_2 [FILE_3]

DESCRIPTION

   ‘ncbo’ performs binary operations on variables in FILE_1 and the
corresponding variables (those with the same name) in FILE_2 and stores
the results in FILE_3.  The binary operation operates on the entire
files (modulo any excluded variables).  *Note Missing Values::, for
treatment of missing values.  One of the four standard arithmetic binary
operations currently supported must be selected with the ‘-y OP_TYP’
switch (or long options ‘--op_typ’ or ‘--operation’).  The valid binary
operations for ‘ncbo’, their definitions, corresponding values of the
OP_TYP key, and alternate invocations are:
“Addition”
     Definition: FILE_3 = FILE_1 + FILE_2
     Alternate invocation: ‘ncadd’
     OP_TYP key values: ‘add’, ‘+’, ‘addition’
     Examples: ‘ncbo --op_typ=add 1.nc 2.nc 3.nc’, ‘ncadd 1.nc 2.nc
     3.nc’
“Subtraction”
     Definition: FILE_3 = FILE_1 - FILE_2
     Alternate invocations: ‘ncdiff’, ‘ncsub’, ‘ncsubtract’
     OP_TYP key values: ‘sbt’, ‘-’, ‘dff’, ‘diff’, ‘sub’, ‘subtract’,
     ‘subtraction’
     Examples: ‘ncbo --op_typ=- 1.nc 2.nc 3.nc’, ‘ncdiff 1.nc 2.nc 3.nc’
“Multiplication”
     Definition: FILE_3 = FILE_1 * FILE_2
     Alternate invocations: ‘ncmult’, ‘ncmultiply’
     OP_TYP key values: ‘mlt’, ‘*’, ‘mult’, ‘multiply’, ‘multiplication’
     Examples: ‘ncbo --op_typ=mlt 1.nc 2.nc 3.nc’, ‘ncmult 1.nc 2.nc
     3.nc’
“Division”
     Definition: FILE_3 = FILE_1 / FILE_2
     Alternate invocation: ‘ncdivide’
     OP_TYP key values: ‘dvd’, ‘/’, ‘divide’, ‘division’
     Examples: ‘ncbo --op_typ=/ 1.nc 2.nc 3.nc’, ‘ncdivide 1.nc 2.nc
     3.nc’
Care should be taken when using the shortest form of key values, i.e.,
‘+’, ‘-’, ‘*’, and ‘/’.  Some of these single characters may have
special meanings to the shell (1).  Place these characters inside quotes
to keep them from being interpreted (globbed) by the shell (2).  For
example, the following commands are equivalent
     ncbo --op_typ=* 1.nc 2.nc 3.nc # Dangerous (shell may try to glob)
     ncbo --op_typ='*' 1.nc 2.nc 3.nc # Safe ('*' protected from shell)
     ncbo --op_typ="*" 1.nc 2.nc 3.nc # Safe ('*' protected from shell)
     ncbo --op_typ=mlt 1.nc 2.nc 3.nc
     ncbo --op_typ=mult 1.nc 2.nc 3.nc
     ncbo --op_typ=multiply 1.nc 2.nc 3.nc
     ncbo --op_typ=multiplication 1.nc 2.nc 3.nc
     ncmult 1.nc 2.nc 3.nc # First do 'ln -s ncbo ncmult'
     ncmultiply 1.nc 2.nc 3.nc # First do 'ln -s ncbo ncmultiply'
   No particular argument or invocation form is preferred.  Users are
encouraged to use the forms which are most intuitive to them.

   Normally, ‘ncbo’ will fail unless an operation type is specified with
‘-y’ (equivalent to ‘--op_typ’).  You may create exceptions to this rule
to suit your particular tastes, in conformance with your site's policy
on “symbolic links” to executables (files of a different name point to
the actual executable).  For many years, ‘ncdiff’ was the main binary
file operator.  As a result, many users prefer to continue invoking
‘ncdiff’ rather than memorizing a new command (‘ncbo -y SBT’) which
behaves identically to the original ‘ncdiff’ command.  However, from a
software maintenance standpoint, maintaining a distinct executable for
each binary operation (e.g., ‘ncadd’) is untenable, and a single
executable, ‘ncbo’, is desirable.  To maintain backward compatibility,
therefore, NCO automatically creates a symbolic link from ‘ncbo’ to
‘ncdiff’.  Thus ‘ncdiff’ is called an “alternate invocation” of ‘ncbo’.
‘ncbo’ supports many additional alternate invocations which must be
manually activated.  Should users or system adminitrators decide to
activate them, the procedure is simple.  For example, to use ‘ncadd’
instead of ‘ncbo --op_typ=add’, simply create a symbolic link from
‘ncbo’ to ‘ncadd’ (3).  The alternatate invocations supported for each
operation type are listed above.  Alternatively, users may always define
‘ncadd’ as an “alias” to ‘ncbo --op_typ=add’ (4).

   It is important to maintain portability in NCO scripts.  Therefore we
recommend that site-specfic invocations (e.g., ‘ncadd’) be used only in
interactive sessions from the command-line.  For scripts, we recommend
using the full invocation (e.g., ‘ncbo --op_typ=add’).  This ensures
portability of scripts between users and sites.

   ‘ncbo’ operates (e.g., adds) variables in FILE_2 with the
corresponding variables (those with the same name) in FILE_1 and stores
the results in FILE_3.  Variables in FILE_1 or FILE_2 are “broadcast” to
conform to the corresponding variable in the other input file if
necessary(5).  Now ‘ncbo’ is completely symmetric with respect to FILE_1
and FILE_2, i.e., FILE_1 - FILE_2 = - (FILE_2 - FILE_1.

   Broadcasting a variable means creating data in non-existing
dimensions by copying data in existing dimensions.  For example, a two
dimensional variable in FILE_2 can be subtracted from a four, three, or
two (not one or zero) dimensional variable (of the same name) in
‘file_1’.  This functionality allows the user to compute anomalies from
the mean.  In the future, we will broadcast variables in FILE_1, if
necessary to conform to their counterparts in FILE_2.  Thus, presently,
the number of dimensions, or “rank”, of any processed variable in FILE_1
must be greater than or equal to the rank of the same variable in
FILE_2.  Of course, the size of all dimensions common to both FILE_1 and
FILE_2 must be equal.

   When computing anomalies from the mean it is often the case that
FILE_2 was created by applying an averaging operator to a file with
initially the same dimensions as FILE_1 (often FILE_1 itself).  In these
cases, creating FILE_2 with ‘ncra’ rather than ‘ncwa’ will cause the
‘ncbo’ operation to fail.  For concreteness say the record dimension in
‘file_1’ is ‘time’.  If FILE_2 was created by averaging FILE_1 over the
‘time’ dimension with the ‘ncra’ operator (rather than with the ‘ncwa’
operator), then FILE_2 will have a ‘time’ dimension of size 1 rather
than having no ‘time’ dimension at all (6).  In this case the input
files to ‘ncbo’, FILE_1 and FILE_2, will have unequally sized ‘time’
dimensions which causes ‘ncbo’ to fail.  To prevent this from occurring,
use ‘ncwa’ to remove the ‘time’ dimension from FILE_2.  See the example
below.

   ‘ncbo’ never operates on coordinate variables or variables of type
‘NC_CHAR’ or ‘NC_STRING’.  This ensures that coordinates like (e.g.,
latitude and longitude) are physically meaningful in the output file,
FILE_3.  This behavior is hardcoded.  ‘ncbo’ applies special rules to
some CF-defined (and/or NCAR CCSM or NCAR CCM fields) such as ‘ORO’.
See *note CF Conventions:: for a complete description.  Finally, we note
that ‘ncflint’ (*note ncflint netCDF File Interpolator::) is designed
for file interpolation.  As such, it also performs file subtraction,
addition, multiplication, albeit in a more convoluted way than ‘ncbo’.

   Beginning with NCO version 4.3.1 (May, 2013), ‘ncbo’ supports “group
broadcasting”.  Group broadcasting means processing data based on group
patterns in the input file(s) and automatically transferring or
transforming groups to the output file.  Consider the case where FILE_1
contains multiple groups each with the variable V1, while FILE_2
contains V1 only in its top-level (i.e., root) group.  Then ‘ncbo’ will
replicate the group structure of FILE_1 in the output file, FILE_3.
Each group in FILE_3 contains the output of the corresponding group in
FILE_1 operating on the data in the single group in FILE_2.  An example
is provided below.

EXAMPLES

   Say files ‘85_0112.nc’ and ‘86_0112.nc’ each contain 12 months of
data.  Compute the change in the monthly averages from 1985 to 1986:
     ncbo   86_0112.nc 85_0112.nc 86m85_0112.nc
     ncdiff 86_0112.nc 85_0112.nc 86m85_0112.nc
     ncbo --op_typ=sub 86_0112.nc 85_0112.nc 86m85_0112.nc
     ncbo --op_typ='-' 86_0112.nc 85_0112.nc 86m85_0112.nc
These commands are all different ways of expressing the same thing.

   The following examples demonstrate the broadcasting feature of
‘ncbo’.  Say we wish to compute the monthly anomalies of ‘T’ from the
yearly average of ‘T’ for the year 1985.  First we create the 1985
average from the monthly data, which is stored with the record dimension
‘time’.
     ncra 85_0112.nc 85.nc
     ncwa -O -a time 85.nc 85.nc
The second command, ‘ncwa’, gets rid of the ‘time’ dimension of size 1
that ‘ncra’ left in ‘85.nc’.  Now none of the variables in ‘85.nc’ has a
‘time’ dimension.  A quicker way to accomplish this is to use ‘ncwa’
from the beginning:
     ncwa -a time 85_0112.nc 85.nc
We are now ready to use ‘ncbo’ to compute the anomalies for 1985:
     ncdiff -v T 85_0112.nc 85.nc t_anm_85_0112.nc
Each of the 12 records in ‘t_anm_85_0112.nc’ now contains the monthly
deviation of ‘T’ from the annual mean of ‘T’ for each gridpoint.

   Say we wish to compute the monthly gridpoint anomalies from the zonal
annual mean.  A “zonal mean” is a quantity that has been averaged over
the longitudinal (or X) direction.  First we use ‘ncwa’ to average over
longitudinal direction ‘lon’, creating ‘85_x.nc’, the zonal mean of
‘85.nc’.  Then we use ‘ncbo’ to subtract the zonal annual means from the
monthly gridpoint data:
     ncwa -a lon 85.nc 85_x.nc
     ncdiff 85_0112.nc 85_x.nc tx_anm_85_0112.nc
This examples works assuming ‘85_0112.nc’ has dimensions ‘time’ and
‘lon’, and that ‘85_x.nc’ has no ‘time’ or ‘lon’ dimension.

   Group broadcasting simplifies evaluation of multiple models against
observations.  Consider the input file ‘cmip5.nc’ which contains
multiple top-level groups ‘cesm’, ‘ecmwf’, and ‘giss’, each of which
contains the surface air temperature field ‘tas’.  We wish to compare
these models to observations stored in ‘obs.nc’ which contains ‘tas’
only in its top-level (i.e., root) group.  It is often the case that
many models and/or model simulations exist, whereas only one
observational dataset does.  We evaluate the models and obtain the bias
(difference) between models and observations by subtracting ‘obs.nc’
from ‘cmip5.nc’.  Then ‘ncbo’ "broadcasts" (i.e., replicates) the
observational data to match the group structure of ‘cmip5.nc’,
subtracts, and then stores the results in the output file, ‘bias.nc’
which has the same group structure as ‘cmip5.nc’.
     % ncbo -O cmip5.nc obs.nc bias.nc
     % ncks -H -v tas -d time,3 bias.nc
     /cesm/tas
     time[3] tas[3]=-1
     /ecmwf/tas
     time[3] tas[3]=0
     /giss/tas
     time[3] tas[3]=1

   As a final example, say we have five years of monthly data (i.e.,
60 months) stored in ‘8501_8912.nc’ and we wish to create a file which
contains the twelve month seasonal cycle of the average monthly anomaly
from the five-year mean of this data.  The following method is just one
permutation of many which will accomplish the same result.  First use
‘ncwa’ to create the five-year mean:
     ncwa -a time 8501_8912.nc 8589.nc
Next use ‘ncbo’ to create a file containing the difference of each
month's data from the five-year mean:
     ncbo 8501_8912.nc 8589.nc t_anm_8501_8912.nc
Now use ‘ncks’ to group together the five January anomalies in one file,
and use ‘ncra’ to create the average anomaly for all five Januarys.
These commands are embedded in a shell loop so they are repeated for all
twelve months:
     for idx in {1..12}; do # Bash Shell (version 3.0+) 
       idx=`printf "%02d" ${idx}` # Zero-pad to preserve order
       ncks -F -d time,${idx},,12 t_anm_8501_8912.nc foo.${idx}
       ncra foo.${idx} t_anm_8589_${idx}.nc
     done
     for idx in 01 02 03 04 05 06 07 08 09 10 11 12; do # Bourne Shell
       ncks -F -d time,${idx},,12 t_anm_8501_8912.nc foo.${idx}
       ncra foo.${idx} t_anm_8589_${idx}.nc
     done
     foreach idx (01 02 03 04 05 06 07 08 09 10 11 12) # C Shell
       ncks -F -d time,${idx},,12 t_anm_8501_8912.nc foo.${idx}
       ncra foo.${idx} t_anm_8589_${idx}.nc
     end
Note that ‘ncra’ understands the ‘stride’ argument so the two commands
inside the loop may be combined into the single command
     ncra -F -d time,${idx},,12 t_anm_8501_8912.nc foo.${idx}
Finally, use ‘ncrcat’ to concatenate the 12 average monthly anomaly
files into one twelve-record file which contains the entire seasonal
cycle of the monthly anomalies:
     ncrcat t_anm_8589_??.nc t_anm_8589_0112.nc

   ---------- Footnotes ----------

   (1) A naked (i.e., unprotected or unquoted) ‘*’ is a wildcard
character.  A naked ‘-’ may confuse the command line parser.  A naked
‘+’ and ‘/’ are relatively harmless.

   (2) The widely used shell Bash correctly interprets all these special
characters even when they are not quoted.  That is, Bash does not
prevent NCO from correctly interpreting the intended arithmetic
operation when the following arguments are given (without quotes) to
‘ncbo’: ‘--op_typ=+’, ‘--op_typ=-’, ‘--op_typ=*’, and ‘--op_typ=/’

   (3) The command to do this is ‘ln -s -f ncbo ncadd’

   (4) The command to do this is ‘alias ncadd='ncbo --op_typ=add'’

   (5) Prior to NCO version 4.3.1 (May, 2013), ‘ncbo’ would only
broadcast variables in FILE_2 to conform to FILE_1.  Variables in FILE_1
were _never_ broadcast to conform to the dimensions in FILE_2.

   (6) This is because ‘ncra’ collapses the record dimension to a size
of 1 (making it a “degenerate” dimension), but does not remove it,
while, unless ‘-b’ is given, ‘ncwa’ removes all averaged dimensions.  In
other words, by default ‘ncra’ changes variable size though not rank,
while, ‘ncwa’ changes both variable size and rank.


File: nco.info,  Node: ncchecker netCDF Compliance Checker,  Next: ncclimo netCDF Climatology Generator,  Prev: ncbo netCDF Binary Operator,  Up: Reference Manual

4.4 ‘ncchecker’ netCDF Compliance Checker
=========================================

SYNTAX
     ncchecker
     [-D DBG] [-i DRC_IN]
     [--tests=TST_LST]
     [-x] [-v VAR[,...]] [--version]
     [INPUT-FILES]

DESCRIPTION

   As of version 5.2.2 (March, 2024), NCO comes with the ‘ncchecker’
script.  This command checks files for compliance with best practices
rules and recommendations from various data and metadata standards
bodies.  These include the Climate & Forecast (CF) Metadata Conventions
and the NASA Dataset Interoperability Working Group (DIWG)
recommendations.  Only a small subset (six tests) of CF or DIWG
recommendations are currently supported.  The number of tests
implemented, or, equivalently, of recommendations checked, is expected
to grow.

   ‘ncchecker’ reads each data file in INPUT-FILES, in DRC_IN, or piped
through standard input.  It performs the checks requested in the
‘--tests=TST_LST’ option, if any (otherwise it performs all tests), and
writes the results to ‘stdout’.  The command supports some standard NCO
options, including increasing the verbosity level with ‘-D DBG_LVL’,
excluding variables with ‘-x -v VAR_LST’, variable subsetting with ‘-v
VAR_LST’, and printing the version with ‘--version’.  The output
contains counts of the location and number of failed tests, or prints
"SUCCESS" for tests with no failures.

EXAMPLES

     ncchecker in1.nc in2.nc # Run all tests on two files
     ncchecker -v var1,var2 in1.nc # Check only two variables
     ncchecker *.nc # Glob input files via wildcard
     ls *.nc | ncchecker # Input files via stdin
     ncchecker --dbg=2 *.nc # Debug ncchecker
     ncchecker --tests=nan,mss *.nc # Select only two tests
     ncchecker --tests=xtn,tm,nan,mss,chr,bnd *.nc # Change test ordering
     ncchecker file:///Users/zender/in_zarr4#mode=nczarr,file # Check Zarr object(s)


File: nco.info,  Node: ncclimo netCDF Climatology Generator,  Next: ncecat netCDF Ensemble Concatenator,  Prev: ncchecker netCDF Compliance Checker,  Up: Reference Manual

4.5 ‘ncclimo’ netCDF Climatology Generator
==========================================

SYNTAX
     ncclimo [-3] [-4] [-5] [-6] [-7]
     [-a WNT_MD] [-C CLM_MD] [-c CASEID] [--cmp CMP_SNG]
     [-d DBG_LVL] [--d2f] [--dpf=DPF] [--dpt_fl=DPT_FL] [-E YR_PRV] [-e YR_END]
     [-f FML_NM] [--fl_fmt=FL_FMT] [--glb_avg] [--glb_stt=GLB_STT]
     [-h HST_NM] [-i DRC_IN] [-j JOB_NBR] [-L DFL_LVL] [-l LNK_FLG]
     [-m MDL_NM] [--mth_end=MTH_END] [--mth_srt=MTH_SRT]
     [-n NCO_OPT] [--npo] [--no_cll_msr] [--no_frm_trm] [--no_ntv_tms] [--no_stg_grd] [--no_stdin]
     [-O DRC_RGR] [-o DRC_OUT] [-P PRC_TYP] [-p PAR_TYP] [--qnt=QNT_PRC]
     [-R RGR_OPT] [-r RGR_MAP] [-S YR_PRV] [-s YR_SRT]
     [--seasons=CSN_LST] [--sgs_frc=SGS_FRC] [--split] [--sum_scl=SUM_SCL]
     [-t THR_NBR] [--tpd=TPD] [--uio] [-v VAR_LST] [--var_xtr=VAR_XTR] [--version]
     [--vrt_out=VRT_FL] [--vrt_xtr=VRT_XTR]
     [-X DRC_XTN] [-x DRC_PRV] [--xcl_var]
     [-Y RGR_XTN] [-y RGR_PRV] [--ypf=YPF_MAX]
     [INPUT-FILES]

DESCRIPTION

   In climatology generation mode, ‘ncclimo’ ingests "raw" data
consisting of interannual sets of files, each containing sub-daily
(diurnal), daily, monthly, or yearly averages, and from these produces
climatological daily, monthly, seasonal, and/or annual means.
Alternatively, in timeseries reshaping (aka "splitter") mode, ‘ncclimo’
will subset and temporally split the input raw data timeseries into
per-variable files spanning the entire period.  ‘ncclimo’ can optionally
(call ‘ncremap’ to) regrid all output files in either mode.  Unlike the
rest of NCO, ‘ncclimo’ and ‘ncremap’ are shell scripts, not compiled
binaries(1).  As of NCO 4.9.2 (February, 2020), the ‘ncclimo’ and
‘ncremap’ scripts export the environment variable
‘HDF5_USE_FILE_LOCKING’ with a value of ‘FALSE’.  This prevents failures
of these operators that can occur with some versions of the underlying
HDF library that attempt to lock files on file systems that cannot, or
do not, support it.

   There are five (usually) required options (‘-c’, ‘-s’, ‘-e’, ‘-i’,
and ‘-o’)) to generate climatologies, and many more options are
available to customize the processing.  Options are similar to ‘ncremap’
options.  Standard ‘ncclimo’ usage for climatology generation looks like
     ncclimo            -c caseid -s srt_yr -e end_yr -i drc_in -o drc_out
     ncclimo -m mdl_nm  -c caseid -s srt_yr -e end_yr -i drc_in -o drc_out
     ncclimo -v var_lst -c caseid -s srt_yr -e end_yr -i drc_in -o drc_out
     ncclimo --case=caseid --start=srt_yr --end=end_yr --input=drc_in --output=drc_out
   In climatology generation mode, ‘ncclimo’ constructs the list of
input filenames from the arguments to the caseid, date, and model-type
options.  As of NCO version 4.9.4 (September, 2020), ‘ncclimo’ can
produce climatologies of high-frequency input data supplied via standard
input, positional command-line options, or directory contents, all input
methods traditionally supported only in splitter mode.  Instead of using
the ‘caseid’ option to help generate the input filenames as it does for
normal (monthly) climos, ‘ncclimo’ uses the ‘caseid’ option, when
provided, to rename the output files for high-frequency climos.
     # Generate diurnal climos from high-frequency CMIP6 timeseries
     cd ${drc_in};ls ${caseid}*.h4.nc | ncclimo --clm_md=hfc \
       -c ${caseid} --yr_srt=2001 --yr_end=2002 --drc_out=${HOME}

   ‘ncclimo’ automatically switches to timeseries reshaping mode if it
receives a list of files from ‘stdin’, or, alternatively, placed as
positional arguments (after the last command-line option), or if neither
of these is done and no CASEID is specified, in which case it assumes
all ‘*.nc’ files in DRC_IN constitute the input file list.
     # Split monthly timeseries into CMIP-like timeseries
     cd ${drc_in};ls ${caseid}*.h4.nc | ncclimo              -v=T \
       --ypf=1 --yr_srt=56 --yr_end=76 --drc_out=${HOME}
     # Split high-frequency timeseries into CMIP-like timeseries
     cd ${drc_in};ls ${caseid}*.h4.nc | ncclimo --clm_md=hfs -v=T \
       --ypf=1 --yr_srt=56 --yr_end=76 --drc_out=${HOME}

   Options for ‘ncclimo’ and ‘ncremap’ come in both short
(single-letter) and long forms.  The handful of long-option synonyms for
each option allows the user to imbue the commands with a level of
verbosity and precision that suits her taste.  A complete description of
all options is given below, in alphabetical order of the short option
letter.  Long option synonyms are given just after the letter.  When
invoked without options, ‘ncclimo’ and ‘ncremap’ print a succinct table
of all options and some examples.  All valid options for both operators
are listed in their command syntax above but, for brevity, options that
‘ncclimo’ passes straight through to ‘ncremap’ are only fully described
in the table of ‘ncremap’ options.
‘-a DEC_MD (--dec_md, --dcm_md, --december_mode, --dec_mode)’
     Winter mode aka December mode determines the start and end months
     of the climatology and the type of NH winter seasonal average.  Two
     valid arguments are ‘jfd’ (default, or synonyms ‘sdd’ and ‘JFD’)
     and ‘djf’ (or synonyms ‘scd’ and ‘DJF’).  DJF-mode is the same as
     SCD-mode which stands for "Seasonally Continuous December".  The
     first month used is December of the year before the start year
     specified with ‘-s’.  The last month is November of the end year
     specified with ‘-e’.  In DJF-mode the Northern Hemisphere winter
     seasonal climatology will be computed with sets of the three
     consecutive months December, January, and February (DJF) where the
     calendar year of the December months is always one less than the
     calendar year of January and February.  JFD-mode is the same as
     SDD-mode which stands for "Seasonally Discontinuous December".  The
     first month used is January of the specified start year.  The last
     month is December of the end year specified with ‘-e’.  In JFD-mode
     the Northern Hemisphere winter seasonal climatology will be
     computed with sets of the three non-consecutive months January,
     February, and December (JFD) from each calendar year.

‘-C CLM_MD (--clm_md, --climatology_mode, --mode, --climatology)’
     Climatology mode.  Valid values for CLM_MD are ‘ann’ (or synonyms
     ‘annual’, ‘annual’, ‘yearly’, or ‘year’) for annual-mode
     climatologies, ‘dly’ (or synonyms ‘daily’, ‘doy’, or ‘day’) for
     daily-mode climatologies, ‘hfc’ (or synonyms ‘high_frequency_climo’
     or ‘hgh_frq_clm’) for high-frequency (diurnally resolved) climos,
     ‘hfs’ (or synonyms ‘high_frequency_splitter’ or ‘hgh_frq_spl’) for
     high-frequency splitting, and ‘mth’ (or synonyms ‘month’ or
     ‘monthly’) for monthly climotologies.  The value indicates the
     timespan of each input file for annual and monthly climatologies.
     The default mode is ‘mth’, which means input files are monthly
     averages.  Use ‘ann’ when the input files are a series of annual
     means (a common temporal resolution for ice-sheet simulations).
     The value ‘dly’ is used only input files whose temporal resolution
     is daily or finer, and when the desired output is a day-of-year
     climatology where the means are output for each day of a 365 day
     year.  Day-of-year climatologies are uncommon, yet useful for
     showing daily variability.  The value ‘hfc’ indicates a
     high-frequency climatology where the output will be a traditional
     set of climatological monthly, seasonal, or annual means similar to
     monthly climos, except that each file will have the same number of
     timesteps-per-day as the input data to resolve the diurnal cycle.
     The value ‘hfs’ indicates a high-frequency splitting operation
     where an interannual input timeseries will be split into regular
     size segments of a given number of years, similar to CMIP
     timeseries.

     The climatology generator and splitter do not require that
     daily-mode input files begin or end on daily boundaries.  These
     tools hyperslab the input files using the date information required
     to performed their analysis.  This facilitates analyzing datasets
     with varying numbers of days per input file.

     Explicitly specifying ‘--clm_md=mth’ serves a secondary purpose,
     namely invoking the default setting on systems that control
     ‘stdin’.  When ‘ncclimo’ detects that ‘stdin’ is not attached to
     the terminal (keyboard) it automatically expects a list of files on
     ‘stdin’.  Some environments, however, hijack ‘stdin’ for their
     purposes and thereby confuse ‘ncclimo’ into expecting a list
     argument.  Users have encountered this issue when attempting to run
     ‘ncclimo’ in Python parallel environments, via inclusion in
     ‘crontab’, and in ‘nohup’-mode (whatever that is!).  In such cases,
     explicitly specify ‘--clm_md=mth’ (or ‘ann’ or ‘day’) to persuade
     ‘ncclimo’ to compute a normal climatology.

‘-c CASEID (--case, --caseid, --case_id)’
     Simulation name, or any input filename for non-CESM'ish files.  The
     use of CASEID is required in climate generation mode (unless
     equivalent information is provided through other options), where
     CASEID is used to construct both input and output filenames.  For
     CESM'ish input files like
     ‘famipc5_ne30_v0.3_00001.cam.h0.1980-01.nc’, specify ‘-c
     famipc5_ne30_v0.3_00001’.  The ‘.cam.’ and ‘.h0.’ bits are added
     internally to produce the input filenames.  Modify these via the
     ‘-m MDL_NM’ and ‘-h HST_NM’ options if needed.

     For input files named slightly differently than standard CESM'ish
     names, supply the filename (excluding the path component) as the
     CASEID and then ‘ncclimo’ will attempt to parse that by matching to
     a database of regular expressions known to be favored by various
     other datasets.  These expressions are all various formats of dates
     at the end of the filenames, and adhere to the general convention
     PREFIX[.-]YYYY[-]MM[-]DD[-]SSSSS.SUFFIX.  The particular formats
     currently supported, as of NCO version 5.1.6 (May, 2023) are:
     PREFIX‘_YYYYMM’.SUFFIX, PREFIX‘.YYYY-MM’.SUFFIX,
     PREFIX‘.YYYY-MM-01’.SUFFIX, and PREFIX‘.YYYY-MM-01-00000’.SUFFIX.
     For example, input files like ‘merra2_198001.nc’ (i.e., the six
     digits that precede the suffix are YYYYMM-format), specify ‘-c
     merra2_198001.nc’ and the prefix (‘merra2’) will be automatically
     abstracted and used to template and generate all the filenames
     based on the specified YR_SRT and YR_END.
          ncclimo -c merra2_198001.nc --start=1980 --end=1999 --drc_in=${drc}
          ncclimo -c cesm_1980-01.nc --start=1980 --end=1999 --drc_in=${drc}
          ncclimo -c eamxx_1980-01-00000.nc --start=1980 --end=1999 --drc_in=${drc}
     Please tell us any common dataset filename regular expressions that
     you would like added to ‘ncclimo’'s internal database.

     The ‘--caseid=CASEID’ option is not mandatory in the
     High-Frequency-Splitter (CLM_MD=‘hfs’) and
     High-Frequency-Climatology (CLM_MD=‘hfc’) modes.  Those modes
     expect all input filenames to be entered from the command-line so
     there is no internal need to create filenames from the CASEID
     variable.  Instead, when CASEID is specified in a high-freqency
     mode, its value is used to name the output files in a similar
     manner to the ‘-f FML_NM’ option.

‘-D DBG_LVL (--dbg_lvl, --dbg, --debug, --debug_level)’
     Specifies a debugging level similar to the rest of NCO.  If DBG_LVL
     = 1, ‘ncclimo’ prints more extensive diagnostics of its behavior.
     If DBG_LVL = 2, ‘ncclimo’ prints the commands it would execute at
     any higher or lower debugging level, but does not execute these
     commands.  If DBG_LVL > 2, ‘ncclimo’ prints the diagnostic
     information, executes all commands, and passes-through the
     debugging level to the regridder (‘ncks’) for additional
     diagnostics.

‘--d2f (--d2f, --d2s, --dbl_flt, --dbl_sgl, --double_float)’
     This switch (which takes no argument) causes ‘ncclimo’ to invoke
     ‘ncremap’ with the same switch, so that ‘ncremap’ converts all
     double precision non-coordinate variables to single precision in
     the regridded file.  This switch has no effect on files that are
     not regridded.  To demote the precision in such files, use ‘ncpdq’
     to apply the ‘dbl_flt’ packing map to the file directly.

‘--dpf=DPF (--dpf, --days_per_file)’
     The number of days-per-file in files ingested by ‘ncclimo’.  It can
     sometimes be difficult for ‘ncclimo’ to infer the number of
     days-per-file in high-frequency input files, i.e., those with 1 or
     more timesteps-per-day.  In such cases, users may override the
     inferred value by explicitly specifying ‘--dpf=DPF’.

‘--dpt_fl=DPT_FL (--dpt_fl, --depth_file, --mpas_fl, --mpas_depth)’
     The ‘--dpt_fl=DPT_FL’ triggers the addition of a depth coordinate
     to MPAS ocean datasets that will undergo regridding.  ‘ncclimo’
     passes this option through to ‘ncremap’, and this option has no
     effect when ‘ncclimo’ does not invoke ‘ncremap’.  The ‘ncremap’
     documentation contains the full description of this option.

‘-e END_YR (--end_yr, --yr_end, --end_year, --year_end, --end)’
     End year (example: 2000).  By default, the last month is December
     of the specified end year.  If ‘-a scd’ is specified, the last
     month used is November of the specified end year.

‘-f FML_NM (--fml_nm, --fml, --family, --family_name)’
     Family name (nickname) of output files.  In climate generation
     mode, output climo file names are constructed by default with the
     same CASEID as the input files.  The FML_NM, if supplied, replaces
     CASEID in output climo names, which are of the form
     ‘FML_NM_XX_YYYYMM_YYYYMM.nc’ where XX is the month or seasonal
     abbreviation.  Use ‘-f FML_NM’ to simplify long names, avoid
     overlap, etc.  Example values of FML_NM are ‘control’,
     ‘experiment’, and (for a single-variable climo) ‘FSNT’.  In
     timeseries reshaping mode, FML_NM will be used, if supplied, as an
     additional string in the output filename.  For example, specifying
     ‘-f control’ would cause ‘T_000101_000912.nc’ to be instead named
     ‘T_control_000101_000912.nc’.

‘-h HST_NM (--hst_nm, --history_name, --history)’
     History volume name of file used to generate climatologies.  This
     referring to the HST_NM character sequence used to construct input
     file names: ‘caseid.mdl_nm.’HST_NM‘.YYYY-MM.nc’.  By default input
     climo file names are constructed from the CASEID of the input
     files, together with the model name MDL_NM (specified with ‘-m’)
     and the date range.  Use ‘-h HST_NM’ to specify alternative history
     volumes.  Examples include ‘h0’ (default, works for CAM,
     CLM/CTSM/ELM), ‘h1’, and ‘h’ (for CISM).

‘-i DRC_IN (--drc_in, --in_drc, --dir_in, --input)’
     Directory containing all monthly mean files to read as input to the
     climatology.  The use of DRC_IN is mandatory in climate generation
     mode and is optional in timeseries reshaping mode.  In timeseries
     reshaping mode, ‘ncclimo’ uses all netCDF files (meaning files with
     suffixes ‘.nc’, ‘.nc3’, ‘.nc4’, ‘.nc5’, ‘.nc6’, ‘.nc7’, ‘.cdf’,
     ‘.hdf’, ‘.he5’, or ‘.h5’) in DRC_IN to create the list of input
     files when no list is provided through ‘stdin’ or as positional
     arguments to the command-line.

‘-j JOB_NBR (--job_nbr, --job_number, --jobs)’
     The JOB_NBR parameter controls the parallelism granularity of both
     timeseries reshaping (aka splitting) and climatology generation.
     These modes parallelize over different types of tasks, so we
     describe the effects of JOB_NBR separately, first for
     climatologies, then for splitting.  However, for both modes,
     JOB_NBR specifies the total number of simultaneous processes to run
     in parallel either on the local node for Background parallelism, or
     across all the nodes for MPI parallelism (i.e., JOB_NBR is the
     simultaneous total across all nodes, it is not the simultaneous
     number per node).

     For climatology generation, JOB_NBR specifies the number of
     averaging tasks to perform simultaneously on the local node for
     Background parallelism, or spread across all nodes for
     MPI-parallelism.  By default ‘ncclimo’ sets JOB_NBR = 12 for
     background parallelism mode.  This number ensures that monthly
     averages for all individual months complete more-or-less
     simultaneously, so that all seasonal averages can then be computed.
     However, many nodes are too small to simultaneously average
     multiple distinct months (January, February, etc.).  Hence JOB_NBR
     may be set to any factor of 12, i.e., 1, 2, 3, 4, 6, or 12.  For
     Background parallelism, setting JOB_NBR = 4 causes four-months to
     be averaged at one time.  After three batches of four-months
     complete, the climatology generator then moves on to seasonal
     averaging and regridding.  In MPI-mode, ‘ncclimo’ defaults to
     JOB_NBR = ND_NBR unless the user explicitly sets JOB_NBR to a
     different value.  For the biggest jobs, when a single-month nearly
     exhausts the RAM on a node, this default value JOB_NBR = ND_NBR
     ensures that each node gets only one job at a time.  To override
     this default for MPI-parallelism, set JOB_NBR >= ND_NBR otherwise
     some nodes will be idle for the entire time.  If a node can handle
     averaging three distinct months simultaneously, then try JOB_NBR =
     3*ND_NBR.  Never set JOB_NBR > 12 in climatology mode, since there
     are at most only twelve jobs that can be performed in parallel.

     For splitting, JOB_NBR specifies the number of simultaneous
     subsetting processes to spawn during parallel execution for both
     Background and MPI-parallelism.  In both parallelism modes
     ‘ncclimo’ spawns processes in batches of JOB_NBR jobs, then waits
     for those processes to complete.  Once a batch finishes, ‘ncclimo’
     spawns the next batch.  For Background-parallelism, all jobs are
     spawned to the local node.  For MPI-parallelism, all jobs are
     spawned in round-robin fashion to all available nodes until JOB_NBR
     jobs are running.  Rinse, lather, repeat until all variables have
     been split.  The splitter chooses its default value of JOB_NBR
     based on on the parallelism mode.  For Background parallelism,
     JOB_NBR defaults to the number of variables to be split, so that
     not specifying JOB_NBR results in launching VAR_NBR simultaneous
     splitter tasks.  This scales well to over a hundred variables in
     our tests (2).  In practice, splitting timeseries consumes minimal
     memory, since ‘ncrcat’ (which underlies the splitter) only holds
     one record (timestep) of a variable in memory *note Memory
     Requirements::.

     However, if splitting consumes so much RAM (e.g., because variables
     are large and/or the number of jobs is large) that a single node
     can perform only one or a few subsetting jobs at a time, then it is
     reasonable to use MPI-mode to split the datasets.  For
     MPI-parallelism, JOB_NBR defaults to the number of nodes requested.
     This helps prevent users from overloading nodes with too many jobs.
     Usually, however, nodes can usually subset (and then regrid, if
     requested) multiple variables simultaneously.  In summary, the
     splitter defaults to JOB_NBR = VAR_NBR in Background mode, and to
     JOB_NBR = NODE_NBR in MPI mode.  Subject to the availability of
     adequate RAM, expand the number of jobs-per-node by increasing
     JOB_NBR until overall throughput peaks.

     The main throughput bottleneck in timeseries reshaping mode is I/O.
     Increasing JOB_NBR may reduce throughput once the maximum I/O
     bandwidth of the node is reached, due to contention for I/O
     resources.  Regridding requires math that can relieve some I/O
     contention and allows for some throughput gains with increasing
     JOB_NBR.  One strategy that seems sensible is to set JOB_NBR equal
     to the number of nodes times the number of cores per node, and
     increase or decrease as necessary until throughput peaks.

‘-L (--dfl_lvl, --dfl, --deflate)’
     Activate deflation (i.e., lossless compress, see *note Deflation::)
     with the ‘-L DFL_LVL’ short option (or with the same argument to
     the ‘--dfl_lvl’ or ‘--deflate’ long options).  Specify deflation
     level DFL_LVL on a scale from no deflation (DFL_LVL = 0, the
     default) to maximum deflation (DFL_LVL = 9).

‘-l (--lnk_flg, --link_flag)’
‘--no_amwg_link (--no_amwg_link, --no_amwg_links, --no_amwg, --no_AMWG_link, --no_AMWG_links)’
‘--amwg_link (--amwg_link, --amwg_links, --AMWG_link, --AMWG_links)’
     These options turn-on or turn-off the linking of E3SM/ACME-climo to
     AMWG-climo filenames.  AMWG omits the YYYYMM components of climo
     filenames, resulting in shorter names.  By default ‘ncclimo’
     symbolically links the full E3SM/ACME filename (which is always)
     created to a file with the shorter (AMWG) name whose creation is
     optional.  AMWG diagnostics scripts can produce plots directly from
     the linked AMWG filenames.  The ‘-l’ (and ‘--lnk_flg’ and
     ‘--link_flag’ long-option synonmyms) are true options that require
     an argument of either ‘Yes’ or ‘No’.  The remaining synonyms are
     switches that take no arguments.  The ‘--amwg_link’ switch and its
     synonyms cause the creation of symbolic links with AMWG filenames.
     The ‘--no_amwg_link’ switch and its synonyms prevent the creation
     of symbolic links with AMWG filenames.  If you do not need AMWG
     filenames, turn-off linking to reduce file proliferation in the
     output directories.

‘-m MDL_NM (--mdl_nm, --mdl, --model_name, --model)’
     Model name (as embedded in monthly input filenames).  Default is
     ‘cam’.  Other options are ‘clm2’, ‘ocn’, ‘ice’, ‘cism’, ‘cice’,
     ‘pop’.

‘-n NCO_OPT (nco_opt, nco, nco_options)’
     Specifies a string of options to pass-through unaltered to ‘ncks’.
     NCO_OPT defaults to ‘--no_tmp_fl’.  Note that ‘ncclimo’ passes its
     NCO_OPT to ‘ncremap’.  This can cause unexpected results, so use
     the front-end options to ‘ncclimo’ when possible, rather than
     attempting to subvert them with NCO_OPT.

     As of NCO version 5.3.1 (released January, 2025) users at select
     supercomputer centers can access the latest versions of NCO scripts
     directly from C. Zender's build directories.  These build
     directories usually contain the latest NCO snapshot, and so may be
     unstable and are not for everyday use.  This feature is instead
     intended to allow users to test and provide feedback on new
     features before they are distributed in a public release.  Invoking
     ‘ncclimo’ and ‘ncremap’ with the ‘--npo’ (or long-option equivalent
     ‘--nco_path_override’) flag as _the first option_ (i.e., as
     ‘${1}’), causes these scripts to utilize the binary NCO executables
     in Zender's build directories on the following machines at the
     national supercomputer centers in parentheses: ‘acme1’ (LLNL),
     ‘andes’ (ORNL), ‘chrysalis’ (ANL), ‘compy’ (PNNL), ‘derecho’
     (NCAR), ‘frontier’ (ORNL), and ‘perlmutter’ (NERSC).

          If the latest features that you want are actually implemented in the
          NCO binaries or library, then it may suffice to invoke your
          own version of the scripts, as in the first examples below.
          However, if the latest features that you want are implemented in the
          scripts themselves, then you can either invoke Zender's scripts
          directly (as in the second set of examples), or copy those scripts
          into your own executable path:
          # Access Zender's latest binaries from your default scripts
          ncremap --npo -m map.nc in.nc out.nc 
          ncclimo --npo -P elm -c ${caseid} ...
          # Access Zender's latest binaries from your Zender's latest scripts 
          ~zender/bin/ncremap --npo -m map.nc in.nc out.nc 
          ~zender/bin/ncclimo --npo -P elm -c ${caseid} ...
     In both cases, the ‘--npo’ flag must be the first option invoked.

‘-O DRC_RGR (--drc_rgr, --rgr_drc, --dir_rgr, --regrid)’
     Directory to hold regridded climo files.  Regridded climos are
     placed in DRC_OUT unless a separate directory for them is specified
     with ‘-O’ (NB: capital "O").

‘--no_cll_msr (--no_cll_msr, --no_cll, --no_cell_measures, --no_area)’
     This switch (which takes no argument) controls whether ‘ncclimo’
     and ‘ncremap’ add measures variables to the extraction list along
     with the primary variable and other associated variables.  See
     *note CF Conventions:: for a detailed description.

‘--no_frm_trm (--no_frm_trm, --no_frm, --no_formula_terms)’
     This switch (which takes no argument) controls whether ‘ncclimo’
     and ‘ncremap’ add formula variables to the extraction list along
     with the primary variable and other associated variables.  See
     *note CF Conventions:: for a detailed description.

‘--glb_avg (--glb_avg, --global_average) (deprecated)’
‘--rgn_avg (--rgn_avg, --region_average)’
     When introduced in NCO version 4.9.1 (released December, 2019),
     this switch (which takes no argument) caused the splitter to output
     global horizontally spatially averaged timeseries files instead of
     raw, native-grid timeseries.  This switch changed behavior in NCO
     version 5.1.1 (released November, 2022).  It now causes the
     splitter to output three horizontally spatially averaged
     timeseries.  First is the global average (as before), next is the
     northern hemisphere average, followed by the southern hemisphere
     average.  The three timeseries are now saved in a two-dimensional
     (time by region) array with a "region dimension" named ‘rgn’.  The
     region names are stored in the variable named ‘region_name’.

     As of NCO version 5.2.3 (released March, 2024), this switch works
     with sub-gridscale fractions, such as are common in surface models
     like ELM and CLM.  The correct weights for (SGS) fraction will
     automatically be applied so long as ‘ncclimo’ is invoked with ‘-P
     PRC_TYP’.  Otherwise the field containing the (SGS) fraction must
     be supplied with ‘--sgs_frc=SGS_FRC’.

     This switch only has effect in timeseries splitting mode.  This is
     useful, for example, to quickly diagnose the behavior of ongoing
     model simulations prior to a full-blown analysis.  Thus the spatial
     mean files will be in the same location and have the same name as
     the native grid timeseries would have been and had, respectively.
     Note that this switch does not alter the capability of also
     outputting the full regridded timeseries, if requested, at the same
     time.

     Because the switch now outputs global and regional averages, the
     best practice is to invoke with ‘--rgn_avg’ instead of ‘--glb_avg’.
     NCO version 5.2.8 (released September, 2024) superseded this switch
     by introducing support for more general global/regional statistics
     timeseries output via the ‘--glb_stt’/‘--rgn_stt’ options.

‘--glb_stt (--glb_stt, --global_statistic)’
‘--rgn_stt (--rgn_stt, --region_statistic)’
     NCO version 5.2.8 (released September, 2024) introduced the
     ‘--glb_stt’/‘--rgn_stt’ options (or long options equivalents
     ‘--hms_stt’, ‘--regional_statistic’, and ‘--global_statistic’) to
     support more general global/regional statistics timeseries output.
     These options allow the user to choose which statistic, sums or
     averages, to output with global/regional timeseries for all
     variables.  Set RGN_STT to ‘avg’, ‘average’, or ‘mean’ to output
     timeseries of the global/regional mean statistic.  Set RGN_STT to
     ‘sum’, ‘total’, ‘ttl’, or ‘integral’ to output timeseries of the
     global/regional sum.  The option ‘--rgn_stt=avg’ is equivalent
     setting the ‘--rgn_avg’ switch (which may eventually be
     deprecated).  When invoked with ‘--rgn_stt=sum’ the averaged field
     is multiplied by the sum of the AREA variable.  For area-intensive
     fields (e.g., fluxes per unit area) this results in the total net
     flux over the area.  However, the field must employ the same areal
     units as the AREA variable for this to be true.  For example,
     fields given in inverse square meters would need to employ an AREA
     variable in square meters.  Unfortunately, many people love non-SI
     units so that is rarely the case!  For example, ELM and CLM archive
     AREA in a field named ‘area’ whose units are square kilometers, so
     a scale factor of one million is needed to correct the sum for many
     variables.  EAM and CAM also archive AREA in a field named ‘area’,
     though in unis of inverse steradians, which would require a
     different scale factor to match the sums of area-intensive fields.

     That is why ‘ncclimo’ introduced a second new option
     ‘--sum_scl=SUM_SCL’, in NCO version 5.2.8 (released September,
     2024).  The long option equivalents are ‘--scl_fct’, ‘--sum_scale’,
     and ‘--scale_factor’.  When RGN_STT is ‘sum’, the SUM_SCL scale
     factor multiplies the integrated field value, which allows the user
     to generate timeseries in the desired units for any field.
     Consider these prototypical examples to generate global timeseries
     of common geophysical statistics from ESM output:
          # Timeseries of global GPP in grams/s for ELM/CLM:
          ncclimo -P elm --split --rgn_stt=sum --sum_scl=1.0e6 -v GPP ...
          # Timeseries of global GPP in GT C/yr for ELM/CLM:
          ncclimo -P elm --split --rgn_stt=sum --sum_scl=1.0e6*3600*24*365/1.0e12 -v GPP ...
          # Timeseries of global column vapor in kg for EAM/CAM:
          ncclimo -P eam --split --rgn_stt=sum --sum_scl=6.37122e6^2 -v TMQ ...
     All three examples set RGN_STT to ‘sum’ in order to activate the
     SUM_SCL factor.  The first example scales (multiplies) the mean of
     all global timeseries (here, ‘GPP’ for concreteness) by one
     million.  This factor converts the ELM or CLM ‘area’ variable from
     square kilometers to square meters, appropriate to to integrating
     fields like ‘GPP’ whose fluxes are per square meter.  The output
     timeseries of ‘GPP’ would then be in gC s-1.  The second example
     sets the scale factor to convert the global ‘GPP’ statistic to
     units of GT C yr-1.  The third example shows how to convert areal
     sums in sterradians (which EAM and CAM use for ‘area’) to square
     meters.  This factor converts atmospheric variables from global
     mean mass per square meter to global total mass.  The
     ‘--glb_stt=sum’/‘--sum_scl’ procedure is model- and
     variable-specific and we are open to suggestions to make it more
     useful.

     As of NCO version 5.3.0 (December, 2025) ‘ncclimo’ automatically
     outputs additional metrics with global statistics.  The output
     files containing the global timeseries also contain the variable
     ‘valid_area_per_gridcell’.  This field is equivalent to the product
     of the AREA variable and the SGS_FRC variable (if any).  Thus for
     ELM/CLM/CTSM, this field equals ‘area’ times ‘landfrac’, while for
     EAM/CAM this variable simply equals ‘area’.  The output files also
     contain the AREA and SGS_FRC variables separately.  The presence of
     these variables in output allows downstream processors (e.g.,
     ‘zppy’) to generate additional masks and weights for rescaling the
     statistics.  For example, these fields can be used to rescale
     global sums into any units desired.

‘--no_ntv_tms (--no_ntv_tms, --no_ntv, --no_native, --remove_native)’
     This switch (which takes no argument) controls whether the splitter
     retains native grid split files, which it does by default, or
     deletes them.  ‘ncclimo’ can split model output from multi-variable
     native grid files into per-variable timeseries files and regrid
     those onto a so-called analysis grid.  That is the typical format
     in which Model Intercomparison Projects (MIPs) request and
     disseminate contributions.  When the data producer has no use for
     the split timeseries on the native grid, he/she can invoke this
     flag to cause ‘ncclimo’ to delete the native grid timeseries (not
     the raw native grid datafiles).  This functionality is implemented
     by first creating the native grid timeseries, regridding it, and
     then overwriting the native grid timeseries with the regridded
     timeseries.  Thus the regridded files will be in the same location
     and have the same name as the native grid timeseries would have
     been and had, respectively.

‘--no_stg_grd (--no_stg_grd, --no_stg, --no_stagger, --no_staggered_grid)’
     This switch (which takes no argument) controls whether regridded
     output will contain the staggered grid coordinates ‘slat’, ‘slon’,
     and ‘w_stag’ (*note Regridding::).  By default the staggered grid
     is output for all files regridded from a Cap (aka FV) grid, except
     when the regridding is performed as part of splitting (reshaping)
     into timeseries.

‘-o DRC_OUT (--drc_out, --out_drc, --dir_out, --output)’
     Directory to hold computed (output) native grid climo files.
     Regridded climos are also placed here unless a separate directory
     for them is specified with ‘-O’ (NB: capital "O").

‘-p PAR_TYP (--par_typ, --par_md, --parallel_type, --parallel_mode, --parallel)’
     Specifies the parallelism mode for files.  The options are serial
     mode (‘-p srl’, ‘-p serial’, or ‘-p nil’), background mode
     parallelism (‘-p bck’ or ‘-p background’)), and MPI parallelism
     (‘-p mpi’ or ‘-p MPI’).  The default is background-mode
     parallelism.  The default PAR_TYP is ‘background’, which means
     ‘ncclimo’ spawns up to twelve (one for each month) parallel
     processes at a time.  See discussion below under Memory
     Considerations.

‘--qnt=QNT_PRC (--ppc, --ppc_prc, --precision, --qnt, --quantize)’
     Specifies the precision of the Precision-Preserving Compression
     algorithm (*note Precision-Preserving Compression::).  A positive
     integer is interpreted as the Number of Significant Digits for the
     Bit-Grooming algorithm, and is equivalent to specifying ‘--qnt
     default=QNT_PRC’ to a binary operator.  A positive or negative
     integer preceded by a period, e.g., ‘.-2’ is interpreted as the
     number of Decimal Significant Digits for the rounding algorithm and
     is equivalent to specifying ‘--qnt default=.QNT_PRC’ to a binary
     operator.  This option applies one precision algorithm and a
     uniform precision for the entire file.  To specify
     variable-by-variable precision options, pass the desired options as
     a quoted string directly with ‘-n NCO_OPT’, e.g., ‘-n '--qnt
     FSNT,TREFHT=4 --qnt CLOUD=2'’.

‘-R RGR_OPT (rgr_opt, regrid_options)’
     Specifies a string of options to pass-through unaltered to ‘ncks’.
     RGR_OPT defaults to ‘-O --no_tmp_fl’.

‘-r RGR_MAP (--rgr_map, --regrid_map, --map)’
     Regridding map.  Unless ‘-r’ is specified ‘ncclimo’ produces only a
     climatology on the native grid of the input datasets.  The RGR_MAP
     specifies how to (quickly) transform the native grid into the
     desired analysis grid.  ‘ncclimo’ will (call ‘ncremap’ to) apply
     the given map to the native grid climatology and produce a second
     climatology on the analysis grid.  Options intended exclusively for
     the regridder may be passed as arguments to the ‘-R’ switch.  See
     below the discussion on regridding.

‘--mth_srt=MTH_SRT (--mth_srt, --srt_mth, --month_start, --start_month)’
‘--mth_end=MTH_END (--mth_end, --end_mth, --month_end, --end_month)’
     Start month (example: 4), and end month (example: 11).  The
     starting month of monthly timeseries extracted by the splitter
     defaults to January of the specified start year, and the ending
     month defaults to December of the specified end year.  As of NCO
     version 4.9.8, released in March, 2021, the splitter mode of
     ‘ncclimo’ accepts user-specified start and end months with the
     ‘--mth_srt’ and ‘--mth_end’ options, respectively.  Months are
     input as one-based integers so January is 1 and December is is 12.
     To extract 14-month timeseries from individual monthly input files
     one could use, e.g.,
          ncclimo --yr_srt=1 --yr_end=2 --mth_srt=4 --mth_end=5 ...
     Note that ‘mth_srt’ and ‘mth_end’ only affect the splitter, and
     that they play no role in climatology generation.

‘-s SRT_YR (--srt_yr, --yr_srt, --start_year, --year_start, --start)’
     Start year (example: 1980).  By default, the first month used is
     January of the specified start year.  If ‘-a scd’ is specified, the
     first month used will be December of the year before the start year
     (to allow for contiguous DJF climos).

‘--seasons=CSN_LST (--seasons, --csn_lst, --csn)’
     Seasons for ‘ncclimo’ to compute in monthly climatology generation
     mode.  The list of seasons, CSN_LST, is a comma-separated,
     case-insensitive, unordered subset of the abbreviations for the
     eleven (so far) defined seasons: ‘jfm’, ‘amj’, ‘jas’, ‘ond’, ‘on’,
     ‘fm’, ‘djf’, ‘mam’, ‘jja’, ‘son’, and ‘ann’.  By default
     ‘CSN_LST=mam,jja,son,djf’.  Moreover, ‘ncclimo’ automatically
     computes the climatological annual mean, ‘ANN’, is always computed
     when MAM, JJA, SON, and DJF are all requested (which is the
     default).  The ANN computed automatically is the time-weighted
     average of the four seasons, rather than as the time-weighted
     average of the twelve monthly climatologies.  Users who need ANN
     but not DJF, MAM, JJA, and SON should instead explicitly specify
     ANN as a season in CSN_LST.  The ANN computed as a season is the
     time-weighted average of the twelve monthly climatologies, rather
     than the time-weighted average of four seasonal climatologies.
     Specifying the four seasons and ANN in CSN_LST (e.g.,
     ‘CSN_LST=mam,jja,son,djf,ann’) is legal though redundant and
     wasteful.  It cause ANN to be computed twice, first as the average
     of the twelve monthly climatologies, then as the average of the
     four seasons.  The special value ‘CSN_LST=none’ turns-off
     computation of seasonal (and annual) climatologies.
          ncclimo --seasons=none ...            # Produce only monthly climos
          ncclimo --seasons=mam,jja,son,djf ... # Monthly + MAM,JJA,SON,DJF,ANN
          ncclimo --seasons=jfm,jas,ann ...     # Monthly + JFM,JAS,ANN
          ncclimo --seasons=fm,on ...           # Monthly + FM,ON

‘--split (--split, --splitter, --tms_flg, --timeseries)’
     This switch (which takes no argument) explicitly instructs
     ‘ncclimo’ to split the input multi-variable raw datasets into
     per-variable timeseries spanning the entire period.  The ‘--split’
     switch, and its synonyms ‘--splitter’, ‘--tms_flg’, and
     ‘--timeseries’, were introduced in NCO version 5.0.4 (released
     December, 2021).  Previously, the splitter was automatically
     invoked whenever the input files were provided via ‘stdin’,
     globbing, or positional command-line arguments, with some
     exceptions.  That older method became ambiguous and untenable once
     it was decided to also allow climos to be generated from files
     provided via ‘stdin’, globbing, or positional command-line
     arguments.  Now there are three methods to invoke the splitter:
     1) Use the ‘--split’ flag: this is the most explicit way to invoke
     the splitter.  2) Select ‘clm_md=hfs’: the high-frequency splitter
     mode by definition invokes the splitter, so a more explicit option
     than this is not necessary.  3) Set the years-per-file option,
     e.g., ‘--ypf=25’: the ‘ypf_max’ option is only useful to the
     splitter, and has thus been used in many scripts.  Since this
     option still causes the splitter to be invoked, those will perform
     as before the API change.

     These three splitter invocations methods are non-exclusive, i.e.,
     more than one can be used, and there is no harm in doing so.  While
     the API change in version 5.0.4 does proscribe the former practice
     of passively invoking the splitter by simply piping files to
     ‘stdin’ or similar, it enables much more flexibility for future
     features, including the possibility of automatically generating
     timeseries filenames for the splitter, and of piping files to
     ‘stdin’ or similar for climo generation.

‘--no_stdin (--no_stdin, --no_inp_std, --no_redirect, --no_standard_input)’
     First introduced in NCO version 4.8.0 (released May, 2019), this
     switch (which takes no argument) disables checking standard input
     (aka ‘stdin’) for input files.  This is useful because ‘ncclimo’
     and ‘ncremap’ may mistakenly expect input to be provided on ‘stdin’
     in environments that use ‘stdin’ for other purposes.  Some
     non-interactive environments (e.g., ‘crontab’, ‘nohup’, Azure CI,
     CWL), may use standard input for their own purposes, and thus
     confuse NCO into thinking that you provided the input files names
     via the ‘stdin’ mechanism.  In such cases users may disable the
     automatic checks for standard input by explicitly invoking the
     ‘--no_stdin’ flag.  This switch is usually not required for jobs in
     an interactive shell.  Interactive SLURM shells can also commandeer
     ‘stdin’, as is the case on the ‘DOE’ machine named Chrysalis.  This
     behavior appears to vary depending on the SLURM implementation.
          ncclimo --no_stdin -v T -s 2000 -e 2001 --ypf=10 -i in -o out

‘-t THR_NBR (--thr_nbr, --thr, --thread_number, --threads)’
     Specifies the number of threads used per regridding process (*note
     OpenMP Threading::).  The NCO regridder scales well to 8-16
     threads.  However, regridding with the maximum number of threads
     can interfere with climatology generation in parallel climatology
     mode (i.e., when PAR_TYP = ‘mpi’ or ‘bck’).  Hence ‘ncclimo’
     defaults to THR_NBR=2.

‘--tpd=TPD (--tpd_out, --tpd, --timesteps_per_day)’
     Normally, the number of timesteps-per-day in files ingested by
     ‘ncclimo’.  It can sometimes be difficult for ‘ncclimo’ to infer
     the number of timesteps-per-day in high-frequency input files,
     i.e., those with 1 or more timesteps-per-day.  In such cases, users
     may override the inferred value by explicitly specifying
     ‘--tpd=TPD’.

     The value of TPD_OUT in daily-average climatology mode ‘clm_md=dly’
     (which is generally not used outside of ice-sheet models) is
     different, and actually refers to the number of timesteps per day
     that ncclimo will output, regardless of its value in the input
     files.  Hence in daily-average mode (only), we refer to this
     variable as TPD_OUT.

     The climatology output from input files at daily or sub-daily
     resolution is, by default, averaged to daily resolution, i.e.,
     TPD_OUT=1.  If the number of timesteps per day in each input file
     is TPD_IN, then the user may select any value of TPD_OUT that is
     smaller than and integrally divides TPD_IN.  For example, an input
     timeseries with TPD_IN=8 (i.e., 3-hourly resolution), can be used
     to produce climatological output at 3, 6, or 12-hourly resolution
     by setting TPD_OUT to 8, 4, or 2, respectively.  This option only
     takes effect in daily-average climatology mode.

     For full generality, the ‘--tpd’ option should probably be split
     into separate options ‘--tpd_in’ and ‘--tpd_out’.  However, because
     it is unlikely that anyone will need to specify these to different
     values, we leave only one option.  If this hinders you, please let
     us know and we will split the options.

‘-v VAR_LST (--var_lst, --var, --vars, --variables, --variable_list)’
     Variables to subset or to split.  Same behavior as *note Subsetting
     Files::.  The use of VAR_LST is optional in climo-generation mode.
     We suggest using this feature to test whether an ‘ncclimo’ command,
     especially one that is lengthy and/or time-consuming, works as
     intended on one or a few variables with, e.g., ‘-v T,FSNT’ before
     generating the full climatology (by omitting this option).
     Invoking this switch was required in the original splitter released
     in version 4.6.5 (March, 2017), and became optional as of version
     4.6.6 (May, 2017).  This option is recommended in timeseries
     reshaping mode to prevent inadvertently copying the results of an
     entire model simulation.  Regular expressions are allowed so, e.g.,
     ‘PREC.?’ extracts the variables ‘PRECC,PRECL,PRECSC,PRECSL’ if
     present.  Currently in reshaping mode all matches to a regular
     expression are placed in the same output file.  We hope to remove
     this limitation in the future.

‘--var_xtr=VAR_XTR (--var_xtr, --var_xtr, --var_extra, --variables_extra, --extra_variables)’
     The ‘--var_xtr’ option causes ‘ncclimo’ to include the extra
     variables list in VAR_XTR in every timeseries split from the raw
     data.  This is useful when extra variables are desired in
     timeseries.  There are no limits on the extra variables--they may
     be of any rank and may be timeseries themselves.  One useful
     application of this option, is to ensure that the area variable is
     included with each timeseries, e.g., ‘--var_xtr=area’.

‘--version (--version, --vrs, --config, --configuration, --cnf)’
     This switch (which takes no argument) causes the operator to print
     its version and configuration.  This includes the copyright notice,
     URLs to the BSD and NCO license, directories from which the NCO
     scripts and binaries are running, and the locations of any separate
     executables that may be used by the script.

‘--xcl_var (--xcl_var, --xcl, --exclude, --exclude_variables)’
     This flag (which takes no argument) changes VAR_LST, as set by the
     ‘--var_lst’ option, from an extraction list to an exclusion list so
     that variables in VAR_LST will not be processed, and variables not
     in VAR_LST will be processed.  Thus the option ‘-v VAR_LST’ must
     also be present for this flag to take effect.  Variables explicitly
     specified for exclusion by ‘--xcl --vars=VAR_LST[,...]’ need not be
     present in the input file.  Previously, this switch has always
     woked in climo mode.  As of NCO version 5.2.5 (July, 2024), this
     switch also works in timeseries mode.

‘--ypf_max YPF_MAX (--ypf, --years, --years_per_file)’
     Specifies the maximum number of years-per-file output by
     ‘ncclimo’'s splitting operation.  When ‘ncclimo’ subsets and splits
     a collection of input files spanning a timerseries, it places each
     subset variable in its own output file.  The maximum length, in
     years, of each output file is YPF_MAX, which defaults to
     YPF_MAX=50.  If an input timeseries spans 237 years and YPF_MAX=50,
     then ‘ncclimo’ will generate four output files of length 50 years
     and one output file of length 37 years.  Note that invoking this
     option _causes_ ‘ncclimo’ to enter timeseries reshaping mode.  In
     fact, one _must_ use ‘--ypf’ to turn-on splitter mode when the
     input files are specified by using the ‘-i drc_in’ method.
     Otherwise it would be ambiguous whether to generate a climatology
     from or to split the input files.

Timeseries Reshaping mode, aka Splitting
----------------------------------------

This section of the ‘ncclimo’ documentation applies only to resphaping
mode, whereas all subsequent sections apply to climatology generation
mode.  In splitter mode, ‘ncclimo’ _reshapes_ the input so that the
outputs are continuous timeseries of each variable taken from all input
files.  As of NCO version 5.0.4 (released December, 2021), ‘ncclimo’
enters splitter mode when invoked with the ‘--split’ switch (or its
synonyms ‘--splitter’, ‘--tms_flg’, or ‘--timeseries’) or with the
‘--ypf_max’ option.  Then ‘ncclimo’ will create per-variable timeseries
from the list of files supplied via ‘stdin’, or, alternatively, placed
as positional arguments (after the last command-line option), or if
neither of these is done and no CASEID is specified, in which case it
assumes all ‘*.nc’ files in DRC_IN constitute the input file list.
These examples invoke reshaping mode in the four possible ways:
     # Pipe list to stdin
     cd $drc_in;ls *mdl*000[1-9]*.nc | ncclimo --split -v T,Q,RH -s 1 -e 9 -o $drc_out
     # Redirect list from file to stdin
     cd $drc_in;ls *mdl*000[1-9]*.nc > foo;ncclimo --split -v T,Q,RH -s 1 -e 9 -o $drc_out < foo
     # List as positional arguments
     ncclimo --split -v T,Q,RH -s 1 -e 9 -o $drc_out $drc_in/*mdl*000[1-9]*.nc
     # Glob directory
     ncclimo --split -v T,Q,RH -s 1 -e 9 -i $drc_in -o $drc_out
   Assuming each input file is a monthly average comprising the
variables T, Q, and RH, then the output will be files
‘T_000101_000912.nc’, ‘Q_000101_000912.nc’, and ‘RH_000101_000912.nc’.
When necessary, the output is split into segments each containing no
more than YPF_MAX (default 50) years of input, i.e.,
‘T_000101_005012.nc’, ‘T_005101_009912.nc’, ‘T_010001_014912.nc’, etc.

MPAS-O/SI/LI considerations
---------------------------

MPAS ocean and ice models currently have their own (non-CESM'ish) naming
convention that guarantees output files have the same names for all
simulations.  By default ‘ncclimo’ analyzes the "timeSeriesStatsMonthly"
analysis member output (tell us if you want options for other analysis
members).  ‘ncclimo’ and ‘ncremap’ recognize input files as being
MPAS-style when invoked with ‘-P mpas’ or with the more expressive
synonym ‘--prc_typ=mpas’.  The the generic ‘-P mpas’ invocation works
for generating climatologies for any MPAS model.  However, some
regridder options are model-specific and therefore it is smarter to
specify which MPAS model produced the input data with ‘-P
mpasatmosphere’, (or ‘-P mpasa’ for short), ‘-P mpasocean’, (or ‘-P
mpaso’ for short), ‘-P mpasseaice’, (or ‘-P mpassi’ for short), or ‘-P
mali’, like this:
     ncclimo -P mpasa  -c $case -s 1980 -e 1983 -i $drc_in -o $drc_out # MPAS-A
     ncclimo -P mpaso  -c $case -s 1980 -e 1983 -i $drc_in -o $drc_out # MPAS-O
     ncclimo -P mpassi -c $case -s 1980 -e 1983 -i $drc_in -o $drc_out # MPAS-SI
     ncclimo -P mali   -c $case -s 1980 -e 1983 -i $drc_in -o $drc_out # MPAS-LI
   As of June 2024 and NCO version 5.2.5, ‘ncclimo’ updated its MPAS
dataset filename construction option.  Previously it constructed MPAS
monthly datasets names like this:
‘$mdl_nm.hist.am.timeSeriesStatsMonthly.$YYYY-$MM-01.nc’.  where MDL_NM
is the canonical MPAS component name, e.g., ‘mpaso’.  This yielded names
consistent with E3SM v1 output like
‘mpaso.hist.am.timeSeriesStatsMonthly.0001-02-01.nc’, and
‘mpascice.hist.am.timeSeriesStatsMonthly.0001-02-01.nc’.  Now ‘ncclimo’
prepends the CASEID, if present, to the filename.  This yields names
consistent with E3SM v2 and v3 output like
‘v2.LR.historical_0101.mpaso.hist.am.timeSeriesStatsMonthly.0001-02-01.nc’,
and
‘v2.LR.historical_0101.mpassi.hist.am.timeSeriesStatsMonthly.0001-02-01.nc’.
To read MPAS filenames with other patterns, simply pipe the filenames to
‘ncclimo’: ‘ls *mpas*hist | ncclimo ...’.

   Raw output data from all MPAS models does not contain missing value
attributes (3).  These attributes must be manually added before sending
the data as input to ‘ncclimo’ or ‘ncremap’.  We recommend that
simulation producers annotate all floating point variables with the
appropriate ‘_FillValue’ prior to invoking ‘ncclimo’.  Run something
like this once in the history-file directory:
     for fl in `ls hist.*` ; do
       ncatted -O -t -a _FillValue,,o,d,-9.99999979021476795361e+33 ${fl}
     done
   If/when MPAS-O/I generates the ‘_FillValue’ attributes itself, this
step can and should be skipped.  All other ‘ncclimo’ features like
regridding (below) are invoked identically for MPAS as for CAM/CLM users
although under-the-hood ‘ncclimo’ does do some special pre-processing
(dimension permutation, metadata annotation) for MPAS.  A five-year
oEC60to30 MPAS-O climo with regridding to T62 takes less than 10 minutes
on the machine ‘rhea’.

Annual climos
-------------

Not all model or observed history files are created as monthly means.
To create a climatological annual mean from a series of annual mean
inputs, select ‘ncclimo’'s annual climatology mode with the ‘-C ann’
option:
     ncclimo -C ann -m cism -h h -c caseid -s 1851 -e 1900 -i drc_in -o drc_out
   The options ‘-m mdl_nm’ and ‘-h hst_nm’ (that default to ‘cam’ and
‘h0’, respectively) tell ‘ncclimo’ how to construct the input filenames.
The above formula names the files ‘caseid.cism.h.1851-01-01-00000.nc’,
‘caseid.cism.h.1852-01-01-00000.nc’, and so on.  Annual climatology mode
produces a single output file (or two if regridding is selected), and in
all other respects behaves the same as monthly climatology mode.

Regridding Climos and Other Files
---------------------------------

‘ncclimo’ will (optionally) regrid during climatology generation and
produce climatology files on both native and analysis grids.  This
regridding is virtually free, because it is performed on idle
nodes/cores after monthly climatologies have been computed and while
seasonal climatologies are being computed.  This load-balancing can save
half-an-hour on ne120 datasets.  To regrid, simply pass the desired
mapfile name with ‘-r map.nc’, e.g., ‘-r
maps/map_ne120np4_to_fv257x512_aave.20150901.nc’.  Although this should
not be necessary for normal use, you may pass any options specific to
regridding with ‘-R opt1 opt2’.

   Specifying ‘-O DRC_RGR’ (NB: uppercase ‘O’) causes ‘ncclimo’ to place
the regridded files in the directory DRC_RGR.  These files have the same
names as the native grid climos from which they were derived.  There is
no namespace conflict because they are in separate directories.  These
files also have symbolic links to their AMWG filenames.  If ‘-O DRC_RGR’
is not specified, ‘ncclimo’ places all regridded files in the native
grid climo output directory, DRC_OUT, specified by ‘-o DRC_OUT’ (NB:
lowercase ‘o’).  To avoid namespace conflicts when both climos are
stored in the same directory, the names of regridded files are suffixed
by the destination geometry string obtained from the mapfile, e.g.,
‘*_climo_fv257x512_bilin.nc’.  These files also have symbolic links to
their AMWG filenames.
     ncclimo -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out
     ncclimo -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out -r map_fl
     ncclimo -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out -r map_fl -O drc_rgr
The above commands perform a climatology without regridding, then with
regridding (all climos stored in DRC_OUT), then with regridding and
storing regridded files separately.  Paths specified by DRC_IN, DRC_OUT,
and DRC_RGR may be relative or absolute.  An alternative to regridding
during climatology generation is to regrid afterwards with ‘ncremap’,
which has more special features built-in for regridding.  To use
‘ncremap’ to regrid a climatology in DRC_OUT and place the results in
DRC_RGR, use something like
     ncremap -I drc_out -m map.nc -O drc_rgr
     ls drc_out/*climo* | ncremap -m map.nc -O drc_rgr
   See *note ncremap netCDF Remapper:: for more details (including
MPAS!).

Extended Climatologies
----------------------

‘ncclimo’ supports two methods for generating extended climatologies:
Binary and Incremental.  Both methods lengthen a climatology without
requiring access to all the raw monthly data spanning the time period.
The binary method combines, with appropriate weighting, two previously
computed climatologies into a single climatology.  No raw monthly data
are employed.  The incremental method computes a climatology from raw
monthly data and (with appropriate weighting) combines that with a
previously computed climatology that ends the month prior to raw data.
The incremental method was introduced in NCO version 4.6.1 (released
August, 2016), and the binary method was introduced in NCO version 4.6.3
(released December, 2016).

   Both methods, binary and incremental, compute the so-called "extended
climo" as a weighted mean of two shorter climatologies, called the
"previous" and "current" climos.  The incremental method uses the
original monthly input to compute the curent climo, which must
immediately follow in time the previous climo which has been
pre-computed.  The binary method use pre-computed climos for both the
previous and current climos, and these climos need not be sequential nor
chronological.  Both previous and current climos for both binary and
incremental methods may be of any length (in years); their weights will
be automatically adjusted in computing the extended climo.

   The use of pre-computed climos permits ongoing simulations (or
lengthy observations) to be analyzed in shorter segments combined
piecemeal, instead of requiring all raw, native-grid data to be
simultaneously accessible.  Without extended climatology capability,
generating a one-hundred year climatology requires that one-hundred
years of monthly data be available on disk.  Disk-space requirements for
large datasets may make this untenable.  Extended climo methods permits
a one-hundred year climo to be generated as the weighted mean of, say,
the current ten year climatology (weighted at 10%) combined with the
pre-computed climatology of the previous 90-years (weighted at 90%).
The 90-year climo could itself have been generated incrementally or
binary-wise, and so on.  Climatologies occupy at most 17/(12N) the
amount of space of N years of monthly data, so the extended methods
vastly reduce disk-space requirements.

   Incremental mode is selected by specifying ‘-S’, the start year of
the pre-computed, previous climo.  The argument to ‘-S’) is the previous
climo start year.  That, together with the current climo end year,
determines the extended climo range.  ‘ncclimo’ assumes that the
previous climo ends the month before the current climo begins.  In
incremental mode, ‘ncclimo’ first generates the current climatology from
the current monthly input files then weights that current climo with the
previous climo to produce the extended climo.

   Binary mode is selected by specifying both ‘-S’ and ‘-E’, the end
year of the pre-computed, previous climo.  In binary mode, the previous
and current climatologies can be of any length, and from any
time-period, even overlapping.  Most users will run extended clmos the
same way they run regular climos in terms of parallelism and regridding,
although that is not required.  Both climos must treat Decembers same
way (or else previous climo files will not be found), and if subsetting
(i.e., ‘-v var_lst’) is performed, then the subset must remain the same,
and if nicknames (i.e., ‘-f fml_nm’) are employed, then the nickname
must remain the same.

   As of 20161129, the ‘climatology_bounds’ attributes of extended
climos are incorrect.  This is a work in progress...

   Options:
‘-E YR_END_PRV (--yr_end_prv, --prv_yr_end, --previous_end)’
     The ending year of the previous climo.  This argument is required
     to trigger binary climatologies, and should not be used for
     incremental climatologies.

‘-S YR_SRT_PRV (--yr_srt_prv, --prv_yr_srt, --previous_start)’
     The starting year of the previous climo.  This argument is required
     to trigger incremental climatologies, and is also mandatory for
     binary climatologies.

‘-X DRC_XTN (--drc_xtn, --xtn_drc, --extended)’
     Directory in which the extended native grid climo files will be
     stored for an extended climatology.  Default value is DRC_PRV.
     Unless a separate directory is specified (with ‘-Y’) for the
     extended climo on the analysis grid, it will be stored in DRC_XTN,
     too.

‘-x DRC_PRV (--drc_prv, --prv_drc, --previous)’
     Directory in which the previous native grid climo files reside for
     an incremental climatology.  Default value is DRC_OUT.  Unless a
     separate directory is specified (with ‘-y’) for the previous climo
     on the analysis grid, it is assumed to reside in DRC_PRV, too.

‘-Y DRC_RGR_XTN (--drc_rgr_xtn, --drc_xtn_rgr, --extended_regridded, --regridded_extended)’
     Directory in which the extended analysis grid climo files will be
     stored in an incremental climatology.  Default value is DRC_XTN.

‘-y DRC_RGR_PRV (--drc_rgr_prv, --drc_prv_rgr, --regridded_previous, --previous_regridded)’
     Directory in which the previous climo on the analysis grid resides
     in an incremental climatology.  Default value is DRC_PRV.

   Incremental method climatologies can be as simple as providing a
start year for the previous climo, e.g.,
     ncclimo -v FSNT,AODVIS -c caseid -s 1980 -e 1981 -i raw -o clm -r map.nc
     ncclimo -v FSNT,AODVIS -c caseid -s 1982 -e 1983 -i raw -o clm -r map.nc -S 1980
   By default ‘ncclimo’ stores all native and analysis grid climos in
one directory so the above "just works".  There are no namespace clashes
because all climos are for distinct years, and regridded files have a
suffix based on their grid resolution.  However, there can be only one
set of AMWG filename links due to AMWG filename convention.  Thus AMWG
filename links, if any, point to the latest extended climo in a given
directory.

   Many researchers segregate (with ‘-O DRC_RGR’) native-grid from
analysis-grid climos.  Incrementally generated climos must be consistent
in this regard.  In other words, all climos contributing to an extended
climo must have their native-grid and analysis-grid files in the same
(per-climo) directory, or all climos must segregate their native from
their analysis grid files.  Do not segregate the grids in one climo, and
combine them in another.  Such climos cannot be incrementally
aggregated.  Thus incrementing climos can require from zero to four
additional options that specify all the previous and extended
climatologies for both native and analysis grids.  The example below
constructs the current climo in CRR, then combines the weighted average
of that with the previous climo in PRV, and places the resulting
extended climatology in XTN.  Here the native and analysis climos are
combined in one directory per climo:
     ncclimo -v FSNT,AODVIS -c caseid -s 1980 -e 1981 -i raw -o prv -r map.nc
     ncclimo -v FSNT,AODVIS -c caseid -s 1982 -e 1983 -i raw -o clm -r map.nc \
             -S 1980 -x prv -X xtn
   If the native and analysis grid climo directories are segregated,
then those directories must be specified, too:
     ncclimo -v FSNT,AODVIS -c caseid -s 1980 -e 1981 -i raw -o prv -O rgr_prv -r map.nc
     ncclimo -v FSNT,AODVIS -c caseid -s 1982 -e 1983 -i raw -o clm -O rgr -r map.nc \
             -S 1980 -x prv -X xtn -y rgr_prv -Y rgr_xtn

   ‘ncclimo’ does not know whether a pre-computed climo is on a native
grid or an analysis grid, i.e., whether it has been regridded.  In
binary mode, ‘ncclimo’ may be pointed to two pre-computed native grid
climatologies, or to two pre-computed analysis grid climatologies.  In
other words, it is not necessary to maintain native grid climatologies
for use in creating extended climatologies.  It is sufficient to
generate climatologies on the analysis grid, and feed them to ‘ncclimo’
in binary mode, without a mapping file:
     ncclimo -c caseid -S 1980 -E 1981 -x prv -s 1980 -e 1981 -i crr -o clm

Coupled Runs
------------

‘ncclimo’ works on all E3SM/ACME and CESM models.  It can simultaneously
generate climatologies for a coupled run, where climatologies mean both
native and regridded monthly, seasonal, and annual averages as per
E3SM/ACME specifications (which mandate the inclusion of certain helpful
metadata and provenance information).  Here are template commands for a
recent simulation:
     caseid=20160121.A_B2000ATMMOD.ne30_oEC.titan.a00
     drc_in=/scratch/simulations/$caseid/run
     drc_out=${DATA}/acme
     map_atm=${DATA}/maps/map_ne30np4_to_fv129x256_aave.20150901.nc
     map_lnd=$map_atm
     map_ocn=${DATA}/maps/map_oEC60to30_to_t62_bilin.20160301.nc
     map_ice=$map_ocn
     ncclimo -p mpi -c $caseid -m cam  -s 2 -e 5 -i $drc_in -r $map_atm -o $drc_out/atm
     ncclimo        -c $caseid -m clm2 -s 2 -e 5 -i $drc_in -r $map_lnd -o $drc_out/lnd
     ncclimo -p mpi -m mpaso           -s 2 -e 5 -i $drc_in -r $map_ocn -o $drc_out/ocn 
     ncclimo        -m mpassi          -s 2 -e 5 -i $drc_in -r $map_ice -o $drc_out/ice
   Atmosphere and ocean model output is typically larger than land and
ice model output.  These commands recognize that by using different
parallelization strategies that may (‘rhea’ standard queue) or may not
(‘cooley’, or ‘rhea’'s ‘bigmem’ queue) be required, depending on the
fatness of the analysis nodes, as explained below.

Memory Considerations
---------------------

It is important to employ the optimal ‘ncclimo’ parallelization strategy
for your computer hardware resources.  Select from the three available
choices with the ‘-p PAR_TYP’ switch.  The options are serial mode (‘-p
srl’, ‘-p serial’, or ‘-p nil’), background mode parallelism (‘-p bck’,
or ‘-p background’), and MPI parallelism (‘-p mpi’ or ‘-p MPI’).  The
default is background-mode parallelism.  This is appropriate for lower
resolution (e.g., ne30L30) simulations on most nodes at high-performance
computer centers.  Use (or at least start with) serial mode on personal
laptops/workstations.  Serial mode requires twelve times less RAM than
the parallel modes, and is much less likely to deadlock or cause OOM
(out-of-memory) conditions on your personal computer.  If the available
RAM (plus swap) is < 12*4*‘sizeof(’monthly input file‘)’, then try
serial mode first (12 is the optimal number of parallel processes for
monthly climos, the computational overhead is a factor of four).  EAM-SE
ne30L30 output is about 1 GB/month so each month requires about 4 GB of
RAM.  EAM-SE ne30L72 output (with LINOZ) is about 10 GB/month so each
month requires about 40 GB RAM.  EAM-SE ne120 output is about
12 GB/month so each month requires about 48 GB RAM.  The computer does
not actually use all this memory at one time, and many kernels compress
RAM usage to below what top reports, so the actual physical usage is
hard to pin-down, but may be a factor of 2.5-3.0 (rather than a factor
of four) times the size of the input file.  For instance, my 16 GB 2014
MacBookPro successfully runs an ne30L30 climatology (that requests 48 GB
RAM) in background mode.  However the laptop is slow and unresponsive
for other uses until it finishes (in 6-8 minutes) the climos.
Experiment and choose the parallelization option that performs best.

   Serial-mode, as its name implies, uses one core at a time for climos,
and proceeds sequentially from months to seasons to annual
climatologies.  Serial mode means that climos are performed serially,
while regridding still employs OpenMP threading (up to 16 cores) on
platforms that support it.  By design each month and each season is
independent of the others, so all months can be computed in parallel,
then each season can be computed in parallel (using monthly
climatologies), from which annual average is computed.  Background
parallelization mode exploits this parallelism and executes the climos
in parallel as background processes on a single node, so that twelve
cores are simultaneously employed for monthly climatologies, four for
seasonal, and one for annual.  The optional regridding will employ, by
default, up to two cores per process.  The MPI parallelism mode executes
the climatologies on different nodes so that up to (optimally) twelve
nodes compute monthly climos.  The full memory of each node is available
for each individual climo.  The optional regridding employs, by default,
up to eight cores per node in MPI-mode.  MPI-mode or serial-mode must be
used to process ne30L72 and ne120L30 climos on all but the fattest DOE
nodes.  An ne120L30 climo in background mode on ‘rhea’ (i.e., on one
128 GB compute node) fails due to OOM.  (Unfortunately OOM errors do not
produce useful return codes so if your climo processes die without
printing useful information, the cause may be OOM).  However the same
climo in background-mode succeeds when executed on a single big-memory
(1 TB) node on ‘rhea’ (use ‘-lpartition=gpu’, as shown below).  Or
MPI-mode can be used for any climatology.  The same ne120L30 climo will
also finish blazingly fast in background mode on ‘cooley’ (i.e., on one
384 GB compute node), so MPI-mode is unnecessary on ‘cooley’.  In
general, the fatter the memory, the better the performance.

Single, Dedicated Nodes at LCFs
-------------------------------

The basic approach above (running the script from a standard terminal
window) that works well for small cases can be unpleasantly slow on
login nodes of LCFs and for longer or higher resolution (e.g., ne120)
climatologies.  As a baseline, generating a climatology of 5 years of
ne30 (~1x1 degree) EAM-SE output with ‘ncclimo’ takes 1-2 minutes on
‘rhea’ (at a time with little contention), and 6-8 minutes on a 2014
MacBook Pro.  To make things a bit faster at LCFs, request a dedicated
node (this only makes sense on supercomputers or clusters with
job-schedulers).  On ‘rhea’ or ‘titan’, which use the PBS scheduler, do
this with
     # Standard node (128 GB), PBS scheduler
     qsub -I -A CLI115 -V -l nodes=1 -l walltime=00:10:00 -N ncclimo
     # Bigmem node (1 TB), PBS scheduler
     qsub -I -A CLI115 -V -l nodes=1 -l walltime=00:10:00 -lpartition=gpu -N ncclimo
   The equivalent requests on ‘cooley’ or ‘mira’ (Cobalt scheduler) and
‘cori’ or ‘titan’ (SLURM scheduler) are:
     # Cooley node (384 GB) with Cobalt
     qsub -I -A HiRes_EarthSys --nodecount=1 --time=00:10:00 --jobname=ncclimo
     # Cori node (128 GB) with SLURM
     salloc  -A acme --nodes=1 --partition=debug --time=00:10:00 --job-name=ncclimo
Flags used and their meanings:
‘-I’
     Submit in interactive mode.  This returns a new terminal shell
     rather than running a program.
‘--time’
     How long to keep this dedicated node for.  Unless you kill the
     shell created by the ‘qsub’ command, the shell will exist for this
     amount of time, then die suddenly.  In the above examples,
     10 minutes is requested.
‘-l nodes=1’
     PBS syntax (e.g., on ‘rhea’) for nodes.
‘--nodecount 1’
     Cobalt syntax (e.g., on ‘cooley’) for nodes.
‘--nodes=1’
     SLURM syntax (e.g., on ‘cori’ or ‘edison’) for nodes.  These
     scheduler-dependent variations request a quantity of nodes.
     Request 1 node for Serial or Background-mode, and up to 12 nodes
     for MPI-mode parallelism.  In all cases ‘ncclimo’ will use multiple
     cores per node if available.
‘-V’
     Export existing environmental variables into the new interactive
     shell.  This may not actually be needed.
‘-q name’
     Queue name.  This is needed for locations like ‘edison’ that have
     multiple queues with no default queue.
‘-A’
     Name of account to charge for time used.
   Acquiring a dedicated node is useful for any workflow, not just
creating climos.  This command returns a prompt once nodes are assigned
(the prompt is returned in your home directory so you may then have to
‘cd’ to the location you meant to run from).  Then run your code with
the basic ‘ncclimo’ invocation.  The is faster because the node is
exclusively dedicated to ‘ncclimo’.  Again, ne30L30 climos only require
< 2 minutes, so the 10 minutes requested in the example is excessive and
conservative.  Tune it with experience.

12 node MPI-mode Jobs
---------------------

The above parallel approaches will fail when a single node lacks enough
RAM (plus swap) to store all twelve monthly input files, plus extra RAM
for computations.  One should employ MPI multinode parallelism ‘-p mpi’
on nodes with less RAM than 12*3*‘sizeof(’input‘)’.  The longest an
ne120 climo will take is less than half an hour (~25 minutes on ‘edison’
or ‘rhea’), so the simplest method to run MPI jobs is to request
12-interactive nodes using the above commands (though remember to add
‘-p mpi’), then execute the script at the command line.

   It is also possible, and sometimes preferable, to request
non-interactive compute nodes in a batch queue.  Executing an MPI-mode
climo (on machines with job scheduling and, optimally, 12 nodes) in a
batch queue can be done in two commands.  First, write an executable
file which calls the ‘ncclimo’ script with appropriate arguments.  We do
this below by echoing to a file, ‘ncclimo.pbs’.
     echo "ncclimo -p mpi -c $caseid -s 1 -e 20 -i $drc_in -o $drc_out" > ncclimo.pbs
   The only new argument here is ‘-p mpi’ that tells ‘ncclimo’ to use
MPI parallelism.  Then execute this command file with a 12 node
non-interactive job:
     qsub -A CLI115 -V -l nodes=12 -l walltime=00:30:00 -j oe -m e -N ncclimo \
          -o ncclimo.out ncclimo.pbs
   This script adds new flags: ‘-j oe’ (combine output and error streams
into standard error), ‘-m e’ (send email to the job submitter when the
job ends), ‘-o ncclimo.out’ (write all output to ‘ncclimo.out’).  The
above commands are meant for PBS schedulers like on ‘rhea’.  Equivalent
commands for ‘cooley’/‘mira’ (Cobalt) and ‘cori’/‘edison’ (SLURM) are
     # Cooley (Cobalt scheduler)
     /bin/rm -f ncclimo.err ncclimo.out
     echo '#!/bin/bash' > ncclimo.cobalt
     echo "ncclimo -p mpi -c $caseid -s 1 -e 20 -i $drc_in -o $drc_out" >> ncclimo.cobalt
     chmod a+x ncclimo.cobalt
     qsub -A HiRes_EarthSys --nodecount=12 --time=00:30:00 --jobname ncclimo \
          --error ncclimo.err --output ncclimo.out --notify zender@uci.edu ncclimo.cobalt
     
     # Cori/Edison (SLURM scheduler)
     echo "ncclimo -p mpi -c $caseid -s 1 -e 20 -i $drc_in -o $drc_out -r $map_fl" \
           > ncclimo.pbs
     chmod a+x ncclimo.slurm
     sbatch -A acme --nodes=12 --time=03:00:00 --partition=regular --job-name=ncclimo \
            --mail-type=END --error=ncclimo.err --output=ncclimo.out ncclimo.slurm
   Notice that Cobalt and SLURM require the introductory
shebang-interpreter line (‘#!/bin/bash’) which PBS does not need.  Set
only the scheduler batch queue parameters mentioned above.  In MPI-mode,
‘ncclimo’ determines the appropriate number of tasks-per-node based on
the number of nodes available and script internals (like load-balancing
for regridding).  Hence do not set a tasks-per-node parameter with
scheduler configuration parameters as this could cause conflicts.

What does ‘ncclimo’ do?
-----------------------

For monthly climatologies (e.g., JAN), ‘ncclimo’ passes the list of all
relevant January monthly files to NCO's ‘ncra’ command, which averages
each variable in these monthly files over their time-dimension (if it
exists) or copies the value from the first month unchanged (if no
time-axis exists).  Seasonal climos are then created by taking the
average of the monthly climo files using ‘ncra’.  To account for
differing numbers of days per month, the ‘ncra’ ‘-w’ flag is used,
followed by the number of days in the relevant months.  For example, the
MAM climo is computed with ‘ncra -w 31,30,31 MAR_climo.nc APR_climo.nc
MAY_climo.nc MAM_climo.nc’ (details about file names and other
optimization flags have been stripped here to make the concept easier to
follow).  The annual (ANN) climo is then computed as a weighted average
of the seasonal climos.

Assumptions, Approximations, and Algorithms (AAA) Employed:
-----------------------------------------------------------

A climatology embodies many algorithmic choices, and regridding from the
native to the analysis grid involves still more choices.  A separate
method should reproduce the ‘ncclimo’ and NCO answers to round-off
precision if it implements the same algorithmic choices.  For example,
‘ncclimo’ agrees to round-off with AMWG diagnostics when making the same
(sometimes questionable) choices.  The most important choices have to do
with converting single- to double-precision (SP and DP, respectively),
treatment of missing values, and generation/application of regridding
weights.  For concreteness and clarity we describe the algorithmic
choices made in processing a EAM-SE monthly output into a climatological
annual mean (ANN) and then regridding that.  Other climatologies (e.g.,
daily to monthly, or annual-to-climatological) involve similar choices.

   E3SM/ACME (and CESM) computes fields in DP and outputs history (not
restart) files as monthly means in SP.  The NCO climatology generator
(‘ncclimo’) processes these data in four stages.  Stage N accesses input
only from stage N-1, never from stage N-2 or earlier.  Thus the
(on-disk) files from stage N determine the highest precision achievable
by stage N+1.  The general principal is to perform math (addition,
weighting, normalization) in DP and output results to disk in the same
precision in which they were input from disk (usually SP).  In Stage 1,
NCO ingests Stage 0 monthly means (raw EAM-SE output), converts SP input
to DP, performs the average across all years, then converts the answer
from DP to SP for storage on-disk as the climatological monthly mean.
In Stage 2, NCO ingests Stage 1 climatological monthly means, converts
SP input to DP, performs the average across all months in the season
(e.g., DJF), then converts the answer from DP to SP for storage on-disk
as the climatological seasonal mean.  In Stage 3, NCO ingests Stage 2
climatological seasonal means, converts SP input to DP, performs the
average across all four seasons (DJF, MAM, JJA, SON), then converts the
answer from DP to SP for storage on-disk as the climatological annual
mean.

   Stage 2 weights each input month by its number of days (e.g., 31 for
January), and Stage 3 weights each input season by its number of days
(e.g., 92 for MAM).  E3SM/ACME runs EAM-SE with a 365-day calendar, so
these weights are independent of year and never change.  The treatment
of missing values in Stages 1-3 is limited by the lack of missing value
tallies provided by Stage 0 (model) output.  Stage 0 records a value as
missing if it is missing for the entire month, and present if the value
is valid for one or more timesteps.  Stage 0 does not record the missing
value tally (number of valid timesteps) for each spatial point.  Thus a
point with a single valid timestep during a month is weighted the same
in Stages 1-4 as a point with 100% valid timesteps during the month.
The absence of tallies inexorably degrades the accuracy of subsequent
statistics by an amount that varies in time and space.  On the positive
side, it reduces the output size (by a factor of two) and complexity of
analyzing fields that contain missing values.  Due to the ambiguous
nature of missing values, it is debatable whether they merit efforts to
treat them more exactly.

   The vast majority of fields undergo three promotion/demotion cycles
between EAM-SE and ANN.  No promotion/demotion cycles occur for history
fields that EAM-SE outputs in DP rather than SP, nor for fields without
a time dimension.  Typically these fields are grid coordinates (e.g.,
longitude, latitude) or model constants (e.g., CO2 mixing ratio).  NCO
never performs any arithmetic on grid coordinates or non-time-varying
input, regardless of whether they are SP or DP.  Instead, NCO copies
these fields directly from the first input file.  Stage 4 uses a mapfile
to regrid climos from the native to the desired analysis grid.
E3SM/ACME currently uses mapfiles generated by ‘ESMF_RegridWeightGen’
(ERWG) and by TempestRemap.

   The algorithmic choices, approximations, and commands used to
generate mapfiles from input gridfiles are separate issues.  We mention
only some of these issues here for brevity.  Input gridfiles used by
E3SM/ACME until ~20150901, and by CESM (then and currently, at least for
Gaussian grids) contained flaws that effectively reduced their
precision, especially at regional scales, and especially for Gaussian
grids.  E3SM/ACME (and CESM) mapfiles continue to approximate grids as
connected by great circles, whereas most analysis grids (and some
models) use great circles for longitude and small circles for latitude.
The great circle assumption may be removed in the future.  Constraints
imposed by ERWG during weight-generation ensure that global integrals of
fields undergoing conservative regridding are exactly conserved.

   Application of weights from the mapfile to regrid the native data to
the analysis grid is straightforward.  Grid fields (e.g., latitude,
longitude, area) are not regridded.  Instead they are copied (and area
is reconstructed if absent) directly from the mapfile.  NCO ingests all
other native grid (source) fields, converts SP to DP, and accumulates
destination gridcell values as the sum of the DP weight (from the sparse
matrix in the mapfile) times the (usually SP-promoted-to-DP) source
values.  Fields without missing values are then stored to disk in their
original precision.  Fields with missing values are treated (by default)
with what NCO calls the "conservative" algorithm.  This algorithm uses
all valid data from the source grid on the destination grid once and
only once.  Destination cells receive the weighted valid values of the
source cells.  This is conservative because the global integrals of the
source and destination fields are equal.  See *note ncremap netCDF
Remapper:: for more description of the conservative and of the optional
("renormalized") algorithm.

EXAMPLES

   How does one create a climo from a collection of monthly non-CESM'ish
files?  This is a two-step procedure: First be sure the names are
arranged with a YYYYMM-format date preceding the suffix (usually ‘.nc’).
Then give _any_ monthly input filename to ‘ncclimo’.  Consider the
MERRA2 collection, for example.  As retrieved from NASA, MERRA2 files
have names like ‘svc_MERRA2_300.tavgM_2d_aer_Nx.200903.nc4’.  While the
sub-string ‘200903’ is easy to recognize as a month in YYYYMM format,
other parts (specifically the ‘300’ code) of the filename also change
with date.  We can use Bash regular expressions to extract dates and
create symbolic links to simpler filenames with regularly patterned
YYYYMM strings like ‘merra2_200903.nc4’:
     for fl in `ls *.nc4` ; do
     # Convert svc_MERRA2_300.tavgM_2d_aer_Nx.YYYYMM.nc4 to merra2_YYYYMM.nc4
         sfx_out=`expr match "${fl}" '.*_Nx.\(.*.nc4\)'`
         fl_out="merra2_${sfx_out}"
         ln -s ${fl} ${fl_out}
     done
   Then call ‘ncclimo’ with any standard format filename, e.g.,
‘merra2_200903.nc4’, as as the CASEID:
     ncclimo -c merra2_200903.nc4 -s 1980 -e 2016 -i $drc_in -o $drc_out

   In the default monthly climo generation mode, ‘ncclimo’ expects each
input file to contain one single record that is the monthly average of
all fields.  Another example of of wrangling observed datasets into a
CESMish format is ECMWF Integrated Forecasting System (IFS) output that
contains twelve months per file, rather than the one month per file that
‘ncclimo’ expects.
     for yr in {1979..2016}; do
     # Convert ifs_YYYY01-YYYY12.nc to ifs_YYYYMM.nc
         yyyy=`printf "%04d" $yr`
         for mth in {1..12}; do
             mm=`printf "%02d" $mth`
             ncks -O -F -d time,${mth} ifs_${yyyy}01-${yyyy}12.nc ifs_${yyyy}${mm}.nc
         done
     done
   Then call ‘ncclimo’ with ‘ifs_197901.nc’ as CASEID:
     ncclimo -c ifs_197901.nc -s 1979 -e 2016 -i $drc_in -o $drc_out
   ncclimo does not recognize all combinations imaginable of records per
file and files per year.  However, support can be added for the most
prevalent combinations so that ncclimo, rather than the user, does any
necessary data wrangling.  Contact us if there is a common input data
format you would like supported as a custom option.

   Often one wishes to create a climatology of a single variable.  The
‘-f FML_NM’ option to ‘ncclimo’ makes this easy.  Consider a series of
single-variable climos for the fields ‘FSNT’, and ‘FLNT’
     ncclimo -v FSNT -f FSNT -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out
     ncclimo -v FLNT -f FLNT -c amip_xpt -s 1980 -e 1983 -i drc_in -o drc_out
   These climos use the ‘-f’ option and so their output files will have
no namespace conflicts.  Moreover, the climatologies can be generated in
parallel.

   ---------- Footnotes ----------

   (1) This means that newer (including user-modified) versions of
‘ncclimo’ work fine without re-compiling NCO.  Re-compiling is only
necessary to take advantage of new features or fixes in the NCO
binaries, not to improve ‘ncclimo’.  One may download and give
executable permissions to the latest source at
<https://github.com/nco/nco/tree/master/data/ncclimo> without
re-installing the rest of NCO.

   (2) At least one known environment (the E3SM-Unified Anaconda
environment at NERSC) prevents users from spawning scores of processes
and may report OpenBLAS/pthread or ‘RLIMIT_NPROC’-related errors.  A
solution seems to be executing ‘ulimit -u unlimited’

   (3) We submitted pull-requests to implement the ‘_FillValue’
attribute in all MPAS-ocean output in July, 2020.  The status of this PR
may be tracked at <https://github.com/MPAS-Dev/MPAS-Model/pull/677>.
Once this PR is merged to master, we will do the same for the
MPAS-Seaice and MPAS-Landice models.


File: nco.info,  Node: ncecat netCDF Ensemble Concatenator,  Next: nces netCDF Ensemble Statistics,  Prev: ncclimo netCDF Climatology Generator,  Up: Reference Manual

4.6 ‘ncecat’ netCDF Ensemble Concatenator
=========================================

SYNTAX
     ncecat [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c] [--cmp CMP_SNG]
     [--cnk_byt SZ_BYT] [--cnk_csh SZ_BYT] [--cnk_dmn NM,SZ_LMN]
     [--cnk_map MAP] [--cnk_min SZ_BYT] [--cnk_plc PLC] [--cnk_scl SZ_LMN]
     [-D DBG] [-d DIM,[MIN][,[MAX][,[STRIDE]]] [-F] [--fl_fmt FL_FMT]
     [-G GPE_DSC] [-g GRP[,...]] [--gag] [--gaa ...] [--gad ATT[,...]]
     [-H] [-h] [--hdf] [--hdr_pad NBR] [--hpss]
     [-L DFL_LVL] [-l PATH] [-M] [--md5_digest] [--mrd] [-n LOOP]
     [--no_cll_msr] [--no_frm_trm] [--no_tmp_fl]
     [-O] [-o OUTPUT-FILE] [-p PATH] [--qnt ...] [--qnt_alg ALG_NM]
     [-R] [-r] [--ram_all] [-t THR_NBR] [-u ULM_NM] [--unn]
     [-v VAR[,...]] [-X ...] [-x]
     [INPUT-FILES] [OUTPUT-FILE]

DESCRIPTION

   ‘ncecat’ aggregates an arbitrary number of input files into a single
output file using using one of two methods.  “Record AGgregation” (RAG),
the traditional method employed on (flat) netCDF3 files and still the
default method, stores INPUT-FILES as consecutive records in the
OUTPUT-FILE.  “Group AGgregation” (GAG) stores INPUT-FILES as top-level
groups in the netCDF4 OUTPUT-FILE.  Record Aggregation (RAG) makes
numerous assumptions about the structure of input files whereas Group
Aggregation (GAG) makes none.  Both methods are described in detail
below.  Since ‘ncecat’ aggregates all the contents of the input files,
it can easily produce large output files so it is often helpful to
invoke subsetting simultaneously (*note Subsetting Files::).

   RAG makes each variable (except coordinate variables) in each input
file into a single record of the same variable in the output file.
Coordinate variables are not concatenated, they are instead simply
copied from the first input file to the OUTPUT-FILE.  All INPUT-FILES
must contain all extracted variables (or else there would be "gaps" in
the output file).

   A new record dimension is the glue which binds together the input
file data.  The new record dimension is defined in the root group of the
output file so it is visible to all sub-groups.  Its name is, by
default, "record".  This default name can be overridden with the ‘-u
ULM_NM’ short option (or the ‘--ulm_nm’ or ‘rcd_nm’ long options).

   Each extracted variable must be constant in size and rank across all
INPUT-FILES.  The only exception is that ‘ncecat’ allows files to differ
in the record dimension size if the requested record hyperslab (*note
Hyperslabs::) resolves to the same size for all files.  This allows
easier gluing/averaging of unequal length timeseries from simulation
ensembles (e.g., the CMIP archive).

   Classic (i.e., all netCDF3 and ‘NETCDF4_CLASSIC’) output files can
contain only one record dimension.  ‘ncecat’ makes room for the new glue
record dimension by changing the pre-existing record dimension, if any,
in the input files into a fixed dimension in the output file.  netCDF4
output files may contain any number of record dimensions, so ‘ncecat’
need not and does not alter the record dimensions, if any, of the input
files as it copies them to the output file.

   “Group AGgregation” (GAG) stores INPUT-FILES as top-level groups in
the OUTPUT-FILE.  No assumption is made about the size or shape or type
of a given object (variable or dimension or group) in the input file.
The entire contents of the extracted portion of each input file is
placed in its own top-level group in OUTPUT-FILE, which is automatically
made as a netCDF4-format file.

   GAG has two methods to specify group names for the OUTPUT-FILE.  The
‘-G’ option, or its long-option equivalent ‘--gpe’, takes as argument a
group path editing description GPE_DSC of where to place the results.
Each input file needs a distinct output group name to avoid namespace
conflicts in the OUTPUT-FILE.  Hence ‘ncecat’ automatically creates
unique output group names based on either the input filenames or the
GPE_DSC arguments.  When the user provides GPE_DSC (i.e., with ‘-G’),
then the output groups are formed by enumerating sequential two-digit
numeric suffixes starting with zero, and appending them to the specified
group path (*note Group Path Editing::).  When GPE_DSC is not provided
(i.e., user requests GAG with ‘--gag’ instead of ‘-G’), then ‘ncecat’
forms the output groups by stripping the input file name of any
type-suffix (e.g., ‘.nc’), and all but the final component of the full
filename.
     ncecat --gag 85.nc 86.nc 87.nc 8587.nc # Output groups 85, 86, 87
     ncecat -G 85_ a.nc b.nc c.nc 8589.nc # Output groups 85_00, 85_01, 85_02
     ncecat -G 85/ a.nc b.nc c.nc 8589.nc # Output groups 85/00, 85/01, 85/02

   With both RAG and GAG the OUTPUT-FILE size is the sum of the sizes of
the extracted variables in the input files.  *Note Statistics vs
Concatenation::, for a description of the distinctions between the
various statistics tools and concatenators.  As a multi-file operator,
‘ncecat’ will read the list of INPUT-FILES from ‘stdin’ if they are not
specified as positional arguments on the command line (*note Large
Numbers of Files::).

   Suppress global metadata copying.  By default NCO's multi-file
operators copy the global metadata from the first input file into
OUTPUT-FILE.  This helps to preserve the provenance of the output data.
However, the use of metadata is burgeoning and sometimes one encounters
files with excessive amounts of extraneous metadata.  Extracting small
bits of data from such files leads to output files which are much larger
than necessary due to the automatically copied metadata.  ‘ncecat’
supports turning off the default copying of global metadata via the ‘-M’
switch (or its long option equivalents, ‘--no_glb_mtd’ and
‘--suppress_global_metadata’).

   Consider five realizations, ‘85a.nc’, ‘85b.nc’, ... ‘85e.nc’ of 1985
predictions from the same climate model.  Then ‘ncecat 85?.nc 85_ens.nc’
glues together the individual realizations into the single file,
‘85_ens.nc’.  If an input variable was dimensioned [‘lat’,‘lon’], it
will by default have dimensions [‘record’,‘lat’,‘lon’] in the output
file.  A restriction of ‘ncecat’ is that the hyperslabs of the processed
variables must be the same from file to file.  Normally this means all
the input files are the same size, and contain data on different
realizations of the same variables.

   Concatenating a variable packed with different scales across multiple
datasets is beyond the capabilities of ‘ncecat’ (and ‘ncrcat’, the other
concatenator (*note Concatenation::).  ‘ncecat’ does not unpack data, it
simply _copies_ the data from the INPUT-FILES, and the metadata from the
_first_ INPUT-FILE, to the OUTPUT-FILE.  This means that data compressed
with a packing convention must use the identical packing parameters
(e.g., ‘scale_factor’ and ‘add_offset’) for a given variable across
_all_ input files.  Otherwise the concatenated dataset will not unpack
correctly.  The workaround for cases where the packing parameters differ
across INPUT-FILES requires three steps: First, unpack the data using
‘ncpdq’.  Second, concatenate the unpacked data using ‘ncecat’, Third,
re-pack the result with ‘ncpdq’.

EXAMPLES

   Consider a model experiment which generated five realizations of one
year of data, say 1985.  You can imagine that the experimenter slightly
perturbs the initial conditions of the problem before generating each
new solution.  Assume each file contains all twelve months (a seasonal
cycle) of data and we want to produce a single file containing all the
seasonal cycles.  Here the numeric filename suffix denotes the
experiment number (_not_ the month):
     ncecat 85_01.nc 85_02.nc 85_03.nc 85_04.nc 85_05.nc 85.nc
     ncecat 85_0[1-5].nc 85.nc
     ncecat -n 5,2,1 85_01.nc 85.nc
These three commands produce identical answers.  *Note Specifying Input
Files::, for an explanation of the distinctions between these methods.
The output file, ‘85.nc’, is five times the size as a single INPUT-FILE.
It contains 60 months of data.

   One often prefers that the (new) record dimension have a more
descriptive, context-based name than simply "record".  This is easily
accomplished with the ‘-u ULM_NM’ switch.  To add a new record dimension
named "time" to all variables
     ncecat -u time in.nc out.nc
   To glue together multiple files with a new record variable named
"realization"
     ncecat -u realization 85_0[1-5].nc 85.nc
Users are more likely to understand the data processing history when
such descriptive coordinates are used.

   Consider a file with an existing record dimension named ‘time’.  and
suppose the user wishes to convert ‘time’ from a record dimension to a
non-record dimension.  This may be useful, for example, when the user
has another use for the record variable.  The simplest method is to use
‘ncks --fix_rec_dmn’, and another possibility is to use ‘ncecat’
followed by ‘ncwa’:
     ncecat in.nc out.nc # Convert time to non-record dimension
     ncwa -a record in.nc out.nc # Remove new degenerate record dimension
The second step removes the degenerate record dimension.  See *note
ncpdq netCDF Permute Dimensions Quickly:: and *note ncks netCDF Kitchen
Sink:: for other methods of of changing variable dimensionality,
including the record dimension.


File: nco.info,  Node: nces netCDF Ensemble Statistics,  Next: ncflint netCDF File Interpolator,  Prev: ncecat netCDF Ensemble Concatenator,  Up: Reference Manual

4.7 ‘nces’ netCDF Ensemble Statistics
=====================================

SYNTAX
     nces [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c] [--cb Y1,Y2,M1,M2,TPD]
     [--cmp CMP_SNG] [--cnk_byt SZ_BYT] [--cnk_csh SZ_BYT] [--cnk_dmn NM,SZ_LMN]
     [--cnk_map MAP] [--cnk_min SZ_BYT] [--cnk_plc PLC] [--cnk_scl SZ_LMN]
     [-D DBG] [-d DIM,[MIN][,[MAX][,[STRIDE]]] [-F]
     [-G GPE_DSC] [-g GRP[,...]] [--gaa ...] [--gad ATT[,...]]
     [-H] [-h] [--hdf] [--hdr_pad NBR] [--hpss]
     [-L DFL_LVL] [-l PATH] [-n LOOP]
     [--no_cll_msr] [--no_frm_trm] [--no_tmp_fl] [--nsm_fl|grp] [--nsm_sfx sfx]
     [-O] [-o OUTPUT-FILE] [-p PATH] [--qnt ...] [--qnt_alg ALG_NM]
     [-R] [-r] [--ram_all] [--rth_dbl|flt] [-t THR_NBR] [--unn]
     [-v VAR[,...]] [-w wgt] [-X ...] [-x] [-y OP_TYP]
     [INPUT-FILES] [OUTPUT-FILE]

DESCRIPTION

   ‘nces’ performs gridpoint statistics (including, but not limited to,
averages) on variables across an arbitrary number (an “ensemble”) of
INPUT-FILES and/or of input groups within each file.  Each file (or
group) receives an equal weight by default.  ‘nces’ was formerly (until
NCO version 4.3.9, released December, 2013) known as ‘ncea’ (netCDF
Ensemble Averager)(1).  For example, ‘nces’ will average a set of files
or groups, weighting each file or group evenly by default.  This is
distinct from ‘ncra’, which performs statistics only over the record
dimension(s) (e.g., TIME), and weights each record in each record
dimension evenly.

   The file or group is the logical unit of organization for the results
of many scientific studies.  Often one wishes to generate a file or
group which is the statistical product (e.g., average) of many separate
files or groups.  This may be to reduce statistical noise by combining
the results of a large number of experiments, or it may simply be a step
in a procedure whose goal is to compute anomalies from a mean state.  In
any case, when one desires to generate a file whose statistical
properties are influenced by all the inputs, then use ‘nces’.

   As of NCO version 4.9.4, released in July, 2020, ‘nces’ accepts
user-specified weights with the ‘-w’ (or long-option equivalent ‘--wgt’,
‘--wgt_var’, or ‘--weight’) switch.  The user must specify one weight
per input file on the command line, or the name of a (scalar or
degenerate 1-D array) variable in each input file that contains a single
value to weight that file.  When no weight is specified, ‘nces’ weights
each file (e.g., ensemble) in the INPUT-FILES equally.

   Variables in the OUTPUT-FILE are the same size as the variable
hyperslab in each input file or group, and each input file or group must
be the same size after hyperslabbing (2) ‘nces’ does allow files to
differ in the input record dimension size if the requested record
hyperslab (*note Hyperslabs::) resolves to the same size for all files.
‘nces’ recomputes the record dimension hyperslab limits for each input
file so that coordinate limits may be used to select equal length
timeseries from unequal length files.  This simplifies analysis of
unequal length timeseries from simulation ensembles (e.g., the CMIP3
IPCC AR4 archive).

   ‘nces’ works in one of two modes, file ensembles or group ensembles.
File ensembles are the default (equivalent to the old ‘ncea’) and may
also be explicitly specified by the ‘--nsm_fl’ or ‘--ensemble_file’
switches.  To perform statistics on ensembles of groups, a newer
feature, use ‘--nsm_grp’ or ‘--ensemble_group’.  Members of a group
ensemble are groups that share the same structure, parent group, and
nesting level.  Members must be “leaf groups”, i.e., not contain any
sub-groups.  Their contents usually have different values because they
are realizations of replicated experiments.  In group ensemble mode
‘nces’ computes the statistics across the ensemble, which may span
multiple input files.  Files may contain members of multiple, distinct
ensembles.  However, all ensembles must have at least one member in the
first input file.  Group ensembles behave as an unlimited dimension of
datasets: they may contain an arbitrary and extensible number of
realizations in each file, and may be composed from multiple files.

   Output statistics in group ensemble mode are stored in the parent
group by default.  If the ensemble members are ‘/cesm/cesm_01’ and
‘/cesm/cesm_02’, then the computed statistic will be in ‘/cesm’ in the
output file.  The ‘--nsm_sfx’ option instructs nces to instead store
output in a new child group of the parent created by attaching the
suffix to the parent group's name, e.g., ‘--nsm_sfx='_avg'’ would store
results in the output group ‘/cesm/cesm_avg’:
     nces --nsm_grp                  mdl1.nc mdl2.nc mdl3.nc out.nc
     nces --nsm_grp --nsm_sfx='_avg' mdl1.nc mdl2.nc mdl3.nc out.nc

   *Note Statistics vs Concatenation::, for a description of the
distinctions between the statistics tools and concatenators.  As a
multi-file operator, ‘nces’ will read the list of INPUT-FILES from
‘stdin’ if they are not specified as positional arguments on the command
line (*note Large Numbers of Files::).

   Like ‘ncra’ and ‘ncwa’, ‘nces’ treats coordinate variables as a
special case.  Coordinate variables are assumed to be the same in all
ensemble members, so ‘nces’ simply copies the coordinate variables that
appear in ensemble members directly to the output file.  This has the
same effect as averaging the coordinate variable across the ensemble,
yet does not incur the time- or precision- penalties of actually
averaging them.  ‘ncra’ and ‘ncwa’ allow coordinate variables to be
processed only by the linear average operation, regardless of the
arithmetic operation type performed on the non-coordinate variables
(*note Operation Types::).  Thus it can be said that the three operators
(‘ncra’, ‘ncwa’, and ‘nces’) all average coordinate variables (even
though ‘nces’ simply copies them).  All other requested arithmetic
operations (e.g., maximization, square-root, RMS) are applied only to
non-coordinate variables.  In these cases the linear average of the
coordinate variable will be returned.

EXAMPLES

   Consider a model experiment which generated five realizations of one
year of data, say 1985.  Imagine that the experimenter slightly perturbs
the initial conditions of the problem before generating each new
solution.  Assume each file contains all twelve months (a seasonal
cycle) of data and we want to produce a single file containing the
ensemble average (mean) seasonal cycle.  Here the numeric filename
suffix denotes the realization number (_not_ the month):
     nces 85_01.nc 85_02.nc 85_03.nc 85_04.nc 85_05.nc 85.nc
     nces 85_0[1-5].nc 85.nc
     nces -n 5,2,1 85_01.nc 85.nc
These three commands produce identical answers.  *Note Specifying Input
Files::, for an explanation of the distinctions between these methods.
The output file, ‘85.nc’, is the same size as the inputs files.  It
contains 12 months of data (which might or might not be stored in the
record dimension, depending on the input files), but each value in the
output file is the average of the five values in the input files.

   In the previous example, the user could have obtained the ensemble
average values in a particular spatio-temporal region by adding a
hyperslab argument to the command, e.g.,
     nces -d time,0,2 -d lat,-23.5,23.5 85_??.nc 85.nc
In this case the output file would contain only three slices of data in
the TIME dimension.  These three slices are the average of the first
three slices from the input files.  Additionally, only data inside the
tropics is included.

   As of NCO version 4.3.9 (released December, 2013) ‘nces’ also works
with groups (rather than files) as the fundamental unit of the ensemble.
Consider two ensembles, ‘/ecmwf’ and ‘/cesm’ stored across three input
files ‘mdl1.nc’, ‘mdl2.nc’, and ‘mdl3.nc’.  Ensemble members would be
leaf groups with names like ‘/ecmwf/01’, ‘/ecmwf/02’ etc.  and
‘/cesm/01’, ‘/cesm/02’, etc.  These commands average both ensembles:
     nces --nsm_grp mdl1.nc mdl2.nc mdl3.nc out.nc
     nces --nsm_grp --nsm_sfx='_min' --op_typ=min -n 3,1,1 mdl1.nc out.nc
     nces --nsm_grp -g cesm -v tas -d time,0,3 -n 3,1,1 mdl1.nc out.nc
     nces --nsm_grp mdl1.nc mdl2.nc mdl3.nc out.nc
     nces --nsm_grp --nsm_sfx='_min' --op_typ=min -n 3,1,1 mdl1.nc out.nc
     nces --nsm_grp -g cesm -v tas -d time,0,3 -n 3,1,1 mdl1.nc out.nc
   The first command stores averages in the output groups ‘/cesm’ and
‘/ecmwf’, while the second stores minima in the output groups
‘/cesm/cesm_min’ and ‘/ecmwf/ecmwf_min’: The third command demonstrates
that sub-setting and hyperslabbing work as expected.  Note that each
input file may contain different numbers of members of each ensemble, as
long as all distinct ensembles contain at least one member in the first
file.

   As of NCO version 4.9.4, released in July, 2020, ‘nces’ accepts
user-specified weights with the ‘-w’ (or long-option equivalent ‘--wgt’,
‘--wgt_var’, or ‘--weight’) switch:
     # Construct input variables with values of 1 and 2
     ncks -O -M -v one ~/nco/data/in.nc ~/1.nc
     ncrename -O -v one,var ~/1.nc
     ncap2 -O -s 'var=2' ~/1.nc ~/2.nc

     # Three methods of weighting input files unevenly
     # 1. Old-method: specify input files multiple times
     # 2. New-method: specify one weight per input file
     # 3. New-method: specify weight variable in each input file
     nces -O ~/1.nc ~/2.nc ~/2.nc ~/out.nc # Clumsy, limited to integer weights
     nces -O -w 1,2 ~/1.nc ~/2.nc ~/out.nc # Flexible, works for any weight
     nces -O -w var ~/1.nc ~/2.nc ~/out.nc # Flexible, works for any weight
     # All three methods produce same answer: var=(1*1+2*2)/3=5/3=1.67
     ncks ~/out.nc

   ---------- Footnotes ----------

   (1) The old ncea command was deprecated in NCO version 4.3.9,
released December, 2013.  NCO will attempt to maintain
back-compatibility and work as expected with invocations of ‘ncea’ for
as long as possible.  Please replace ‘ncea’ by ‘nces’ in all future
work.

   (2) As of NCO version 4.4.2 (released February, 2014) ‘nces’ allows
hyperslabs in all dimensions so long as the hyperslabs resolve to the
same size.  The fixed (i.e., non-record) dimensions should be the same
size in all ensemble members both before and after hyperslabbing,
although the hyperslabs may (and usually do) change the size of the
dimensions from the input to the output files.  Prior to this, ‘nces’
was only guaranteed to work on hyperslabs in the record dimension that
resolved to the same size.


File: nco.info,  Node: ncflint netCDF File Interpolator,  Next: ncks netCDF Kitchen Sink,  Prev: nces netCDF Ensemble Statistics,  Up: Reference Manual

4.8 ‘ncflint’ netCDF File Interpolator
======================================

SYNTAX
     ncflint [-3] [-4] [-5] [-6] [-7] [-A] [-C] [-c]  [--cmp CMP_SNG]
     [--cnk_byt SZ_BYT] [--cnk_csh SZ_BYT] [--cnk_dmn NM,SZ_LMN]
     [--cnk_map MAP] [--cnk_min SZ_BYT] [--cnk_plc PLC] [--cnk_scl SZ_LMN]
     [-D DBG] [-d DIM,[MIN][,[MAX][,[STRIDE]]] [--fl_fmt FL_FMT]
     [-F] [--fix_rec_crd] [-G GPE_DSC] [-g GRP[,...]] [--gaa ...] [--gad ATT[,...]]
     [-H] [-h] [--hdr_pad NBR] [--hpss]
     [-i VAR,VAL3] [-L DFL_LVL] [-l PATH] [-N]
     [--no_cll_msr] [--no_frm_trm] [--no_tmp_fl]
     [-O] [-o FILE_3] [-p PATH] [--qnt ...] [--qnt_alg ALG_NM]
     [-R] [-r] [--ram_all] [-t THR_NBR] [--unn] [-v VAR[,...]]
     [-w WGT1[,WGT2]] [-X ...] [-x]
     FILE_1 FILE_2 [FILE_3]

DESCRIPTION

   ‘ncflint’ creates an output file that is a linear combination of the
input files.  This linear combination is a weighted average, a
normalized weighted average, or an interpolation of the input files.
Coordinate variables are not acted upon in any case, they are simply
copied from FILE_1.

   There are two conceptually distinct methods of using ‘ncflint’.  The
first method is to specify the weight each input file contributes to the
output file.  In this method, the value VAL3 of a variable in the output
file FILE_3 is determined from its values VAL1 and VAL2 in the two input
files according to VAL3 = WGT1*VAL1 + WGT2*VAL2 .  Here at least WGT1,
and, optionally, WGT2, are specified on the command line with the ‘-w’
(or ‘--weight’ or ‘--wgt_var’) switch.  If only WGT1 is specified then
WGT2 is automatically computed as WGT2 = 1 − WGT1.  Note that weights
larger than 1 are allowed.  Thus it is possible to specify WGT1 = 2 and
WGT2 = -3.  One can use this functionality to multiply all values in a
given file by a constant.

   As of NCO version 4.6.1 (July, 2016), the ‘-N’ switch (or long-option
equivalents ‘--nrm’ or ‘--normalize’) implements a variation of this
method.  This switch instructs ‘ncflint’ to internally normalize the two
supplied (or one supplied and one inferred) weights so that WGT1 =
WGT1/(WGT1 + WGT2 and WGT2 = WGT2/(WGT1 + WGT2 and .  This allows the
user to input integral weights, say, and to delegate the chore of
normalizing them to ‘ncflint’.  Be careful that ‘-N’ means what you
think, since the same switch means something quite different in ‘ncwa’.

   The second method of using ‘ncflint’ is to specify the interpolation
option with ‘-i’ (or with the ‘--ntp’ or ‘--interpolate’ long options).
This is the inverse of the first method in the following sense: When the
user specifies the weights directly, ‘ncflint’ has no work to do besides
multiplying the input values by their respective weights and adding
together the results to produce the output values.  It makes sense to
use this when the weights are known _a priori_.

   Another class of problems has the “arrival value” (i.e., VAL3) of a
particular variable VAR known _a priori_.  In this case, the implied
weights can always be inferred by examining the values of VAR in the
input files.  This results in one equation in two unknowns, WGT1 and
WGT2: VAL3 = WGT1*VAL1 + WGT2*VAL2 .  Unique determination of the
weights requires imposing the additional constraint of normalization on
the weights: WGT1 + WGT2 = 1.  Thus, to use the interpolation option,
the user specifies VAR and VAL3 with the ‘-i’ option.  ‘ncflint’ then
computes WGT1 and WGT2, and uses these weights on all variables to
generate the output file.  Although VAR may have any number of
dimensions in the input files, it must represent a single, scalar value.
Thus any dimensions associated with VAR must be “degenerate”, i.e., of
size one.

   If neither ‘-i’ nor ‘-w’ is specified on the command line, ‘ncflint’
defaults to weighting each input file equally in the output file.  This
is equivalent to specifying ‘-w 0.5’ or ‘-w 0.5,0.5’.  Attempting to
specify both ‘-i’ and ‘-w’ methods in the same command is an error.

   ‘ncflint’ does not interpolate variables of type ‘NC_CHAR’ and
‘NC_STRING’.  This behavior is hardcoded.

   By default ‘ncflint’ interpolates or multiplies record coordinate
variables (e.g., time is often stored as a record coordinate) not other
coordinate variables (e.g., latitude and longitude).  This is because
‘ncflint’ is often used to time-interpolate between existing files, but
is rarely used to spatially interpolate.  Sometimes however, users wish
to multiply entire files by a constant that does not multiply any
coordinate variables.  The ‘--fix_rec_crd’ switch was implemented for
this purpose in NCO version 4.2.6 (March, 2013).  It prevents ‘ncflint’
from multiplying or interpolating any coordinate variables, including
record coordinate variables.

   Depending on your intuition, ‘ncflint’ may treat missing values
unexpectedly.  Consider a point where the value in one input file, say
VAL1, equals the missing value MSS_VAL_1 and, at the same point, the
corresponding value in the other input file VAL2 is not misssing (i.e.,
does not equal MSS_VAL_2).  There are three plausible answers, and this
creates ambiguity.

   Option one is to set VAL3 = MSS_VAL_1.  The rationale is that
‘ncflint’ is, at heart, an interpolator and interpolation involving a
missing value is intrinsically undefined.  ‘ncflint’ currently
implements this behavior since it is the most conservative and least
likely to lead to misinterpretation.

   Option two is to output the weighted valid data point, i.e., VAL3 =
WGT2*VAL2 .  The rationale for this behavior is that interpolation is
really a weighted average of known points, so ‘ncflint’ should weight
the valid point.

   Option three is to return the _unweighted_ valid point, i.e., VAL3 =
VAL2.  This behavior would appeal to those who use ‘ncflint’ to estimate
data using the closest available data.  When a point is not bracketed by
valid data on both sides, it is better to return the known datum than no
datum at all.

   The current implementation uses the first approach, Option one.  If
you have strong opinions on this matter, let us know, since we are
willing to implement the other approaches as options if there is enough
interest.

EXAMPLES

   Although it has other uses, the interpolation feature was designed to
interpolate FILE_3 to a time between existing files.  Consider input
files ‘85.nc’ and ‘87.nc’ containing variables describing the state of a
physical system at times ‘time’ = 85 and ‘time’ = 87.  Assume each file
contains its timestamp in the scalar variable ‘time’.  Then, to linearly
interpolate to a file ‘86.nc’ which describes the state of the system at
time at ‘time’ = 86, we would use
     ncflint -i time,86 85.nc 87.nc 86.nc

   Say you have observational data covering January and April 1985 in
two files named ‘85_01.nc’ and ‘85_04.nc’, respectively.  Then you can
estimate the values for February and March by interpolating the existing
data as follows.  Combine ‘85_01.nc’ and ‘85_04.nc’ in a 2:1 ratio to
make ‘85_02.nc’:
     ncflint -w 0.667 85_01.nc 85_04.nc 85_02.nc
     ncflint -w 0.667,0.333 85_01.nc 85_04.nc 85_02.nc

   Multiply ‘85.nc’ by 3 and by −2 and add them together to make
‘tst.nc’:
     ncflint -w 3,-2 85.nc 85.nc tst.nc
This is an example of a null operation, so ‘tst.nc’ should be identical
(within machine precision) to ‘85.nc’.

   Multiply all the variables except the coordinate variables in the
file ‘emissions.nc’ by by 0.8:
     ncflint --fix_rec_crd -w 0.8,0.0 emissions.nc emissions.nc scaled_emissions.nc
The use of ‘--fix_rec_crd’ ensures, e.g., that the ‘time’ coordinate, if
any, is not scaled (i.e., multiplied).

   Add ‘85.nc’ to ‘86.nc’ to obtain ‘85p86.nc’, then subtract ‘86.nc’
from ‘85.nc’ to obtain ‘85m86.nc’
     ncflint -w 1,1 85.nc 86.nc 85p86.nc
     ncflint -w 1,-1 85.nc 86.nc 85m86.nc
     ncdiff 85.nc 86.nc 85m86.nc
Thus ‘ncflint’ can be used to mimic some ‘ncbo’ operations.  However
this is not a good idea in practice because ‘ncflint’ does not broadcast
(*note ncbo netCDF Binary Operator::) conforming variables during
arithmetic.  Thus the final two commands would produce identical results
except that ‘ncflint’ would fail if any variables needed to be
broadcast.

   Rescale the dimensional units of the surface pressure ‘prs_sfc’ from
Pascals to hectopascals (millibars)
     ncflint -C -v prs_sfc -w 0.01,0.0 in.nc in.nc out.nc
     ncatted -a units,prs_sfc,o,c,millibar out.nc


File: nco.info,  Node: ncks netCDF Kitchen Sink,  Next: ncpdq netCDF Permute Dimensions Quickly,  Prev: ncflint netCDF File Interpolator,  Up: Reference Manual

4.9 ‘ncks’ netCDF Kitchen Sink
==============================

SYNTAX
     ncks [-3] [-4] [-5] [-6] [-7] [-A] [-a] [--area_wgt] [-b FL_BNR]
     [-C] [-c] [--cdl] [--chk_bnd] [--chk_chr] [--chk_map] [--chk_mss] [--chk_nan] [--chk_xtn]
     [--cmp CMP_SNG] [--cnk_byt SZ_BYT] [--cnk_csh SZ_BYT] [--cnk_dmn NM,SZ_LMN]
     [--cnk_map MAP] [--cnk_min SZ_BYT] [--cnk_plc PLC] [--cnk_scl SZ_LMN]
     [-D DBG] [-d DIM,[MIN][,[MAX][,[STRIDE]]]
     [-F] [--fix_rec_dmn DIM] [--fl_fmt FL_FMT] [--fmt_val FORMAT] [--fpe]
     [-G GPE_DSC] [-g GRP[,...]] [--gaa ...] [--gad ATT[,...]] [--grp_xtr_var_xcl]
     [-H] [-h] [--hdn] [--hdr_pad NBR] [--hpss] [--hrz FL_HRZ]
     [--is_hrz VAR] [--jsn] [--jsn_fmt LVL] [-L DFL_LVL] [-l PATH]
     [-M] [-m] [--map MAP-FILE] [--md5] [--mk_rec_dmn DIM]
     [--no_blank] [--no_cll_msr] [--no_frm_trm] [--no_tmp_fl]
     [-O] [-o OUTPUT-FILE] [-P] [-p PATH] [--prn_fl PRINT-FILE]
     [-Q] [-q] [--qnt ...] [--qnt_alg ALG_NM]
     [-R] [-r] [--rad] [--ram_all] [--rgr ...] [--rnr=wgt]
     [-s FORMAT] [--s1d] [-u] [--unn] [-V] [-v VAR[,...]] [--vrt VRT-FILE]
     [-X ...] [-x] [--xml] INPUT-FILE [[OUTPUT-FILE]]

DESCRIPTION

   The nickname "kitchen sink" is a catch-all because ‘ncks’ combines
most features of ‘ncdump’ and ‘nccopy’ with extra features to extract,
hyperslab, multi-slab, sub-set, and translate into one versatile
utility.  ‘ncks’ extracts (a subset of the) data from INPUT-FILE,
regrids it according to MAP-FILE if specified, then writes in netCDF
format to OUTPUT-FILE, and optionally writes it in flat binary format to
‘fl_bnr’, and optionally prints it to screen.

   ‘ncks’ prints netCDF input data in ASCII, CDL, JSON, or NcML/XML text
formats to ‘stdout’, like (an extended version of) ‘ncdump’.  By default
‘ncks’ prints CDL format.  Option ‘-s’ (or long options ‘--sng_fmt’ and
‘--string’) permits the user to format data using C-style format
strings, while option ‘--cdl’ outputs CDL, option ‘--jsn’ (or ‘json’)
outputs JSON, option ‘--trd’ (or ‘traditional’) outputs "traditional"
format, and option ‘--xml’ (or ‘ncml’) outputs NcML.  The "traditional"
tabular format is intended to be easy to search for the data you want,
one datum per screen line, with all dimension subscripts and coordinate
values (if any) preceding the datum.  ‘ncks’ exposes many flexible
controls over printed output, including CDL, JSON, and NcML.

   Options ‘-a’, ‘--cdl’, ‘-F’, ‘--fmt_val’, ‘-H’, ‘--hdn’, ‘--jsn’,
‘-M’, ‘-m’, ‘-P’, ‘--prn_fl’, ‘-Q’, ‘-q’, ‘-s’, ‘--trd’, ‘-u’, ‘-V’, and
‘--xml’ (and their long option counterparts) control the presence of
data and metadata and their formatted location and appearance when
printed.

   ‘ncks’ extracts (and optionally creates a new netCDF file comprised
of) only selected variables from the input file (similar to the old
‘ncextr’ specification).  Only variables and coordinates may be
specifically included or excluded--all global attributes and any
attribute associated with an extracted variable are copied to the screen
and/or output netCDF file.  Options ‘-c’, ‘-C’, ‘-v’, and ‘-x’ (and
their long option synonyms) control which variables are extracted.

   ‘ncks’ extracts hyperslabs from the specified variables (‘ncks’
implements the original ‘nccut’ specification).  Option ‘-d’ controls
the hyperslab specification.  Input dimensions that are not associated
with any output variable do not appear in the output netCDF. This
feature removes superfluous dimensions from netCDF files.

   ‘ncks’ will append variables and attributes from the INPUT-FILE to
OUTPUT-FILE if OUTPUT-FILE is a pre-existing netCDF file whose relevant
dimensions conform to dimension sizes of INPUT-FILE.  The append
features of ‘ncks’ are intended to provide a rudimentary means of adding
data from one netCDF file to another, conforming, netCDF file.  If
naming conflicts exist between the two files, data in OUTPUT-FILE is
usually overwritten by the corresponding data from INPUT-FILE.  Thus,
when appending, the user should backup OUTPUT-FILE in case valuable data
are inadvertantly overwritten.

   If OUTPUT-FILE exists, the user will be queried whether to
“overwrite”, “append”, or “exit” the ‘ncks’ call completely.  Choosing
“overwrite” destroys the existing OUTPUT-FILE and create an entirely new
one from the output of the ‘ncks’ call.  Append has differing effects
depending on the uniqueness of the variables and attributes output by
‘ncks’: If a variable or attribute extracted from INPUT-FILE does not
have a name conflict with the members of OUTPUT-FILE then it will be
added to OUTPUT-FILE without overwriting any of the existing contents of
OUTPUT-FILE.  In this case the relevant dimensions must agree (conform)
between the two files; new dimensions are created in OUTPUT-FILE as
required.  When a name conflict occurs, a global attribute from
INPUT-FILE will overwrite the corresponding global attribute from
OUTPUT-FILE.  If the name conflict occurs for a non-record variable,
then the dimensions and type of the variable (and of its coordinate
dimensions, if any) must agree (conform) in both files.  Then the
variable values (and any coordinate dimension values) from INPUT-FILE
will overwrite the corresponding variable values (and coordinate
dimension values, if any) in OUTPUT-FILE (1).

   Since there can only be one record dimension in a file, the record
dimension must have the same name (though not necessarily the same size)
in both files if a record dimension variable is to be appended.  If the
record dimensions are of differing sizes, the record dimension of
OUTPUT-FILE will become the greater of the two record dimension sizes,
the record variable from INPUT-FILE will overwrite any counterpart in
OUTPUT-FILE and fill values will be written to any gaps left in the rest
of the record variables (I think).  In all cases variable attributes in
OUTPUT-FILE are superseded by attributes of the same name from
INPUT-FILE, and left alone if there is no name conflict.

   Some users may wish to avoid interactive ‘ncks’ queries about whether
to overwrite existing data.  For example, batch scripts will fail if
‘ncks’ does not receive responses to its queries.  Options ‘-O’ and ‘-A’
are available to force overwriting existing files, and appending
existing variables, respectively.

Options specific to ‘ncks’
--------------------------

The following summarizes features unique to ‘ncks’.  Features common to
many operators are described in *note Shared features::.

‘-a’
     Switches ‘-a’, ‘--abc’, and ‘--alphabetize’ _turn-off_ the default
     alphbetization of extracted fields in ‘ncks’ only.  These switches
     are misleadingly named and were deprecated in ‘ncks’ as of NCO
     version 4.7.1 (December, 2017).

     This is the default behavior so these switches are no-ops included
     only for completeness.  By default, NCO extracts, prints, and
     writes specified output variables to disk in alphabetical order.
     This tends to make long output lists easier to search for
     particular variables.  Again, no option is necessary to write
     output in alphabetical order.  Until NCO version 4.7.1 (December,
     2017), ‘ncks’ used the ‘-a’, ‘--abc’, or ‘--alphabetize’ switches
     to _turn-off_ the default alphabetization.  These names were
     counter-intuitive and needlessly confusing.  As of NCO version
     4.7.1, ‘ncks’ uses the new switches ‘--no_abc’, ‘--no-abc’,
     ‘--no_alphabetize’, or ‘--no-alphabetize’, all of which are
     equivalent.  The ‘--abc’ and ‘--alphabetize’ switches are now
     no-ops, i.e., they write the output in the unsorted order of the
     input.  The ‘-a’ switch is now completely deprecated in favor of
     the clearer long option switches.

‘-b ‘file’’
     Activate native machine binary output writing to binary file
     ‘file’.  Also ‘--fl_bnr’ and ‘--binary-file’.  Writing packed
     variables in binary format is not supported.  Metadata is never
     output to the binary file.  Examine the netCDF output file to see
     the variables in the binary file.  Use the ‘-C’ switch, if
     necessary, to avoid wanting unwanted coordinates to the binary
     file:
          % ncks -O -v one_dmn_rec_var -b bnr.dat -p ~/nco/data in.nc out.nc
          % ls -l bnr.dat | cut -d ' ' -f 5 # 200 B contains time and one_dmn_rec_var
          200
          % ls -l bnr.dat
          % ncks -C -O -v one_dmn_rec_var -b bnr.dat -p ~/nco/data in.nc out.nc
          % ls -l bnr.dat | cut -d ' ' -f # 40 B contains one_dmn_rec_var only
          40

‘--cal’
     As of NCO version 4.6.5 (March, 2017), ‘ncks’ can print
     human-legible calendar strings corresponding to time values with
     UDUnits-compatible date units of the form time-since-basetime,
     e.g., ‘days since 2000-01-01’ and a CF calendar attribute, if any.
     Enact this with the ‘--calendar’ (also ‘--cln’, ‘--prn_lgb’, and
     ‘--datestamp’) option when printing in any mode.  Invoking this
     option when DBG_LVL >= 1 in CDL mode prints both the value and the
     calendar string (one in comments):
          zender@aerosol:~$ ncks -D 1 --cal -v tm_365 ~/nco/data/in.nc
          ...
            variables:
              double tm_365 ;
                tm_365:units = "days since 2013-01-01" ; // char
                tm_365:calendar = "365_day" ; // char
          
            data:
              tm_365 = "2013-03-01"; // double value: 59
          ...
          zender@aerosol:~$ ncks -D 1 -v tm_365 ~/nco/data/in.nc
          ...
              tm_365 = 59; // calendar format: "2013-03-01"
          ...
     This option is similar to the ‘ncdump’ ‘-t’ option.  As of NCO
     version 4.6.8 (August, 2017), ‘ncks’ CDL printing supports
     finer-grained control of date formats with the ‘--dt_fmt=DT_FMT’
     (or ‘--date_format’) option.  The DT_FMT is an enumerated integer
     from 0-3.  Values DT_FMT=0 or 1 correspond to the short format for
     dates that are the default.  The value DT_FMT=2 requests the
     "regular" format for dates, DT_FMT=3 requests the full ISO-8601
     format with the "T" separator and the comma:
          ncks -H -m -v time_bnds -C --dt_fmt=value ~/nco/data/in.nc
          # Value:    Output:
          # 0,1       1964-03-13 09:08:16        # Default, short format
          # 2         1964-03-13 09:08:16.000000 # Regular format
          # 3         1964-03-13T09:08:16.000000 # ISO8601 'T' format
     Note that ‘--dt_fmt’ automatically implies ‘--cal’ makes that
     options superfluous.

     As of NCO version 4.9.4 (September, 2020), invoking the ‘--dt_fmt’
     option now applies equally well to JSON and XML output as to CDL
     output:
          % ncks -d time,0 -v time --cdl --dt_fmt=3 ~/nco/data/in.nc
          ...
          time = "1964-03-13T21:09:0.000000" ;
          ...
          % ncks -d time,0 -v time --json --dt_fmt=3 ~/nco/data/in.nc
          ...
          "data": ["1964-03-13T21:09:0.000000"]
          ...
          % ncks -d time,0 -v time --xml --dt_fmt=3 ~/nco/data/in.nc
          ...
          <ncml:values separator="*">1964-03-13T21:09:0.000000</ncml:values>
          ...

‘--chk_map’
     As of NCO version 4.9.0 (December, 2019), invoking ‘--chk_map’
     causes ‘ncks’ to evaluate the quality of regridding weights in the
     map-file provided as INPUT-FILE.  This option works with map-files
     (not grid-files) in ESMF/CMIP6-compliant format (i.e., a sparse
     matrix variable named ‘S’ and coordinates ‘[xy][ab]_[cv]’.  When
     invoked with the additional ‘--area_wgt’ option, the evaluation
     statistics are area-weighted and thus exactly represent the
     global-mean/min/max/mebs/rms/sdn biases expected when regridding a
     globally uniform field.  This tool makes it easier to objectively
     assess weight-generation algorithms, and will hopefully assist in
     their improvement.  Thanks to Mark Taylor of Saturday Night Live
     (SNL) and Paul Ullrich of UC Davis for this suggestion and early
     prototypes.
          $ ncks --chk_map map.nc            # Unweighted statistics
          $ ncks --chk_map --dbg=2 map.nc    # Additional diagnostics
          $ ncks --chk_map --area_wgt map.nc # Area-weighted statistics

     The map-checker performs numerous checks and reports numerous
     statistics, probably more than you care about.  Be assured that
     each piece of provided information has in the past proved useful to
     developers of weight-generation and regridding algorithms.  Most of
     the time, users can learn whether the examined map is of sufficient
     quality for their purposes by examing only a few of these
     statistics.  Before defining these primary statistics, it is
     helpful to understand the meaning of the weight-array S (stored in
     a map-file as the variable ‘S’), and the terminology of rows and
     columns.

     A remapping (aka regridding) transforms a field on an input grid to
     an an output grid while conserving to the extent possible or
     desired the local and global properties of the field.  The map S is
     a matrix of M rows and N columns of weights, where M is the number
     of gridcells (or degrees of freedom, DOFs) in the destination grid,
     and N is the number of gridcells (or DOFs) in the source grid.  An
     individual weight S(M,N) represents the fractional contribution to
     destination gridcell M by source gridcell N.  By convention the
     weights are normalized to sum to unity in each row (destination
     gridcell) that completely overlaps the input grid.  Thus the
     weights in a single row are all equivalent to the fractional
     destination areas that the same destination gridcell (we will drop
     the DOF terminology hereafter for conciseness) receives from each
     source gridcell.  Regardless of the values of the individual
     weights, it is intuitive that their row-sum should never exceed
     unity because that would be physically equivalent to an output
     gridcell receiving more than its own area from the source grid.
     Map-files typically store these row-sum statistics for each
     destination gridcell in the ‘frac_b’ variable described further
     below.

     Likewise the weights in a single column represent the fractional
     destination areas that a single source gridcell contributes to
     every output gridcell.  Each output gridcell in a column may have a
     different area so column-sums need not, and in general do not, sum
     to unity.  However, a source gridcell ought to contribute to the
     destination grid a total area equal to its own area.  Thus a
     constraint on column-sums is that their weights, themselves
     weighted by the destination gridcell area corresponding to each
     row, should sum exactly to the source gridcell area.  In other
     words, the destination-area-weighted column-sum divided by the
     source gridcell area would be unity (in a perfect first order map)
     for every source gridcell that completely overlaps valid
     destination gridcells.  Map-files typically store these
     area-weighted-column-sum-ratio statistics for each gridcell in the
     ‘frac_a’ variable described further below.

     Storing the entire weight-matrix S is unnecessary because only a
     relative handful of gridcells in the source grid contribute to a
     given destination gridcell, and visa versa.  Instead, map-files
     store only the non-zero S(M,N), and encode them as a sparse-matrix.
     Storing S as a sparse matrix rather than a full matrix reduces
     overall storage sizes by a factor on the order of the ratio of the
     product of the grid sizes to their sum, or about 10,000 for grids
     with horizontal resolution near one degree, and more for finer
     resolutions.  The sparse-matrix representation is a one-dimensional
     array of weights ‘S’, together with two ancillary arrays, ‘row’ and
     ‘column’, that contain the one-dimensional row and column indices,
     respectively, corresponding to the destination and source gridcells
     of the associated weight.  By convention, map-files store the row
     and column indices using the 1-based convention in common use in
     the 1990s when regridding software was all written in Fortran.  The
     map-checker prints cell locations with 1-based indices as well:
          % ncks --chk_map map_ne30np4_to_cmip6_180x360_nco.20190601.nc
          Characterization of map-file map_ne30np4_to_cmip6_180x360_nco.20190601.nc
          Cell triplet elements : [Fortran (1-based) index, center latitude, center longitude]
          Sparse matrix size n_s: 246659
          Weight min S(190813):  5.1827201764857658e-25 from cell \
                                 [33796,-45.7998,+136.437] to [15975,-45.5,+134.5]
          Weight max S( 67391):  1.0000000000000000e+00 from cell \
                                 [33671,-54.4442,+189.645] to [12790,-54.5,+189.5]
          Ignored weights (S=0.0): 0
          ...
     Here the map-file weights span twenty-five orders of magnitude.
     This may seem large though in practice is typical for
     high-resolution intersection grids.  The Fortran-convention index
     of each weight extreme is followed by its geographic latitude and
     longitude.  Reporting the locations of extrema, and of gridcells
     whose metrics miss their target values by more than a specificied
     tolerance, are prime map-checker features.

     As mentioned above, the two statistics most telling about map
     quality are the weighted column-sums FRAC_A and the row-sums
     FRAC_B.  The short-hand names for what these metrics quantify are
     Conservation and Consistency, respectively.  Conservation means the
     total fraction of an input gridcell that contributes to the output
     grid.  For global input and output grids that completely tile the
     sphere, the entirety of each input gridcell should contribute
     (i.e., map to) the output grid.  The same concept that applies
     locally to conservation of a gridcell value applies globally to the
     overall conservation of an input field.  Thus a perfectly
     conservative mapping between global grids that tile the sphere
     would have FRAC_A = 1.0 for every input gridcell, and for the mean
     of all input gridcells.

     The map-checker computes Conservation (FRAC_A) from the stored
     variables ‘S’, ‘row’, ‘column’, ‘area_a’, and ‘area_b’ in the
     map-file, and then compares those values to the ‘frac_a’ values (if
     any) on-disk, and warns of any disagreements (2).  By definition,
     conservation is perfect to first order if the sum of the
     destination-gridcell-area-weighted weights (which is an area)
     equals the source gridcell area, and so their ratio (FRAC_A) is
     unity.  Computing the area-weighted-column-sum-ratios and comparing
     those FRAC_A to the stored ‘frac_a’ catches any discrepancies.  The
     analysis sounds an alarm when discrepancies exceed a tolerance
     (currently 5.0e-16).  More importantly, the map-checker reports the
     summary statistics of the computed FRAC_A metrics and their imputed
     errors, including the grid mean, minimum, maximum, mean-absolute
     bias, root-mean-square bias, and standard deviation.
          % ncks --chk_map map_ne30np4_to_cmip6_180x360_nco.20190601.nc
          ...
          Conservation metrics (column-sums of area_b-weighted weights normalized by area_a) and errors---
          Perfect metrics for global Grid B are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
          frac_a avg: 1.0000000000000000 = 1.0-0.0e+00 // Mean
          frac_a min: 0.9999999999991109 = 1.0-8.9e-13 // Minimum in grid A cell [45328,+77.3747,+225]
          frac_a max: 1.0000000000002398 = 1.0+2.4e-13 // Maximum in grid A cell [47582,+49.8351,+135]
          frac_a mbs: 0.0000000000000096 =     9.6e-15 // Mean absolute bias from 1.0
          frac_a rms: 0.0000000000000167 =     1.7e-14 // RMS relative to 1.0
          frac_a sdn: 0.0000000000000167 =     1.7e-14 // Standard deviation
          ...
     The values of the ‘frac_a’ metric are generally imperfect (not 1.0)
     for global grids.  The bias is the deviation from the target metric
     shown in the second floating-point column in each row above (e.g.,
     8.9e-13).  These biases should be vanishingly small with respect to
     unity.  Mean biases as large as 1.0e-08 may be considered
     acceptable for off-line analyses (i.e., a single regridding of raw
     data) though the acceptable tolerance should be more stringent for
     on-line use such as in a coupler where forward and reverse mappings
     may be applied tens of thousands of times.  The mean biases for
     such on-line regridding should be close to 1.0e-15 in order for
     tens-of-thousands of repetitions to still conserve mass/energy to
     full double-precision.

     The minimum and maximum gridcell biases indicate the worst
     performing locations of the mapping.  These are generally much (a
     few orders of magnitude) greater than the mean biases.  Observe
     that the minimum and maximum biases in the examples above and below
     occur at longitudes that are multiples of 45 degrees.  This is
     characteristic of mappings to/from for cube-square grids whose
     faces have edges, and thus additional complexity, at multiples of
     45 degrees.  This illustrates how intersection grid geometry
     influences biases.  More complex, finer-scale structures, produce
     greater biases.  The Root-Mean-Square (RMS) and standard deviation
     metrics characterize the distribution of biases throughout the
     entire intersection grid, and are thus complementary information to
     the minimum and maximum biases.

     Consistency expresses the total fraction of an output gridcell that
     receives contributions from the input grid.  Thus Consistency is
     directly analogous to Conservation, only applied to the output
     grid.  Conservation is the extent to which the mapping preserves
     the local and grid-wide integrals of input fields, while
     Consistency is the extent to which the mapping correctly aligns the
     input and output grids so that each destination cell receives the
     appropriate proportion of the input integrals.  The mapping will
     produce an acceptably faithful reproduction of the input on the
     output grid only if all local and global Conservation and
     Consistency metrics meet the acceptable error tolerances.

     The map-checker computes the Consistency (FRAC_B) as row-sums of
     the weights stored in ‘S’ and compares these to the stored values
     of ‘frac_b’.  (Note how the definition of weights S(M,N) as the
     fractional contribution to destination gridcell M by source
     gridcell N makes calculation of FRAC_B almost trivial in comparison
     to FRAC_A).  Nevertheless, ‘frac_b’ in the file may differ from the
     computed row-sum for example if the map-file generator artificially
     limits the stored ‘frac_b’ value for any cell to 1.0 for those
     row-sums that exceed 1.0.  The map-checker raises an alarm when
     discrepancies between computed and stored ‘frac_b’ exceed a
     tolerance (currently 5.0e-16).  There are semi-valid reasons a
     map-generator might do this, so this does not necessarily indicate
     an error.  The alarm simply informs the user that applying the
     weights will lead to a slightly different Consistency than
     indicated by the stored ‘frac_b’.

     As with ‘frac_a’, the values of ‘frac_b’ are generally imperfect
     (not 1.0) for global grids:
          % ncks --chk_map map_ne30np4_to_cmip6_180x360_nco.20190601.nc
          ...
          Consistency metrics (row-sums of weights) and errors---
          Perfect metrics for global Grid A are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
          frac_b avg: 0.9999999999999999 = 1.0-1.1e-16 // Mean
          frac_b min: 0.9999999999985523 = 1.0-1.4e-12 // Minimum in grid B cell [59446,+75.5,+45.5]
          frac_b max: 1.0000000000004521 = 1.0+4.5e-13 // Maximum in grid B cell [63766,+87.5,+45.5]
          frac_b mbs: 0.0000000000000065 =     6.5e-15 // Mean absolute bias from 1.0
          frac_b rms: 0.0000000000000190 =     1.9e-14 // RMS relative to 1.0
          frac_b sdn: 0.0000000000000190 =     1.9e-14 // Standard deviation
          ...
     This example shows that FRAC_B has the greatest local errors at
     similar boundaries (multiples of 45 degrees longitude) as FRAC_A.
     It is typical for Conservation and Consistency to degrade in
     intricate areas of the intersection grid, and these areas occur at
     multiples of 45 degrees longitude for cubed-sphere mappings.

     The map-checker will produce area-weighted metrics when invoked
     with the ‘--area_wgt’ flag, e.g., ‘ncks --area_wgt in.nc’.
     Area-weighted statistics show the exact local and global results to
     expect with real-world grids in which large
     consistency/conservation errors in small gridcells may be less
     important than smaller errors in larger gridcells.  Global-weighted
     mean statistics will of course differ from unweighted statistics,
     although the minimum and maximum do not change:
          % ncks --area_wgt map_ne30np4_to_cmip6_180x360_nco.20190601.nc
          ...
          Conservation metrics (column-sums of area_b-weighted weights normalized by area_a) and errors---
          Perfect metrics for global Grid B are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
          frac_a avg: 1.0000000000000009 = 1.0+8.9e-16 // Area-weighted mean
          frac_a min: 0.9999999999999236 = 1.0-7.6e-14 // Minimum in grid A cell [12810,+3.44654,+293.25]
          frac_a max: 1.0000000000001146 = 1.0+1.1e-13 // Maximum in grid A cell [16203,-45.7267,+272.31]
          frac_a mbs: 0.0000000000000067 =     6.7e-15 // Area-weighted mean absolute bias from 1.0
          frac_a rms: 0.0000000000000102 =     1.0e-14 // Area-weighted RMS relative to 1.0
          frac_a sdn: 0.0000000000000103 =     1.0e-14 // Standard deviation
          
          Consistency metrics (row-sums of weights) and errors---
          Perfect metrics for global Grid A are avg = min = max = 1.0, mbs = rms = sdn = 0.0:
          frac_b avg: 1.0000000000000047 = 1.0+4.7e-15 // Area-weighted mean
          frac_b min: 0.9999999999998442 = 1.0-1.6e-13 // Minimum in grid B cell [48415,+44.5,+174.5]
          frac_b max: 1.0000000000002611 = 1.0+2.6e-13 // Maximum in grid B cell [16558,-44.5,+357.5]
          frac_b mbs: 0.0000000000000065 =     6.5e-15 // Area-weighted mean absolute bias from 1.0
          frac_b rms: 0.0000000000000129 =     1.3e-14 // Area-weighted RMS relative to 1.0
          frac_b sdn: 0.0000000000000133 =     1.3e-14 // Standard deviation
          ...
     The examples above show no outstanding differences (besides
     rounding) between the unweighted and area-weighted statistics.  The
     absence of degradation between the global unweighted statistics
     (further up the page) and the global weighted statistics (just
     above) demonstrates there are no important correlations between
     local weight biases and gridcell areas.  The area-weighted mean
     FRAC_B statistic deserves special mention.  Its value is the exact
     factor by which the mapping will shift the global mean of a
     spatially uniform input field.  This metric is, therefore, first
     among equals when evaluating the quality of maps under
     consideration for use in time-stepping models where global
     conservation (e.g., of mass or energy) is crucial.

     As of NCO version 4.9.2 (March, 2020), adding the ‘--frac_b_nrm’
     flag changes the map-checker into a read-write algorithm that first
     diagnoses the map-file statistics described above and then
     re-writes the weights (and weight-derived statistics FRAC_A and
     FRAC_B) to compensate or "fix" issues that poor-quality input grids
     can cause.  Input grids can and often do have regions that are not
     tiled by any portion of any input gridcell.  For example, many FV
     ocean grids (such as MPAS) are empty (have no gridcells) in land
     regions beyond the coasts.  Some FV ocean grids have gridcells
     everywhere and mask (i.e., screen-out) the non-ocean gridcells by
     setting the mask value to zero.  Both these designs are perfectly
     legal.  What is illegal, yet sometimes encountered in practice, is
     overlapping gridcells on the same input grid.  Such an input grid
     is said to be self-overlapping.

     The surface topography dataset grid
     ‘SCRIPgrid_1km-merge-10min_HYDRO1K-merge-nomask_c130402.nc’
     (hereafter the HYDRO1K grid for short) used by E3SM and CESM is
     self-overlapping.  Weight-generators that receive the same input
     location twice might (if they do not take precaustions to idenfity
     the issue, which no known weight-generators do) double-weight the
     self-overlapped region(s).  In other words, self-overlapping input
     grids can lead weight-generators to produce values FRAC_B >> 1.0.
     Applying these weights would lead to exaggerated values on the
     destination grid.

     The best solution to this issue is to adjust the input grid to
     avoid self-overlap.  However, this solution may be difficult or
     impractical where the original data, producer, or algorithm are
     unavailable or unclear.  In such cases, the ‘--frac_b_nrm’ flag
     provides a workaround.  Please understand that ‘ncks --frac_b_nrm
     map.nc’ is designed to alter ‘map.nc’ in-xsplace, so backup the
     original file first.
          % ncks --frac_b_nrm map_hydro1k_to_ne1024np4_nco.20200301.nc
          ...
          ...

‘--chk_bnd’
     As of NCO version 5.2.0 (February, 2022), ‘ncks’ can report all
     coordinates that lack a corresponding ‘bounds’ attribute.  This
     check complies with CF Conventions and with NASA's Dataset
     Interoperability Working Group (DIWG
     (https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group)).
     CF requires that coordinate variables that describe a continuous
     (not discrete) axis contain a "bounds" attribute that points to a
     variable marking the edges of each gridcell (in time, space, or
     other dimensions).  This option reports which coordinates lack the
     required ‘bounds’ attribute, so that a file can be easily checked
     for compliance with the convention:
          $ ncks --chk_bnd in.nc
          ncks: WARNING nco_chk_bnd() reports coordinate Lat does not contain "bounds" attribute
          ncks: WARNING nco_chk_bnd() reports coordinate Lon does not contain "bounds" attribute
          ncks: INFO nco_chk_bnd() reports total number of coordinates without "bounds" attribute is 2

‘--chk_chr’
     The _identifiers_ in a netCDF file are the set of dimension, group,
     variable, and attribute names it contains.  As of NCO version 5.1.8
     (September, 2023), ‘ncks’ can report all identifiers that violate
     the CF Convention that identifiers "should begin with a letter and
     be composed of letters, digits, and underscores."  System or
     library-defined identifiers (such as ‘_FillValue’) are not subject
     to this (user-land) rule.  NASA's Dataset Interoperability Working
     Group (DIWG
     (https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group))
     supports this convention.  This option reports which identifiers do
     not comply with this convention, so that a file can be easily
     checked for compliance with the DIWG recommendation and the
     underlying CF Convention:
          $ ncks --chk_chr ~/nco/data/in.nc
          ...
          ncks: WARNING nco_chk_chr() reports variable att_var_spc_chr attribute name "at_in_name@" is not CF-compliant
          ncks: WARNING nco_chk_chr() reports variable name "var_nm-dash" is not CF-compliant
          ncks: WARNING nco_chk_chr() reports variable var_nm-dash attribute name "att_nm-dash" is not CF-compliant
          ncks: INFO nco_chk_chr() reports total number of identifiers with CF non-compliant names is 26

‘--chk_mss’
     As of NCO version 5.1.8 (September, 2023), ‘ncks’ can report all
     variables and groups that contain a ‘missing_value’ attribute.
     NASA's Dataset Interoperability Working Group (DIWG
     (https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group))
     notes that the ‘missing_value’ attribute has been semi-deprecated,
     and recommends that it should not be used in new Earth Science data
     products.  This option reports which variables (and groups) contain
     a ‘missing_value’ attribute, so that a file can be easily checked
     for compliance with the DIWG recommendation:
          $ ncks --chk_mss ~/nco/data/in.nc
          ncks: WARNING nco_chk_mss() reports variable fll_val_mss_val contains "missing_value" attribute
          ncks: WARNING nco_chk_mss() reports variable one_dmn_rec_var_missing_value contains "missing_value" attribute
          ...
          ncks: WARNING nco_chk_mss() reports variable rec_var_int_mss_val_flt contains "missing_value" attribute
          ncks: INFO nco_chk_mss() reports total number of variables and/or groups with "missing_value" attribute is 11

‘--chk_nan’
     As of NCO version 4.8.0 (May, 2019), ‘ncks’ can locate ‘NaN’ or
     ‘NaNf’ in double- and single-precision floating-point variables,
     respectively.  NCO prints the location of the first ‘NaN’ (if any)
     encountered in each variable.  NASA's Dataset Interoperability
     Working Group (DIWG
     (https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group))
     notes that the ‘missing_value’ attribute has been semi-deprecated,
     and recommends that it should not be used in new Earth Science data
     products.  This option reports allows users to easily check whether
     all the floating point variables in a file comply with the DIWG
     recommendation:
          $ ncks --chk_nan ~/nco/data/in_4.nc
          ncks: WARNING nco_chk_nan() reports variable /nan_arr has first NaNf at hyperslab element 1
          ncks: WARNING nco_chk_nan() reports variable /nan_scl has first NaNf at hyperslab element 0
          ncks: INFO nco_chk_nan() reports total number of floating-point variables with NaN elements is 2
     Thanks to Matthew Thompson of NASA for originally suggesting this
     feature.

‘--chk_xtn’
     A filename _extension_ is the suffix that follows the final period
     ‘.’ in a filename.  For example, the suffix of ‘in.file.nc’ is
     ‘nc’.  NASA's Dataset Interoperability Working Group (DIWG
     (https://wiki.earthdata.nasa.gov/display/ESDSWG/Dataset+Interoperability+Working+Group))
     recommends that "files created with the HDF5, HDF-EOS5, or netCDF
     APIs should have filename extensions \"h5\", \"he5\", or \"nc\",
     respectively."  As of NCO version 5.1.9 (November, 2023), ‘ncks’
     can report all filenames that violate this DIWG recommendation.
     This option reports which filenames do not comply with this
     convention.  If a file appears to be mis-labeled, e.g., the
     extension is ‘.h5’ but the file contents match HDF5-EOS structure,
     that will also be reported.
          zender@spectral:~$ ncks --chk_xtn ~/nco/data/in.nc
          zender@spectral:~$ ncks --chk_xtn ~/in.nc4
          ncks: WARNING nco_chk_xtn() reports filename extension "nc4" is non-compliant
          ncks: HINT rename file with "nc" rather than "nc4" extension
          ncks: INFO nco_chk_xtn() reports total number of non-compliant filename extensions is 1

‘--fix_rec_dmn’
     Change record dimension DIM in the input file into a fixed
     dimension in the output file.  Also ‘--no_rec_dmn’.  Before NCO
     version 4.2.5 (January, 2013), the syntax for ‘--fix_rec_dmn’ did
     not permit or require the specification of the dimension name DIM.
     This is because the feature only worked on netCDF3 files, which
     support only one record dimension, so specifying its name was
     unnecessary.  netCDF4 files allow an arbitrary number of record
     dimensions, so the user must specify which record dimension to fix.
     The decision was made that starting with NCO version 4.2.5
     (January, 2013), it is always required to specify the dimension
     name to fix regardless of the netCDF file type.  This keeps the
     code simple, and is symmetric with the syntax for ‘--mk_rec_dmn’,
     described next.

     As of NCO version 4.4.0 (January, 2014), the argument ‘all’ may be
     given to ‘--fix_rec_dmn’ to convert _all_ record dimensions to
     fixed dimensions in the output file.  Previously, ‘--fix_rec_dmn’
     only allowed one option, the name of a single record dimension to
     be fixed.  Now it is simple to simultaneously fix all record
     dimensions.  This is useful (and nearly mandatory) when flattening
     netCDF4 files that have multiple record dimensions per group into
     netCDF3 files (which are limited to at most one record dimension)
     (*note Group Path Editing::).

‘--fpe’
     As of NCO version 5.3.3 (March, 2025), ‘ncks’ accepts the flag
     ‘--fpe’ (or long-option equivalent ‘--floating_point_exception’) to
     report the status of all IEEE FPE status after command execution.
     This can be useful to verify whether NCO produced any
     arithmetically questionable results:
          % ncks --fpe -O -7 -L 1 -C -v ppc_zro_ngt_nan_flt --qnt_alg=gbr \
                 --qnt default=3 --cmp='shf|zst' in.nc out.nc
          ncks: Successfully set FE_INVALID|FE_DIVBYZERO|FE_OVERFLOW exception flags
          ncks: FE_DIVBYZERO is not raised
          ncks: FE_INEXACT is raised
          ncks: FE_INVALID is not raised
          ncks: FE_OVERFLOW is not raised
          ncks: FE_UNDERFLOW is not raised
     Note that ‘FE_INEXACT’ can be ignored since it is triggered by
     common numbers like ‘1.0/3.0’ and ‘sqrt(2.0)’.  The other
     conditions may warrant further investigation.  Note that the
     regridder _is_ ‘ncks’, and can be passed this flag with ‘ncreamp
     --nco_opt='--fpe'’.

‘--hdn’
     As of NCO version 4.4.0 (January, 2014), the ‘--hdn’ or ‘--hidden’
     options print hidden (aka special) attributes.  This is equivalent
     to ‘ncdump -s’.  Hidden attributes include: ‘_Format’,
     ‘_DeflateLevel’, ‘_Shuffle’, ‘_Storage’, ‘_ChunkSizes’,
     ‘_Endianness’, ‘_Fletcher32’, and ‘_NOFILL’.  Previously ‘ncks’
     ignored all these attributes in CDL/XML modes.  Now it prints these
     attributes as appropriate in all modes.  As of NCO version 4.4.6
     (September, 2014), ‘--hdn’ also prints the extended file format
     (i.e., the format of the file or server supplying the data) as
     ‘_SOURCE_FORMAT’.  As of NCO version 4.6.1 (August, 2016), ‘--hdn’
     also prints the hidden attributes ‘_NCProperties’, ‘_IsNetcdf4’,
     and ‘_SuperblockVersion’ for netCDF4 files so long as NCO is linked
     against netCDF library version 4.4.1 or later.  Users are referred
     to the Unidata netCDF Documentation
     (http://www.unidata.ucar.edu/software/netcdf/docs), or the man
     pages for ‘ncgen’ or ‘ncdump’, for detailed descriptions of the
     meanings of these hidden attributes.

‘--cdl’
     As of NCO version 4.3.3 (July, 2013), ‘ncks’ can print extracted
     data and metadata to screen (i.e., ‘stdout’) as valid CDL (network
     Common data form Description Language).  CDL is the human-readable
     "lingua franca" of netCDF ingested by ‘ncgen’ and excreted by
     ‘ncdump’.  As of NCO version 4.6.9 (September, 2017), ‘ncks’ prints
     CDL by default, and the "traditional" mode must be explicitly
     selected with ‘--trd’.  Compare ‘ncks’ "traditional" with CDL
     printing:
          zender@roulee:~$ ncks --trd -v one ~/nco/data/in.nc
          one: type NC_FLOAT, 0 dimensions, 1 attribute, chunked? no, compressed? no, packed? no
          one size (RAM) = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes
          one attribute 0: long_name, size = 3 NC_CHAR, value = one
          
          one = 1 
          
          zender@roulee:~$ ncks --cdl -v one ~/nco/data/in.nc
          netcdf in {
          
            variables:
              float one ;
              one:long_name = "one" ;
          
            data:
              one = 1 ;
          
          } // group /
     Users should note the NCO's CDL mode outputs successively more
     verbose additional diagnostic information in CDL comments as the
     level of debugging increases from zero to two.  For example
     printing the above with ‘-D 2’ yields
          zender@roulee:~$ ncks -D 2 --cdl -v one ~/nco/data/in.nc
          netcdf in {
            // ncgen -k classic -b -o in.nc in.cdl
          
            variables:
              float one ; // RAM size = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes, ID = 147
                one:long_name = "one" ; // char
          
            data:
              one = 1 ; 
          
          } // group /

     ‘ncgen’ converts CDL-mode output into a netCDF file:
          ncks -v one ~/nco/data/in.nc > ~/in.cdl
          ncgen -k netCDF-4 -b -o ~/in.nc ~/in.cdl
          ncks -v one ~/in.nc
     The HDF4 version of ‘ncgen’, often named ‘hncgen’, ‘h4_ncgen’, or
     ‘ncgen-hdf’, (usually) converts netCDF3 CDL into an HDF file:
          cd ~/nco/data
          ncgen      -b -o hdf.hdf hdf.cdl # HDF ncgen is sometimes named...ncgen
          ncgen      -b -o in.hdf  in.cdl  # Fails: Some valid netCDF CDL breaks HDF ncgen
          hncgen     -b -o hdf.hdf hdf.cdl # HDF ncgen is hncgen in some RPM packages
          h4_ncgen   -b -o hdf.hdf hdf.cdl # HDF ncgen is h4_ncgen in Anaconda packages
          ncgen-hdf  -b -o hdf.hdf hdf.cdl # HDF ncgen is ncgen-hdf in some Debian packages
          hdp dumpsds hdf.hdf              # ncdump/h5dump-equivalent for HDF4
          h4_ncdump dumpsds hdf.hdf        # ncdump/h5dump-equivalent for HDF4
     Note that HDF4 does not support netCDF-style groups, so the above
     commands fail when the input file contains groups.  Only netCDF4
     and HDF5 support groups.  In our experience the HDF ‘ncgen’
     command, by whatever name installed, is not robust and fails on
     many valid netCDF3 CDL constructs.  The HDF4 version of ‘ncgen’
     will definitely fail on the default NCO input file
     ‘nco/data/in.cdl’.  The NCO source code distribution provides
     ‘nco/data/hdf.cdl’ that works with the HDF4 version of ‘ncgen’, and
     can be used to test HDF files.

‘--mk_rec_dmn DIM’
     Change existing dimension DIM to a record dimension in the output
     file.  This is the most straightforward way of changing a dimension
     to a/the record dimension, and works fine in most cases.  See *note
     ncecat netCDF Ensemble Concatenator:: and *note ncpdq netCDF
     Permute Dimensions Quickly:: for other methods of changing variable
     dimensionality, including the record dimension.

‘-H’
     Toggle (turn-on or turn-off) default behavior of printing data (not
     metadata) to screen or copying data to disk.  Also activated using
     ‘--print’ or ‘--prn’.  By default ‘ncks’ prints all metadata but no
     data to screen when no netCDF OUTPUT-FILE is specified.  And if
     OUTPUT-FILE is specified, ‘ncks’ copies all metadata and all data
     to it.  In other words, the printing/copying default is
     context-sensitive, and ‘-H’ toggles the default behavior.  Hence,
     use ‘-H’ to turn-off copying data (not metadata) to an output file.
     (It is occasionally useful to write all metadata to a file, so that
     the file has allocated the required disk space to hold the data,
     yet to withold writing the data itself).  And use ‘-H’ to turn-on
     printing data (not metadata) to screen.  Unless otherwise specified
     (with ‘-s’), each element of the data hyperslab prints on a
     separate line containing the names, indices, and, values, if any,
     of all of the variables dimensions.  The dimension and variable
     indices refer to the location of the corresponding data element
     with respect to the variable as stored on disk (i.e., not the
     hyperslab).
          % ncks --trd -C -v three_dmn_var in.nc
          lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0
          lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1
          lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2
          ...
          lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21
          lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22
          lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23
     Printing the same variable with the ‘-F’ option shows the same
     variable indexed with Fortran conventions
          % ncks -F -C -v three_dmn_var in.nc
          lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0
          lon(2)=90 lev(1)=100 lat(1)=-90 three_dmn_var(2)=1
          lon(3)=180 lev(1)=100 lat(1)=-90 three_dmn_var(3)=2
          ...
     Printing a hyperslab does not affect the variable or dimension
     indices since these indices are relative to the full variable (as
     stored in the input file), and the input file has not changed.
     However, if the hyperslab is saved to an output file and those
     values are printed, the indices will change:
          % ncks --trd -H -d lat,90.0 -d lev,1000.0 -v three_dmn_var in.nc out.nc
          ...
          lat[1]=90 lev[2]=1000 lon[0]=0 three_dmn_var[20]=20
          lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21
          lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22
          lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23
          % ncks --trd -C -v three_dmn_var out.nc
          lat[0]=90 lev[0]=1000 lon[0]=0 three_dmn_var[0]=20
          lat[0]=90 lev[0]=1000 lon[1]=90 three_dmn_var[1]=21
          lat[0]=90 lev[0]=1000 lon[2]=180 three_dmn_var[2]=22
          lat[0]=90 lev[0]=1000 lon[3]=270 three_dmn_var[3]=23

‘--is_hrz’
     This option checks whether the single variable VAR is a "horizontal
     variable".  In this context, horizontal variables are single level
     variables that have horizontal dimensions (in latitude and
     longitude) yet no vertical dimension and no other dimensions except
     possibly the temporal dimension (time).  Introduced in NCO version
     5.3.4 (June, 2025), this capability helps ‘ncclimo’ determine
     whether a variable is suitable for creating a regional average
     timeseries from.  Due to this context, coordinate variables
     (including latitude and longitude) are not identified as horiztonal
     variables.  Furthermore, variables identified as horizontal may
     only have two or three dimensions, and those dimensions must be in
     this list: ‘lat’, ‘lon’, ‘ncol’, ‘nCells’, ‘time’, and ‘Time’.
     This E3SM-centric list of dimension names is simply a starting
     point.  Please contact Charlie if you would like this list
     expanded.
          zender@spectral:~$ ncks --is_hrz three_dmn_rec_var ~/nco/data/in.nc
          Yes
          zender@spectral:~$ ncks --is_hrz one ~/nco/data/in.nc
          No

‘--jsn, --json’
     As of NCO version 4.6.2 (November, 2016), ‘ncks’ can print
     extracted metadata and data to screen (i.e., ‘stdout’) as JSON,
     JavaScript Object Notation, defined here (http://www.json.org).
     ‘ncks’ supports JSON output more completely, flexibly, and robustly
     than any other tool to our knowledge.  With ‘ncks’ one can
     translate entire netCDF3 and netCDF4 files into JSON, including
     metadata and data, using all NCO's subsetting and hyperslabbing
     capabilities.  NCO uses a JSON format we developed ourselves,
     during a year of discussion among interested researchers.  Some
     refer to this format as NCO-JSON, to disambiguate it from other
     JSON formats for netCDF data.  Other projects have since adopted,
     and some can generate, NCO-JSON.  Projects that support NCO-JSON
     include ERDDAP
     (<https://coastwatch.pfeg.noaa.gov/erddap/index.html>, choose
     output filetype ‘.ncoJson’ from this table
     (https://coastwatch.pfeg.noaa.gov/erddap/griddap/documentation.html#fileType))
     and CF-JSON (<http://cf-json.org>).

     Behold JSON output in default mode:
          zender@aerosol:~$ ncks --jsn -v one ~/nco/data/in.nc
          {
            "variables": {
              "one": {
                "type": "float",
                "attributes": {
                  "long_name": "one"
                },
                "data": 1.0
              }
            }
          }
     NCO converts to (using commonsense rules) and prints all NC_TYPEs
     as one of three atomic types distinguishable as JSON values:
     ‘float’, ‘string’, and ‘int’ (3).  Floating-point types (‘NC_FLOAT’
     and ‘NC_DOUBLE’) are printed with a decimal point and at least one
     signficant digit following the decimal point, e.g., ‘1.0’ rather
     than ‘1.’ or ‘1’.  Integer types (e.g., ‘NC_INT’, ‘NC_UINT64’) are
     printed with no decimal point.  String types (‘NC_CHAR’ and
     ‘NC_STRING’) are enclosed in double-quotes.

     The JSON specification allows many possible output formats for
     netCDF files.  NCO developers implemented a working prototype in
     Octoboer, 2016 and, after discussing options with interested
     parties here
     (https://sourceforge.net/p/nco/discussion/9829/thread/8c4d7e72),
     finalized the emitted JSON syntax a few weeks later.  The resulting
     JSON backend supports three levels of pedanticness, ordered from
     more concise, flexible, and human-readable to more verbose,
     restrictive, and 1-to-1 reproducible.  JSON-specific switches
     access these modes and other features.  Each JSON configuration
     option automatically triggers JSON printing, so that specifying
     ‘--json’ in addition to a JSON configuration option is redundant
     and unnecessary.

     Request a specific format level with the pedantic level argument to
     the ‘--jsn_fmt LVL’ option.  As of NCO version 4.6.3 (December,
     2016), the option formerly known as ‘--jsn_att_fmt’ was renamed
     simply ‘--jsn_fmt’.  The more general name reflects the fact that
     the option controls all JSON formatting, not just attribute
     formatting.  As of version 4.6.3, NCO defaults to demarcate inner
     dimensions of variable data with (nested) square brackets rather
     than printing data as an unrolled single dimensional array.  An
     array with C-ordered dimensionality [2,3,4] prints as:
          % ncks --jsn -v three_dmn_var ~/nco/data/in.nc
          ...
          "data": [[[0.0, 1.0, 2.0, 3.0], [4.0, 5.0, 6.0, 7.0], [8.0, 9.0, 10.0,11.0]], [[12.0, 13.0, 14.0, 15.0], [16.0, 17.0, 18.0, 19.0], [20.0,21.0, 22.0, 23.0]]]
          ...
          % ncks --jsn_fmt=4 -v three_dmn_var ~/nco/data/in.nc
          ...
          "data": [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0,22.0, 23.0]
          ...
     One can recover the former behavior (and omit the brackets) by
     adding four to the base pedantic level LVL (as shown above).
     Besides the potential offset of four, LVL may take one of three
     values between 0-2:
        • LVL = 0 is the default mode, and is also explicitly selectable
          with ‘--jsn_fmt=0’.  All values are output without the
          original NC_TYPE token.  This allows attributes to print as
          JSON name-value pairs, rather than as more complex objects:
               % ncks --jsn_fmt=0 -v att_var ~/nco/data/in_grp.nc
               ...
               "att_var": {
                 "shape": ["time"],
                 "type": "float",
                 "attributes": {
                   "byte_att": [0, 1, 2, 127, -128, -127, -2, -1],
                   "char_att": "Sentence one.\nSentence two.\n",
                   "short_att": 37,
                   "int_att": 73,
                   "long_att": 73,
                   "float_att": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.010],
                   "double_att": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.0100010]
                 },
                 "data": [10.0, 10.10, 10.20, 10.30, 10.40101, 10.50, 10.60, 10.70, 10.80, 10.990]
               ...

          This least pedantic mode produces the most easily read
          results, and suffices for many (most?)  purposes.  Any
          downstream parser is expected to assign an appropriate type as
          indicated by JSON syntax rules.  Because the original
          attributes' ‘NC_TYPE’ are not output, a downstream parser may
          not exactly reproduce the input file datatypes.  For example,
          whether the original attribute string was stored as ‘NC_CHAR’
          or ‘NC_STRING’ will be unknown to a downstream parser.
          Distinctions between ‘NC_FLOAT’ and ‘NC_DOUBLE’ are similarly
          lost, as are all distinctions among the integer types.

          In our experience, these distinctions are immaterial for
          attributes, which are intended for metadata not for
          large-scale storage.  Type-distinctions can, however,
          significantly impact the size of variable data, responsible
          for nearly all the storage required by datasets.  For
          instance, storing or transferring an ‘NC_SHORT’ field as
          ‘NC_DOUBLE’ would waste a factor of four in space or
          bandwidth.  This is why NCO _always_ prints the ‘NC_TYPE’ of
          variable data.  Downstream parsers can (but are not required
          to) take advantage of the variable's ‘NC_TYPE’ to choose the
          most efficient storage type.

          The Shape member of the variable object is an ordered array of
          dimension names such as ‘"shape": ["lat","lon"]’, similar to
          its use in NcML. Each name corresponds to a previously defined
          Dimension object that, taken together, define the rank, shape,
          and size of the variable.  Variables are assumed to be scalar
          by default.  Hence the Shape member is mandatory for arrays,
          and is always omitted for scalars (by contrast, NcML requires
          an empty shape string to indicate scalars).

        • LVL = 1 is a medium-pedantic level that prints all attributes
          as objects (with explicit types) unless the attribute type
          match the simplest default JSON value types.  In other words,
          attributes of type ‘NC_FLOAT’, ‘NC_CHAR’, ‘NC_SHORT’, and
          ‘NC_BYTE’ are printed as objects with an explicit type so that
          parsers do not use the default type.  Attributes of type
          ‘NC_DOUBLE’, ‘NC_STRING’, and ‘NC_INT’ are printed as JSON
          arrays, as in the LVL = 0 above:
               % ncks --jsn_fmt=1 -v att_var ~/nco/data/in.nc
               ...
               "att_var": {
                 "shape": ["time"],
                 "type": "float",
                 "attributes": {
                   "byte_att": { "type": "byte", "data": [0, 1, 2, 127, -128, -127, -2, -1]},
                   "char_att": "Sentence one.\nSentence two.\n",
                   "short_att": { "type": "short", "data": 37},
                   "int_att": 73,
                   "long_att": 73,
                   "float_att": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.010],
                   "double_att": { "type": "double", "data": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.0100010]}
                 },
                 "data": [10.0, 10.10, 10.20, 10.30, 10.40101, 10.50, 10.60, 10.70, 10.80, 10.990]
               ...
          The attributes of type ‘NC_BYTE’, ‘NC_SHORT’, and ‘NC_DOUBLE’
          are printed as JSON objects that comprise an ‘NC_TYPE’ and a
          value list, because their values could conceivably not be
          representable, or would waste space, if interpreted as
          ‘NC_INT’ or ‘NC_FLOAT’, respectively.  All other attributes
          may be naturally mapped to the type indicated by the JSON
          syntax of the value, where numbers are assumed to correspond
          to ‘NC_FLOAT’ for floating-point, ‘NC_INT’ for integers, and
          ‘NC_CHAR’ or ‘NC_STRING’ for strings.  This minimal increase
          in verbosity allows a downstream parser to re-construct the
          original dataset with nearly identical attributes types to the
          original.

        • LVL = 2 is the most pedantic mode, and should be used when
          preserving all input types (e.g., to ensure exact
          reproducibility of the input file) is important.  This mode
          always prints attributes as JSON objects with a type value so
          that any downstream parser can (though it need not) guarantee
          exact reproduction of the original dataset:
               % ncks --jsn_fmt=2 -v att_var ~/nco/data/in.nc
               ...
               "att_var": {
                 "shape": ["time"],
                 "type": "float",
                 "attributes": {
                   "byte_att": { "type": "byte", "data": [0, 1, 2, 127, -128, -127, -2, -1]},
                   "char_att": { "type": "char", "data": "Sentence one.\nSentence two.\n"},
                   "short_att": { "type": "short", "data": 37},
                   "int_att": { "type": "int", "data": 73},
                   "long_att": { "type": "int", "data": 73},
                   "float_att": { "type": "float", "data": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.010]},
                   "double_att": { "type": "double", "data": [73.0, 72.0, 71.0, 70.010, 69.0010, 68.010, 67.0100010]}
                 },
                 "data": [10.0, 10.10, 10.20, 10.30, 10.40101, 10.50, 10.60, 10.70, 10.80, 10.990]
               ...

     That ncks produces correct translations of for all supported
     datatypes may be verified by a JSON syntax checker command like
     ‘jsonlint’.  Please let us know how to improve JSON features for
     your application.

‘-M’
     Turn-on printing to screen or turn-off copying global and group
     metadata.  This includes file summary information and global and
     group attributes.  Also ‘--Mtd’ and ‘--Metadata’.  By default
     ‘ncks’ prints global metadata to screen if no netCDF output file
     and no variable extraction list is specified (with ‘-v’).  Use ‘-M’
     to print global metadata to screen if a netCDF output is specified,
     or if a variable extraction list is specified (with ‘-v’).  Use
     ‘-M’ to turn-off copying of global and group metadata when copying,
     subsetting, or appending to an output file.

     The various combinations of printing switches can be confusing.  In
     an attempt to anticipate what most users want to do, ‘ncks’ uses
     context-sensitive defaults for printing.  Our goal is to minimize
     the use of switches required to accomplish the common operations.
     We assume that users creating a new file or overwriting (e.g., with
     ‘-O’) an existing file usually wish to copy all global and
     variable-specific attributes to the new file.  In contrast, we
     assume that users appending (e.g., with ‘-A’ an explicit variable
     list from one file to another usually wish to copy only the
     variable-specific attributes to the output file.  The switches
     ‘-H’, ‘-M’, and ‘-m’ switches are implemented as toggles which
     reverse the default behavior.  The most confusing aspect of this is
     that ‘-M’ inhibits copying global metadata in overwrite mode and
     causes copying of global metadata in append mode.
          ncks                 in.nc        # Print  VAs and GAs
          ncks          -v one in.nc        # Print  VAs not GAs
          ncks    -M    -v one in.nc        # Print  GAs only
          ncks       -m -v one in.nc        # Print  VAs only
          ncks    -M -m -v one in.nc        # Print  VAs and GAs
          ncks -O              in.nc out.nc # Copy   VAs and GAs
          ncks -O       -v one in.nc out.nc # Copy   VAs and GAs
          ncks -O -M    -v one in.nc out.nc # Copy   VAs not GAs
          ncks -O    -m -v one in.nc out.nc # Copy   GAs not VAs
          ncks -O -M -m -v one in.nc out.nc # Copy   only data (no atts)
          ncks -A              in.nc out.nc # Append VAs and GAs
          ncks -A       -v one in.nc out.nc # Append VAs not GAs
          ncks -A -M    -v one in.nc out.nc # Append VAs and GAs
          ncks -A    -m -v one in.nc out.nc # Append only data (no atts)
          ncks -A -M -m -v one in.nc out.nc # Append GAs not VAs
     where ‘VAs’ and ‘GAs’ denote variable and group/global attributes,
     respectively.

‘-m’
     Turn-on printing to screen or turn-off copying variable metadata.
     Using ‘-m’ will print variable metadata to screen (similar to
     ‘ncdump -h’).  This displays all metadata pertaining to each
     variable, one variable at a time.  This includes information on the
     storage properties of the variable, such as whether it employs
     chunking, compression, or packing.  Also activated using ‘--mtd’
     and ‘--metadata’.  The ‘ncks’ default behavior is to print variable
     metadata to screen if no netCDF output file is specified.  Use ‘-m’
     to print variable metadata to screen if a netCDF output is
     specified.  Also use ‘-m’ to turn-off copying of variable metadata
     to an output file.

‘--no_blank’
     Print numeric representation of missing values.  As of NCO version
     4.2.2 (October, 2012), NCO prints missing values as blanks (i.e.,
     the underscore character ‘_’) by default.  To enable the old
     behavior of printing the numeric representation of missing values
     (e.g., ‘1.0e36’), use the ‘--no_blank’ switch.  Also activated
     using ‘--noblank’ or ‘--no-blank’.

‘-P’
     Print data, metadata, and units to screen.  The ‘-P’ switch is a
     convenience abbreviation for ‘-C -H -M -m -u’.  Also activated
     using ‘--print’ or ‘--prn’.  This set of switches is useful for
     exploring file contents.

‘--prn_fl ‘print-file’’
     Activate printing formatted output to file ‘print-file’.  Also
     ‘--print_file’, ‘--fl_prn’, and ‘--file_print’.  One can achieve
     the same result by redirecting stdout to a named file.  However, it
     is slightly faster to print formatted output directly to a file
     than to stdout:
          ncks --fl_prn=foo.txt --jsn in.nc

‘-Q’
     Print quietly, meaning omit dimension names, indices, and
     coordinate values when printing arrays.  Variable (not dimension)
     indices are printed.  Variable names appear flush left in the
     output:
          zender@roulee:~$ ncks --trd -Q -v three_dmn_rec_var -C -H ~/nco/data/in.nc              
          three_dmn_rec_var[0]=1 
          ...
     This helps locate specific variables in lists with many variables
     and different dimensions.  See also the ‘-V’ option, which omits
     all names and indices and prints only variable values.

‘-q’
     Quench (turn-off) all printing to screen.  This overrides the
     setting of all print-related switches, equivalent to ‘-H -M -m’
     when in single-file printing mode.  When invoked with ‘-R’ (*note
     Retaining Retrieved Files::), ‘ncks’ automatically sets ‘-q’.  This
     allows ‘ncks’ to retrieve remote files without automatically trying
     to print them.  Also ‘--quench’.

‘--rad’
     Retain all dimensions.  When invoked with ‘--rad’ (Retain All
     Dimensions), ‘ncks’ copies each dimension in the input file to the
     output file, regardless of whether the dimension is utilized by any
     variables.  Normally ‘ncks’ discards "orphan dimensions", i.e.,
     dimensions not referenced by any variables.  This switch allows
     users to keep non-referenced dimensions in the workflow.  When
     invoked in printing mode, causes orphaned dimensions to be printed
     (they are not printed by default).  Also ‘--retain_all_dimensions’,
     ‘--orphan_dimensions’, and ‘--rph_dmn’.

‘-s FORMAT’
     String format for text output.  Accepts C language escape sequences
     and ‘printf()’ formats.  Also ‘--string’ and ‘--sng_fmt’.  This
     option is only intended for use with traditional (TRD) printing,
     and thus automatically invokes the ‘--trd’ switch.

‘--fmt_val FORMAT’
     Supply a ‘printf()’-style format for printed output, i.e., in CDL,
     JSON, TRD, or XML modes.  Also ‘--val_fmt’ and ‘--value_format’.
     One use for this option is to reduce the printed precision of
     floating point values:
          # Default printing of original double precision values
          # 0.0,0.1,0.12,0.123,0.1234,0.12345,0.123456,0.1234567,0.12345678,0.123456789
          % ncks -C -v ppc_dbl ~/nco/data/in.nc
          ...
          ppc_dbl = 0, 0.1, 0.12, 0.123, 0.1234, 0.12345, 0.123456, 0.1234567, 0.12345678, 0.123456789 ;
          ...
          # Restrict printing to three digits after the decimal
          % ncks --fmt_val=%.3f -C -v ppc_dbl ~/nco/data/in.nc
          ...
          ppc_dbl = 0., 0.1, 0.12, 0.123, 0.123, 0.123, 0.123, 0.123, 0.123, 0.123 ;
          ...
     The supplied FORMAT only applies to floating point variable values
     (‘NC_FLOAT’ or ‘NC_DOUBLE’), and not to other types or to
     attributes.  For reference, the default ‘printf()’ FORMAT for CDL,
     JSON, TRD, and XML modes is ‘%#.7gf’, ‘%#.7g’, ‘%g’, and ‘%#.7g’,
     respectively, for single-precision data, and, for double-precision
     data is ‘%#.15g’, ‘%#.15g’, ‘%.12g’, and ‘%#.15g’, respectively.
     NCO introduced this feature in version 4.7.3 (March, 2018).  We
     would appreciate your feedback on whether and how to extend this
     feature to make it more useful.

‘--s1d, --sparse, --unpack_sparse, --hrz ‘file’’
     As of NCO version 5.2.0, released in February, 2024, ‘ncks’ can
     help analyze initial condition and restart datasets produced by the
     E3SM ELM and CESM CLM/CTSM land-surface models.  Whereas gridded
     history datasets from these ESMs use a standard gridded data
     format, land-surface "restart files" employ a custom packing format
     that unwinds multi-dimensional data into sparse, 1-D (S1D) arrays
     that are not easily visualized.  ‘ncks’ can convert S1D files into
     gridded datasets where all dimensions are explicitly declared,
     rather than unrolled or "packed".  Invoke this conversion feature
     with the ‘--s1d’ option (or long option equivalents, ‘--sparse’ or
     ‘--unpacksparse’) and, with ‘--hrz_crd FL_HRZ’ (e.g., ‘--hrz_crd
     ‘hrz.nc’’), point to the file that contains the horizontal
     coordinates (that restart files usually omit).  The output file is
     the fully gridded input file, with no loss of information:
          ncks --s1d --hrz=elmv3_history.nc elmv3_restart.nc out.nc
     The output file contains all input variables placed on a lat-lon or
     unstructured grid, with new dimensions for Plant Funtional Type
     (PFT) and multiple elevation classes (MECs).

     The S1D capabilities have steadily grown culminating in major new
     features in NCO version 5.2.9 and version 5.3.0, released in
     October and November, 2024, respectively.

     The ‘--rgr lut_out=$lut_out’ option specifies that only columns of
     specified landunit type(s) should appear in the output for
     column-variables.  The value LUT_OUT is the standard landunit type
     of the column.  Three additional values specify to output
     area-weighted averages of multiple landunit types:
          lut_out Output will be value of column(s) in this andunit 
          0       Not Currently Used
          1       Vegetated or bare soil
          2       Crop
          3       Landice (plain, no MEC)
          4       Landice multiple elevation classes
          5       Deep lake
          6       Wetland
          7       Urban tall building district
          8       Urban high density
          9       Urban medium density
          10      Area-weighted average of all landunit types except MEC glaciers
          13      Area-weighted average of soil+(non-MEC) glacier (types 1+3)
          789     Area-weighted average of all urban types (types 7+8+9)
     This feature is necessarily restricted to restart datasets, e.g.,
          ncks --s1d --rgr lut_out=1   --hrz=hst.nc rst.nc s1d.nc # Output Soil LUT
          ncks --s1d --rgr lut_out=13  --hrz=hst.nc rst.nc s1d.nc # Avg Soil+Glacier
          ncks --s1d --rgr lut_out=789 --hrz=hst.nc rst.nc s1d.nc # Avg Urban
     In conjunction with this capability, the ‘ncks’ S1D functionality
     also outputs two diagnostic variables that contain the
     area-fraction of different landunits and column types.  The
     ‘frc_landunit’ field contains the gridcell fraction occupied by
     every Landunit type and has dimension ‘landunit’ of size 10.  Index
     = 0 is the sum of all landunit fractions and should be 1.0 over all
     gridcells with any land fraction.  Indexes 1 through 9 are the
     standard Landunit types: 1 = Vegetated or bare soil; 2 = Crop; 3 =
     Landice (plain, no MEC); 4 = Landice multiple elevation classes; 5
     = Deep lake; 6 = Wetland; 7 = Urban tall building district; 8 =
     Urban high density; 9 = Urban medium density.

     The ‘frc_column’ field contains the gridcell fraction occupied by
     various columns (or groups of columns).  It has dimension
     ‘landunit’ of size 16 for datasets with Multiple Elevation Classes
     MECs and size 6 for datasets without MECs.  For datasets with MECs:
     index = 0 is soil column, index = 1 is MEC == 1, indexes 2..10 are
     remaining MEC columns, index 11 is sub-total of MEC columns, index
     = 12 is deep lake column, index = 13 is wetland column, index = 14
     is subtotal of natural (soil, MECs, lakes, wetlands) columns, and
     index = 15 is subtotal of urban columns.  For non-MEC datasets,
     index = 0 is soil column, index = 1 is glaciated column, index = 2
     is deep lake column, index = 3 is wetland column, index = 4 is
     subtotal of natural (soil, glacier, lakes, wetlands) columns, and
     index = 5 is subtotal of urban columns.

     S1D can now grid snow-related variables into a top-down
     (ocean-like) vertical grid that many think is more intuitive.  By
     default the land system models ELM, CLM, and CTSM store the
     negative of the number of active snow layers in the variable
     ‘SNLSNO’.  Restart files for these models store the active snow
     layer butted-up against the lowest layers in the snow-level
     dimension (so that they are continguous with soil layers to
     simplify hydrologic calculations).  This makes good modeling sense
     though also makes snow variables in restart files difficult to
     visualize.  By default S1D now uses ‘SNLSNO’, if present, to unpack
     active layers of snow variables into a top-layer first, downwards
     order, increasing with depth.  Inactive layers are underneath the
     bottom (i.e., where they reside physically).  The resulting snow
     variables appear like ocean state variables over uneven bathymetry,
     with missing values underneath.  We call this "snow-ocean" ordering
     to contrast it with the on-disk storage order of snow variables.
          ncks --s1d --rgr snw_ocn --hrz=hst.nc rst.nc s1d.nc # Snow-ocean order
          ncks --s1d --rgr no_snw_ocn --hrz=hst.nc rst.nc s1d.nc # Input order

‘--ssh, --secret’
     Print summary of ‘ncks’ hidden features.  These hidden or secret
     features are used mainly by developers.  They are not supported for
     general use and may change at any time.  This demonstrates
     conclusively that I cannot keep a secret.  Also ‘--ssh’ and
     ‘--scr’.

‘--trd, --traditional’
     From 1995-2017 ‘ncks’ dumped the ASCII text representation of
     netCDF files in what we now call "traditional" mode.  Much of this
     manual contains output printed in traditional mode, which places
     one value per line, with complete dimensional information.
     Traditional-mode metadata output includes lower-level information,
     such as RAM usage and internal variable IDs, than CDL.  While this
     is useful for some developers and user, CDL has, over the years,
     become more useful than traditional mode for most users.  As of NCO
     version 4.6.9 (September, 2017) CDL became the default printing
     mode.  Traditional printing mode is accessed via the ‘--trd’
     option.

‘-u, --units’
     Toggle the printing of a variable's ‘units’ attribute, if any, with
     its values.  Also ‘--units’.

‘-V’
     Print variable values only.  Do not print variable and dimension
     names, indices, and coordinate values when printing arrays.
          zender@roulee:~$ ncks --trd -V -v three_dmn_rec_var -C -H ~/nco/data/in.nc
          1
          ...
     See also the ‘-Q’ option, which prints variable names and indices,
     but not dimension names, indices, or coordinate values when
     printing arrays.  Using ‘-V’ is the same as specifying ‘-Q
     --no_nm_prn’.

‘--xml, --ncml’
     As of NCO version 4.3.3 (July, 2013), ‘ncks’ can print extracted
     data and metadata to screen (i.e., ‘stdout’) as XML in NcML, the
     netCDF Markup Language.  ‘ncks’ supports XML more completely than
     ‘ncdump -x’.  With ‘ncks’ one can translate entire netCDF3 and
     netCDF4 files into NcML, including metadata and data, using all
     NCO's subsetting and hyperslabbing capabilities.  Compare ‘ncks’
     "traditional" with XML printing:
          zender@roulee:~$ ncks --trd -v one ~/nco/data/in.nc
          one: type NC_FLOAT, 0 dimensions, 1 attribute, chunked? no, compressed? no, packed? no
          one size (RAM) = 1*sizeof(NC_FLOAT) = 1*4 = 4 bytes
          one attribute 0: long_name, size = 3 NC_CHAR, value = one
          
          one = 1 
          
          zender@roulee:~$ ncks --xml -v one ~/nco/data/in.nc
          <?xml version="1.0" encoding="UTF-8"?>
          <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/home/zender/nco/data/in.nc">
            <variable name="one" type="float" shape="">
              <attribute name="long_name" separator="*" value="one" />
              <values>1.</values>
            </variable>
          </netcdf>
     XML-mode prints variable metadata and, as of NCO version 4.3.7
     (October, 2013), variable data and, as of NCO version 4.4.0
     (January, 2014), hidden attributes.  That ncks produces correct
     NcML translations of CDM files for all supported datatypes is
     verified by comparison to output from Unidata's ‘toolsUI’ Java
     program.  Please let us know how to improve XML/NcML features.

     ‘ncks’ provides additional options to configure NcML output:
     ‘--xml_no_location’, ‘--xml_spr_chr’, and ‘--xml_spr_nmr’.  Every
     NcML configuration option automatically triggers NcML printing, so
     that specifying ‘--xml’ in addition to a configuration option is
     redundant and unnecessary.  The ‘--xml_no_location’ switch prevents
     output of the NcML ‘location’ element.  By default the location
     element is printed with a value equal to the location of the input
     dataset, e.g., ‘location="/home/zender/in.nc"’.  The
     ‘--xml_spr_chr’ and ‘--xml_spr_nmr’ options customize the strings
     used as NcML separators for attributes and variables of
     character-type and numeric-type, respectively.  Their default
     separators are ‘*’ and "‘ ’" (a space):
          zender@roulee:~$ ncks --xml -d time,0,3 -v two_dmn_rec_var_sng in.nc
          ...
             <values separator="*">abc*bcd*cde*def</values>
           ...
           zender@roulee:~$ ncks --xml_spr_chr=', ' -v two_dmn_rec_var_sng in.nc
          ...
          <values separator=", ">abc, bcd, cde, def, efg, fgh, ghi, hij, jkl, klm</values>
          ...
          zender@roulee:~$ ncks --xml -v one_dmn_rec_var in.nc
          ...
          <values>1 2 3 4 5 6 7 8 9 10</values>
          ...
          zender@roulee:~$ ncks --xml_spr_nmr=', ' -v one_dmn_rec_var in.nc
          ...
          <values separator=", ">1, 2, 3, 4, 5, 6, 7, 8, 9, 10</values>
          ...
     Separator elements for strings are a thorny issue.  One must be
     sure that the separator element is not mistaken as a portion of the
     string.  NCO attempts to produce valid NcML and supplies the
     ‘--xml_spr_chr’ option to work around any difficulties.  NCO
     performs precautionary checks with ‘strstr(VAL,SPR)’ to identify
     presence of the separator string (SPR) in data (VAL) and, when it
     detects a match, automatically switches to a backup separator
     string (‘*|*’).  However limitations of ‘strstr()’ may lead to
     false negatives when the separator string occurs in data beyond the
     first string in multi-dimensional ‘NC_CHAR’ arrays.  Hence, results
     may be ambiguous to NcML parsers.  If problems arise, use
     ‘--xml_spr_chr’ to specify a multi-character separator that does
     not appear in the string array and that does not include an NcML
     formatting characters (e.g., commas, angles, quotes).

* Menu:

* Filters for ncks::

   ---------- Footnotes ----------

   (1) Those familiar with netCDF mechanics might wish to know what is
happening here: ‘ncks’ does not attempt to redefine the variable in
OUTPUT-FILE to match its definition in INPUT-FILE, ‘ncks’ merely copies
the values of the variable and its coordinate dimensions, if any, from
INPUT-FILE to OUTPUT-FILE.

   (2) As of version 5.1.1 (November 2022), the map checker diagnoses
from the global attributes ‘map_method’, ‘no_conserve’, or ‘noconserve’
(in that order, if present) whether the mapping weights are intended to
be conservative (as opposed to, e.g., bilinear).  Weights deemed
non-conservative by design are no longer flagged with dire _WARNING_
messages.

   (3) The JSON boolean atomic type is not (yet) supported as there is
no obvious netCDF-equivalent to this type.


File: nco.info,  Node: Filters for ncks,  Prev: ncks netCDF Kitchen Sink,  Up: ncks netCDF Kitchen Sink

4.9.1 Filters for ‘ncks’
------------------------

We encourage the use of standard UNIX pipes and filters to narrow the
verbose output of ‘ncks’ into more precise targets.  For example, to
obtain an uncluttered listing of the variables in a file try
     ncks --trd -m in.nc | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort
   A Bash user could alias the previous filter to the shell command
‘ncvarlst’ as shown below.  More complex examples could involve command
line arguments.  For example, a user may frequently be interested in
obtaining the value of an attribute, e.g., for textual file examination
or for passing to another shell command.  Say the attribute is
‘purpose’, the variable is ‘z’, and the file is ‘in.nc’.  In this
example, ‘ncks --trd -m -v z’ is too verbose so a robust ‘grep’ and
‘cut’ filter is desirable, such as
     ncks --trd -M -m in.nc | grep -E -i "^z attribute [0-9]+: purpose" | cut -f 11- -d ' ' | sort
   The filters are clearly too complex to remember on-the-fly so the
entire procedure could be implemented as a shell command or function
called, say, ‘ncattget’
     function ncattget { ncks --trd -M -m ${3} | grep -E -i "^${2} attribute [0-9]+: ${1}" | cut -f 11- -d ' ' | sort ; }
   The shell ‘ncattget’ is invoked with three arugments that are, in
order, the names of the attribute, variable, and file to examine.
Global attributes are indicated by using a variable name of ‘global’.
This definition yields the following results
     % ncattget purpose z in.nc
     Height stored with a monotonically increasing coordinate
     % ncattget Purpose Z in.nc
     Height stored with a monotonically increasing coordinate
     % ncattget history z in.nc
     % ncattget history global in.nc
     History global attribute.
   Note that case sensitivity has been turned off for the variable and
attribute names (and could be turned on by removing the ‘-i’ switch to
‘grep’).  Furthermore, extended regular expressions may be used for both
the variable and attribute names.  The next two commands illustrate this
by searching for the values of attribute ‘purpose’ in all variables, and
then for all attributes of the variable ‘z’:
     % ncattget purpose .+ in.nc
     1-D latitude coordinate referred to by geodesic grid variables
     1-D longitude coordinate referred to by geodesic grid variables
     ...
     % ncattget .+ Z in.nc
     Height
     Height stored with a monotonically increasing coordinate
     meter

   Extended filters are best stored as shell commands if they are used
frequently.  Shell commands may be re-used when they are defined in
shell configuration files.  These files are usually named ‘.bashrc’,
‘.cshrc’, and ‘.profile’ for the Bash, Csh, and Sh shells, respectively.
     # NB: Untested on Csh, Ksh, Sh, Zsh! Send us feedback!
     # Bash shell (/bin/bash), .bashrc examples
     # ncattget $att_nm $var_nm $fl_nm : What attributes does variable have?
     function ncattget { ncks --trd -M -m ${3} | grep -E -i "^${2} attribute [0-9]+: ${1}" | cut -f 11- -d ' ' | sort ; }
     # ncavg $var_nm $fl_nm : What is mean of variable?
     function ncavg { ncwa -y avg -O -C -v ${1} ${2} ~/foo.nc ; ncks --trd -H -C -v ${1} ~/foo.nc | cut -f 3- -d ' ' ; }
     # ncavg $var_nm $fl_nm : What is mean of variable?
     function ncavg { ncap2 -O -C -v -s "foo=${1}.avg();print(foo)" ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
     # ncdmnlst $fl_nm : What dimensions are in file?
     function ncdmnlst { ncks --cdl -m ${1} | cut -d ':' -f 1 | cut -d '=' -s -f 1 ; }
     # ncdmnsz $dmn_nm $fl_nm : What is dimension size?
     function ncdmnsz { ncks --trd -m -M ${2} | grep -E -i ": ${1}, size =" | cut -f 7 -d ' ' | uniq ; }
     # ncgrplst $fl_nm : What groups are in file?
     function ncgrplst { ncks -m ${1} | grep 'group:' | cut -d ':' -f 2 | cut -d ' ' -f 2 | sort ; }
     # ncmax $var_nm $fl_nm : What is maximum of variable?
     function ncmax { ncwa -y max -O -C -v ${1} ${2} ~/foo.nc ; ncks --trd -H -C -v ${1} ~/foo.nc | cut -f 3- -d ' ' ; }
     # ncmax $var_nm $fl_nm : What is maximum of variable?
     function ncmax { ncap2 -O -C -v -s "foo=${1}.max();print(foo)" ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
     # ncmdn $var_nm $fl_nm : What is median of variable?
     function ncmdn { ncap2 -O -C -v -s "foo=gsl_stats_median_from_sorted_data(${1}.sort());print(foo)" ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
     # ncmin $var_nm $fl_nm : What is minimum of variable?
     function ncmin { ncap2 -O -C -v -s "foo=${1}.min();print(foo)" ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
     # ncrng $var_nm $fl_nm : What is range of variable?
     function ncrng { ncap2 -O -C -v -s "foo_min=${1}.min();foo_max=${1}.max();print(foo_min,\"%f\");print(\" to \");print(foo_max,\"%f\")" ${2} ~/foo.nc ; }
     # ncmode $var_nm $fl_nm : What is mode of variable?
     function ncmode { ncap2 -O -C -v -s "foo=gsl_stats_median_from_sorted_data(${1}.sort());print(foo)" ${2} ~/foo.nc | cut -f 3- -d ' ' ; }
     # ncrecsz $fl_nm : What is record dimension size?
     function ncrecsz { ncks --trd -M ${1} | grep -E -i "^Root record dimension 0:" | cut -f 10- -d ' ' ; }
     # nctypget $var_nm $fl_nm : What type is variable?
     function nctypget { ncks --trd -m -v ${1} ${2} | grep -E -i "^${1}: type" | cut -f 3 -d ' ' | cut -f 1 -d ',' ; }
     # ncunits $att_val $fl_nm : Which variables have given units?
     function ncunits { ncks --trd -m ${2} | grep -E -i " attribute [0-9]+: units.+ ${1}" | cut -f 1 -d ' ' | sort ; }
     # ncvardmnlst $var_nm $fl_nm : What dimensions are in a variable?
     function ncvardmnlst { ncks --trd -m -v ${1} ${2} | grep -E -i "^${1} dimension [0-9]+: " | cut -f 4 -d ' ' | sed 's/,//' ; }
     # ncvardmnlatlon $var_nm $fl_nm : Does variable contain both lat and lon dimensions?
     function ncvardmnlatlon { flg=`ncks -C -v ${1} -m ${2} | grep -E -i "${1}\(" | grep -E "lat.*lon|lon.*lat"` ; [[ ! -z "$flg" ]] && echo "Yes, ${1} has both lat and lon dimensions" || echo "No, ${1} does not have both lat and lon dimensions" ; }
     # ncvarlst $fl_nm : What variables are in file?
     function ncvarlst { ncks --trd -m ${1} | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort ; }
     # ncvarrnk $var_nm $fl_nm : What is variable rank?
     function ncvarrnk { ncks --trd -m -C -v $1 $2 | grep -E -i ^$1: | cut -f 4 -d ' ' ; }
     
     # Csh shell (/bin/csh), .cshrc examples (derive others from Bash definitions):
     ncattget() { ncks --trd -M -m -v ${3} | grep -E -i "^${2} attribute [0-9]+: ${1}" | cut -f 11- -d ' ' | sort ; }
     ncdmnsz() { ncks --trd -m -M ${2} | grep -E -i ": ${1}, size =" | cut -f 7 -d ' ' | uniq ; }
     ncvarlst() { ncks --trd -m ${1} | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort ; }
     ncrecsz() { ncks --trd -M ${1} | grep -E -i "^Record dimension:" | cut -f 8- -d ' ' ; }
     
     # Sh shell (/bin/sh), .profile examples (derive others from Bash definitions):
     ncattget() { ncks --trd -M -m ${3} | grep -E -i "^${2} attribute [0-9]+: ${1}" | cut -f 11- -d ' ' | sort ; }
     ncdmnsz() { ncks --trd -m -M ${2} | grep -E -i ": ${1}, size =" | cut -f 7 -d ' ' | uniq ; }
     ncvarlst() { ncks --trd -m ${1} | grep -E ': type' | cut -f 1 -d ' ' | sed 's/://' | sort ; }
     ncrecsz() { ncks --trd -M ${1} | grep -E -i "^Record dimension:" | cut -f 8- -d ' ' ; }

EXAMPLES

   View all data in netCDF ‘in.nc’, printed with Fortran indexing
conventions:
     ncks -F in.nc

   Copy the netCDF file ‘in.nc’ to file ‘out.nc’.
     ncks in.nc out.nc
   Now the file ‘out.nc’ contains all the data from ‘in.nc’.  There are,
however, two differences between ‘in.nc’ and ‘out.nc’.  First, the
‘history’ global attribute (*note History Attribute::) will contain the
command used to create ‘out.nc’.  Second, the variables in ‘out.nc’ will
be defined in alphabetical order.  Of course the internal storage of
variable in a netCDF file should be transparent to the user, but there
are cases when alphabetizing a file is useful (see description of ‘-a’
switch).

   Copy all global attributes (and no variables) from ‘in.nc’ to
‘out.nc’:
     ncks -A -x ~/nco/data/in.nc ~/out.nc
   The ‘-x’ switch tells NCO to use the complement of the extraction
list (*note Subsetting Files::).  Since no extraction list is explicitly
specified (with ‘-v’), the default is to extract all variables.  The
complement of all variables is no variables.  Without any variables to
extract, the append (‘-A’) command (*note Appending Variables::) has
only to extract and copy (i.e., append) global attributes to the output
file.

   Copy/append metadata (not data) from variables in one file to
variables in a second file.  When copying/subsetting/appending files (as
opposed to printing them), the copying of data, variable metadata, and
global/group metadata are now turned OFF by ‘-H’, ‘-m’, and ‘-M’,
respectively.  This is the opposite sense in which these switches work
when _printing_ a file.  One can use these switches to easily replace
data or metadata in one file with data or metadata from another:
     # Extract naked (data-only) copies of two variables
     ncks -h -M -m -O -C -v one,three_dmn_rec_var ~/nco/data/in.nc ~/out.nc
     # Change values to be sure original values are not copied in following step
     ncap2 -O -v -s 'one*=2;three_dmn_rec_var*=0' ~/nco/data/in.nc ~/in2.nc
     # Append in2.nc metadata (not data!) to out.nc
     ncks -A -C -H -v one,three_dmn_rec_var ~/in2.nc ~/out.nc
   Variables in ‘out.nc’ now contain data (not metadata) from ‘in.nc’
and metadata (not data) from ‘in2.nc’.

   Print variable ‘three_dmn_var’ from file ‘in.nc’ with default
notations.  Next print ‘three_dmn_var’ as an un-annotated text column.
Then print ‘three_dmn_var’ signed with very high precision.  Finally,
print ‘three_dmn_var’ as a comma-separated list:
     % ncks --trd -C -v three_dmn_var in.nc
     lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0
     lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1
     ...
     lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23
     % ncks --trd -s '%f\n' -C -v three_dmn_var in.nc
     0.000000
     1.000000
     ...
     23.000000
     % ncks --trd -s '%+16.10f\n' -C -v three_dmn_var in.nc
        +0.0000000000
        +1.0000000000
     ...
       +23.0000000000
     % ncks --trd -s '%f, ' -C -v three_dmn_var in.nc
     0.000000, 1.000000, ..., 23.000000,
Programmers will recognize these as the venerable C language ‘printf()’
formatting strings.  The second and third options are useful when
pasting data into text files like reports or papers.  *Note ncatted
netCDF Attribute Editor::, for more details on string formatting and
special characters.

   As of NCO version 4.2.2 (October, 2012), NCO prints missing values as
blanks (i.e., the underscore character ‘_’) by default:
     % ncks --trd -C -H -v mss_val in.nc
     lon[0]=0 mss_val[0]=73
     lon[1]=90 mss_val[1]=_
     lon[2]=180 mss_val[2]=73
     lon[3]=270 mss_val[3]=_
     % ncks -s '%+5.1f, ' -H -C -v mss_val in.nc
     +73.0, _, +73.0, _,
   To print the numeric value of the missing value instead of a blank,
use the ‘--no_blank’ option.

   ‘ncks’ prints in a verbose fashion by default and supplies a number
of switches to pare-down (or even spruce-up) the output.  The interplay
of the ‘-Q’, ‘-V’, and (otherwise undocumented) ‘--no_nm_prn’ switches
yields most desired verbosities:
     % ncks -v three_dmn_rec_var -C -H ~/nco/data/in.nc
     time[0]=1 lat[0]=-90 lon[0]=0 three_dmn_rec_var[0]=1 
     % ncks -Q -v three_dmn_rec_var -C -H ~/nco/data/in.nc              
     three_dmn_rec_var[0]=1 
     % ncks -V -v three_dmn_rec_var -C -H ~/nco/data/in.nc
     1
     % ncks -Q --no_nm_prn -v three_dmn_rec_var -C -H ~/nco/data/in.nc
     1
     % ncks --no_nm_prn -v three_dmn_rec_var -C -H ~/nco/data/in.nc
     1 -90 0 1

   One dimensional arrays of characters stored as netCDF variables are
automatically printed as strings, whether or not they are
NUL-terminated, e.g.,
     ncks -v fl_nm in.nc
The ‘%c’ formatting code is useful for printing multidimensional arrays
of characters representing fixed length strings
     ncks -s '%c' -v fl_nm_arr in.nc
Using the ‘%s’ format code on strings which are not NUL-terminated (and
thus not technically strings) is likely to result in a core dump.

   Create netCDF ‘out.nc’ containing all variables, and any associated
coordinates, except variable ‘time’, from netCDF ‘in.nc’:
     ncks -x -v time in.nc out.nc
   As a special case of this, consider how to remove a variable such as
‘time_bounds’ that is identified in a CF Convention (*note CF
Conventions::) compliant ‘ancillary_variables’, ‘bounds’, ‘climatology’,
‘coordinates’, or ‘grid_mapping’ attribute.  NCO subsetting assumes the
user wants all ancillary variables, axes, bounds and coordinates
associated with all extracted variables (*note Subsetting Coordinate
Variables::).  Hence to exclude a ‘ancillary_variables’, ‘bounds’,
‘climatology’, ‘coordinates’, or ‘grid_mapping’ variable while retaining
the "parent" variable (here ‘time’), one must use the ‘-C’ switch:
     ncks -C -x -v time_bounds in.nc out.nc
   The ‘-C’ switch tells the operator _NOT_ to necessarily include all
the CF ancillary variables, axes, bounds, and coordinates.  Hence the
output file will contain ‘time’ and not ‘time_bounds’.

   Extract variables ‘time’ and ‘pressure’ from netCDF ‘in.nc’.  If
‘out.nc’ does not exist it will be created.  Otherwise the you will be
prompted whether to append to or to overwrite ‘out.nc’:
     ncks -v time,pressure in.nc out.nc
     ncks -C -v time,pressure in.nc out.nc
The first version of the command creates an ‘out.nc’ which contains
‘time’, ‘pressure’, and any coordinate variables associated with
PRESSURE.  The ‘out.nc’ from the second version is guaranteed to contain
only two variables ‘time’ and ‘pressure’.

   Create netCDF ‘out.nc’ containing all variables from file ‘in.nc’.
Restrict the dimensions of these variables to a hyperslab.  The
specified hyperslab is: the fifth value in dimension ‘time’; the
half-open range LAT > 0. in coordinate ‘lat’; the half-open range LON <
330. in coordinate ‘lon’; the closed interval 0.3 < BAND < 0.5 in
coordinate ‘band’; and cross-section closest to 1000. in coordinate
‘lev’.  Note that limits applied to coordinate values are specified with
a decimal point, and limits applied to dimension indices do not have a
decimal point *Note Hyperslabs::.
     ncks -d time,5 -d lat,,0.0 -d lon,330.0, -d band,0.3,0.5
     -d lev,1000.0 in.nc out.nc

   Assume the domain of the monotonically increasing longitude
coordinate ‘lon’ is 0 < LON < 360.  Here, ‘lon’ is an example of a
wrapped coordinate.  ‘ncks’ will extract a hyperslab which crosses the
Greenwich meridian simply by specifying the westernmost longitude as MIN
and the easternmost longitude as MAX, as follows:
     ncks -d lon,260.0,45.0 in.nc out.nc
   For more details *Note Wrapped Coordinates::.

