forked from cjcoats/ioapi-3.2
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathV32-CHANGES.txt
538 lines (384 loc) · 20.7 KB
/
V32-CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
Announcing RELEASE CANDIDATE 1 for I/O API VERSION 3.2
There are a number of new features in Models-3 I/O API Version 3.2,
which are summarized in the list list below. Please download and
test; if you have problems or suggestions, please contact the
I/O API author, Carlie J. Coats, Jr., Ph.D. <cjcoats@email.unc.edu>
Contributions and suggestions will of course be acknowledged on the
NewStuff/ChangeLog page.
*) Availability: now on Github
*) NetCDF4 Issues
*) Changes to Module M3UTILIO
> New Routines
> Fortran-90 Generic Routines
*) PNCF/MPI Distributed I/O option
*) New Module MODATTS3:
> CF-compliant geospatial metadata option
> Geospatial-Transform metadata option
> "Standard" CMAQ and SMOKE metadata options.
> "Unstructured Text" metadata option
*) New Module MODGCTP
*) New Module MODNCFIO
> Resolve netCDF-vs-PnetCDF issues
> New "read raw netCDF" routines
*) New Module MODWRFIO
*) New and/or OpenMP Parallel "m3tools" programs
*) Hacks for "gfortran"
*) CygWin/MS-Windows, gcc/gfortran/Mac OSX support
*) Other Issues
See the I/O API ChangeLog or the I/O API Version 3.x Summary for links
to these features:
<https://www.cmascenter.org/ioapi/documentation/3.1/html/NEWSTUFF.html>
<https://www.cmascenter.org/ioapi/documentation/3.1/html/v3stuff.html>
Note that the I/O API follows a "rolling release" model, and works
very hard at maintaining upward compatibility, especially at the
source level. New versions typically are driven by a combination
of needed new modeling capabilities and by changes in external
software such as netCDF:
2.2 -> 3.0: netCDF v2:v3 changes; support for "free" F-90
source format; introduction of F-90 INTERFACEs.
3.0 -> 3.1: change MXVARS3 from 120 to 2048; more F-90
INTERFACEs and "generic" routines.
3.1 -> 3.2: netCDF v3:v4 changes; more F-90 changes;
"raw netCDF" routines for SMOKE-4; "standard" CMAQ metadata;
PnetCDF distributed I/O; new MODULEs and their associated
functionality.
Within each version, there are typically two reasons for
changes in the release: bug fixes; and experimental new
features.
----------------------------------------------------------------------
AVAILABILITY: now on Github
Release Candidate 1 of I/O API Version 3.2 is now available
both as a gzipped "tar-ball" from the CMAS web-site, and also
via "git" from GitHub.
From the CMAS web-site, you can download the tar-ball from
<https://www.cmascenter.org/ioapi/download/ioapi-3.2.tar.gz>
On GitHub, the web-site is <https://github.com/cjcoats/ioapi-3.2>
and you can install the source code using the command
git clone https://github.com/cjcoats/ioapi-3.2
See
<https://www.cmascenter.org/ioapi/documentation/3.1/html/AVAIL.html#v32>
----------------------------------------------------------------------
NetCDF-4 Issues
Evidently, starting with netCDF-Fortran 4.4.2, UCAR in their
wisdom decided to silently eliminate the
CALL NC*() [e.g., CALL NCCLOS( CDFID, IERR ) ]
Fortran interfaces in terms of which the I/O API was originally
implemented, in favor of the
IERR = NF_*()
forms introduced with netCDF-3. [If you can find any
documentation on the web indicating that they were making
this change I would enjoy seeing it; I can't find any.]
I/O API Version 3.2 was tediously re-coded to replace all 943
of these calls by the newer IERR=NF_*() Fortran calls that
are still supported. Prior editions of the I/O API will have
link errors with netCDF-Fortran 4.4.2 or later, because routines
such as NCCLOS() are no longer in that library.
Note that netCDF-4 now requires separate netCDF-C (which is
now just called "netCDF"), which has library "libnetcdf.a"
and also netCDF-Fortran, which has library "libnetcdff.a".
With netCDF-4 , your "link" command will need the following
flags to find both of these libraries:
-lnetcdff -lnetcdf
Note also that netCDF Versions 4.x have lots of additional build
options, that (if turned on) will require a complex set of
additional libraries in "Makefile"s and all other model-building
systems. It is recommended that you disable these options by
adding the following command-line flags below to your netCDF
"configure" command:
--disable-netcdf-4 --disable-dap
----------------------------------------------------------------------
CHANGES to Module M3UTILIO
See
<https://www.cmascenter.org/ioapi/documentation/3.1/html/M3UTILIO.html>
New Routines and INTERFACEs
DBLLIST()
GETNUM1(), GETREAL1() GETDBLE1(): variants on GETNUM() etc.
with no range-checking
LASTTIME(): overflow-safe "last time step in a sequence"
routine
F90 Generic Routines and INTERFACEs
ENVLIST: generic "get a comma-list from environment" routine
ENVGET: generic "get a value from environment" routine
FINDKEY: generic "find key-tuple in sorted index"
GETVAL: generic "prompt user for value, with or without
range-checking
PMATVEC: 1-D and 2-D OpenMP parallel "multiply by sparse
incidence matrix"
SORTI: generic "indexed-key-tuple quicksort"
----------------------------------------------------------------------
PNCF/MPI Distributed I/O option
See
<https://www.cmascenter.org/ioapi/documentation/3.1/html/BUFFERED.html#pncf>
The I/O API can now be configured to use PnetCDF (from Argonne
National Laboratory) for distributed I/O for CMAQ, on a file-by-file
basis. Distributed I/O is only supported for gridded files on the
cross-point grid; to turn it on, prefix the path-name by a "MPI:"
prefix, as in the example below:
setenv CHEMCONC3D MPI:/mydir/cmaq.conc.US15_CRO.2015233.ncf
Enabling this option is a build-time option, turned on by using
"ioapi/Makefile.pncf" in conjunction with an "mpif90" enabled
binary type ${BIN}; due to vagaries in the installation of various
MPI versions, you may have to customize the corresponding
"Makeinclude.${BIN}" file
ioapi/Makeinclude.Linux2_x86_64gfortmpi
ioapi/Makeinclude.Linux2_x86_64ifortmpi
ioapi/Makeinclude.Linux2_x86_64pgmpi
ioapi/Makeinclude.Linux2_x86_64sunmpi
or by adding -DIOAPI_PNCF=1 to the DEFINEFLAGS "make"-variable
of your "ioapi/Makefile".
Related data- and processor-distribution state and
routines are found in new MODULE MODPDATA.
See
<https://www.cmascenter.org/ioapi/documentation/3.1/html/BUFFERED.html#pncf>
for environment variables needed for using distributed-parallel I/O
The original concept and the prototype code are due to Dr. David
Wong, US EPA. The data structures and algorithms in new MODULE
MODPDATA are his; the rest of the code has been extensively extended
and revised, to meet the needs of proper I/O API integration and
Models-3 software engineering standards.
----------------------------------------------------------------------
NEW Module MODATTS3
See
<https://www.cmascenter.org/ioapi/documentation/3.1/html/MODATTS3.html>
New module MODATTS3 replaces I/O API-3.1 module MATXATTS (one
source incompatibility between these two I/O API versions, but
hopefully not a major one). It provides routines and MODULE
variables for the following sorts of metadata:
Matrix-attribute routines add extra file attributes
to describe input and output grids for matrix-files
CF-convention geospatial-metadata routines optionally make
newly-created files CF compliant. This is turned on by
"setenv IOAPI_CFMETA YES". (CF metadata is used by the global
climate community as well as by a number of third-party
applications; see
<http://cfconventions.org/>
CMAQ-metadata routines add the current version of "Standard CMAQ
metadata" to the headers of newly created files. This is turned
on by "setenv IOAPI_CMAQMETA [ENV | <path>]" or by calling
INITCMAQ(). The current version of "Standard CMAQ metadata" is
defined primarily in terms of the environment variables used to
control a CMAQ run.
Unstructured-text metadata routines add multiple lines of
80-column text from either INITMDATA()-arguments or from a text
file, to the headers of newly created files. This is turned on
by "setenv IOAPI_TEXTMETA <path>" or by calling INITMTEXT(),
and uses only the non-comment, non-blank lines for this purpose.
Note that UNIX/Linux filter-utility "fold" can be used to create
80-column input...
SMOKE-metadata routines are not currently implemented
(pending a definition of "Standard SMOKE Metadata");
eventually, they should function similarly.
----------------------------------------------------------------------
NEW Module MODGCTP
See
<https://www.cmascenter.org/ioapi/documentation/3.1/html/MODGCTP.html>
This module contains:
* PARAMETERs for GCTP spheroid names and indices
* INTERFACEs for many geospatial-transform routines (moved
from MODULE M3UTILIO)
* New (parallel!) geospatial transform and grid-to-grid
interpolation routines:
> M3TOGTPZ() for easy GCTP initialization
> GRID2XY() generic transform: grid-node to X-Y
> XY2XY() generic transform: array of points to X-Y
> GRID2INDX(), PNTS2INDX(), and INDXMULT(): optimized/
parallelized bilinear interpolation package
> INTERFACEs for the LAMBERT family of single-precision
single-point transform-routines
----------------------------------------------------------------------
NEW Module MODNCFIO
See
<https://www.cmascenter.org/ioapi/documentation/3.1/html/MODNCFIO.html>
Due to inconsistencies between the vendor supplied INCLUDE files for
both netCDF and PnetCDF (see above; the latter defined some but not
all of the needed netCDF definitions), it was not possible to use
"#ifdef..." to select between them as suggested by the initial
distributed-I/O draft code. The new module puts all the definitions
in a consistent form in one place (with conditional compilation, to
determine whether or not PnetCDF is active).
This module also supplies new routines for use with a specified
(non-timestepped) "raw netCDF" file.
* CREATENC() creates a new "raw netCDF" file, according to
the file-description supplied as arguments.
* DESCNCVAR() returns the list of variables and their types
and dimensions for a specified file, and
* READNCVAR() reads a specified 1-D, 2-D, 3-D, or 4D variable
or timestep-of-variable from a specified file.
* WRITENCVAR() writesa specified 1-D, 2-D, 3-D, or 4D variable
or timestep-of-variable to a specified file.
Examples of such files are EDGAR and GEIA global emissions
and GEOS[Chem] met/chem files: see
<http://edgar.jrc.ec.europa.eu/>,
<http://www.geiacenter.org/access>,
<http://acmg.seas.harvard.edu/geos/>
----------------------------------------------------------------------
NEW Module MODWRFIO
See
<https://www.cmascenter.org/ioapi/documentation/3.1/html/MODWRFIO.html>
This module supplies new routines for reading "WRF netCDF" files
(writing is yet to be implemented).
Presently, only single-file operation is supported: for multiple
files, you must CLOSEWRF() any currently-open WRF file before
opening and reading or writing another:
* OPENWRF() opens the WRF-netCDF format file with specified
logical name for read or read/write.
* CRTWRF() not yet implemented (stub only)
* READWRF() reads the specified timestep of the specified
1-D, 2-D, or 3-D variable from the current WRF-netCDF file.
* WRITEWRF() not yet implemented (stub only)
* CLOSEWRF() closes the current WRF-netCDF file.
----------------------------------------------------------------------
NEW Module MODMPASFIO
See
<https://www.cmascenter.org/ioapi/documentation/3.1/html/MODMPASFIO.html>
This module supplies new routines and data structures for opening,
creating, reading, writing, and manipulating "MPAS netCDF" files and
their (unstructured) grids.
See <https://mpas-dev.github.io/files/documents/MPAS-MeshSpec.pdf>
for details about these files.
----------------------------------------------------------------------
NEW and/or OpenMP PARALLEL "m3tools" Programs
See
<https://www.cmascenter.org/ioapi/documentation/3.1/html/AA.html#tools>
DATE-AND-TIME MANIPULATION PROGRAMS for Scripting: these echo the
program's result without any extraneous stuff, and so work very
well in shell-scripts. They are carefully coded to be safe from
integer-overflow, and support "yesterday", "today", and "tomorrow"
as command-line arguments:
> datshift YYYYMMDD NDAYS -- update Greg-date YYYYMMDD by NDAYS
> greg2jul YYYYMMDD -- writes Julian date YYYYDDD
> jul2greg YYYYDDD -- writes Greg-date YYYYMMDD
> juldiff YYYYDDD ZZZZEEE -- number of days from Julian date
YYYYDDD to Julian date ZZZZEEE
> julshift YYYYDDD NDAYS -- update Julian date YYYYDDD by NDAYS
> timeshift YYYYDDD.HHMMSS IIMMSS -- update date&time
YYYYDDD:HHMMSS by time step
IIMMSS
For example, look at the following, where note the back-quotes
(the "evaluate this" operator) in "set"-statements, and note
the ":e" and ":r" filename-extension and filename-root operators
(these split off the "after the last period" and "before the
last period" parts of names):
set gdate = `greg2jul today`
> puts today's Gregorian date into shell-variable "gdate"
set kdate = `greg2jul 20150103`
> converts Jan. 3, 2015 to Julian date YYYYDDD format and
> puts it into shell-variable "kdate"
set idate = `julshift today 7`
> puts Julian date for this day next week into
> shell-variable "idate"
set days = `juldiff tomorrow yesterday`; echo $days
> echoes -2
timeshift 2014029.120000 183000
> echoes 2014030.063000
set foo = `timeshift 2014029.120000 183000`
> puts 2014030.063000 into shell-variable "foo"
echo $foo:r # echoes 2014030
echo $foo:e # echoes 063000
OTHER NEW "m3tools" PROGRAMS.
bcwndw, camxtom3, dayagg, findwndw, gridprobe, insertgrid,
m3probe, m3totxt, vertimeproc, vertintegral, wrfgriddesc,
wrftom3
OpenMP PARALLEL "m3tools" programs:
bcwndw, dayagg, findwndw, m3agmask, m3agmax, m3combo, m3cple,
m3interp, m3tproc, mtxcalc mtxcple, presz, vertimeproc,
vertintegral, vertot
These programs now USE MODATTS3 for matrix-file attributes:
mtxbuild, mtxcalc, mtxcple
These programs now USE MODGCTP for geospatial transforms and
interpolation:
findwndw, gridprobe, insertgrid, latlon, m3cple, m3interp,
mtxcalc, projtool
----------------------------------------------------------------------
Hacks for "gfortran"
Recent versions of "gfortran" insist on breaking compatibility with
all other compilers we use (and with earlier versions of
"gfortran"), not accepting command-line-argument routines IARGC()
and GETARG() (which are an industry standard Fortran extension),
and accepting only Fortran-2003 routines GET_COMMAND_ARGUMENT()
and COMMAND_ARGUMENT_COUNT() . IARGC() and GETARG() are now
conditionally implemented internally in "init3.F", depending upon
preprocessor-definition -DNEED_ARGS=1 which is now set in
Makeinclude.Linux2_x86, Makeinclude.Linux2_x86gfort,
Makeinclude.Linux2_x86_64, and Makeinclude.Linux2_x86_64gfort.
Depending upon "gfortran" version (which will be dependent upon your
Linux distribution and its version), you either need to have it, or
need *not* to have it. Edit the appropriate "Maekinclude* file
if needed...
----------------------------------------------------------------------
CygWn support under MS-Windows
The following "Makeinclude" files are now available,
to build the I/O API using "gcc" and "gfortran" under
CygWin (see <https://cygwin.com/index.html>):
Makeinclude.WIN_x86gfort: 32-bit library/executables using
Cygwin "gcc" and "gfortran"
Makeinclude.WIN_x86_64mingw64: 4-bit library/executables using
MingW-64 "gcc" and "gfortran"
I'm having trouble building netCDF-Fortran under these environments,
and would welcome suggestions on that subject.
----------------------------------------------------------------------
Other Issues
Environment variables of length up to 64K are now supported
internally. Note that your operating system may well barf over
this kind of thing--the POSIX standard only mandates up to 512
bytes ;-(
Many source files have been converted to Fortran-90 "free"
source format.
"Makefile"s are no longer in the tar-ball or "git clone" result:
they must be customized anyway (you need to copy the appropriate
"Makefile.*" to "Makefile" and then customize it for your
installation-directory, etc.). Now, an I/O API update won't wipe
out the customization-work you have done.
INTEGER Overflow Issues:
Note that time intervals coded HHMMSS will suffer overflow
after a little more than 24.5 years; however, I/O API routines
CURREC(), CURRSTEP(), JSTEP3(), LASTTIME(), NEXTTIME() are now
carefully coded so as to avoid such overflow, and can be used
for time periods as long as several thousand years.
Programs that use starting date and time and run-duration
(formatted YYYDDD, HHMMSS, and HMMSS) as run-control parameters
will necessarily have the matching overflow problems; it is
recommended that starting and ending dates and times be used
instead. This style is safe for multi-century runs... and
is aided/supported by SUBROUTINE LASTTIME().
CLIMATOLOGY-Year Support:
Support for 365-day (no leap-year) climate modeling years
is now available, in addition to the 360-day climatology year
version available since May 2002. Note that these are
SPECIALIZED BUILDS and do not mix with "vanilla" I/O API builds.
To build these versions of the library, customize your
"ioapi/Makefile" by adding -DIO_365=1 or -DIO_360=1
to its DEFINEFLAGS "make"-variable.
Fixup in GCTP package: introduced scratch variables into
"ioapi/gctp.f" so that a correct INTERFACE block can be placed
into MODULE M3UTILIO.
Standards-violations fixes for getyn(), getstr(), getreal(),
getnum(), getmenu(), getdble(), promptmfile(), promptffile(),
year4(), skipl(), where the author who modified these routines
to avoid prompting the user had violated coding standards.
The relevant Models-3 coding standards state:
YOU WILL CHECK STATUS of systems-calls and other calls that
may fail (e.g., ALLOCATE, get-environment, netCDF and MPI
calls) and handle the error appropriately (whether quit or
return success/failure status); and
YOU WILL WRITE MEANINGFUL LOG MESSAGES adequate to locate and
diagnose the error for the above, in cases of failures.
Updated and error-checked documentation. All source files and
HTML pages now have Subversion IDs in their header comments.
64-bit INTEGER(8)-key search and sort routines. New SORTINC()
routines look only at the first M characters of the sort-key.
Subroutine LASTTIME() returns the ending date&time for
time step sequence SDATE:STIME:TSTEP:NSTEPS and is safe for
very long time step sequences (even multi-century!).
Added PARAMETER SINUGRD3=11 for the Sinusoidal Equal Area map
projection to INCLUDE-file PARMS3.EXT (and therefore to
MODULE M3UTILIO).
----------------------------------------------------------------------
ACKNOWLEDGEMENT
Many thanks to the following for help getting a cleanly-working
release out:
George Delic, Ph.D., HiPERiSM Consulting LLC
<http://www.hiperism.com/>
Edward Anderson, Lockheed Martin for US EPA.
B.H.Baek and Francis S. Binkowski, UNC Institute for the Environment
Chris Nolte and Tanya Spero, US EPA