@ekluzek thanks for opening this issue. Ideally we'll coordinate with CAM developers (@gold2718, @brian-eaton and others) on the user interface so we have consistency between CAM and CTSM. My initial thinking was that we'd get rid of the ability to have a ":I" designation and instead have a new hist_* variable (like hist_instantaneous) with one value per stream (similar to hist_dov2xy, etc.). But I'm not tied to that idea.

@ekluzek thanks for opening this issue. Ideally we'll coordinate with CAM developers (@gold2718, @brian-eaton and others) on the user interface so we have consistency between CAM and CTSM. My initial thinking was that we'd get rid of the ability to have a ":I" designation and instead have a new hist_* variable (like hist_instantaneous) with one value per stream (similar to hist_dov2xy, etc.). But I'm not tied to that idea.

There, actually already is a variable that sets the averaging type per stream "hist_avgflag_pertape". And the only thing that you can't have is ":I" fields with the other averaging types (:A, :X, :M). So it's actually ok to have A, X, and M mixed up on one stream. So it seems like the current user interface is fine, we just need additional checking to make sure I type averaging isn't mixed up with any of the other types.

Thanks for pointing out hist_avgflag_pertape – I wasn't aware of that. That gets us most of the way there. I'd still argue for removing the :I designator, because it feels like it adds conceptual and code complexity to allow that designator but then make sure it's only used in very limited circumstances – circumstances that could also be achieved via hist_avgflag_pertape.

So allow :A, :M, and :X, but disallow :I (it that tape isn't instantaneous). That does make sense. That would still keep the current interface and just add additional checking to make sure it was correct.

Copying some notes from https://docs.google.com/document/d/1r1WUXONjEKjKqECDkML72gtQnpj-EjGQE1_a6EryhY8 . These notes were from 2020-06-19, so a few days before this issue was opened:

Gokhan & Mariana feel this should actually get attention at fairly high priority.

Since it doesn’t seem feasible to have multiple time dimensions, at least in the short-term (because it sounds like this would require NetCDF4): One proposed implementation would be to require components to separate time-averaged and instantaneous fields into separate files.

We’d like input from CAM SEs on how hard it would be to do this (or some other solution to this problem).

CTSM would also have to do some development work for this. I looked quickly (git grep -n 'avgflag=' | grep -v "avgflag='A'") and it looks like the only default-active instantaneous field is SNOW_PERSISTENCE. Assuming we can think of a way around that one, and that we can also get rid of / change the handful of history variables whose default is instantaneous: I think the main things this would require would be (1) changing some infrastructure and documentation so that users specify instantaneous on a per-file basis rather than a per-field basis; and (2) allowing for a larger number of history files, because in practice we seem to often want instantaneous as well as average fields at a few different time frequencies and/or spatial averages.

I [Bill Sacks] wonder about another option, which would be: instantaneous fields are saved in the time step that is closest to the midpoint of the averaging period, rather than at the end of the averaging period. i.e., those instantaneous fields would still apply at the time given in the history file – but that time would now be in the middle of the averaging period (e.g., month) rather than at the end of the period. I think this would be easier to implement in CTSM, but I’m not sure if it would be acceptable for analyses.

One thing we'll need to consider is how to deal with the cmip6 user mods. @olyson can help with this.

At today's SE meeting we mentioned that there may need to be some coordination with FATES too, although @rgknox suggested this may not be too difficult and can be added to code here

As per ESCOMP/CESM#194 (comment) - time_bounds can / should be left off of files that just have instantaneous values.

See also discussion here ESCOMP/CESM#194 (comment) about possible naming of the different streams.

Particularly if we are going to rename the streams (e.g., "ha1", "ha2"; "hi1", "hi2" - as per some ongoing discussion in ESCOMP/CESM#194 ) – would this also be a good time to rename our "clm2" output files to "ctsm" (or at least drop the "2" from "clm2")?

As @phillips-ad noted in ESCOMP/CAM#159 (comment), there are a few other date fields on CTSM as well as CAM output that should be changed to stay consistent with the centered time for the :X :M :X field streams: in CAM these are date and datesec; it looks like in CTSM they are mcdate and mcsec; maybe also mdcur, mscur and nstep?

Does anyone know enough about the use of these different time fields to be able to say definitively which ones should be changed to be consistent with the centered time?

Another averaging type that will require new handling for time is the Lxxxxx: averages that came in not too long ago. They should be on their own files and should be the closest local time to the center of the interval that is used in the average. There could be two such times and in that case I suppose the first one should be used.

I'm tentatively assigning this issue to myself based on this morning's conversation. Assignment may change after we meet with @billsacks

From discussion today: @slevis-lmwg 's initial feeling is to keep history-related namelist flags as they currently are, relying on hist_avgflag_pertape. But if that starts to feel confusing, then we could revisit the idea of having a separate set of namelist variables for instantaneous files.

How I see the order of tasks:

• Split instantaneous variables into hi files
• Disallow :I as recommended in these earlier suggestions
• Change time definitions to adhere to the rules outlined in the opening post of this issue
• Verify consistent behavior in FATES, MOSART, ...?

Also, @billsacks you mentioned communication with the CAM group. Maybe I keep Cecile in the loop?

Sounds great. The similar CAM issue is ESCOMP/CAM#159. In addition to Cecile, I would reach out to some of the CAM SEs - at least Cheryl Craig and Jesse Nusbaumer and maybe Courtney Peverley for starters - to see if they have been thinking any more about this.

From a first look at the code:
Subr. htapes_fieldlist determines total number of history tapes.
First task is to separate instantaneous variables into hi files, so I will modify this subroutine to add hi files for instantaneous variables.

Wrapping my head around what the code does now and how I can best use the existing infrastructure.

This is my current plan of first steps in this PR, which may differ from what we agreed above, though I'm pretty sure it's close. Comments always welcome:

• Only SNOW_PERSISTENCE is set to avgflag = 'I'. Seems most sensible to first change this field to avgflag = 'A' rather than keeping it uniquely different from all other fields.
• Users can then use the existing fincl and hist_avgflag_pertape infrastructure to manually add history tape(s) with instantaneous-only fields.
• I will eliminate the instantaneous option from fincl and from hist_addfld1d and 2d, to prevent inadvertent mixing of instantaneous fields with other fields in a single tape.

Update: Plan hasn't changed. I just wanted to correct the above statement about SNOW_PERSISTENCE. Turns out there are also five inactive fields set to 'I'.

CAM just completed this work. So now we are free to copy Courtney's work in

ESCOMP/CAM#903

CAM divides it into

hXi and hXa files.