is this an important test if water isotopes don't work in CTSM?

@wwieder good question. I'm not sure.

Keeping these tests in place does keep the parts of water isotopes working that is in place. Which should help if someone can get back to water isotopes. And we haven't had these tests fail that often, so it hasn't been onerous.

I am going to leave it alone for a bit and move forward with other things. And I'm figuring out where it breaks which might lead to an easy solution. And a reasonable thing might be to bring in ctsm5.2.0 even with this one failing thing if there isn't an easy fix.

I tracked this back based on tags and it works in alpha-ctsm5.2.mksrf.23_ctsm5.1.dev168 bucould t fails in the next tag we made on ctsm5.2 of alpha-ctsm5.2.mksrf.23_ctsm5.1.dev171. These means it fails somewhere in the update between ctsm5.1.dev168 and ctsm5.1.dev171. In tracking that I find that it works in ctsm5.1.dev170. So the cold-start change in ctsm5.1.dev170 in PR #2355 seems to be the culprit here.

This test is a cold-start test so that could be, but the change is just a small change in initial Temperatures which shouldn't matter for water isotopes.. And yet it somehow does...

I verified that doing the following simple change in #2417 gets this test to pass.

diff --git a/src/biogeophys/TemperatureType.F90 b/src/biogeophys/TemperatureType.F90
index ab310650c..6450f3f31 100644
--- a/src/biogeophys/TemperatureType.F90
+++ b/src/biogeophys/TemperatureType.F90
@@ -741,7 +741,7 @@ subroutine InitCold(this, bounds, &
                   end if
                end if
             else
-               this%t_soisno_col(c,1:nlevgrnd) = 272._r8
+               this%t_soisno_col(c,1:nlevgrnd) = 274._r8
                if (use_excess_ice .and. (lun%itype(l) == istsoil .or. lun%itype(l) == istcrop)) then
                   this%t_soisno_col(c,1:nlevgrnd) = SHR_CONST_TKFRZ - 5.0_r8 !needs to be below freezing to properly initiate excess ice
                end if

@ekluzek is suggesting this an expected fail and will talk to @billsacks to see if it's something we need to modify the test case for? @wwieder assumes this is caused by fractionation caused by freezing water (somehow).

This issue reminds me of two issues that I worked on in the last few months; they may be related to this:
#2041
#2366

Thanks @ekluzek for the detailed notes here along with all of your investigations so far! Thanks also @slevis-lmwg for the pointers to similar issues, and @wwieder for the good questions. I'm happy to look into this a bit. I have a couple of ideas of things that might fix this that I want to try initially.

Regarding @wwieder 's question:

is this an important test if water isotopes don't work in CTSM?

I have two answers to that:

(1) The water isotope tests help ensure that the ~ 6 months of work that got water tracers / isotopes partially working don't get undone. From that perspective, these tests could be dropped if/when the feeling becomes that it is very unlikely that the water tracer / isotope work will ever be finished... though there's a part of me that would be tempted to explore re-shifting my priorities to work on this for a year and try to get it wrapped up if it came to that (if this had enough support to do so).

(2) Most – maybe all? – recent failures in isotope tests, including this one, are "canary in the coal mine" sorts of failures. That is, even though the failures are in water isotope tests, they're really indicative of bigger problems in the model that have nothing to do with water isotopes (in my view). And this is really tied in with why implementing water tracers in CTSM is hard in the first place: CTSM's hydrology is not careful / rigorous about its numerics. Implementing water tracers in a maintainable way requires more rigor with the hydrology numerics. Many (all?) of the problems we've been running into have been related to state variables getting updated in a way that should bring them to exactly zero but in fact they are ending up roundoff-level different from zero. That seems to be the situation here. Without the tracer/isotope-related checks, the model continues to run, but that doesn't necessarily mean that it's running correctly. Here, for example, you have a snocan value of 0.4 x 10^-295. That is effectively zero, and the model should have set that to exactly zero. If we have any code like if (snocan > 0) then X else Y then this code currently isn't working right under conditions like this. So under this perspective, my feeling is that we should be thankful that these tests are catching issues with CTSM's hydrology numerics that may be causing incorrect operation of the model, so we shouldn't be quick to throw them out.

This test is a cold-start test so that could be, but the change is just a small change in initial Temperatures which shouldn't matter for water isotopes.. And yet it somehow does...

That is indeed weird. My guess is that this is just triggering a different evolution of the system that makes this issue appear when (by chance) it didn't appear before, but maybe there's something weirder going on here.

@wwieder assumes this is caused by fractionation caused by freezing water (somehow).

Scientifically I see how that could make sense, but so far we don't have any fractionation implemented, so that wouldn't explain this.

I have resolved this in #2457 . The change is to adjust the tolerance for the truncation that @slevis-lmwg introduced in #2053. Note that this will probably change answers, maybe for many tests. (Note that ctsm5.1.dev141, which was the tag associated with #2053, had answer changes, so I expect this adjustment to have similar answer changes.)

Thanks for engaging here, Bill. Much appreciated

Thanks for engaging here, Bill. Much appreciated

Happy to help. As I said to Erik, I feel invested in keeping the water tracer stuff as working as possible so that one day in the uncertain future it could all come together....