I {hate,heart} NaN

For two week's I've intermittently been tracking down an issue where suddenly a flow profile goes from turbulent/sane to a uniform flow profile in a single time step.

For awhile now I've been running with a custom banded solver that permits

• mixed precision factorization
• use of approximate factorizations, and
• iterative refinement.

Relative to ZGBSVX this has been a net win. I get firm control over the quality of my linear system solutions and save a factor of 10 (yes, 10) over LAPACK's Cadillac banded solve.

It turns out that I didn't harden my one-off solver against NaNs. Ah, NaNs. Those lovely things that can make puzzling things like the following happen to a simulation:

The problem wasn't the NaN showing up in a computation, as I'd notice that. The problem was NaN showing up in a residual computation. Necessarily, that NaN failed a residual tolerance check. Because the tolerance check was false, I'd kick back a linear solution consisting of all zeros from an initial solution residual calculation that I'd performed.

What happened next was a comedy of my best intentions. I had a forcing constraint set so that if my solution failed to match my freestream, the state profile was forced to match. As my solution was all zeros, that NaN residual combined with the forcing caused my solution to become the freestream in a single time step. And suddenly my simulations went from sane to uniform instantaneously.

This came to me on the motorcycle while riding home after a rough day.

I simultaneously love and hate the beauty that is floating point.