For two week's I've intermittently been tracking down an issue where suddenly a flow profile goes from turbulent/sane to a uniform flow profile in a single time step.
For awhile now I've been running with a custom banded solver that permits
- mixed precision factorization
- use of approximate factorizations, and
- iterative refinement.
It turns out that I didn't harden my one-off solver against NaNs. Ah, NaNs. Those lovely things that can make puzzling things like the following happen to a simulation:
What happened next was a comedy of my best intentions. I had a forcing constraint set so that if my solution failed to match my freestream, the state profile was forced to match. As my solution was all zeros, that NaN residual combined with the forcing caused my solution to become the freestream in a single time step. And suddenly my simulations went from sane to uniform instantaneously.
This came to me on the motorcycle while riding home after a rough day.
I simultaneously love and hate the beauty that is floating point.