I often find that a crash in CALCOT (with this kind of error message) is the thing that finally stops a run that has gone unstable for other reasons. In other words, the problem probably isn't anything to do with CALCOT.
If you take a look at the output for a few time-steps before the crash, you'll probably see some rather large fluctuations in the free surface (and, since you're using sigma coordinates, the intermediate planes).
If you can attach your test case, then we might be able to see where the problem lies. What specifically are you trying to do in this test case?