Do you use a fisheye or a non-fisheye objective?
All my panos use a Samyang 8mm vertically on a Pentax K-7. That lens works extremely well in IR once allowance is made for the different focus of IR wavelengths. That was adjusted by loosening the focus ring and setting it so that infinity on the ring matched what the sensor sees. Then focus is largely done by looking at the subject and setting the ring to (say) 3m or so.
Previously I used the Sigma 10-20mm, but that tends to have poor contrast in IR due to some internal reflections. (That's because some surfaces that are black in visible light are actually metallic looking in IR - but I digress...)