Once again, thank you so much for the hints and the food for thought, zygot! This is much appreciated.
I, too, assume that TxVOCM and TxVOD in the Xilinx article refer to the Tx=transmitter VO=voltage output for CM=common mode ("bias") and D=differential. After all, they make statements about what their (Xilinx') hardware is able to handle. Furthermore, this seems reasonable due to the way they define the limit on the input voltage and the criterion for rebiasing.
I agree that the over-/undershoot conditions and the AC nature of the signal make this more involved than comparing static voltage levels. However, I think those assumptions are reasonable given the "low" speed at which I operate the link: ~100MHz, cf. my earlier posting. Over-/undershoot can only happen when Tx and Rx are separated by a transmission line, i.e. when you have a substantial phase delay between the two. The given configuration (cable and operating frequency) resembles more of a DC than high-speed digital.. 😉
Looking at the datasheet again, I have to correct myself. The information on VCCO=3.3V is given in Table 1 right below the line that you refer to: the upper limit for VIN is 2.625V. Strictly speaking, this puts the device at risk of being damaged if the FIN1108 would output its maximum 3.3...3.6V. However, the nominal maximum output is --as you say-- only 1.6V with up to 1.825V still being covered by its specification. So no damage under practical (and reasonably to assume) operating conditions.
Last but not least, I was able to tweak the design and run part of it with the DIFF_TERM=TRUE. I have yet to confirm full function, but: the clock seems quite stable. (I compare the input clock with an internally generated clock of the same frequency.) It looks like the cable is not the culprit, but rather it was the missing termination.