Hi JColvin,
You are right, TXAOUT and TXBOUT are not the issue. The chip has 3 GND pins and connect to DD2 3 GND pins.
Here is the bench set up, 800MHzx8. TXAOUT/TXBOUT unconnected, trigger on falling edge nCS. SI signal has a lot of glitches, sometimes TFEMPTY signal also glitched. I think more GND connections to the board will be better but not sure.