RTL8211 Ethernet PHY issue on Genesys 2 Kintex-7 FPGA

[Questioner]

Please help me determine if I have an issue with my (brand new) Genesys2 eval board.
I am trying to use the Ethernet PHY, but the signals coming out of it are not as I expect. I have used other FPGA's with other PHY's so I have a bit of experience here.

The RGMII data-valid signal, RXCTL, appears to be un-driven between packets.

Below is a screen capture of my oscilloscope, which is probing the RXCTL(pin 13 of the PHY) signal. More exactly, I probed resistor R220 net ETH_RX_CTL.  I am using the constraints file provided by digilent, so the RGMII interface is running at 1.5 volts. The link speed is 100 Megabit, and I am sending ping packets onto the network. The oscilloscope has been configured to persist the signal across many captures so you can see what is happening with the signal outside of the data-valid window.

 

You can see that before the packet is placed on the RGMII lines, the signal appears to be floating, then is driven low by the RTL8211E-VL PHY, then we see an ~8uS asserting to indicate a packet should be read on the data lines, then low, and then again floating. 

I am unable to use the RXCTL signal as a proper data-valid envelope to sample the data lines appropriately. I see no other signals that would indicate a packet should start being sampled. I have proven this by setting up an "ILA" logic analyzer within the chip. Since I have little confidence in the clock as well (I am unable to see it on my scope) I am sampling the MII signals using an internal 200Mhz clock.  Here is a capture using the logic analyzer:

 

You can clearly see that the input signal p_i_mii_rxdvalid (RXCTL) is being randomly sampled high and low. The FPGA logic cells cannot distinguish the difference between the floating signal seen on the scope and the low/high moments so all I am seeing is digital noise. 

I have tried enabling pull-down as well as pull-up resistors on the nets in the constraints file, but this made absolutely no difference.

I still have the out-of-box demo loaded into the Flash of the eval board. When I load it, I see the same behavior on all the IO pins as with my project. However, the demo is somehow able to get an IP address and run the echo service and I am able to ping it. The response time seems terrible, but maybe that is the soft-core CPU being used for packet processing. I am unable to figure out how the example project is getting absolutely any data out of this.

 

Any insight into what is wrong with the PHY and why it is not producing proper digital signals is very appreciated. A solution is even better!

 

Thank you all!

[asmi]

There is no way RXCTL is floating, because there is a pull-down (which acts as a strap resistor):

 

I would recommend trying to use eval version of 1G Ethernet Subsystem IP (you can generate an eval license for free at any time) and see if it works. Since demo app works (and yes, it does respond slowly, I've brought it up as well over here years ago and was told that it's normal), I would expect it to work too, if that's the case, that will exclude HW problem.

Edited October 18, 2023 by asmi

[zygot]

You don't provide any clues as to how you are doing 100 Mbsp link speed.

What I can tell you from the information provided so far is that the PHY comes up on power-on configured to do 1000 Mbsp and will auto-negotiate with another PHY at that speed. My guess is that you need to modify some of the PHY registers using the serial interface in order to do 10/100 Mbsp link speeds. Certainly having access to the registers might help you debug your PHY interface. Tri-speed Ethernet IP from FPGA vendors use a clock switch to support the three link speeds as there is a difference in operation.

Likely, the soft-processor project that you can run is using the MDI interface to configure the PHY.

Potentially, the PHY status indicator LEDs on the Genesys2 might be helpful.

No one needs to be constrained by obtuse and licensed FPGA vendor IP to use the Ethernet PHYs on Digilent FPGA boards. Edited October 18, 2023 by zygot

[Questioner]

Asmi and Zygot, thank you both for your responses.

Floating: I completely agree, it should NOT be floating as R222 should hold the line down. However, you see my scope shot, do you have any idea how we get to that mid-state level? rxdata[0] does the same thing (while the 3 other data bits look correct).

I do not intend on using a different MAC, as we have been using our own bare-metal raw-ethernet MAC for years with many different PHYs and switches. My goal was to rebuild some FIFO cores for a Kintex7 from previous designs, but have been derailed by this strange PHY behavior.

100 Megabit: That's the easy part, I am in control of the device on the other end of the Ethernet cable :)  I have a direct cable between the Genesys2 to a Windows10 computer with an Ethernet port dedicated to this project. I configured windows to only run that adapter at 100Megabit so the Genesys PHY and PC PHY have to establish a link at 100Megabit. I have verified this with both lights and clock speeds.

If the PHY had come up in Gigabit mode, the difference would be that the data pins would run at double-data-rate and the clock would be a bit faster. The signaling should still be clean.

 

Any other thoughts on what I am seeing on the scope?

Do either one of you two have a Genesys board handy and could probe R220 for me please?  I agree that a working example project should prove that the hardware "works" but "working-OK" is not the same thing as "working-correctly". There is no way a digital signal should be "floating" or driven to a middle-voltage level in a digital system, yet you can see on my scope that this is exactly what I am seeing.

 

Thank you very much for the fast responses!

 

[Questioner]

Asmi and Zygot, thank you both for your responses.

Floating: I completely agree, it should NOT be floating as R222 should hold the line down. However, you see my scope shot, do you have any idea how we get to that mid-state level? rxdata[0] does the same thing (while the 3 other data bits look correct).

I do not intend on using a different MAC, as we have been using our own bare-metal raw-ethernet MAC for years with many different PHYs and switches. My goal was to rebuild some FIFO cores for a Kintex7 from previous designs, but have been derailed by this strange PHY behavior.

100 Megabit: That's the easy part, I am in control of the device on the other end of the Ethernet cable :)  I have a direct cable between the Genesys2 to a Windows10 computer with an Ethernet port dedicated to this project. I configured windows to only run that adapter at 100Megabit so the Genesys PHY and PC PHY have to establish a link at 100Megabit. I have verified this with both lights and clock speeds.

If the PHY had come up in Gigabit mode, the difference would be that the data pins would run at double-data-rate and the clock would be a bit faster. The signaling should still be clean.

 

Any other thoughts on what I am seeing on the scope?

Do either one of you two have a Genesys board handy and could probe R220 for me please?  I agree that a working example project should prove that the hardware "works" but "working-OK" is not the same thing as "working-correctly". There is no way a digital signal should be "floating" or driven to a middle-voltage level in a digital system, yet you can see on my scope that this is exactly what I am seeing.

 

Thank you very much for the fast responses!

 

[zygot]

I have a couple of Genesys2 boards that I use for various projects using the onboard Ethernet PHY. I've had no issues with either one though my PHY IP needs to use IDELAY on receiver RGMII data pins using the default configuration. I don't use Vivado Ethernet IP or a soft processor for any my designs, just the default 1 GbE link speed and DDR primitives. I don't use a MAC, especially the encrypted MACs that are embedded in FPGA vendor IP. I also have no use for 10/100 link speeds, but that isn't exactly your problem.

Since you are able to use your Genesys2 Ethernet PHY using someone else's bitstream, there's no point to anyone probing anything on a different board as you are telling me that your board is working... you just don;t know why.

Your assumptions about how Ethernet works is a bit naive. 10, 100, and 1000 Mbsps Ethernet each use different modulation schemes. Clocking the RGMII interface at different frequencies isn't sufficient to using all of the link speeds. I never hurts to learn about the interfaces that you want to use in a design. You don't need to know all of the packet types and structures to use an Ethernet PHY on an ( non-ZYNQ ) FPGA board in order to do basic point-point communication between boards or even your Genesys2 to a PC.
All of the relevant hardware layer, protocol and packet information is readily available on-line. You don't need to be an expert ( I'm certainly not ) but understanding basic concepts won't hurt either. FPGA vendors are all happy to provide an alternative, as long as you are dependent on their IP. I advise against that approach if you want to do anything useful with Etherent connectivity for a number of projects.

I recently created an Ethernet Appliance using the XEM7320 and a SYZYGY Ethernet PHY pod that allows one to transfer data at 1 GbE data rates between an FPGA board and a PC using a USB 3.0 interface. No OS Ethernet processes involved. No soft-processors. No ZYNQ software development. No Soft-processor software development. No MACs. No Vivado Ethernet IP. I like to have full control of my FPGA projects and I hate doing unnecessary work. I've done similar projects using PCIe instead of USB PC connectivity. Because of the way that PCIe works on modern PCs, USB is a lot easier to work with and USB 3.0 is plenty fast enough if implemented correctly.

Perhaps you know more about this than I do. In that case I can't help you.

I'm not sure about this but what you are seeing on the scope could be associated with the power saving setting in the PHY. I've never had a reason to look at the receiver control signal on a scope. Regardless, I'm pretty sure that this has nothing to do with your design not working as you intend it to. Edited October 18, 2023 by zygot

[Questioner]

Zygot, thank you for your thoughts.

However, can anyone probe their Genesys2 board at R220 for me while receiving Ethernet packets?
Electrically the signal is incorrect, even if a packet squeezes through once in a while.
Also if anyone knows which registers in the PHY deal with power management or signal integrity (not the clock delays) I would appreciate a pointer to them.
 

[zygot]

Fortunately for you, you don't need to sign an NDA to get documentation on how the Ethernet PHY on the Genesys2 or Nexys Video boards work. This wasn't the case with the ubiquitous Marvell PHYs that appears on other FPGA boards from almost all FPGA vendors. Download the datasheet. The first step in using a device is reading about how it operates. The Marvel PHY information can be gleaned from software driver code and a few other sources, but the Realtek documentation is freely available.

[edit] CORRECTION! Evidently, ReadTek doesn't provide its datasheet freely either. I believe that I got the datasheet from another FPGA board vendor.

Reading about the RTL8211E-VL PHY registers and power-on configuration might be informative.

I'm not sure why you believe that "electrically, the pin behavior is incorrect". If after reading the datasheet and you haven't figured this out you still think that the PHY is mis-behaving, you could try asking Realtek support. You aren't driving the PHY outputs are you? Edited October 18, 2023 by zygot

[zygot]

I looked at the information that I have for the RTL8211 and don't see a power saving bit in any of the configuration registers. The device might power down if the cable is disconnected. Broken off plastic retention clips on the cable could do this. Make sure that ETH_PHYRESET_N isn't getting asserted.

The RTL8211E supports Energy Efficient Ethernet. You might want to read about that. One way of interpreting your scope picture would be the PHY entering a low-power mode when there's nothing to transmit over the RGMII RX pins. Sounds plausible...
Edited October 18, 2023 by zygot

[asmi]

I can probe my board a bit later (I need to find it in the storage), but I support zygot's advice to read the datasheet for the PHY (you can find it online, or drop your email to my PM and I will send it to you), as it's got quite un-orthodox MDIO interface (compared to other 1Gig PHYs I worked with), which is why I avoid using it in my board designs - my current favourite is Marvell 88E1510/1518 (for 3.3 and 1.8 V respectively).

 

Edited October 18, 2023 by asmi

[Questioner]

Thank you Asmi! I would very much appreciate it.

I have used quite a few of the Marvell parts, they are rock-solid, and have the fastest packet propagation of all the PHY's (and switches) I tested when working on some industrial applications. Their Ethernet switches are so fast they start sending a packet out the proper port as soon as they know the MAC address, they don't wait to buffer the whole packet first. Only down side is a whole bunch of NDA's and company agreements and your company name printed all over the datasheet.

With some more messing around and checking, I realized on the RTL8211E I can trigger on the RX pins to capture the preamble (0x5) AND data-valid. When I do so I can actually find the data, but it's a bit of a hack, and that data-valid buzzes up and down between 1 and 0 endlessly at all other times. I would very much like to hear that I have a defective chip and not that this is normal behavior. Marvell certainly never did any hokey stuff like this.

 

[asmi]

Yeah that is certainly a weird device. That said, I've mostly moved on from 1G Ethernet towards something faster, currently looking at MaxLinear GPY212 2.5G Ethernet PHY as 2.5x bandwidth for pretty much the same price is very attractive.

[zygot]

I'm betting that Questioner is chasing a ghost and that there's nothing to see here. Part of this is based on the scope picture and comments in the original post. What I see in the scope screen shot doesn't look like an output tied to GND through a 4.7 Kohm resistor going into a high-impedance state. What I see is a valid active signal tens of us after some quiescent state and even longer before it goes back to a quiescent state; no sigh of mis-behavior. The demo bitstream wouldn't be "managing to send packets, somehow" if the PHY or board was defective. From what I've read in the PHY datasheets, these devices don't strictly adhere to 'normal' logic level standards, especially when sensing pin states for power=on configuration of internal registers.

I don't have any projects that use 10/100 link speeds, but all of my projects using the Genesys2 or Nexys Video boards with the RTL8211E-VL PHY are (eventually) rock-solid on all of them.

Personally, I use an FPGA board that serves 1 GbE packets continuously, with <300ns inter-packet delay, as a testbed to verify that new FPGA projects using an Ethernet PHY on a different board has a robust PHY interface. The DUT just echos packets as received with a small delay so that there is full-duplex data transfer during most of the packet transfer. These test involve PHYs from Marvel, Realtek and TI and FPGA devices from multiple vendors, and generations spanning Spartan3 to UltraScale.. If one of them enters a low-power quiescent state when the test isn't running, I really don't care about that. The only time I've had to resort to using the MDI is for UltraScale platforms using Vivado IP due to the bizarre way that the IP works ( or when a board with a PHY comes up in an unusable state so that the I have to program it for desired operation). Every FPGA board with a 1 GbE PHY, regardless of FPGA vendor has a Tester and DUT bitstream to make settign up testing as easy as possible.

If you can report seeing random failures during actual packet transmission, with pictures, then perhaps there's something to discuss. Edited October 20, 2023 by zygot

[zygot]

The MaxLinear devices are certainly interesting. Marvel makes 1/2.5/5/10 GbE PHYs that work over copper as well. Asmi, have you tried any of those?

I looked into MaxLinear as they are available in distribution but haven't been able to contact anyone who will send me the required information. 2.5G Ethernet provides a nice data rate for FPGA/PC connectivity. Let me know if you make any of GPY212 designs. Populated boards would be preferred.

[asmi]

On 10/20/2023 at 10:46 AM, zygot said:

Marvel makes 1/2.5/5/10 GbE PHYs that work over copper as well. Asmi, have you tried any of those?

No, as none of those have publicly available datasheets. The only multi-gig PHY of theirs which does is 88E2010/2040L, which is a 5-speed 5G/2.5G/1G/100M/10M PHY. But they are kind of pricey ($40 per chip), and my home network is only 2.5G/10G, so I have no use for 5G anyway at this point.

On 10/20/2023 at 10:46 AM, zygot said:

I looked into MaxLinear as they are available in distribution but haven't been able to contact anyone who will send me the required information. 2.5G Ethernet provides a nice data rate for FPGA/PC connectivity. Let me know if you make any of GPY212 designs. Populated boards would be preferred.

The datasheet is publicly available, and if you sign up with acccout, you can download additional stuff like appnotes, reference designs, software, etc. I've recently designed a board with Artix-7 100T in 484 ball package and an SODIMM module (infact later today I'm going to begin assembling a second respin of those boards after fixing some issues with the first revision), so my plan is to build an addon board with GPY212B1VC. I've already bought a few of those chips, so it's just a matter of spinning up the small addon board. I like that this device has internal DC-DC buck for the core power, so it can be powered for a single 3.3 V rail.

This is what rev A of my Artix board looks like after some "fixes" with bodges and stuff :)

It's got 3 high-speed connectors, two on the left are for regular IO (diff pairs on the bottom connector, and single-ended on the top one), the one on the top of the board has all MGTs broken out with their clocks, and a single IO byte lane worth of regular IO pins for auxiliary stuff like I2C buses, control pins and things like that.

 

Edited October 21, 2023 by asmi