Implimenting Etherent on the Zybo Z7 development board

[evers4]

I'm working with a Zybo Z7 development board and trying to implement Ethernet. I'm new to Ethernet and have been looking for an example project that utilizes it.

The example design for the (AXI 1G/2.5G Ethernet Subsystem) is quite bare bones and I don't see how to configure it for the Zybo. Addtionally, I'm looking connect the Ethernet Subsystem to an existing project that generates bytes. I'm attempting to have those bytes streamed to a PC. My plan is to utiize a python script to capture this data.

 

Thank you so much!

Chris

[evers4]

I'm attempting now to modify the Python script to request an arbitrary amount of data/bytes from the FPGA. Since I'm not sending any commands to the FPGA, such as how many bytes to send I tried removing the call to client.send(). However, when I run this code the script hangs. Is it possible to have one way communication here by only calling client.recv()?

 

import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(("192.168.1.10", 7)) # connect to the server; enter your server's IP and port here

# number of bytes we're requesting from the server; change this as desired
num_bytes = 5000
 
# arbitrary packet size, max number of bytes we'll receive at once
packet_size = 256 #256
 
# note: little endian is important, requirement for Zynq-7000 to easily translate the sent number to an int without reordering
#print(f"requesting {num_bytes.to_bytes(2, 'little')} bytes")
#client.send(num_bytes.to_bytes(2, 'little')) # request {num_bytes} bytes in response
 
# loop while calling recv to receive data from the client until the expected number of bytes has been successfully transferred
received = 0
while received < num_bytes:  
    data = client.recv(packet_size)
    for d in range(len(data)):
        print(f"data[{d}] ({data[d]})")
    received += len(data)
if received == num_bytes:
    print("All data received!")
else:
    print("Missing data!")

[artvvb]

You would also need to modify echo.c on the board side. The way the blog code is structured makes it so that tcp_write is called from the recv_callback. If the server never receives a byte count to return, it never tries to send anything back to the client. You might call tcp_write (and set up the globals that manage how much is in the buffer) from like the interrupt handler servicing the AXI master, or just from main for testing purposes. You'd also want to make sure that the server doesn't try to send data before the connection is established, either by waiting for an accept_callback to occur before sending anything, or by starting the python script before running the app on the board.

[evers4]

Regarding the fix to the Ethernet Example,

I replaced the calls to tcp_sndbuf(tpcb) with tcp_mss(tpcb) following the updated blog post. However, I'm still seeing errors when requesting more than 1446. Do you still see this problem on your end, if not I'm wondering what could be different with my configuration.

 

[evers4]

Great news,

the original goal has mostly been accomplished at this point, thank you! I'm receiving data over Ethernet from the custom IP in the firmware, and better still, so far it appears correct. In main() I'm sending data once a connection is made via the accept callback.

My question now is how can I detect an active connection. The accept callback increments the connection variable, but it doesn't decrement it. So I'm wondering how to reliably detect that an active connection exists.

The way the C code works is it fills a ring buffer with data, from the IP in the firmware, and pushes that out to an active connection as long as new data exists. I'm currently looping on 3 conditions: an active connection, data available, & a state I call sending. I'm using the connection variable in the accept callback to determine if a connection is active. But, this value does seem to track disconnects.

 

err_t accept_callback(void *arg, struct tcp_pcb *newpcb, err_t err)
{
    static int connection = 1;
    xil_printf("Echo: c_pcb=%u\n\r",(u32)newpcb);

    /* set the receive callback for this connection */
    tcp_recv(newpcb, recv_callback);

    /* set the sent callback for this connection */
    tcp_sent(newpcb, sent_callback); // <- add this

    /* just use an integer number indicating the connection id as the
       callback argument */
    tcp_arg(newpcb, (void*)(UINTPTR)connection);

    /* increment for subsequent accepted connections */
    connection++;

    return ERR_OK;
}

[artvvb]

It looks like there are more callbacks that could also be used listed out in the tcp.h header - lines 60-134 for function prototypes and 414-421 for callback registration functions. The callback registration function implementations in tcp.c also have more information/documentation in their function headers. I'd take a look at either figuring out if tcp_err is called for the kind of disconnect event you are looking at, or implementing a timeout with tcp_poll.

[evers4]

HI again artvvb,

let me update you on the status of the project:

I'm able to receive large quantities of data from the FPGA over Ethernet using the (tcp_write/tcp_mss). I've tested packet sizes up to 65500, with num_bytes up to 1 million. I'm seeing error rates of approximately 0.4%. For example, for 1,000,000 bytes requested I see 4374 errors. I see that the packet size throttles as the ring buffer on the FPGA empties (packet_size=[65500, 65500, 5012, 1446, 1446, 1446, 1446, ...]).

I tied out the tcp_err function, however it doesn't call the supplied callback when using client.close() in the script, nor when physically unplugging the Zybo Ethernet port. The problem I'm trying to solve is that when relaunching the script I have to reboot the FPGA. It would be great too if I could disconnect/reconnect the Zybo Ethernet port though.

When I run use the code from the revised forum post I still receive errors when requesting over 1446 bytes. Which I just noticed matches the received packet size I'm observing as the ring buffer empties. Do you still see error when running the revised code?

Edited December 21, 2023 by evers4

[artvvb]

I copied the code back in from scratch, and yes, I see the bug again. Try the following edits to the python script?

import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
client.connect(("10.0.0.128", 7)) # connect to the server; enter your server's IP and port here 

# number of bytes we're requesting from the server; change this as desired 
num_bytes = 2000
 
# arbitrary packet size, max number of bytes we'll receive at once 
packet_size = 500
 
# note: little endian is important, requirement for Zynq-7000 to easily translate the sent number to an int without reordering 
print(f"requesting {num_bytes.to_bytes(2, 'little')} bytes") 
client.send(num_bytes.to_bytes(2, 'little')) # request {num_bytes} bytes in response 
 
# loop while calling recv to receive data from the client until the expected number of bytes has been successfully transferred 
received = 0 
errors = 0
value = 0
while received < num_bytes:  
    data = client.recv(packet_size) 
    for d in range(len(data)):
        if data[d] != 0xff & value: # validate data 
            print(f"Error, data[{d}] ({data[d]}) != {0xff & value}") 
            errors += 1
        value += 1
    received += len(data)
    print(received, len(data))
if errors == 0:
    print("All data received matched the expected values!")
else:
    print(f"{errors} errors")

This tracks the "value" separately from the index of the data in the buffer - changing packet sizes away from 256 or receiving amounts of data other than 256 in one block before would have caused the value that the script uses to compare against received data not to match. I'm out most of the week but will take a look at updating the post when I'm back.

Thanks,

Arthur

[artvvb]

Hi @evers4

The blog post has been updated with corrected sources. The original "fix", to use tcp_mss instead of tcp_sndbuf, was not required, as the bug turned out to be a combo of how the python script was validating the data coming back as well as some additional issues with pointer/buffer management. There's some possibility the buffer issue in C was causing your more recent issues; I was seeing integer underflows causing some data to be re-sent. A push_data helper function was added to the blog that looks to address this. Apologies, and thanks again.

Happy new year,

Arthur

[evers4]

Hi and happy new year,

thanks for following up on the example and updating the ethernet example. I don't think the updates will directly effect my current project since it has diverged somewhat since we started working together.

The general issue I’d like to work out with you is debugging the data path from the Firmware to remote PC. I’m receiving what looks like mostly valid data, but with occasional hiccups. I’m still seeing a data rate error of about 0.4%.

Among the few issues I’m still trying to resolve pertains towards the firmware that generates the data forwarded out the Zybo board’s ethernet port.

For some reason I’m seeing one of my debug signals (m00_axi_bresp[1:0]) being factored out of my synthesized design. I’d like to debug this particular signal using the embedded logic analyzer. Its unclear to me why this is happening, since this block (AxiDeviceMasterSlave) is a barebones IP generated using the Vivado IP Package Manger. An AXI-Lite example design with a master and slave port.

I did check the path of this signal, it begins from a Xilinx IP called AXI Memory Interconnect, and ends within the generated IP being captured by a register. Is there any reason why this signal could be factored out? And how can I tell the tools to keep this one?

[artvvb]

Happy new year!

I'd recommend checking the elaborated design to make sure, but does the bresp register drive any other logic? It could be getting removed during synthesis due to Vivado thinking it is unimportant and unused - this is a normal step that the tools take and is important for reuse of modules, as, otherwise, plenty of unused logic could remain and eat up FPGA resources.

One suggestion for keeping the signal around, according to this thread, https://stackoverflow.com/questions/30676741/how-to-cheat-synthesis-to-keep-an-unused-signal-without-using-special-compile, would be to use it to drive some otherwise unused FPGA pins, maybe on the Pmod headers.

[evers4]

Hi artvvb,

I've managed to resolve the issue with the factored out signal. As well as another problem involving the ILA. I found that I couldn't connect to the ILA debug core when programming the FPGA in Vivado, only when I ran the Vitis Project (programming the FPGA with both the bitstream and app) could I connect. I suspect that this is related to having the Zynq in the design which provides the Fabric clock driving the core.

Now, I'm having success with the entire data path. From fabric to remote PC. However, I'm seeing errors in the data received. The error rate is low and is observed when sending large quantities of it (I'm sending 64MB at a time). What I'm finding is the combined summation of the len argument in sent_callback is greater than what I'd expect. I'm suspecting that this could be due to TCP retransmissions. As it turns out the callback indicates an acknowledgement from the receiver side of data to be sent, not that it was successfully transmitted.

I'm still sending a counting value 0->UINT32_MAX_VALUE, and verifying that each value received is value[i+1]=value[i] (Except on overflow). That data is mostly correct, but there are some segments with errors. I'm suspecting data reordering, retransmission, or transmission errors.

I'd like to try using the UDP protocol and compare the results. Is this straight forward? How would I go about modifying the Vitis project & script?

[artvvb]

On 1/24/2024 at 4:08 PM, evers4 said:

I've managed to resolve the issue with the factored out signal. As well as another problem involving the ILA. I found that I couldn't connect to the ILA debug core when programming the FPGA in Vivado, only when I ran the Vitis Project (programming the FPGA with both the bitstream and app) could I connect. I suspect that this is related to having the Zynq in the design which provides the Fabric clock driving the core.

I think the clock should not start up until after the Zynq PS has been configured, yes. It ought to be possible to program the bitstream from Vivado and program only the PS from Vitis - some checkboxes in the Run and Debug Configurations allow you to bypass bitstream programming.

On 1/24/2024 at 4:08 PM, evers4 said:

Now, I'm having success with the entire data path. From fabric to remote PC. However, I'm seeing errors in the data received. The error rate is low and is observed when sending large quantities of it (I'm sending 64MB at a time). What I'm finding is the combined summation of the len argument in sent_callback is greater than what I'd expect. I'm suspecting that this could be due to TCP retransmissions. As it turns out the callback indicates an acknowledgement from the receiver side of data to be sent, not that it was successfully transmitted.

Interesting, I'll need to do some debugging to confirm. Haven't had a chance the last few days.

On 1/24/2024 at 4:08 PM, evers4 said:

I'm still sending a counting value 0->UINT32_MAX_VALUE, and verifying that each value received is value[i+1]=value[i] (Except on overflow). That data is mostly correct, but there are some segments with errors. I'm suspecting data reordering, retransmission, or transmission errors.

I'd like to try using the UDP protocol and compare the results. Is this straight forward? How would I go about modifying the Vitis project & script?

I'm not sure how to go about setting up UDP - the performance measurement application templates might have some hints.

[evers4]

I’d like to share my design, but I’m not sure how to send it to you due to the size.

I’ve migrated the project to UDP, though I’m having trouble receiving all the data from the FPGA. The FPGA is periodically sending 64 MB of data to a python script listening form a remote pc.

What happening is, a function on the FPGA loops calling the Xilinx function udp_sendto(), transferring a packet at a time, until 64MB have been processed. After this I wait 1 second and do this again. After 4-11 times of running my function I get an error in putty over the UART “pack dropped, no space”.

It looks like, there is a transmit buffer somewhere that is being full. I don’t know how to flush it. Actually, I’m not sure why its not being flushed in my function, because after every call to udp_sendto() I free the pbuf passed to it using pbuf_free().

I’ve attached the relevant parts of the Vitis code & the entire python script. If we can find a way to transfer the entire Vitis/Vivado project ill do that. The guts of the code is in main.c & echo.c, and the project is a modified version of the Xilinx echo server example project.scratch_pad_script.py

Vitis Project Code.7z

[artvvb]

To try to answer one of your questions today:

Sharing projects is kind of a huge topic... Digilent has used these two sets of scripts to get projects packaged up for git - https://github.com/Digilent/digilent-vivado-scripts, https://github.com/Digilent/digilent-vitis-scripts - but there may be issues with various tool versions and certain unsupported features (RTL modules aren't handled automatically for example). GitHub's file size limits for files included in releases are pretty lenient, so it might be possible to attach full project archives for both Vivado and Vitis if you created a repo there. Starting points in the menus for creating ZIPs are File -> Project -> Archive in Vivado and File -> Export in Vitis, at least for 2021.1. The nice thing about using the auto export tools is that, at least in Vivado, it'll collect up all external dependencies like IPs and board files, and they can at least theoretically cut out a lot of the build outputs and other random files that show up in project directories.