Board not responding (for 2 out of 3 boards)

[nbv]

Hi 

for about a year I have running 2 mcc 152 boards (addresses 0 and 1) and one mcc 118 (in address 2) on a raspberry pi 3 without any problems. 

Operating System: Raspbian GNU/Linux 10 (buster)
Kernel: Linux 5.10.103-v7l+
Architecture: arm

today I got the two 152:

daqhats.hats.HatError: Addr 0: boards Board not responding

daqhats.hats.HatError: Addr 1: boards Board not responding

using the MCC DAQ HAT Manager app, both boards do respond to clicking on open but no error message appears. The 118 in address 2 opens without a problem. 

clicking on "list devices" shows all three of them (see below).

I ran sudo daqhats_read_eeproms in the terminal and received "Found EEPROM at address 0" (and the same for addresses 1 and 2).

reading EEPROMs from the MCC DAQ HAT Manager app shows the same (see below).

any suggestion for a solution? is there a way to run a test on the boards without unmounting them?

thanks

[Fausto]

Hello @nbv.

Are you using the same RPi 3 module with these new MCC 152 boards or a different RPi module?

Please confirm that both SPI and I2C options are enabled via the OS main menu - Preferences - Raspberry Pi Configuration - Interfaces tab.

Disconnect any wiring to the MCC 152 boards, restart the OS, and repeat your troubleshooting steps.  If the issue continues, then you will need to unmount the boards and test the MCC 152 boards individually.

Regards,

Fausto

[nbv]

Hi Fausto,

thanks for your reply. 

first, a correction - I am using Raspberry Pi 4 (not 3).

both SPI and I2C options are indeed enabled. the setup worked perfectly asi-is a week ago.

without disconnecting the wiring I did the following:

- Rebooted

- Unmounted all three boards

- Remounted only one and tried again - no success.

- Installed everything on a new Raspberry pi 4 based on your github readme (side note: installing on the latest OS is a nightmare) and connected the MCC 152. As with the other (older OS) pi, the HAT manager app reads the chip in address 0 but the board is not responding.

* I can try to repeat everything without the wires if you think this could make a difference. 

UPDATE: Tried with the wiring off. same result. the board is not responding. it is (like before) identified in its address when asking the app for a list of devices and the EEPROM is also found. But it can not be opened (with using my script and also with the HAT manager app).

 

 

Edited March 4 by nbv

[Fausto]

Hello @nbv.

So that I understand what transpired with your hardware configuration and testing, please confirm whether or not the following information is correct.

You began with a RPi 4 module (Buster OS installed) that had no daq hat board mounted.  You then installed a MCC 152 (with no jumper on W2; address 0) followed by a second MCC 152 (with jumper at A0 on W2; address 1).  Afterward, you installed a MCC 118 (with jumper at A1 on W2; address 2).  With both SPI and I2C options enabled and the MCC DAQ HAT library v1.4.0.8 installed, this hardware configuration worked with all three MCC daq hat boards and the MCC DAQ HAT Manager application.  

Next, you removed all three MCC daq hats from the RPi 4 module and then mounted only one of those boards.  You ran the software and the board would not respond as expected.  Was the MCC daq hat board over-flexed when it was removed from the RPi module?  Can you think of anything else that may have happened when you removed the three MCC daq hat boards from the original RPi 4 configuration?

For the next configuration, you mounted one of those MCC 152 boards on a new RPi 4 module (running Bookworm OS version) and the MCC DAQ HAT library v1.4.0.8 (or the v1.5.0.0 beta?).  With the address jumper configured for address 0, the MCC 152 continued to not respond as expected.

 

Regards,

Fausto

[nbv]

Hi, 

not exactly. I'll explain it again chronologically. 

1) a new RPi 4 (not sure what is the name of the OS but it was the one recommended by RPi at the time) was setup as per instructions.

2) 3 MCC boards were mounted and installed (MCC 152 on address 0, MCC 152 on address 1, MCC 118 on address 2). all jumpers and OS configurations as you wrote. The setup worked perfectly well for more than a year both with the manager app and with my scripts. 

3) one day (this last Monday) it simply didn't work anymore. it didn't work in the way I explained in the original question i.e. "board not responding" (both of the MCC 152 on address 0 and 1, the MCC 118 continued to work).

4) The only thing that happened between the last time it worked and this Monday was the weekend during which the pi was turned off. I usually turn the pi off over the weekend. so the boards were not moved or removed or flexed. I only started manipulating them after the "not responding" error. 

5) as part of troubleshooting I unmounted all three boards and mounted back only the first one (mcc 152 address 0) - it didn't work. 

6) I tried it on another RPi - it didn't work. 

7) I tried it again on both pi's after disconnecting all the wires. - didn't work. 

8) NEW UPDATE: took one other mcc 152 scavenged from another system and connected it to my RPi 4 in the original setup (without any jumper in address 0) - this works perfectly well. next step, I took the second mcc 152 (the one that was before in address 1) and mounted it on top of the "new" mcc 152. now both of them work. I continued and mounted the mcc 118 in address 2 (as before) and it also works (as it did before).

9) my (strange) conclusion is: for whatever hardware-related reason, the mcc 152 in address 0 failed. it was identified and the EEPROM can be read but it could not respond (could not be opened in the manager app). This caused the second mcc 152 in address 1 to also not respond in the same exact way, although it is not damaged. ones I put in the same address but with a "new" mcc 152 below it, both respond. Even stranger - this problem didn't affect the mcc 118 in address 2.

 

does this make any sense?

Edited March 6 by nbv

[Fausto]

Hello @nbv.

Thank you for the chronological clarifications and additional troubleshooting.  That is strange indeed.  I'd ask you to swap the original MCC 152 in address 1 with the scavenged MCC 152 in address 0 just to see if the issue returned, but why disrupt what's working now.  You tested multiple RPi modules and multiple MCC 152 boards in the various address locations.  It seems to be an intermittent issue, but not sure of the root cause.  Regrettably, I don't have any other suggestions at this time.  Keep us posted if you find anything else.

Regards,

Fausto

[Nick Wright]

On 3/6/2024 at 5:06 AM, nbv said:

Hi, 

not exactly. I'll explain it again chronologically. 

1) a new RPi 4 (not sure what is the name of the OS but it was the one recommended by RPi at the time) was setup as per instructions.

2) 3 MCC boards were mounted and installed (MCC 152 on address 0, MCC 152 on address 1, MCC 118 on address 2). all jumpers and OS configurations as you wrote. The setup worked perfectly well for more than a year both with the manager app and with my scripts. 

3) one day (this last Monday) it simply didn't work anymore. it didn't work in the way I explained in the original question i.e. "board not responding" (both of the MCC 152 on address 0 and 1, the MCC 118 continued to work).

4) The only thing that happened between the last time it worked and this Monday was the weekend during which the pi was turned off. I usually turn the pi off over the weekend. so the boards were not moved or removed or flexed. I only started manipulating them after the "not responding" error. 

5) as part of troubleshooting I unmounted all three boards and mounted back only the first one (mcc 152 address 0) - it didn't work. 

6) I tried it on another RPi - it didn't work. 

7) I tried it again on both pi's after disconnecting all the wires. - didn't work. 

8) NEW UPDATE: took one other mcc 152 scavenged from another system and connected it to my RPi 4 in the original setup (without any jumper in address 0) - this works perfectly well. next step, I took the second mcc 152 (the one that was before in address 1) and mounted it on top of the "new" mcc 152. now both of them work. I continued and mounted the mcc 118 in address 2 (as before) and it also works (as it did before).

9) my (strange) conclusion is: for whatever hardware-related reason, the mcc 152 in address 0 failed. it was identified and the EEPROM can be read but it could not respond (could not be opened in the manager app). This caused the second mcc 152 in address 1 to also not respond in the same exact way, although it is not damaged. ones I put in the same address but with a "new" mcc 152 below it, both respond. Even stranger - this problem didn't affect the mcc 118 in address 2.

 

does this make any sense?

The MCC 152 is different from the other MCC HATs in that it uses an I2C IO expander on the Pi I2C bus to implement the digital I/O.  The other HATs all use SPI only.  If the IO expander on a 152 gets damaged it can cause issues on the I2C bus, which would show up as problems communicating with other MCC 152s but not different models of HATs.

The HAT detection is performed by reading the EEPROMs on a dedicated ID I2C bus (separate from the one used for the IO expanders) so a damaged board can still be detected but communications do not work.

The most likely causes for damage to the IO expander are voltage related, due to things such as:

• Overvoltage on an IO pin (like connecting a 5V input signal when the MCC 152 is configured to 3.3V)
• Undervoltage on an IO pin (such as connecting signal and ground backwards)

[nbv]

Thanks Nick,

to follow up on what Fausto wrote, one last test I did was to connect the damaged mcc 152 in address 1 on top of a working mcc 152 in address 0. the result is again that both do not respond. i.e. the damaged mcc 152 prevents the working one from responding regardless of which one is in address 0.