MCC152 I2C Not Responding

[Display Name]

I have an four MCC 152 HATs. Two of them appear to be bad as described below. I connect one at a time to my CM4 on the stock CM4 I/O board. Daqhats version 1.4.0.8 on RPi OS bookworm (lite version). I've tried different CM4s and different I/O boards. These are not the issue. It is the HATs. I can run `daqhats_list_boards` and, for every HAT, installed by itself, I get:
 

Found 1 board(s):
Address: 0
Type: MCC 152
Hardware version: 2
Name: MCC 152 Analog Output / Digital I/O HAT

Similarly, `sudo daqhats_read_eeproms` returns:
 

Reading...
Found EEPROM at address 0
Done 

However, upon attempting to create an `mcc152()` instance in Python, I get `daqhats.hats.HatError: Addr 0: Board not responding.` for two of the HATs. This led me to run

i2cdetect -y 1

For one of the bad HAT, this command returns very slowly as if a timeout is being reached for every address scanned. With multiple hats, nothing is detected.

For the other bad HAT, the `i2cdetect` command will show other HATs if installed but not this bad HAT.

In all cases, the digital scope trace of GPIO 2 and GPIO 3 shows typical I2C traffic. The KiCAD file for the RPi I/O board shows the I2C pins (GPIO 2/3) going directly to the CM4. Again, I've tried multiple CM4s and I/O boards and it is not the RPi end. When these bad HATs are replaced, everything works as intended. I do not know what they do once they reach the MCC 152, though, after probing, they do not seem to go directly to the PCAL9554B expander.

The first bad HAT was connected to a rotary encoder. When I noticed the HAT was not responding to I2C, I replaced it with another board. It was working fine until I connected the rotary encoder to it. At that point it became the second bad HAT. The rotary encoder does not appear to be outputting any high voltages (it's powered by 24 VDC) when connected to a scope. Actually, the encoder is not outputting anything on its 8-bit output at all.

Anyway, if the DIO was too high, would this cause the I2C issue?

This seems to be a common issue based on this post and this post.

Is there a newer hardware version coming out?

[Display Name]

@Nick Wright and @Fausto, I've figured out the issue. The problem was that I have a connector that hot-plugs the encoder into the DAQ as pictured below.

As mentioned in previous posts, my sensors are active 24 VDC, and the inrush current on the power lines when hot-plugged was coupling transient voltage into the signal lines (see scope trace below). These transient voltage spikes were causing the PCAL9554B on the MCC 152 to fail. The first symptom noticed under these failures was lack of I2C communication. (As an aside, I was able to purchase this IC and repair the failed MCC 152 HATs, which I am now using as sacrificial test boards.)

The solution was a custom PCB that attaches each signal line to a rail-to-rail transient voltage suppression (TVS) diode (see photo below). The TVS diode array used was a Semtech SRDA70-4. This diode array is "rail-to-rail" and the clamping voltage is set by the reference supplied and the diode forward bias voltage. I am referencing mine to 3.3 V. Note that this will put transients on the 3.3 V source. For this reason, it is important not to connect this reference to the CM4 3.3 V GPIO pin (I tried this first, and the 3.3 V transients caused the CM4 to reboot after every hot-plug attempt). I am now pulling the 3.3 V reference from the dedicated output from the 3.3 V DC-DC converter on the RPi I/O board. Note that this converter is for the PCIe connection. I am not using PCIe, so am pulling the 3.3 V power from a PCIe extender cable connected to the I/O board. In the near future, I plan to develop my own I/O board with this dedicated 3.3 V trace to a terminal block (without the other PCIe traces). For now, pulling from the PCIe extender cable is sufficient. For anyone interested in this fix, the custom TVS PCB KiCAD project can be found here. The custom board is not meant to mount in the HAT stack, feel free to modify and use as needed.

[Fausto]

Hello @Display Name.

If your two MCC 152 boards do not work and the other two MCC 152 boards work with the encoder, then most likely the two nonfunctioning boards are damaged.  I will send you a private message for replacements (if within the 1-year warranty period).

I am not aware of a new hardware version of the MCC 152 to be released.

Regards,

Fausto

[Display Name]

To any users stumbling upon this thread. It is the MCC 152s failing. The encoder is an NPN open-collector, active low output device that appears to be working properly and cannot apply voltage to the digital input terminals on the HATs and therefore cannot overload them. The PCAL9554B is the point of failure. It should be able to handle an open-collector output and its data sheet stated that it has "weak pull-up" resistors, which means they are likely of large resistance and should easily limit current through the open-collector circuit. See IR images below.

Image FLIR6767.jpg (the first image) is the HAT that most recently failed. Its PCAL9554B IC is approaching 200 degrees C with nothing connected to it. The PCAL9554B immediately cools when the W3 jumper is pulled and immediately heats up when it is reinserted. The failure symptom with this HAT is that it is not detected on the I2C bus but other HATs are. I'm guessing that, when left with these heating conditions, further internal failure results, eventually progressing to internal shorts as found with the other HAT resulting in total I2C bus corruption.

Image FLIR6771.jpg is of the HAT that failed first. I first noticed that, when powered down, pin 1 of the PCAL9554B had a very low resistance to ground. When powered and trying to assign an address to this HAT, the A0 pin was held low (~0.2 V). As you can see in the IR image, the logic level shifter is also very hot, as there is significant current being drawn by pin 1 of the PCAL9554B (very low resistance). Note that the failure symptom of this HAT is to corrupt the entire I2C bus. Nothing is detected when scanning the bus, and every address is scanned very slowly as if a timeout is being reached with every address.

Edited March 6 by Display Name

[Fausto]

Hello @Display Name.

Thank you for sharing the results of your investigation.

Kindly provide a wiring diagram of how the MCC 152 is connected to your circuit.  Please include the voltages involved and any other details.  Also, please confirm that your code does not accidentally put the MCC 152's IO into output mode when connected to the encoder. 

Regards,

Fausto

[Display Name]

@Fausto, no problem. A detailed description of my setup is below.

The rotary encoder is a Koyo TRD-NA256NWD from Automation Direct. The data sheet is here. Wiring is shown in the image below (network connections not shown):

The CM4 IO Board is the stock, open-source board from Raspberry Pi. Note that powering the IO board via 24 VDC is acceptable per the data sheet:

Quote

The +12V input feeds the +12V PCIe slot, the external PSU connector and the fan connector directly. If these aren’t being used then a wider input supply is possible (+7.5V to +28V).

The encoder has active low open collector outputs, and, when working properly, cannot source voltage through the outputs connected to the MCC 152. The encoder's open collectors can sink a maximum of 32 mA each. The PCAL9554B on the MCC 152 has 100 kOhm pull-up resistors so the sinking current should be in the micro-amps. I have verified the encoder in question is working properly per the oscilloscope image below:

My code that utilizes the MCC 152 to monitor the encoder is below (the HAT initialization is shown with the comments):

import time
from enum import Enum
from typing import Literal

from redis import Redis

from daqhats import DIOConfigItem, mcc152


class Orientation(Enum):
    UP: str = "encoder wheel is up"
    DOWN: str = "encoder wheel is down"


class EncoderInstructions(Enum):
    STOP: str = "stop encoder data acquisition"


class Encoder:
    ENCODER_WHEEL_CIRCUMFERENCE_FEET: float = 1.000

    def __init__(self, address: int = 0, resolution_bits: int = 8):
        self._cache_connection = Redis(decode_responses=True)
        self._mcc152_connection = mcc152(address)

        self._counts_per_revolution: int = 2**resolution_bits
        self._minimum_encoder_angle_increment_degrees: float = (
            360 / self._counts_per_revolution
        )
        self._max_binary_position: int = self._counts_per_revolution - 1
        self._encoder_orientation: Literal[None, Orientation.UP, Orientation.DOWN] = (
            None
        )
        self._direction_scaler: Literal[None, -1, 1] = None

        self._gray_to_binary_hashmap: dict = {
            gray_code: self._gray_code_to_binary(gray_code)
            for gray_code in range(self._counts_per_revolution)
        }

        self._cache_connection.set("encoder_instruction", "")
		
        # Initialize HAT here
        # Start with the default configuration
        # 	All channels are input
        # 	Pull-up resistors are enabled
        self._mcc152_connection.dio_reset()
        # Encoder is active low and default configuration above sets
        # all inputs to non-inverting. Invert all inputs so they read
        # as active high.
        self._mcc152_connection.dio_config_write_port(
          DIOConfigItem.INPUT_INVERT, int(b"11111111", 2)
        )
        # Default configuration is non-latching.
        # Encoder state is 8-bit gray code, so a single bit changes
        # at a time, and the position state depends on all bits. Latch
        # on state change to store actual 8-bit state to prevent errors
        # when position changes before/during a read.
        self._mcc152_connection.dio_config_write_port(
            DIOConfigItem.INPUT_LATCH, int(b"11111111", 2)
        )
        # Interrupt signal is not needed and costly, disable it.
        self._mcc152_connection.dio_config_write_port(
            DIOConfigItem.INT_MASK, int(b"11111111", 2)
        )

    @property
    def encoder_orientation(self) -> Literal[None, Orientation.UP, Orientation.DOWN]:
        return self._encoder_orientation

    @encoder_orientation.setter
    def encoder_orientation(
        self,
        encoder_orientation: Literal[Orientation.UP, Orientation.UP],
    ):
        self._encoder_orientation = encoder_orientation

        if encoder_orientation == Orientation.UP:
            self._direction_scaler = -1
        elif encoder_orientation == Orientation.DOWN:
            self._direction_scaler = 1

    def _gray_code_to_binary(self, value: int) -> int:
        value ^= value >> 4
        value ^= value >> 2
        value ^= value >> 1

        return value

    def _binary_position_to_angle(self, binary_position: int) -> float:
        angle: float = binary_position * self._minimum_encoder_angle_increment_degrees

        return angle

    def acquire_data(self, start_time: float):
        assert self._direction_scaler is not None

        encoder_rotation_count: int = 0
        encoder_position_degrees: float = 0
        allow_data_acquisition: bool = True
        last_binary_position: None | int = None

        while allow_data_acquisition:
          	# Latching enables, so read all bits (i.g., port)
            gray_coded_position: int = self._mcc152_connection.dio_input_read_port()
            time_of_measurement: float = time.time() - start_time
            binary_coded_position: int = self._gray_to_binary_hashmap[
                gray_coded_position
            ]

            if last_binary_position is not None:
                change_in_position: int = binary_coded_position - last_binary_position
            else:
                change_in_position: int = 0

            if change_in_position or last_binary_position is None:
                if change_in_position <= -0.80 * self._max_binary_position:
                    encoder_rotation_count += 1
                elif change_in_position >= 0.80 * self._max_binary_position:
                    encoder_rotation_count -= 1

                angle = self._binary_position_to_angle(binary_coded_position)
                encoder_position_degrees = self._direction_scaler * (
                    encoder_rotation_count * 360 + angle
                )
                last_binary_position = binary_coded_position

                self._cache_connection.set(
                    "encoder_position_time_seconds", time_of_measurement
                )
                self._cache_connection.set(
                    "encoder_position_degrees", encoder_position_degrees
                )

                instruction = self._cache_connection.get("encoder_instruction")

            if instruction == EncoderInstructions.STOP.value:
                allow_data_acquisition = False
                self._cache_connection.set("encoder_instruction", "")

 

Edited March 7 by Display Name

[Fausto]

Thank you for the additional information, scope trace, and code.

Is the DIO voltage (W3 jumper) configured to 3.3V or 5V?

 

 

[Nick Wright]

@Display Name could you also please capture an analog scope trace on one of the encoder signals to check for the voltage levels, spikes, etc?