Waveforms Live - Digilent Instrumentation Protocol possible inconsistency

[Ryan Minnig]

I am toying around with implementing parts of the Instrumentation Protocol on a product I'm developing.

I have the web (non-local) version of waveforms live running in a browser window and connected to the Digilent Agent 1.0.1.

During the initial connection process when clicking "OPEN" from the Configuration Menu page I see the website (thru the Agent) send:

{"mode":"JSON"}\r\n

My device then responds with:

{"mode":"JSON","statusCode":0,"wait":500}\r\n

The website then sends:

{"device":[{"command":"enumerate"}]}

please note there is no "\r\n".

This seems to be an inconsistency when referencing 

https://reference.digilentinc.com/reference/software/digilent-instrumentation-protocol/protocol

as some commands including the trailing \r\n and some do not.

If there is no terminating character for the JSON data, does anyone have a good way to key parsing of the incoming data without having to implement a full parser on every character received?  Note that since I'm using the Digilent Agent, my device is connected on a USB COM port at 1250000baud, so my embedded device receives an interrupt on every character received.  I have looked at the existing OpenScopeMZ firmware source code from Github and while my project is written in C vs. the OpenScopeMZ in C++, the JSON parser is a little more complicated than I want to port to my device at the moment.

My goal ultimately is to have a subset of the functionality with the excellent graphical interface of waveforms live to help in the development of my product.  The ability to have "o-scope" like view of analog data and "GPIO" like triggers for debugging is a dream come true.

 

[Kristoff]

The JSON parser ended up being much larger than we expected

When the OpenScope MZ boots it defaults to 'menu mode' (the interactive menu through the serial terminal).  While in this mode JSON commands are ignored, however we needed a way to programatically get from this mode into JSON mode so WFL can start talking to the hardware.  The 'Enter JSON mode' command ({"mode":"JSON"}\r\n) is a special command that isn't really part of the Digilent Instrumentation Protocol proper.  It is a special command that is not handled by the DIP parser and is used to activate 'JSON mode' (ie the DIP parser).

I'll have Keith confirm this, but if I recall correctly menu mode looks for the /r/n to process user inputs, which is why the /r/n was required on the 'Enter JSON mode' command, but not for any of the other commands.

The 'Command Format' section in the DIP specification says:

Quote

All Digilent Instrumentation Protocol commands must be a JSON object or a chunked transfer:

• JSON Object Must start with a '{' character and ends with a '}' character.
• May be followed by one or more '\r\n'.

Let us know if you have any questions about this.  I'd also love to hear more about what you're working on.

Thanks!

-Kristoff

[Ryan Minnig]

I can certainly understand that.  I did see the qualifying text "may" and was hoping that it was an early version of the documentation that hadn't been updated as development had progressed.

I can also empathize with the \r\n being used in menu mode and not in JSON mode as this would be typical using a terminal program to talk with the OpenScopeMZ.  Therefore, for others that may be interested in some work I'm doing, a possible solution to not being able to trigger on the \r\n may be to count '{' and '}' characters, and when they become equal, that is the end of the incoming JSON data.  As a quick illustration consider the data that gets passed from WaveForms Live (WFL) to the OpenScopeMZ:

{"device":[{"command":"nicGetStatus","adapter":"wlan0"}]}

By counting 2 '{' characters and 2 '}' characters, one can see that the second '}' character makes the incoming '{'  and '}' counts become equal and completes the command.  To reiterate, my specific project is only implementing a subset of the command set for internal use, but this could be expanded to cover a number of interesting projects.

 

[Kristoff]

Counting '{' and '}' is exactly how I did this in the serial library for the Digilent Agent.

The Agent needs to pass UART data from the hardware to WFL as the response to an HTTP request.  The Agent doesn't need to interpret the data but without some processing it can't tell when the data packet is complete, so to avoid waiting for a long (500 ms+) timeout and adding 500+ ms latency to every request the Agent does some basic interpretation.  It looks at the first byte and if it is a '{' it assumes the packet is a JSON object.  If so it starts incrementing a count when it sees a '{' and decremented when it sees a '}'.  When it gets to zero the JSON object is complete and the Agent assumes the entire packet is complete so it returns it to WFL.

Note: This isn't safe for general JSON.  JSON objects could contain strings which could contain '{' or  '}' characters.  To be safe for general JSON you should keep track of a 'state' to indicate if you're in a string or not.  If you are inside string you need to ignore '{' and  '}' characters.  No DIP commands contain strings with '{' or '}' but it is possible that some user defined parameters (wifi network name) could contain those characters.  TBH that scenario was added after this part of the Agent and is something we should fix.  Added a github issue here.

if(resp[0] == '{') {
            //---------- JSON ----------
            qDebug() <<"Incoming Data Looks Like JSON";
            int openBracketCount = 0;

            //Process initial data
            for(int i=0; i< resp.length(); i++) {
                if(resp[i] == '{') {
                    openBracketCount++;
                } else if(resp[i] == '}') {
                    openBracketCount--;
                }
                if(openBracketCount <= 0) {
                    qDebug() << "Serial::fastWriteRead()" << "thread: " << QThread::currentThread() << "Found the end in " << stopWatch.elapsed() << "- Response:" << resp;
                    emit fastWriteReadResponse(resp);
                    mutex.unlock();
                    return resp;
                }
            }