OpenScope and OpenLogger log file utility - export to CSV

[benl]

Here's an OpenScope and OpenLogger log file utility that can export to CSV (TODO: JSON). It's written in Python, but the important part (the log file parsing) is portable across about a dozen other languages.

https://github.com/bdlow/dlog-utils-portable

I've used a combination of the data structure definitions in the Digilent dlog-utils project, some information posted in this forum on the OpenScope data format, and the application of educated guesswork.

IMHO this is an improvement over the current version of dlog-utils, which has various log file parameters hard-coded within even though these parameters are specified within the log file header. It's also more portable (Python 3), smaller and of course handles both log types.

 

This work was made possible through the use of the awesome Kaitai Struct  project to define the log file structure and automatically generate a Python parsing library from that. Kaitai handles all the details of the log formatting including endianness and data types, and presents the log via a very easy to use Python class. The web IDE makes it very easy to poke around the binary log file. Want to use something other than Python? All you need's the `.ksy` file and you're almost there for any of the languages that Kaitai supports: C++/STL, C#, Go, Java, JavaScript, Lua, Perl, PHP, Python and Ruby.

I've a JSON exporting version almost ready, though I probably ought to add some tests first.

@Digilent: can you expand on what the channel map is meant to represent? I had presumed it was the channel number to sample index mapping, e.g. if I recorded channels 2, 3 and 7 in the OpenLogger then I'd have expected the data to have three channels in each sample; and the channel map to look like [2, 3, 7, 0, ...]. However my OpenLogger generates log files with a channel map [1, 2, 3]. Bug?

[benl]

@Digilent: it would be awesome if you could review the KSY definition, or publish an official one.

[sgrobler]

@benl thanks for your message. Python/KaiTai/JSON are things I know nothing about at the moment! I'll have a go at getting up to speed.

@Digilent - its a bit ridiculous that Digilent seem to be happy to produce a logger that creates files that can't be read without 3rd party intervention!

[benl]

@sgrobler: I've tweaked the project to hopefully make it simpler to install the dependencies (Kaitai Struct v0.9). If you have Python 3 installed, it should simply be a matter of:

 

% git clone https://github.com/bdlow/dlog-utils-portable.git
Cloning into 'dlog-utils-portable'...
remote: Enumerating objects: 26, done.
remote: Counting objects: 100% (26/26), done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 26 (delta 8), reused 20 (delta 5), pack-reused 0
Unpacking objects: 100% (26/26), done.
% cd dlog-utils-portable
dlog-utils-portable% ls
LICENSE			dlog.ksy		dlogcsv.py		requirements.txt
README.md		dlog.py			examples

  # set up and activate a Python virtual environment:
dlog-utils-portable% python3 -m venv .venv
dlog-utils-portable% . .venv/bin/activate

  # install the 0.9 runtime in the virtual environment:
(.venv) dlog-utils-portable% pip install -e 'git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git@0e3f6e0#egg=kaitaistruct'
Obtaining kaitaistruct from git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git@0e3f6e0#egg=kaitaistruct
  Cloning https://github.com/kaitai-io/kaitai_struct_python_runtime.git (to revision 0e3f6e0) to ./.venv/src/kaitaistruct
  Did not find branch or tag '0e3f6e0', assuming revision or ref.
Installing collected packages: kaitaistruct
  Running setup.py develop for kaitaistruct
Successfully installed kaitaistruct

  # run it!
(.venv) dlog-utils-portable% ./dlogcsv.py ./examples/openlogger.dlog > openlogger.csv
Header Information
log format: openlogger
stop reason: normal
number of samples: 27097
voltage units: mV
sample rate: 10E+3 Sa/s
delay: 0 s
number of channels: 3
channel map: [1, 2, 3]
(.venv) dlog-utils-portable% less openlogger.csv

 

[sgrobler]

@benl: thanks for posting, appreciate your help. It looks to me like I need to be running Linux to do this?

I downloaded python 3.7.3 for Windows but it does not respond to commands like "git clone ..." or "python3 -m venv .venv"

 

[benl]

The example I posted would work for Linux or Mac with "common" tools installed. As to Windows... can't really help much there.

git's not part of Python, it's used for managing code; you can achieve the same end result here by downloading the ZIP from https://github.com/bdlow/dlog-utils-portable/archive/master.zip and unzipping to a folder.

Virtual environment support is a standard part of Python 3; you can skip that if you like but without virtual environments eventually your Python installation will end up like this: https://xkcd.com/1987/

Ah, of course, in Windows `activate` is a batch script not a shell script: https://www.techcoil.com/blog/how-to-create-a-python-3-virtual-environment-in-windows-10/

 

[sgrobler]

Thanks for trying to help me! Unfortunately I'm just a dumb end-user of the openlogger, not a software developer, so most of your advice has gone over my head! Not to say I am not going to try it out - I do have Ubuntu on my computer, so I'll have a crack.

I'm hoping @digilent will come to the party and provide openlogger buyers with something that lets us get to our logged data without us having to learn how to handle snakes, or to learn the difference between batch and shell scripts ?...

[sgrobler]

On 6/3/2019 at 10:13 PM, sgrobler said:

I'm hoping @digilent will come to the party and provide openlogger buyers with something that lets us get to our logged data ...

Apparently not. Looks like the OpenLogger is half-developed and then abandoned. And the development team are no longer responding to to questions. Not a good look for Digilent imo.

The Crowd Supply project page has the following people which I assume worked on the project - are any of them listening in and could shed some light perhaps?

• Keith Vogel, Hardware/Firmware Engineer
• Sam Kristoff, Software Engineer
• Andrew Holzer, Software Engineer
• Kaitlyn Franz, Alyssa Holzer, James Colvin

[JColvin]

Hi @sgrobler,

The short answer is we are working on it (much like the statement has been for the last few months). The problem at hand is that engineers most familiar with the OpenLogger have retired, changed jobs, or been told from on high to focus on a different project at Digilent, leaving the project to others who are unfamiliar with the firmware and already have full days with their own Digilent tasks.

As a small bit of good news, the engineers more familiar with the firmware are finally getting some more time to help out with the OpenLogger material (rather than something else) and so have been able to help provide some feedback on getting this necessary feature working. Bad news is that change hasn't fully happened yet.

Other short answer of what needs done:

• Modify the parseFileHeader function in the dlog-utils.cpp to recognize when a file header that matches the OpenLogger style shown in the OpenLogger.h so that it supports both OpenScope MZ and OpenLogger
• Modify the convertToCsv function in the dlog-utils.cpp to also supports the OpenLogger with it's own set of timestamp, values, and units for an arbitrary amount of channels (and which channels) that are being sampled. Because of the multiple channels and the continuous logging nature, each conversion will probably need to be "chunked" out so that less powerful computers don't run into memory problems.

Hopefully this gives some insight on the state of things.

Thanks,
JColvin

P.S. - for my part on the OpenLogger project, I mostly just created and populated a number of the pages on the Digilent reference site, in case you were curious

P.P.S. -- @benl, the channel map should be working as you suspect, but clearly this is a bug that hasn't been resolved yet. I'm not familiar enough with the OpenLogger code to know where to look for the error though.

[benl]

Thanks for the update @JColvin; obviously not what we'd like to hear in so far as lack of resources behind the product but the communications is appreciated.

TBH, the dlog-utils code is... not great. The majority of the code is in type conversion and formatting (i.e. not germane to the actual processing of the data); I'm not surprised to hear it's problematic in updating it for OpenLogger as hard-coded assumptions on the data header abound (e.g. endianness; I presume the author is banking on that never changing, which may well be the case but it is in the format spec). As a reference implementation it hides the important data structure information in amongst language-specific type gymnastics. In contrast the Kaitai Struct approach removes all of that, and puts the data format front and centre, is trivially extensible (you update the struct definition and rebuild the library, done), and works "everywhere".

If it were my decision I'd dump the current dlog-utils and start again based on Kaitai Struct, the result would be:

• a proper definition of the data format (rather than users having to reverse engineer the cpp code and troll the forums)
• a couple of dozen of lines of code for the reference Digilent implementation
• and most importantly would be useful/portable in any language/environment that Kaitai Struct supports (C++/STL, C#, Go, Java, JavaScript, Lua, Perl, PHP, Python, Ruby)

As an example, what is implemented in nearly 180 LOC dlog-utils.cpp is under a dozen lines in the dlog-utils-portable Python example (`dlog = Dlog.from_file(args.inputfile)` followed by a `write_csv`), with far greater flexibility in  terms of handling future variations on data formats, and better output formatting ?

 

Given that Digilent have very limited resources for this project it's important they're used wisely, switching to Kaitai Struct is easily the best bang for buck we can ask for.

 

(BTW, it might sound like I'm a shill for Kaitai Struct - nope, I'm just a satisfied user and first discovered it when writing dlog-utils-portable... I once wrote code to process structured binary data in  the same way as dlog-utils, but I've now seen the light ?

[sgrobler]

On 7/16/2019 at 5:47 AM, JColvin said:

been told from on high to focus on a different project at Digilent, leaving the project to others who are unfamiliar with the firmware and already have full days with their own Digilent tasks.

Sounds like Digilent couldn't care less about finishing what they started. I'll certainly think twice before supporting another Digilent project - if this one is anything to go by. You get half of what was promised. OpenLogger is about a useful as tits on a bull if you can't read the data you've collected.

[sgrobler]

any more progress on this issue @JColvin?

[JColvin]

Hi @sgrobler and @benl,

I was informed today that a conversion process of converting a dlog file into csv is now tested and working for OpenLogger and OpenScope MZ and is documented here: https://reference.digilentinc.com/reference/software/waveforms-live/how-to-convert-dlog. If you have any questions on this, I will try to answer them, but may end up deferring to @AndrewHolzer for the technical side of things.

Thanks,
JColvin

[sgrobler]

Thanks for that, glad to see there is some progress coming at last!

But you don't make it easy for us novices! If there is an executable why can't I just download it, rather than have to go through this github stuff

Could you provide some instructions-for-complete-dummies to help me ?

It starts off by saying "Clone this repository" and it gives the command below which looks like a linux command to me.

git clone https://git.digilentinc.com/dsukumar/dlog-utils.git

So If I'm running windows, what do I do?

regards

Steve (github-dummy)

[vicentiu]

Executables will be posted in the next few days.

But if you'd like to build it on Windows you can do one of the following:

• Install git for windows
• Install cygwin
• If you have Win 10, you can install a Linux Subsystem within Windows
  • Ubuntu
  • Other distributions

If you don't want to use git clone, you can download the source repository as a zip file.

[AndrewHolzer]

Hi @sgrobler,

I've build a 64-bit executable for Windows, which you can download from here. Save that into a location you're likely to remember. Refer to the repository README for instructions on how to use the executable.

Regards,
AndrewHolzer

[sgrobler]

Hi Andrew

Many thanks for this, I've been looking forward to this for a long time!
However I'm still not smart enough to get it to work
I've downloaded the executable and tried to run it by first opening a windows7 command window by typing "cmd".

When that brings up a command window, I change directory to where the executable and my OpenLogger .log files are stored and type:
dlog-utils-v2.2.0.exe inputfile.log outputfile.csv

and the response I get is
"dlog-utils-v2.2.0.exe is not recognized as an internal or external command, operable program or batch file"

The readme on the repository says:
Run an example (Windows):

./examples/build/main.exe log.dlog log

[vicentiu]

You have the following options:

1. Have the executable and the log file in the same directory.
2. Use the full path to the executable
   • C:\<path to executable>\dlog-utils-v2.2.0.exe inputfile.log outputfile.csv
3. add C:\<path to executable> to the PATH variable so that windows knows where to search for that executable name

[sgrobler]

2 hours ago, vicentiu said:

You have the following options:

1. Have the executable and the log file in the same directory. 
2. Use the full path to the executable
   • C:\<path to executable>\dlog-utils-v2.2.0.exe inputfile.log outputfile.csv
3. add C:\<path to executable> to the PATH variable so that windows knows where to search for that executable name

Many thanks, for the first time I can access the data I've logged! ??

1 = did not work by itself, 2 = did not work by itself, 3 = once this is done then 1 works. (did not try 2)

The .csv files that are created look like this: (timestamp in seconds in first column, then data in next columns, in my case in units of Volts)

Log Completed Normally
0.00000, 1.74300, 1.63200, 2.03800
0.01000, 1.72400, 1.62700, 2.03000
0.02000, 1.73900, 1.63600, 2.02500

• There is no information like a timestamp to give date\time that logging started - could that be added?

[AndrewHolzer]

Hi @sgrobler,

The OpenLogger doesn't have any sort of RTC or time keeping mechanism. Because of that, the data cannot be timestamped and that functionality cannot be added.

AndrewHolzer

[AndrewHolzer]

Hi all,

Quick update here. I've got binaries built for Mac and Linux (Ubuntu) and have updated the documentation to link to them. You may also download the Windows, Mac or Linux binaries by following the links I just gave you. If you encounter any issues whatsoever, submit in issue on the GitHub page and I'll set to fixing them straight away.

Regards,
AndrewHolzer 

[sgrobler]

On 9/24/2019 at 6:18 AM, AndrewHolzer said:

Hi @sgrobler,

The OpenLogger doesn't have any sort of RTC or time keeping mechanism. Because of that, the data cannot be timestamped and that functionality cannot be added.

AndrewHolzer

Hi @AndrewHolzer,
Would it be possible for a timestamp from the host computer to be added to .log files? It would only need to be the time at which the logging started (eg in the header), ie individual scans don't need to be timestamped. For standalone logging where openlogger is not connected to any host, that would not be possible, but for many situations logging is initiated from a host of some sort. Another option is to let the filename automatically include the initial date and time.

[AndrewHolzer]

Hi @sgrobler,

I think it's entirely possible to include a timestamp in a log header when starting a session from a connected host. I'll put this on my list of additions & fixes, put some thought into a proper solution and get to work on it. I'll personally message you once I've made the changes (and you'll probably see a blog post about them as well).

Regards,
AndrewHolzer

[sgrobler]

Awesome, that wouldl be a really useful feature I believe.