Parsing message #12

Murmele · 2023-01-07T18:39:33Z

parsing messages from vector
fixing problem that bigendian is when the flag is zero not one

… dublicate

… parse the value which might be wrong.

…has to be changed

…is for CANMessages

LinuxDevon · 2023-01-19T22:32:32Z

Just getting back from vacation. I see you marked as draft. Are you ready or have other changes coming down the pipeline?

Murmele · 2023-01-20T07:45:25Z

Hi, I found a big issue in the parsing, this was completely wrong and worked only in specific scenarios. Currently in my hack branch I used a quite performant approach, but is not ready yet.
Not sure what is the best way to get the signals out

Murmele · 2023-01-20T07:46:41Z

bool Message::parseSignals(const uint8_t* data, int size, std::vector<double>& values) const {
        if (!m_prepared)
            return false;

        if (size > 8)
            return false; // not supported yet

        // Only little endian supported yet!
        // All signals must be little endian
        for (const auto& signal: m_signals) {
            if (signal.is_bigendian)
                return false;
        }

        const uint32_t len = size * 8;

        uint64_t d = 0;
        for (int i=0; i < size; i++) {
            d |= ((uint64_t)data[i]) << i * 8;
        }

        uint64_t v = 0;
        for (const auto& signal: m_signals) {
            const uint32_t shiftLeft = (len - (signal.size + signal.start_bit));
            v = d << shiftLeft;
            v = v >> (shiftLeft + signal.start_bit);
            values.push_back(v * signal.factor + signal.offset);
        }
        return true;
    }

This can be extended for bigendian too. But then it does not support longer once.

LinuxDevon

Good catch on the endianess! Not sure how i missed that or flipped it... sorry about the headache with that one. Thank you for the work on this one! Here are a few more comments on it.

Not sure completely sure what you meant about this comment since I didn't see it in the PR.

// Only little endian supported yet!
        // All signals must be little endian
        for (const auto& signal: m_signals) {
            if (signal.is_bigendian)
                return false;
        }

Not sure what is the best way to get the signals out

I think you did a good job with this interface to get them out.

src/dbc.cpp

test/CMakeLists.txt

test/test_dbc.cpp

LinuxDevon · 2023-01-22T22:38:02Z

third_party/bitstream/Readme.md

@@ -0,0 +1,2 @@
+Bitstream reader.


Do we need this extra library? I was hoping to keep the dependencies for this library down to zero. I guess so long as we can keep it single header which I think we can. If we are using this we might want to pull it in with cmake instead.

This is the library from python CAN tools. This are two files, a .c and a .h. I created the cmake file for it to be able to import it to dbc_parser

LinuxDevon · 2023-01-22T22:42:58Z

src/dbc.cpp

-				double min = std::stod(match.str(11));
-				double max = std::stod(match.str(13));
+                double factor;
+                fast_float::from_chars(match.str(7).data(), match.str(7).data() + match.str(7).size(), factor);


I think i commented on the bitstream but do we need this library as well? If this gets you going I am good with it. I might come back and rework it out if the single header fails with the new libraries.

the problem is you have to convert the values to a double, but std::stod considers localization. I did not find another way, and this library is way faster than the std lib, and is only a single header

Ah gotcha. I didn't realize that #8 mattered with this. Learned something new on that.

src/dbc.cpp

include/libdbc/message.hpp

.github/workflows/tests.yml

LinuxDevon · 2023-01-22T22:51:09Z

include/libdbc/message.hpp

+        /*!
+         * \brief prepareMessage
+         * Preparing message to be able to parse signals afterwards. This speeds up parsing
+         */
+        void prepareMessage();


How does sorting them speed up parsing?

in the current approach not. I tried to reimplement the parser like in the python CAN tools, but I was not yet finished. I think the approach I currently have for <8Byte length and little endian is really fast, only 2 shifts + 3 operations (+-).

Murmele · 2023-01-25T16:52:28Z

Good catch on the endianess! Not sure how i missed that or flipped it... sorry about the headache with that one. Thank you for the work on this one! Here are a few more comments on it.

Not sure completely sure what you meant about this comment since I didn't see it in the PR.
// Only little endian supported yet!
        // All signals must be little endian
        for (const auto& signal: m_signals) {
            if (signal.is_bigendian)
                return false;
        }
Not sure what is the best way to get the signals out

I think you did a good job with this interface to get them out.

Sorry I forgott to push. Now

Conflicts: test/test_dbc.cpp

Murmele · 2023-01-28T19:38:28Z

@LinuxDevon fixed the conflict and the test. I have to check when I find the time to implement also bigendian....

Murmele · 2023-02-08T17:25:19Z

@LinuxDevon I cleaned the branch up and added support for big endian as well

Reason: otherwise it might search through the complete string to find a match Files: dbc.cpp

LinuxDevon

Sorry for the delay on this. Looks good! thank you for the work on this one! I'll try and get the format set and cleaned up here soon as we discussed in the other PR.

Murmele · 2023-02-09T22:56:00Z

Hi @LinuxDevon thanks for merging. I am trying now to add ability to parse VAL_ symbols, but this is anymore that easy

LinuxDevon · 2023-02-09T23:25:31Z

@Murmele no worries. I haven't either and was my first attempt at one. Learned as I went. Let me know if you need any help or anything. I should hopefully have time in the up coming weeks to spend some time on this project

Murmele marked this pull request as ready for review January 7, 2023 18:39

Murmele marked this pull request as draft January 7, 2023 18:39

Murmele force-pushed the parsingMessage branch 2 times, most recently from dbcd3cb to 13bec82 Compare January 11, 2023 09:20

Murmele added 9 commits January 12, 2023 17:13

Update dbc.cpp and test_dbc.cpp

8895220

implement < operator for signal to be able to sort them

55ff2d2

initial commit to parse Messages

079a46b

Update message.hpp, dbc.cpp, and 2 more files...

339d427

clear messages, otherwise when reading a second time the messages are…

ed42851

… dublicate

add comment and fix type to remove warning

f031db6

use fast_float instead of std::stod, because std::stod uses locale to…

6d26e81

… parse the value which might be wrong.

due to the error in 19caeec the assumption in the test was wrong. It …

ea8404c

…has to be changed

implement parsing also std::array, because the blf library outputs th…

e09a41c

…is for CANMessages

Murmele force-pushed the parsingMessage branch from 09c887b to e09a41c Compare January 12, 2023 16:13

Merge branch 'master' into parsingMessage

baca5f7

Murmele marked this pull request as ready for review January 12, 2023 16:18

install fast_float

39f8f90

Murmele force-pushed the parsingMessage branch from 2f31a22 to 39f8f90 Compare January 12, 2023 17:37

FastFloat target must be linked so that dbc finds the headers

80f60f4

Murmele force-pushed the parsingMessage branch from d64f851 to 80f60f4 Compare January 12, 2023 17:41

download FastFloat if it was not found

a7b9a75

Murmele force-pushed the parsingMessage branch from 49d0a8d to a7b9a75 Compare January 16, 2023 18:27

Murmele added 4 commits January 18, 2023 08:04

make compatible with 3.16, because labplot uses 3.16

26bb483

include current source dir

eb12351

rename test

e2ec1de

execute docker everytime

25f17ce

Murmele marked this pull request as draft January 19, 2023 12:34

hack

3af2c0c

add tests

d00599c

LinuxDevon reviewed Jan 22, 2023

View reviewed changes

only little endian support right now

2fd90e0

Murmele added 4 commits January 25, 2023 18:26

do not pass a flag to the Contructor, but create a dedicated function

deba3da

comment not needed stuff out for now

c883074

fix tests

876bab9

add status flag to show which error occured

ca34f24

Murmele marked this pull request as ready for review January 28, 2023 19:18

Murmele added 3 commits January 28, 2023 20:22

Merge remote-tracking branch 'refs/remotes/upstream/master'

f7f5741

Conflicts: test/test_dbc.cpp

move parsing tests to test_parseMessage.cpp

4dc7dfe

sort signals otherwise they do not match with the test results

e9e307e

Murmele added 3 commits February 8, 2023 14:48

add support for big endian

c4d55af

sorting not needed anymore

f65b85b

remove comment

70d8510

Murmele added 2 commits February 8, 2023 18:28

bitstream not needed anymore

117deb6

Description: add missing from start, so it only searches from there

b7d7585

Reason: otherwise it might search through the complete string to find a match Files: dbc.cpp

LinuxDevon approved these changes Feb 9, 2023

View reviewed changes

LinuxDevon merged commit 676da12 into LinuxDevon:master Feb 9, 2023

This was referenced Feb 9, 2023

Decoding messages #3

Closed

std::stod considers locale #8

Closed

Parsing message #12

Parsing message #12

Uh oh!

Conversation

Murmele commented Jan 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LinuxDevon commented Jan 19, 2023

Uh oh!

Murmele commented Jan 20, 2023

Uh oh!

Murmele commented Jan 20, 2023

Uh oh!

LinuxDevon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LinuxDevon Jan 22, 2023

Choose a reason for hiding this comment

Uh oh!

Murmele Jan 25, 2023

Choose a reason for hiding this comment

Uh oh!

LinuxDevon Jan 22, 2023

Choose a reason for hiding this comment

Uh oh!

Murmele Jan 25, 2023

Choose a reason for hiding this comment

Uh oh!

LinuxDevon Jan 28, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LinuxDevon Jan 22, 2023

Choose a reason for hiding this comment

Uh oh!

Murmele Jan 25, 2023

Choose a reason for hiding this comment

Uh oh!

Murmele commented Jan 25, 2023

Uh oh!

Murmele commented Jan 28, 2023

Uh oh!

Murmele commented Feb 8, 2023

Uh oh!

LinuxDevon left a comment

Choose a reason for hiding this comment

Uh oh!

Murmele commented Feb 9, 2023

Uh oh!

LinuxDevon commented Feb 9, 2023

Uh oh!

Uh oh!

Murmele commented Jan 7, 2023 •

edited

Loading