Missing data in historical log

Ewon Support Data Mailbox API
#1

Hi we are testing a Flexy 205 to record 9 tags of data at 1 second interval. We noticed some odd behavior in the historical records in the DataMailbox. Every few minutes there is 1 second of missing data and some other odd behavior.

We can currently access ~9 hours of data on the Flexy’s internal memory. The internal memory has no missing data. When accessing data through Data Mailbox we are missing ~1000 seconds of data over that same time period, so 15 minutes of data over 9 hours. Of that missing data 700 seconds is in one missing block, 300 seconds are intermittent missing rows of data and another 70 are partially recorded rows.

When comparing against the log from the internal memory there are no differences in the recorded data other than round-off error.

How can we troubleshoot this? 1 second logs are critical for our current use case.

#2

Hi @K_Hutch,

We would have to take a closer look at this. Can you create a case at support.hms-networks.com and attach a backup of the Ewon with Support Files? You can download this from the Diagnostic > Files Transfer section, upper right-hand corner.

Thank you.

#3

Sure thing. I’ll create a case now.

Thanks

#4

Thank you! Please feel free to share any solutions found here for future readers.

#5

Sure, I’ll try to remember once we reach a resolution.

So far, we have not seen another large block of missing time. It’s possible that was the result of the test bench being powered off for a short time without us knowing.

The partial records of data are likely due to post-processing error, but I’m not 100% certain.

We are still randomly losing 1s of data for all tags every few minutes. Something on the order of 10-12 minutes of total data per day. The only proposed solution so far is to clear out our data mailbox, we’ve reduced that to roughly 1 day or approximately 1M datapoints. So far, that has not changed this intermittent missing data problem.

#6

Ok, we reached a conclusion.

It turns out that our drive creating the data is missing some seconds of data. When reviewing the device logs, it’s clear that the missing seconds were never recorded on the Flexy and it is not an issue with the transfer of information to the Data Mailbox.

The issues presented themselves in part because downloading data from the m2Web account fills in missing seconds of data with the last known datapoints. This is a totally valid approach but caused some confusion on our end when attempting to verify/validate the data.

1 Like
#7

Hi @K_Hutch,

Sorry, I was out of the office last week. I appreciate the update!

#8

Yes, we’ve since done some further testing while logging simultaneously with ABB’s software. There are missing entries in both logs but they don’t align with one another. The only thing we can think is some round off error with milliseconds is causing this. We’ll just have to live with it for now I’m afraid, but it would be nice to know the underlying cause for the missing data

#9

Have you checked the poll rate on the Flexy? Maybe it is slower than the DM update rate?

Are the clocks synchronized between the Flexy and the ABB?

#10

Originally, poll rate was 2x slower than DM rate and we noticed duplicate values. It’s now set to 1000ms which is more or less the same as the 1s DM rate although this is where I’m thinking millisecond round off error could catch up with us. Is there a best practice for the poll rate/DM rate ratio? I saw that data quality checks the last 3 records to determine data quality. Would that mean that we should always set poll rate to 1/3 DM rate? Or am I conflating two separate topics?

The clocks are relatively close. We’ve done some testing with logs from both the drive and the flexy and it appears that the drive and flexy are synchronized within a fraction of a second of one another. The pattern of missing data is also different, so it’s not like an offset between missing data. The flexy misses 1 second of data every 2-3 minutes consistently. The ABB software may go 10-15minutes without missing anything and then miss two seconds in less than a 30s period. The ABB log looks something like the following:
12:00:01
12:00:02
12:00:04
12:00:05
12:00:06
12:00:07
12:00:09
Where seconds 3 and 8 are missing but then is a several minute long gap until the next missing data. They appear to be different reasons for missing data. The concern is that I’m not sure we can write off the missing data in the flexy log as though it never existed on the drive as we originally thought. I think your question about poll rate could have some merit.

#12

Hi Kyle,

I’ll ask the developers about this.

Have you tried a poll rate lower than 1000ms? Like 500ms maybe?

I’m wondering if this is happening because it takes some time for the Modbus response to arrive. If the poll rate is 1000ms and it takes 50ms (for example) for the response, after 20 responses it equals an extra second (50ms x 20ms = 1000ms). This should however show up in the local data and the M2Web data.

#13

Yes, we did try 500ms on one of our devices but it did not correct this issue, we still have drop out every 2 minutes or so.

We do see the same drop outs in the m2web data as the onboard data. The difference comes when we compare to logs from the ABB drive.

#14

That’s strange. I would expect the drop when using a 1000ms poll rate, due to the short delay in time for the node to respond. But if you are polling twice a second, I wouldn’t expect that to occur.

All I can really suggest is trying a faster rate, like 250ms. Otherwise, I would recommend responding to the support case you opened to reopen it so we can escalate the question to the Ewon developers.

#15

Ok, thanks. Sorry, I thought I was reopening the ticket, but I see that is separate.

We’ll give 250ms a try and if not, I’ll reopen the ticket. Thanks again.

#16

You’re welcome.