EN2SE-R "Update Mode" field


#46

This is interesting and by itself seems to point to there being an issue on the bus. Does the monitoring tool perform a more raw logging like the linking device?

I was reviewing some of the old configs and spotted something that could be causing issues can you send me the latest config you are using? I see the timing for some queries set to really low values. You mentioned you were creating new configurations when you tried a single device but if you were still doing this it might be where your issue is coming from.

This was from one of your original configurations shared. It looks like this for only a few transactions.
image


#47

Deryck,

Yes the Monitor software has a “raw data” view.

It is hard to look at so I copy and pasted the raw data into Word and formatted it.

That was what was in the last snip I showed on the Right.

I’ve attached one of the more recent cfg’s.

(not the current one I’m using as I now have a Hamilton Probe (node 6) attached to see if It has any problems with the SubNet.)

Also, as a Test, I performed a Log (Continuous) and watched the SubNet Light. I made sure it stayed green for 10-15 seconds and manually stopped the Log.

As I suspected, the Log data is straight from the Buffer without regard to the buffer Pointer to where data is being stored. (no FIFO).

If you look at some of the log files you can see the beginning of a transmission at the END of the file and it dovetails right at the TOP of the file (where you see the rest of the transmission).

So the Strange “Missing” data we see in the middle of the file happens to be where we STOP logging.

If you can verify this, there should be some Documentation Update to let the users know this…. I’ve spent weeks scratching my head over this one.

HMS_SER_AopConfigurationCompact.cfg (16 KB)


#48

Hi Howard,

I think i found the issue. I’m guessing you kept reusing the same timing settings each time you recreated the configuration. You had 1 transaction, for each node, setup with a 10ms time between broadcast and a 10ms update time. I think this was causing the linking device and communicator to both constantly send that query. This was probably causing the query to be sent out faster then the device could handle potentially messing up the log. It could also have been causing issues with the slave getting requests that it could not answer.

Please try the following config with corrected timing and let me know if this clears up the issue.

HMS_updated_config_27291.cfg (16.0 KB)


#49

Hi Howard,

Were you able to test out the new configuration file?

Deryck


#50

Deryck,

I couldn’t use the file as is.

I have since added a 6th probe to the bus (Variable Cell Density – VCD). And since I didn’t have a new cable to use, I used the one from probe 1 (pH).

So I modified my config for probes 2-6.

Also to test the new probe (From Hamilton – Not Mettler Toledo like the other 5) I only have one Read query configured just to read the “Probe Type” register (sends back 8 registers – 16 bytes).

So far I did increase the Timing but not to the 100 (1 second) interval you suggested.

Since the total time for the 5 reads take about 120ms to complete back to back (with a few ms in between each probe query) I decided to use 300ms for this query. (I could try more).

Below are a few Oscope snips with the scope cursors set to measure time events…

The 5 Process Variable (PV) query’s = 117 ms

Dead time between query’s = 223ms (so 117 + 223 is a little more than 300)

This is the same PV Query = 123ms followed by the other queries (read Diags and Write configs).

This just shows a timing relationship between where the PV’s and other Read/Writes intersect.

You can see there is idle time on the bus….

Still getting a Red Flash on the Subnet Status LED every 20-60 seconds.

Of note, I did an initial config just talking to the new Hamilton Probe.

Watched the Subnet LED for a full 5 minutes without any RED flashes.

FYI regarding the Logs,

You understand my previous e-mail about the “Continuous” logging stopping midway in the buffer and displaying the buffer from 0-512 without regard to FIFO?

If you look at the bottom of the Log file you can see a transmission that is stopped (cut off) mid way…, but is finished at the top of the log file.

Thanks,

Howard


#51

Hi Howard,

I am fairly confident that the timing was playing a major part in this issue. Can you set the timing to the default 100 to get this working before you try to make it shorter? Modbus typically uses a 1 second message delimiter between transactions. Please send me logs and the configuration you are using this will help me keep up with the changes you are making.

If you are taking a continuous log i would expect it to use a FIFO order but I do not have any details on the process. Since the messages are repeating you could have captured data there the start is similar to the end.

Deryck


#52

Deryck,

Attached Config….

Using 1 second for the PV,

Using 5 seconds for the Diags,

Using 10 seconds for the Writes.

Don’t know if you need the .cfx file.

Since I’ve been changing this so Often I decided I need a Spreadsheet to keep track (and Have Final Document) of the Parameters.

(Would be nice if the “Print” function had the option to include all of these parameters for Documentation).

When you say 1 second delimiter between transactions, you mean between the same msg to the same probe?

With the above configuration you see the 1 second delay between burst of messages to the probes.

I don’t have any Logs for this configuration.

The Subnet LED Red flash rate for this configuration is reduced (as expected since the message traffic is reduced).

I’ve seen two red flashes with this config.

The first was 2 minutes after I opened the cabinet (do not know how long before I did that the Subnet LED was green).

The Subsequent Red flash occurred in another 5.5 minutes.

Thanks

Howard

HMS_SER_AopConfigurationCompact6L1.cfg (16 KB)

HMS_SER_AopConfigurationCompact6L1.cfx (35.9 KB)

HMS_Linking_Configuration_Slow.xlsx (249 KB)


#53

Hi Howard,

Thanks for the configuration This looks like it should be good. Has this decreased how often you are getting these errors? If this is only happening every 5.5 minutes it would seem the main issue is resolved. 5 minutes should also be enough time for the entire configuration to have been polled more then once, from this it seems reasonable to think this is not a transaction issue. I think we have discussed this before but could we setup a WebEX so I could help troubleshoot this in real time? Have you caught any of the red flashes in the log?

Do you know what is causing the spikes in your capture? Is the bus still Biased/polarized?

Deryck


#55

Deryck,

If I slow down how often we ask for data it only seems logical that the Error frequency would also be reduced.

This by itself is not a resolution. Using that logic then I could eliminate the errors by only asking for data once per hour which is not realistic.

And one second is probably slower than we want also.

You can tell when the entire configuration has been polled due to the Subnet LED being a Solid Green. (Starts out flashing Green until all messages have been transmitted)

I agree that the Transactions do look like they are ok.

I am unable to capture a red flash in the Log (with this configuration) due to the long time between flashes.

The Raised Reply (I think you are referring to as a “Spike”) is from a different probe from Hamilton we put on the bus to test if that probe has a problem with the HMS as a master.

It seems to work fine (data is always received from this probe) even though the Bus Voltage level looks strange.

I had removed the Biasing resistors a few days ago and they were not on when I took that scope capture.

At one time I had a configuration that only had the Hamilton Probe on the bus and queried every 50 ms. I had no problems with it. (No red flashes and I watched it for > 5 minutes).

We have reached out to the Metler Toledo Probe MFG and they are going to try to repeat our test.

I also have reached out to Hamilton regarding the Bus Voltage level shift… Waiting to hear back from them.

On another note: Yesterday morning when I came in, None of the probes were replying to the HMS Master. (Log file showed timeouts for every query).

I had to power down the probes to get them to talk again.

If you remember, I have seen that a few times before when I had captured that log where it looked like the Master was sending nonsense out on the bus.

At this point I’m thinking that happened again overnight.

There may also have been a Power outage overnight as I saw my PLC I/O alarms were all latched in (which does happen when I cycle power).

I may try a few more of these to see if I can repeat the problem.

Thanks,


#56

We might need to find a way to isolate what is causing the red flashes. Is the plc signalling any faults or do you ever see any missing info? With out know what the cause is it is going to be difficult to resolve.

This might be a tedious process but if you were to setup a configuration with just one device add an additional one until you see the problem come back or just set up a config for each device individually and see if any of them see the issue.

I suspect this would have been an issue with the probes. Unless something was causing the communicator to not send the message over the bus but i haven’t see anything like that before.


#57

I’m leaning toward the MT probes? That’s why we have reached out to them to try to replicate our setup.

Although I’m a little worried about the Probes getting hosed yesterday and requiring a Power Cycle to recover.

I had already done the Single Device on the Network and still had the red flashes (MT probes).

I’m not sure what the rate was… Probably in the 50 ms range.

I’m sure I discussed that in one of my previous e-mails.

This was done right before putting “Just” the Hamilton probe on the Bus. (For Probe Comparison).

As I indicated in the Last E-Mail, the Hamilton Only setup resulted in NO red flashes for > 5 minutes.

Thanks,


#58

Another idea is that this could be an issue with the baud rate you are using. You could try lowering the baud rate this might make the bus more forgiving.


#60

Hi Howard,

I have a suggestion that might help with this issue. I have another customer running into a similar issue but on a custom configuration. It seem to be tied to the message delimiter. Try setting your message delimiter manually as apposed to 0 which calculates the time for 3.5 characters based on the baud rate. The range can be 1-50.

Deryck


#61

Deryck,

Been waiting for Mettler Toledo (Probe MFG) to get back to be with their testing.

I tried as you suggested to change the message Delimiter from the Default of 0 (3.5 chars).

At 38400 baud, the 3.5 chars should compute to about 1 ms.

I saw a 12.48ms query to query interval when it was set to 0:

With it set to 1 (should now be 10ms?) I see a 25.92ms Query to Query interval.

(and what looks like about 4.5 full scope divisions @4ms/div = 18ms between end of reply and the next query).

Where should the 10ms be applied?

It Didn’t improve my error rate (about 1 every 4 minutes or 12 per hour).

Mettler Toledo has a two probe setup testing and has indicated they get timeouts at a much lower rate.

They don’t have an HMS device as the Master and I don’t know what other differences may exist….


#62

This should be the time after one transaction. if it was 0 is calculating the time for the 3.5 characters.

Try setting it to 2 or 3 to give it extra time.

Any change you have captured in the logs or with the scope the error occuring?

Are they saying they are seeing the same issues with another master?

Deryck


#63

I have the Message Delimiter set at 1.

This did increase the time the Query to Query interval from 12 to 25 ms. (which is the 10 ms?)

I did Capture the Timeout on the Scope….

The Top Trace shown in the next 3 snips is the Same Capture… I Moved the Zoom window to look at the last three transmission bursts.

This is a zoom in on a Normal transmission burst (10 queries and 10 replys)

This is the Timeout, 9 Queries and 8 replies….

The Timeout is set for 1 second so I guess it doesn’t do a retry but instead goes on to the Last (10th) query then moves on to repeat the 10 queries again.


#64

Hi Howard,

I have suggested something like this before but looking at the captures with a colleague we are thinking it could just be one device on the bus. Have you tried disconnecting one device at a time from the bus?

Deryck


#65

Yes I have had the config all the way down to just one Probe.

As I said before, Mettler Toledo (MT) has done some testing and has experienced timeouts as well.

Although they have only two probes connected and are only sending 1 query per second to each probe (60 queries per minute per probe).

They are reading the Process Values only so far.

I asked if they could duplicate all the queries we have in our configuration.

Attached is what we have in the HMS so far.

As you can see :

The PV and Function 7 queries are at 1 second. ( 60/1sec x 2queries = 120 queries / minute)

The 3 Diagnostic queries are at 5 seconds. ( 60/5sec x 3queries = 36 queries / minute)

The 3 Write Queries are at 10 seconds. ( 60/10sec x 3queries = 18 queries / minute)

So for 0ne probe we are sending 174 queries per minute.

Their results are at about 0.005% timeout errors per probe.

If I multiply that by 3 (to match our Query count) then by 6 probes the System Timeout rate would be 0.09%.

That equates to a timeout every 20 minutes.

My Config actual is seeing a timeout every 4-5 minutes.

I’m going to wait for their reply (Unfortunately they are in Switzerland so there is sometimes a Day skipped if I don’t send out E-Mails the right time of day.

HMS_Linking_Configuration_Slow.xlsx (257 KB)


#66

Deryck,

So the HMS device Hosed the Probes again (twice).

Attached are the log files,

And snips of the GS Event Timer……

HMS “Log” issue:

13Nov2019:

14Nov2019:

Only probe connected was a Hammilton probe (no the Mettler Toledo.

This is definitely coming from the HMS Device.

Only happens (as far as I can tell) is when I “Click” on the Log function in Logix.

Please find the source of this.

Thanks,

logbbb6.txt (33.6 KB)

logbbb5.txt (36.7 KB)

logbbb4.txt (36.7 KB)

logbbb3.txt (36.7 KB)

logbbb2.txt (36.7 KB)

logbbb1.txt (36.7 KB)

logaaa5.txt (33.7 KB)

logaaa4.txt (36.7 KB)

logaaa3.txt (36.7 KB)

logaaa2.txt (36.7 KB)

logaaa1.txt (36.7 KB)


#67

Hi Howard,

I am reaching out to my colleagues in Sweden, to see if they have any info on where this could be coming from. I do not see this being in any with this setup that would cause the linking device to send the message.

What is the model of the Hammilton probe? When you say only the probe is connected, is it still connected to the same bus/wiring? Have you tried setting up another bus with just one or a few devices?

I remember you testing with an AB7072 before do you see the same issue with that device?