eFive 25 and Cosy 131 Communication Issues

Ewon Support eFive
#1

We are now in the installation phase of a municipal project with an eFive 25 and 7 Cosy 131 VPN routers connected from around the town but we are having major communication issues over the VPN tunnels to the point that we can’t even get a full series of ping requests to come back all positive; usually its 50% or 75% loss.

All 7 tunnels are up and running from the Cosy 131s to the eFive 25.
I can access web config pages from all 7 Cosy 131s by the WAN IP.
WAN IP pings are 100% and response seems good.
1:1 NAT over VPN is configured on each Cosy 131 for at least a PLC and usually a PLC and HMI.

When connected to an engineering system on the local control network and attempting to ping the local IP addresses of the Cosy 131s, the PLCs and HMIs; ping replies always fail either 50% or 75% of the time but there is almost always 1 successful reply with what seems like more latency then there should be (800 to 1200ms typical). We are also seeing issues establishing connections from the PLC programming software (not my department so can’t provide much more info on that at the time).

I have been over this configuration numerous times and everything appears to be configured as it should be but obviously something has to be off because it just isn’t working. There isn’t a lot of information on all these parameters and the eFive setup is pretty screwy with the backdoor FTP configuration method that has to be done so I feel like I have to just be missing something in the parameters but I have spent hours tonight going over things and nothing is improving the situation.

Are their any specific parameters I can check? Why can’t I even get 100% successful pings through a single tunnel?

One other note is this town has their own municipal cable system and it isn’t very good - our local speed tests are maxing out at 3MB down and 1MB up. Does this seem like it would be an issue?

#2

Can you try setting your Flexy and eFive to try and use TCP instead of the default UDP and see if that fixes the issue?

#4

I changed the eFive over to TCP and then changed the first Cosy and could ping the Cosy and PLC with 100% success. I change the second Cosy and again ping it and the PLC behind it with 100% success. Change the third and 100% failure - everything times out. Go back to the first and second and we are right back to 50% to 100% failure of pings.

Also, through a VNC connection to the engineering computer on site; loading Cosy web config pages are horrendously slow essentially displaying the initial spinning “loading…” icon forever. If I hit the WAN IP of the Cosy; the web config loads in about 2 seconds. Everything over the VPN seems incredible slow. The tunnels are having no problems at all staying up but all communication over the VPN seems really bad.

#5

Can you try unchecking this and see if it responds any better?

image
image1908×494 27 KB

#6

That option was checked; I unchecked it restarted the VPN and didn’t notice any change in response success.

The rest of the day was taken due to a lost a Cosy that now refuses to connect to the eFive VPN at all after troubleshooting this issue and I really don’t want to drive 4 hours to do a reset of the device.

I’m going to do a little more testing over the weekend and pull some data together but I’m really hoping to talk to someone on Monday to try and get something going. We have an install with the last 2 Cosy units coming up and things need to be working because they are the most important installs out of the group.

#7

Hello @NControls,

I will be working with you this week on this issue while Tim is out of the office. Looking over this issue i am wondering if traffic is simply not getting routed correctly. Do you have a network diagram showing how the e5 and the cosys are configured along with the NAT 1:1 table?

Where do you see the traffic going if you do a tracert to the WAN IP vs if you go to the LAN? Could you upload a backup of one of the eWons for me to take a look at?

Deryck

#8

I do have a connection diagram with all the relevant information - just updated it this morning. I also have a backup of a 131. I don’t really want to publicly post these though so let me know the best way to exchange these.

Trace Routes are all over the place. We have 5 of the 7 Cosy 131s online; the first Traceroute is the eFive and the remaining 5 are the Cosy 131s. I tried the WAN IPs from our shop and then tried the WAN and LAN from an Engineering system on site. That system does have dual NICs - one on the control network (10.X) and one on the office network (192.X). I’m not sure if that is part of the problem or not. My IPs are masked at the end.

**Traceroutes from our shop to the WAN IP addresses of the eFive/Cosy 131s.**

Tracing route to 74.117.0.XXX over a maximum of 30 hops
  1    <1 ms    <1 ms    <1 ms  192.168.10.1
  2     1 ms     1 ms     1 ms  10.1.10.1
  3     9 ms     9 ms     9 ms  96.120.9.205
  4     9 ms     8 ms     8 ms  96.110.148.65
  5     9 ms     9 ms     8 ms  162.151.220.110
  6    14 ms    16 ms    14 ms  be-45-ar01.lancaster.pa.pitt.comcast.net [96.108.4.89]
  7    40 ms    23 ms    23 ms  be-34-ar01.mckeesport.pa.pitt.comcast.net [69.139.168.141]
  8    29 ms    29 ms    29 ms  be-7016-cr02.ashburn.va.ibone.comcast.net [68.86.91.25]
  9    37 ms    29 ms    28 ms  be-10130-pe04.ashburn.va.ibone.comcast.net [68.86.82.214]
 10    28 ms    29 ms    28 ms  50.248.118.174
 11    29 ms    30 ms    29 ms  be2676.ccr42.dca01.atlas.cogentco.com [154.54.47.165]
 12    34 ms    32 ms    32 ms  be2807.ccr42.jfk02.atlas.cogentco.com [154.54.40.109]
 13    33 ms    32 ms    32 ms  be3295.ccr31.jfk05.atlas.cogentco.com [154.54.80.2]
 14    43 ms    32 ms    32 ms  gordian-group.demarc.cogentco.com [38.104.74.86]
 15    53 ms    43 ms    45 ms  er0-pitbpa.zitomedia.net [74.81.98.226]
 16    50 ms    49 ms    49 ms  74.81.109.2
 17     *        *        *     Request timed out.
 18     *        *        *     Request timed out.
 19     *        *        *     Request timed out.
 20     *        *        *     Request timed out.
 21     *        *        *     Request timed out.
 22     *        *        *     Request timed out.
 23     *        *        *     Request timed out.
 24     *        *        *     Request timed out.
 25     *        *        *     Request timed out.
 26     *        *        *     Request timed out.
 27     *        *        *     Request timed out.
 28     *        *        *     Request timed out.
 29     *        *        *     Request timed out.
 30     *        *        *     Request timed out.


Tracing route to 74.117.0.XXX over a maximum of 30 hops
  1    <1 ms    <1 ms    <1 ms  192.168.10.1
  2     1 ms     1 ms     1 ms  10.1.10.1
  3     8 ms    25 ms    19 ms  96.120.9.205
  4     9 ms     8 ms     8 ms  96.110.148.65
  5     9 ms     8 ms     8 ms  162.151.220.110
  6    15 ms    14 ms    14 ms  be-45-ar01.lancaster.pa.pitt.comcast.net [96.108.4.89]
  7    23 ms    22 ms    22 ms  be-34-ar01.mckeesport.pa.pitt.comcast.net [69.139.168.141]
  8    30 ms    31 ms    29 ms  be-7016-cr02.ashburn.va.ibone.comcast.net [68.86.91.25]
  9    29 ms    29 ms    51 ms  be-10130-pe04.ashburn.va.ibone.comcast.net [68.86.82.214]
 10    37 ms    27 ms    29 ms  50.248.118.174
 11    29 ms    29 ms    30 ms  be2676.ccr42.dca01.atlas.cogentco.com [154.54.47.165]
 12    33 ms    32 ms    31 ms  be2807.ccr42.jfk02.atlas.cogentco.com [154.54.40.109]
 13    44 ms    32 ms    49 ms  be3295.ccr31.jfk05.atlas.cogentco.com [154.54.80.2]
 14    31 ms    32 ms    32 ms  gordian-group.demarc.cogentco.com [38.104.74.86]
 15    45 ms    45 ms    45 ms  er0-pitbpa.zitomedia.net [74.81.98.226]
 16    49 ms    57 ms    49 ms  74.81.109.2
 17    59 ms    58 ms    57 ms  74.117.0.XXX


Tracing route to 74.117.0.XXX over a maximum of 30 hops
  1    <1 ms    <1 ms    <1 ms  192.168.10.1
  2     1 ms     1 ms     1 ms  10.1.10.1
  3    10 ms     9 ms     8 ms  96.120.9.205
  4     8 ms     9 ms     8 ms  96.110.148.65
  5    18 ms    18 ms    17 ms  162.151.220.110
  6    14 ms    25 ms    16 ms  be-45-ar01.lancaster.pa.pitt.comcast.net [96.108.4.89]
  7    23 ms    22 ms    25 ms  be-34-ar01.mckeesport.pa.pitt.comcast.net [69.139.168.141]
  8    32 ms    30 ms    29 ms  be-7016-cr02.ashburn.va.ibone.comcast.net [68.86.91.25]
  9    29 ms    43 ms    28 ms  be-10130-pe04.ashburn.va.ibone.comcast.net [68.86.82.214]
 10    38 ms    28 ms    29 ms  50.248.118.174
 11    30 ms    30 ms    30 ms  be2676.ccr42.dca01.atlas.cogentco.com [154.54.47.165]
 12    33 ms    33 ms    33 ms  be2807.ccr42.jfk02.atlas.cogentco.com [154.54.40.109]
 13    34 ms    37 ms    32 ms  be3295.ccr31.jfk05.atlas.cogentco.com [154.54.80.2]
 14    31 ms    31 ms    32 ms  gordian-group.demarc.cogentco.com [38.104.74.86]
 15    45 ms    49 ms    45 ms  er0-pitbpa.zitomedia.net [74.81.98.226]
 16    50 ms    50 ms    50 ms  74.81.109.2
 17     *        *        *     Request timed out.
 18     *        *        *     Request timed out.
 19     *        *        *     Request timed out.
 20     *        *        *     Request timed out.
 21     *        *        *     Request timed out.
 22     *        *        *     Request timed out.
 23     *        *        *     Request timed out.
 24     *        *        *     Request timed out.
 25     *        *        *     Request timed out.
 26     *        *        *     Request timed out.
 27     *        *        *     Request timed out.
 28     *        *        *     Request timed out.
 29     *        *        *     Request timed out.
 30     *        *        *     Request timed out.


Tracing route to 74.117.0.XXX over a maximum of 30 hops
  1    <1 ms    <1 ms    <1 ms  192.168.10.1
  2     2 ms     1 ms     1 ms  10.1.10.1
  3    18 ms    17 ms     8 ms  96.120.9.205
  4    16 ms     9 ms     8 ms  96.110.148.65
  5     9 ms     8 ms     8 ms  162.151.220.110
  6    15 ms    14 ms    15 ms  be-45-ar01.lancaster.pa.pitt.comcast.net [96.108.4.89]
  7    23 ms    23 ms    22 ms  be-34-ar01.mckeesport.pa.pitt.comcast.net [69.139.168.141]
  8    29 ms    29 ms    29 ms  be-7016-cr02.ashburn.va.ibone.comcast.net [68.86.91.25]
  9    28 ms    27 ms    28 ms  be-10130-pe04.ashburn.va.ibone.comcast.net [68.86.82.214]
 10    28 ms    28 ms    28 ms  50.248.118.174
 11    29 ms    29 ms    29 ms  be2658.ccr41.dca01.atlas.cogentco.com [154.54.47.137]
 12    33 ms    32 ms    32 ms  be2806.ccr41.jfk02.atlas.cogentco.com [154.54.40.105]
 13    32 ms    32 ms    32 ms  be3294.ccr31.jfk05.atlas.cogentco.com [154.54.47.218]
 14    32 ms    32 ms    32 ms  gordian-group.demarc.cogentco.com [38.104.74.86]
 15    51 ms    44 ms    53 ms  er0-pitbpa.zitomedia.net [74.81.98.226]
 16    51 ms    48 ms    49 ms  74.81.109.2
 17    60 ms    56 ms    56 ms  74.117.0.XXX


Tracing route to 74.117.0.XXX over a maximum of 30 hops
  1    <1 ms    <1 ms    <1 ms  192.168.10.1
  2     3 ms     1 ms     1 ms  10.1.10.1
  3     9 ms     8 ms     8 ms  96.120.9.205
  4     9 ms     8 ms     9 ms  96.110.148.65
  5     9 ms    16 ms     8 ms  162.151.220.110
  6    39 ms    24 ms    14 ms  be-45-ar01.lancaster.pa.pitt.comcast.net [96.108.4.89]
  7    24 ms    22 ms    23 ms  be-34-ar01.mckeesport.pa.pitt.comcast.net [69.139.168.141]
  8    30 ms    29 ms    31 ms  be-7016-cr02.ashburn.va.ibone.comcast.net [68.86.91.25]
  9    29 ms    36 ms    28 ms  be-10130-pe04.ashburn.va.ibone.comcast.net [68.86.82.214]
 10    30 ms    29 ms    29 ms  50.248.118.174
 11    37 ms    42 ms    29 ms  be2676.ccr42.dca01.atlas.cogentco.com [154.54.47.165]
 12    42 ms    42 ms    32 ms  be2807.ccr42.jfk02.atlas.cogentco.com [154.54.40.109]
 13    32 ms    32 ms    32 ms  be3295.ccr31.jfk05.atlas.cogentco.com [154.54.80.2]
 14    39 ms    32 ms    34 ms  gordian-group.demarc.cogentco.com [38.104.74.86]
 15    44 ms    44 ms    44 ms  er0-pitbpa.zitomedia.net [74.81.98.226]
 16    49 ms    48 ms    48 ms  74.81.109.2
 17    63 ms    56 ms    56 ms  74.117.0.XXX


Tracing route to 74.117.0.XXX over a maximum of 30 hops
  1    <1 ms    <1 ms    <1 ms  192.168.10.1
  2     1 ms     1 ms     1 ms  10.1.10.1
  3     9 ms     8 ms    10 ms  96.120.9.205
  4     8 ms    10 ms    16 ms  96.110.148.65
  5     9 ms    33 ms    11 ms  162.151.220.110
  6    15 ms    14 ms    24 ms  be-45-ar01.lancaster.pa.pitt.comcast.net [96.108.4.89]
  7    22 ms    22 ms    22 ms  be-34-ar01.mckeesport.pa.pitt.comcast.net [69.139.168.141]
  8    29 ms    29 ms    31 ms  be-7016-cr02.ashburn.va.ibone.comcast.net [68.86.91.25]
  9    28 ms    27 ms    28 ms  be-10130-pe04.ashburn.va.ibone.comcast.net [68.86.82.214]
 10    28 ms    28 ms    28 ms  50.248.118.174
 11    29 ms    36 ms    29 ms  be2658.ccr41.dca01.atlas.cogentco.com [154.54.47.137]
 12    33 ms    32 ms    32 ms  be2806.ccr41.jfk02.atlas.cogentco.com [154.54.40.105]
 13    32 ms    33 ms    32 ms  be3294.ccr31.jfk05.atlas.cogentco.com [154.54.47.218]
 14    31 ms    32 ms    33 ms  gordian-group.demarc.cogentco.com [38.104.74.86]
 15    45 ms    44 ms    44 ms  er0-pitbpa.zitomedia.net [74.81.98.226]
 16    50 ms    58 ms    52 ms  74.81.109.2
 17    74 ms    58 ms    59 ms  74.117.0.XXX


**Trace Routes from the Engineering system to the WAN IP addresses of the eFive/Cosy 131s.**

Tracing route to 74.117.0.XXX over a maximum of 5 hops
  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2    20 ms     8 ms    12 ms  10.204.8.1
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.

Tracing route to 74.117.0.XXX over a maximum of 5 hops
  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2     8 ms     8 ms     8 ms  10.204.8.1
  3    16 ms    17 ms    16 ms  74.117.0.XXX

Tracing route to 74.117.0.XXX over a maximum of 5 hops
  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2     7 ms     8 ms    11 ms  10.204.8.1
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.

Tracing route to 74.117.0.XXX over a maximum of 5 hops
  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2     7 ms     9 ms     8 ms  10.204.8.1
  3    16 ms    20 ms    17 ms  74.117.0.XXX

Tracing route to 74.117.0.XXX over a maximum of 5 hops
  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2     8 ms     7 ms     8 ms  10.204.8.1
  3    19 ms    21 ms    16 ms  74.117.0.XXX

Tracing route to 74.117.0.XXX over a maximum of 5 hops
  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2     8 ms     9 ms     8 ms  10.204.8.1
  3    18 ms    17 ms    24 ms  74.117.0.XXX


**Trace Routes from the Engineering system to the LAN IP addresses of the eFive/Cosy 131s.**

Tracing route to 10.0.0.1 over a maximum of 5 hops
  1    <1 ms    <1 ms    <1 ms  10.0.0.1

Tracing route to 10.0.0.20 over a maximum of 5 hops
  1     *     BASA-ENG [10.0.0.36]  reports: Destination host unreachable.

Tracing route to 10.0.0.27 over a maximum of 5 hops
  1     *     BASA-ENG [10.0.0.36]  reports: Destination host unreachable.

Tracing route to 10.0.0.29 over a maximum of 5 hops
  1  1930 ms  2482 ms  2042 ms  10.0.0.29

Tracing route to 10.0.0.31 over a maximum of 5 hops
  1  1352 ms  1524 ms  1502 ms  10.0.0.31

Tracing route to 10.0.0.33 over a maximum of 5 hops

  1     *     BASA-ENG [10.0.0.36]  reports: Destination host unreachable.
#9

Hello @NControls,

Thank you for the diagram and the backup.

Looking over the traces and configuration with my colleague in Belgium we would like to take a close look at the eFive’s setup. Would you be able to enable the remote support feature on the eFive or could we schedule a time for me and my colleague in Belgium to view it via teamviewer?

Deryck

#10

Remote Support has been enabled on the eFive. I can do Teamviewer at any time if needed here from our shop. I have VNC access to the Engineering system onsite that can put us on the local network for more testing.

Working on this today is my sole focus so I am available all day for troubleshooting and I can be reached at 814-954-7464.

#11
#12

Hello @NControls,

Thanks we will take a look via remote support.

Deryck

#13
#14

Thanks for your help on this Deryck (and Tim). We will be getting into PLC communications tests later today on these 5 Cosy 131s but right now everything seems to be working very good.

As I mentioned through PM we did confirm the source of the broadcast messaging on the network is a VFD on Ethernet/IP and it’s on our list to look into but it’s not exactly our equipment so we’ll see how that pans out. That being said I was hoping you could explain a little more on what you did to block the traffic for future reference. I’m a little confused how you installed another firewall on the device; and wondering why the built in firewall in the device wasn’t used? I’m curious I guess because if this needs to be reversed or re-configured in the future due to a reset of the device or something I would like to have a little more documentation for the situation.

#15

Hello @NControls,

Glad to hear everything is still working fine.

What was setup was ebtables it is used to filter traffic. It is setup to block all multicast and broadcast messages besides ARP. More info on it can be found here.

I did not do the work my self my colleague in Belgium did the setup, but let me know if you have any questions.

Deryck

#17

So last week we brought up our next to last Cosy 131 (10.0.0.17) and everything has been working smoothly across the VPN since last Tuesday. Today we brought up the last Cosy 131 (10.0.0.14) and within a short while of having it online everything went downhill and suddenly tracerts and pings are all very delayed or just timeout again.

It seems too weird to just be a coincidence of timing but really within a short windows of about an hour everything was fine and all responses were normal and then all of the sudden the traffic increase must have started because VPNs started dropping from the eFive sporadically and now it’s like were back to being flooded with broadcast data again or something. I know nothing has been corrected with the VFD that is generating the traffic but that is beyond our control at the moment. I’m assuming that the traffic filter you guys installed is still working but I don’t know how I would validate that.

This isn’t really making any sense to me right now but taking the last one offline doesn’t seem to affect anything so I don’t think it was something with that but the coincidence is just too weird. There isn’t anything else on the network there except the PLC but I really don’t know what to look at here and I don’t understand what exactly was done with it before but it looks like the same exact thing.

#18

Hello,

How many devices are currently connected to the eFive? Also can I go back on and take a look at the ebtables file at some point today and see if I can verify if the settings are still the same?

Thanks,
-Tim

#19

Oddly enough when I got in this morning and did some more testing everything seemed to be working normal. I don’t know what happened from yesterday at 4:00 to this morning but nothing else was changed. Checking again now; everything is still performing as intended as far as I can tell.

As far as devices on the eFive there are 7 active VPN connections and around 20 local connections.

Remote Support is enabled on the eFive again and you can connect if you need to.

#20

Ok if you run into more issues again, let me know and I’ll take a closer look

#21

So we appear to have lost communication with one of our PLCs behind a Cosy 131. Everything else appears to be okay with the Cosy looking at it remotely but upon looking at the config I see that the “NatItf” parameter is set to 0 - disabling NAT and Transparent Forwarding and no matter what I do I can’t get that parameter to change. I have tried removing all of the all of the NAT settings in the GUI, disabling and enabling the NAT functionality and manually changing the NAT parameters but I can not get NatItf to hold any value other than 0.

What would be preventing this parameter from being changed?

#22

I have seen instances when RTEnIpFwrd has been inadvertently changed to 0, which may cause issues like this. Can you confirm that it is set to 1?