EC6133H connection with 14.1s0 firmware

Having some weird issues with an eWon EC6133H. Most of our eWons are running on firmware 14.0s02 (and are doing fine), but a couple are on 14.1s0, and after seemingly running fine for any where from a couple days to a week, all of them have eventually gone offline (from the perspective of eCatcher, for multiple days straight). We have confirmation that the end client has the machine powered on and is using it, but the eWon is not even showing up as Online in eCatcher. We only seem to be noticing this issue with eWons on firmware 14.1s0.

Power cycling the machine seems to have brought it back online for the time being, and I’ve gone online with it and grabbed the “Support Files” from the web interface. There are a lot of openVPN related issues that I can initially see. For your information as well, the machine last went offline (and stayed offline until power cycle) on 8/14/2020 11:41:11AM.

Can you assist with troubleshooting why this eWon went offline in the first place, and how to better prevent it in the future?

Thanks.
Backup.etar.gz (24.8 KB)

As a follow up as well, now that we (for the time being) have the device online, it will still go offline, and then immediately back online again. The realtime logs on the device (screenshot attached) seem to indicate some kind of inactivity threshold has been reached? Can you elaborate as to what is happening?

In the below screenshot, eCatcher indicated the device went offline at 14:41:25 and then back online at 14:41:26.

Can you take a backup with eBuddy and check the box for “Include Support Files”? I have the file you sent, but it’s hard to get the information we need in that format. We need to .tar file. Thanks!

Attached.

eWON.tar (190.5 KB)

This looks like a potential problem with the cellular service, and possibly the signal.

Do all of your Ewons use this APN, iot0719.com.attz? Are there restrictions on how that SIM can be used?

image

Kyle,

Yes, the SIM cards are linked to a particular ATT IoT account that is setup, it is the same for all our eWons, regardless of firmware version. When you are referring to possible signal problems, are you referring to the logs from today, or the issue with it going offline indefinitely starting 8/14, or both?

Is there any reason to anticipate that this is a firmware problem?

Thanks,
Caleb

I don’t believe it’s a firmware problem because we haven’t seen any issues like this involving 14.1. It seems to be dropping the connection pretty regularly though, so I wanted to check if it was due to the carrier possibly limiting the data connection due to it being an IoT plan, but that isn’t the case if they are all on the same plan.

I would recommend switching to ‘Force TCP’ mode to see if that makes a difference. (Go to Setup > System > Storage > Edit COM cfg and change VPNProto to 1 and VPNPortOut to 443)

If that doesn’t work I would recommend doing a factory reset and re-configuring from scratch. Also, make sure the antenna is free of any obstructions or interference.

Kyle,

Thanks for the ‘Force TCP’ parameter idea. I only keep asking about the firmware since we have about 3 devices that are showing similar issues, however our client has indicated that the ATT service in these areas has not been great. Do you think the ‘Force TCP’ parameter will help the eWons from indefinitely disconnecting like it seemed to in our situation? The eWon connected almost immediately after being power cycled, so there was enough of a signal at that time, but the logs would suggest that it was constantly trying to connect to the network and failing, any idea why it was getting hung up?

Caleb

Yes, potentially. I think that in this case as the cellular signal is not especially good, using TCP should improve the reliability.

If it doesn’t, there may be another problem and I think the quickest way to resolve it would be to do a factory reset.

Hi Kyle, you mentioned that “there may be another problem”, was there anything in particular that you noticed in reviewing the logs that could pinpoint why doing a factory reset would resolve the issue? Our customer has a few other identical 1-1 issues like what caleb_dmc has posted, and these devices are all over the US. Doing a factory reset is a costly, both in time and money, effort. So we need to be able to justify, and without a doubt, that doing a factory reset would actually solve the issue, if the ‘Force TCP’ parameter doesn’t work.

No, but from my experience it can save a lot of time by doing a factory reset when a device isn’t running correctly as opposed to hours of troubleshooting trying to pinpoint an issue, that isn’t always found. The Cosy 131 can typically be reset and re-configured in about 5 minutes.

I wouldn’t recommend factory resetting a number of devices that are having identical issues. That sounds like something else is going on. Can you provide a backup with support files? If there is a larger issue going on with the AT&T modems, we’d like to figure out what it is.

MOVED TO STAFF NOTE (175 KB)

Hey Kyle,

Apologize for the delay, we had to wait on the client in order to get this. This is from another one of the machines that is experiencing the same issues of once it disconnects, it just stays disconnected until a power cycle of the EWON.

Hi Chase,

Kyle’s out on vacation today so I’ll try and take a look at this. I saw that it looks like this device has a pretty weak signal strength. If the device is a Flexy, I have some code that can force a device to reboot if it’s offline for more 10 minutes. If you’d like to try that out, I’ll include the code below:

Reboot if not online after 10 minutes.txt (921 Bytes)

It does require that you create a tag called “counter”

Hi Tim / Kyle,

As the topic states, we do not have a Flexy but a Cosy. We’re uploading another device that is seeing the same issue. MOVED TO STAFF NOTE (175 KB)

This is becoming a serious problem. Based on the logs, would you or (Kyle) be able to explain why the EWON seems to work for awhile (albeit a not great connection), but then experience this long period of them not being able to reconnect until the client power cycle’s the EWON?

In addition, would you be able to go into more detail about what / how changing the “Force TCP” parameter is going to help?

Also, as caleb_dmc mentioned, we are seeing a lot of “openVPN” related issues that were occurring during the “down period”. Would you be able to elaborate on those?

The log you sent me is from an Ewon running 14.0s02, and it appears to have a poor cell signal.

It tries to start OpenVPN:
1598372349 25/08/2020 16:19:09 ovpn-OpenVPN process start openvpnitf-OpenVPN process start wanmgt 79336 1073771442

It is not getting a DNS response:
1598372387 25/08/2020 16:19:47 t2m-DNS failure, using static Talk2M access server talk2m-DNS failure, using static Talk2M access server esyncitf 79337 -33240

It restarts the process:
1598373539 25/08/2020 16:38:59 ovpn-OpenVPN process ended openvpnitf-OpenVPN process ended openvpn 79331 1073771437
1598373540 25/08/2020 16:39:00 ovpn-OpenVPN process start openvpnitf-OpenVPN process start wanmgt 79336 1073771442

This just goes on and on in the logs. It doesn’t appear to have a strong enough signal to keep a consistent connection.

It may be able to connect when it restarts because it refreshes the connection to the tower. This also works on a cell phone when you have very poor service if you’ve ever tried it.

I don’t think using TCP is going to make much of a difference due to the signal, at the time I thought it was worth a try. Where is it located? Can you move it to an area with a better signal or use an extension antenna?

Hey Kyle,

Thanks for the response. Is this the same behavior on all 3 of the files that I sent? We’re trying to 100% confirm that the signal is the issue on these devices.

The location of the eWON in each facility varies. Without going too much into the solution, the eWON is on a device that sits on a track, and the device will move up and down this track. At some points along the track, there may be more points of interference than other positions.

Our client has discussed potentially looking into extension antennas / signal boosters, but this is a costly route to go down if it isn’t because of the signal. We wanted to fully confirm that the issues with the eWON not being able to reconnect is due to the signal strength and not something else. They have multiple eWONs all over the USA, so having to go to each individual device to install one of these boosters is, again, expensive.

Yes, all 3 of the logs show the same poor signal (12/31 or less) and the same issues in the logs. I would try using an extension antenna attached to the top of the machine to see if that helps first as it would be the least expensive option.