Delayed mail delivery, mainly towards Microsoft hosted domains

Ewon Support
#1

We are still experiencing mail messages which are not being delivered.
We sent out a bunch of reports at 9:00am CDT today which have not been received by any recipients yet.
We were affected by the outage yesterday, and can confirm that all the messages finally reached us, albeit delayed by as much as 36-40 hours.

In addition, we are unable to access about 3/4 of our HMIs using VNC. This has been reported by another employee of our company, so I won’t rehash that… but that is severely affecting our ability to remotely support our customers.

#2

Hello @mitchell.hein,

This issue was reported as resolved by the development team, I was receiving emails this morning when i tested again. Are you still seeing issues?

Deryck

#3

Actually, I take that back. We are STILL missing messages.
Here’s a snip from a site showing planner messages that were “sent”.

image
image902×140 10 KB

We haven’t received a message from this site since Friday morning.
image
image884×465 31.9 KB

We missed Wednesday, Thursday, Saturday, Sunday, and Monday’s messages.
The device is online and has been the entire time.
Do I need to reboot this eWON too?
Why am I having to reboot these devices to get messages?

#4

Messages have now been delivered since Friday without failure.

We did have two eWON routers which required a reboot to reactivate the VNC connection feature over the weekend, but they were ones which still functioed on Friday.

So, it does appear as though the “event” is past us.

What on earth happened?

It affected every one of our routers, locked us all out of VNC access. and suspended all messages for two days. Office 365 reported it as a security breach on some of our mail accounts. Is there anything more you can tell me? Our I.T. manager is inquiring as to whether we need to migrate from Talk2M to a different network and without an idea of why the outage occurred, I have no idea what to say to him.

#5

Hi Mitchell,

I have asked the development team to take another look at this issue. I am thinking they are seeing further issues with the relay not communicating with other email servers.

Deryck

#6

After remotely rebooting all four of the questionable eWONs, we finally got all our planner scheduled reports to show up. The time clocks didn’t change, the eWONs weren’t offline… so I still don’t understand why the e-mails weren’t making it to us.

#7

Hello Michell,

I’m not sure this issue would have been related to the email issue we saw previously. That was an issue on the email server side not the ewon, I would not expect a reboot to have resolved this.

If you provide a backup of the unit I can check the logs to see if the logs indicate any issue.

Regards,
Deryck

#9

I’m relatively sure it was related. We didn’t have any issues with e-mails not being received until the event on Wed-Thurs. The planners in the eWONs showed that e-mails were sent out and none of us received them. We did not receive e-mails from any of the units until each eWON was rebooted. At that point, all e-mails worked again. The one site we hadn’t rebooted went silent again after Friday until I rebooted it and this morning we received e-mails as expected.

#10

Using ebuddy please provide a backup of the ewon with support files. This should give me a better idea of the issue.

Deryck

#11

Six reports in four eWONs all sent by the Planner.

image
image886×75 6.46 KB

None of them were received by the recipients on the distribution list this morning.

I will grab an eBuddy backup with all support files for you, but if the planner in the eWON says it was sent, I’m pretty sure that’s not a local issue. Hopefully, the logs will direct you to a fix. Our customer is getting quite upset that they don’t have these reports. They need them for government reporting.

#14

As a followup:
Yesterday, I had planned reports that were “sent” vis e-mail at the proper time, as listed by the planner. None of the e-mails was received by the recipients on the distribution list until 5-1/2 to 6 hours after they were sent. The times inside the reports showed that they HAD been created at the correct time, but there was a large delay before they were delivered. The delivery times were spread out over a half-hour, making them appear as if they were released manually from some holding location.

This morning, all reports were received within a minute of when they were sent out and all recipients received them.

Please let me know what the problem was, how it was corrected, and what I should direct HMS to do in the event that this happens again, as it has now happened three times over the last week.

Thanks!

#15

Hello @mitchell.hein,

I was just discussing this issue with the ewon team and it should be resolved. I do not have all the details on the cause but it is my understanding there was an issue with the relay causing it to retain some emails. It has been updated and emails should be getting delivered correctly now. We are still seeing some delay for Gmail addresses but this is a delay in the Gmail server.

Regards,
Deryck

#16

Thank you for the update.
I’ll be sure to let you know if this continues to be an issue.