All,
Looking for understanding on weird internet issue.
We have redundant internet (primary = Comcast, secondary = Frontier), and
redundant Meraki MX60 security appliances. Fallback has been tested for
every scenario and in every sequence. It works. In fact, earlier this
week, Comcast had issues down the street and it failed over to Frontier
just fine.
Last night at 7:41pm I got several messages from the Merakis:
"The security appliance in the DFF-Firewall - Dilgard Frozen Foods
<
https://n159.meraki.com/DFF-Firewall-Dil/n/I-KIjcP/manage/nodes/list> network
switched to using its primary uplink, configured to be uplink Internet 1,
after a period in which the link was unavailable.
There have been a total of 6 failover events detected:
At 07:41 PM EST on Feb 23, the security appliance switched to using
Internet 2 as its uplink.
At 07:41 PM EST on Feb 23, the security appliance switched to using
Internet 1 as its uplink.
At 07:41 PM EST on Feb 23, the security appliance switched to using
Internet 2 as its uplink.
At 07:41 PM EST on Feb 23, the security appliance switched to using
Internet 1 as its uplink.
At 07:42 PM EST on Feb 23, the security appliance switched to using
Internet 2 as its uplink.
At 07:42 PM EST on Feb 23, the security appliance switched to using
Internet 1 as its uplink."
I went online and looked and everything looked fine on the Meraki dashboard
page. Got these at 11:52pm which I did not see until today:
"The security appliance in the DFF-Firewall - Dilgard Frozen Foods
<
https://n159.meraki.com/DFF-Firewall-Dil/n/I-KIjcP/manage/nodes/list> network
switched to using its primary uplink, configured to be uplink Internet 1,
after a period in which the link was unavailable.
There have been a total of 4 failover events detected:
At 11:51 PM EST on Feb 23, the security appliance switched to using
Internet 2 as its uplink.
At 11:52 PM EST on Feb 23, the security appliance switched to using
Internet 1 as its uplink.
At 11:52 PM EST on Feb 23, the security appliance switched to using
Internet 2 as its uplink.
At 11:52 PM EST on Feb 23, the security appliance switched to using
Internet 1 as its uplink."
When I got to work at 6:30 this morning, the plant manager said they could
not get on the internet to get email. I thought, uh oh, this isn't good.
Since we have redundancy, I thought maybe both Merakis were dead.
The warehouse people use thin clients to get to a terminal services
server. I signed on to my desktop and the internet was fine. Signed on to
the server in question and no internet access. Tried 2 other Windows
servers and they accessed the internet just fine, as did a sales rep that
showed up with his laptop. I rebooted the server with the issue and no joy.
At this point I wondered if the IBM i could get to the internet. That
would not be good because the IBM i goes out and retrieves web orders 24x7
and Thursday overnight we get quite a few orders that should have been
delivered this morning. I pinged google.com from a 5250 session and no
joy. Pinged google.com from my desktop just fine. Since rebooting the
Windows server with a problem did not help, I decided rebooting the IBM i
would not help. I was at a loss. Why would 2 devices and 2 devices only
have internet access issues?
Since there were 2 servers that could not access the internet _and_ those
rapid failovers last night, I decided maybe Comcast was the problem. What
could it hurt to power cycle the cable modem? So I did.
After that, everything worked. I don't understand. Assuming Comcast did
indeed have issues last night with multiple rapid failovers and resets, how
would that cause an issue on only 2 devices? Those 2 devices happened to
be active at some exact moment during one of the failovers? The night
supervisor would have been signed on to that Windows server, so it could
have been active. And the IBM i would be checking for orders every 5
minutes, so it too could have been active.
Looking for ideas or an explanation on what actually happened.
(In the meantime, we have opened a case with Meraki to see if the security
appliances can be set to either 1) not recover from a failover back to the
primary uplink automatically, but needing manual intervention, or 2) not
recover from a failover back to the primary uplink until the primary has
been up for XX number of minutes.)
Thanks.
midrange.com
