Sophos’ access points are very sensitive with PoE enabled cable connections. If your access point gets disconnected, disable PoE for the given Ethernet port.
As a long time reader of my blog, you might know that I am struggling a lot with curious issues like broken L2TP/IPSec VPN connections. This time we had some other sort of issue. But first a little bit of background:
With beginning of the year 2020 we moved out from the office of my old employer. My new employer has its offices in the Forum Autovision. With moving out of the old office we took the Sophos UTM firewall and two of the AP15 access points with us. Both of the access points had been working without any problems for years.
In the Forum Autovision we placed the two access in two different buildings (ICW-3 and ICW-6). During setting up the infrastructure we already experienced some odd behavior: The AP15 in ICW-6 could not connect to our firewall. Instead, only the LED were blinking slowly.
We ignored the issue because at the time there had been other difficulties like misconfigured VLAN. We just replaced the access point afterwards in ICW-3 with the one in ICW-6 and vice versa. Everything was working.
Access Point suddenly loses its connection
Fast forward to one week later: Our co-workers in ICW-6 complained about an outage of their WLANs. We took a quick look into the UTM’s wireless log and just saw the following error:
2020:03:10-13:50:39 fw1 awed[10317]: [A40023288xxxxxx] ll_read: dead socket: Resource temporarily unavailable
2020:03:10-13:50:39 fw1 awed[10317]: [A40023288xxxxxx] disconnected. Close socket and kill process.The LED of the given access point just blinked orange. Neither a power recycle nor removing the AP from the UTM’s Wireless > Access Point > Pending list changed this. It has been like the network connection on Layer 1 or 2 has been completely broken.
We removed the AP from the designated network port in the office and connected a notebook. The notebook received an IP from the DHCP. So the network connection itself worked. But after disconnecting the notebook and connecting the AP, the AP started blinking orange again.
Replacing the AP from ICW-6 with the AP in ICW-3 did not change anything. Both access points did not work on the office’s network port.
Issue with Power over Ethernet
During my search for the ll_read: dead socket: Resource temporarily unavailable error above I stumbled upon this thread in the Sophos community:
…
3. Change physical connection (cables) which connects AP to UTM and the power adapter.
sachingurung
4. Make sure you use the provided power adapter or PoE adapter that comes along with the AP. If you are using a different adapter it could cause a socket error.
After connecting one or both access points directly to our Sophos UTM firewall, everything worked again. So this issue was something out of our control. We suspected that one of the ICW-6 building’s core switches did something strange.
We asked the guys from the Wolfsburg AG helpdesk – who did btw a fantastic job -, if they could see anything on their end. The designated network port in the office of ICW-6 was connected to one of their Cisco switches. This very port had Power over Ethernet enabled. Due to unknown reasons to us, with enabled PoE, the port flapped every few seconds. No connection could be established.
This had been the output in the Cisco’s log file:
Mar 11 09:56:55.279 CET: %ILPOWER-7-DETECT: Interface Gi1/0/5: Power Device detected: IEEE PD
Mar 11 09:56:55.845 CET: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/5: Power granted
Mar 11 09:56:56.754 CET: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/5: PD removed
Mar 11 09:56:56.755 CET: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/5: PD removed
Mar 11 09:56:57.583 CET: %ILPOWER-7-DETECT: Interface Gi1/0/5: Power Device detected: IEEE PD
Mar 11 09:56:57.987 CET: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/5: Power granted
Mar 11 09:56:58.755 CET: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/5: PD removedFixing the flapping port issue
In the end, it was easy to fix the flapping port: The helpdesk guys disabled PoE on the Cisco switch and everything worked again instantly.
