silently hopes nothing breaks
1 Like
Iām at 62 Hue bulbs, 1 Wemo, and 41 Kasa plugs integrated.
3 Likes
@Offthewall - I have good news for you! I have experienced the N/A bug.
Iām going to leave it at that for now. I have to get to sleep because I have to take my mother to the airport in the morning. I did a little bit of digging around and saw the following:
- PCAPs are showing that the Sense is still doing itās broadcast to 9999 asking for all devices to report back. My monitor is set to poll every 6 seconds, but yours should be every 2. Based on my findings I am seeing a request every 6 seconds as expected.
- PCAPs are showing that the Kasa device is not responding anymore and is still communicating with Kasa servers and ARP requests.
- Kasa device did have a DHCP Failure today around the time that the issue started.
I have to piece everything together, when everything went haywire and that will take some time. I will work on that over the next couple days.
I donāt think itās your environment. I have all commercial grade Cisco switches, routers, access points, security appliances etc. I do think itās a result of the Kasa firmware or hardware based on my findings so far. I want to figure out if itās just not respoding anymore or if itās some other issue. In my case itās actually responding but only part of the time. Itās like itās half way working.
3 Likes
I donāt quite understand the āDHCP failureā that the Kasa plug had. Are you saying it had an issue when renewing its lease? Everyone having trouble with their KP115s sees them all go ān/aā at the same time, that would point to some external cause. The DHCP failure you mention might be that cause.
EDIT (additional thoughts): I believe the problem may well be in the KP115, as the Sense polling that you see catches the plugs as they are individually restarted.
@demiller9 Iām going to dive into it more and see if thereās any correlation between the DHCP Failure and the issue everyone is having. I assumed early on, if you read back a few posts, that it was either a DHCP issue or a Kasa issue. It might very well be both however based on the below itās not looking like a DHCP issue even though there are recorded failures.
The DHCP failure happened at 0154 Sat July 10th
The first detected issue in the energy monitor was at 1213 On Fri July 9th
The first DHCP lease was at 1544 Tues July 6th
DHCP Leases are 24 Hours
The commonality that Iām seeing on the KP115 but not on the HS300 is random deauthentications. Thereās no specified reason for it either.
This image shows the % of failed connections, number of failed/connected and the reason
1 Like
The KP115s arenāt responding to the polling when the issue happens. Iāve confirmed that with PCAPs. So I really donāt think this issue is a Sense issue. Itās looking like itās a Kasa issue. A power-pull of the devices would indicate that the deviceās internal software is rebooted and it resumes working as expected until something gums it up. Just trying to figure out what is causing it to gum up.
2 Likes
THX, I am waiting with great anticipation for your additional findings!
Whatās serving DHCP in your network?
Edit: Rare that the AP sees the DHCP request, but doesnāt a DHCP response. Assuming the AP can be trusted, this would imply a failure somewhere between the AP and the DHCP server (inclusive). Can you confirm whether or not there is a response from the DHCP server? I.E. a sniff on the DHCP server? Or a span port?
@dennypage The DHCP server is responding the plug just doesnāt want to accept it. The message about the DHCP not responding is from the device level not the AP level. Iām assuming that the device does a discover, the DHCP does an offer and then either the request or subsequent ack fail. When I force a discover on the same network I have yet to see it fail.
I donāt understand. The approve image appears to be a standard error from an AP indicating a client association failure: āClient made a request to the DHCP server, but it did not respond ā¦ type=āNO DHCP responseāā
If you are confident that the DHCP server sent a response, then the response would have to have been lost, either at or before the AP.
@dennypage I understand your confusion. I wish I could see the KP115 logs. I have looked over the logs on the AP and Security Appliance which is in charge of DHCP for this DMZ. The only issues reported for DHCP are from the KP115 devices. I donāt have any other devices (out of 168) in that DMZ that are reporting issues. When I force DHCP renewals I can see the process go through as expected.
Now ā¦ at some point they are renewing a lease because even after the failure their lease renews. So I donāt think this issue is ultimately whatās causing problems. As I laid out in the timeline, it doesnāt appear to be DHCP related, unless the device itself is doing something funky with the DHCP request which if itās locking up could be as a result of a timeout etc.
Lastly, that one screen shot could very well just be an anomaly as that AP is the only AP that works in repeater mode. Iām posting everything in regards to 1 single KP115 for documentation purposes. I havenāt been able to catch this happening in real time yet.
All requests from the single Kasa device that got to the DHCP server were answered. The error in the screenshot you provided was on July 10th at 0150. As you can see the KP115 actually got a renewed lease 2 hours prior. Since 12 hours is the first available renew, it could be that the DHCP server just ignored the request. Again, I donāt have pcaps at that time so Iām only assuming. See below:
I have cross posted this if anyone is interested in chiming in or up-voting:
Based on my findings this is a Kasa issue and not related to Sense. Additionally, I donāt think this is an infrastructure issue and although thereās some strange things from these plugs I still think itās related to firmware. If anyone feels otherwise please let me know.
2 Likes
Youāre running a backhaul network like @Offthewall?
The DHCP server should never ignore a renew request. It should send either an ack or a nak.
It does occur to me that the plugs could have a defect in that if no response was received to a renew request, they fail to fall back into discover. Triggering this to recreate a persistent failure as described above would require either a network or DHCP failure in renew, followed by the plug failing to ever ask again. Sounds like a long shot, but I suppose itās possible. It would explain why itās so rare.
@Offthewall, what is your current lease time, and can you increase it to test? Something like 14400?
Iāve had it at 1 day, then tried 1 hour. Two different DHCP servers. In each case the plugs went n/a after 4 and 5 days.
How do you feel about trying a longer lease? Like 10 days (14400).
1 of 3 APs but Iām seeing this issue on KP115s connected to all APs
By ignore Iām referring to nak. Iām trying to keep it somewhat common folk language. Also my logs donāt show naks so thatās why I say Iām assuming because I donāt have logs or pcaps of that.
One other thing to note is that some people who donāt intently monitor their devices might not see this issue. Let me show you an example of screen shots where it appears that the device is still working, but if you look itās really not, thereās huge gaps. So there could be many people who have KP115s that donāt know that the data is incorrect.
Normal KP115 on an always on device:
Bugged out KP115 on the same device:
I think thatās the issue of the Sense processor being overloaded (why support moved you to a higher polling interval). The issue @Offthewall is having is that the plugs go offline completely until he resets them. Seems to be a completely different issue.
That issue was very different and didnāt result in the n/a either. It was an overloaded device but in that case sense didnāt report any data for any loads and the monitor locked up. This is quite different and mimics that of @Offthewall in regards to the n/a and the Kasa device not responding to the broadcast udp. Additionally, this is only happening to my KP115s and not the HS300 or the virtual KS110 via HA.
Maybe he can chime in and report if he gets no data at all or some spotty data.
1 Like
I feel like we dive down all kinds of paths here but hereās what I can definitively say:
The KP115s based on the n/a issue
I can say without a doubt:
- The sense monitor is sending broadcast UDP to 9999 according to the polling time
- The plug is not responding or sporadically responding
- HS300 devices donāt have this issue
- HS300 and KP115 have different firmware versions
- If it was the monitor the HS300s would have the same issue
If more people had PCAPs and could verify this it would be nice. In order to trouble shoot we have to eliminate one or the other. We canāt keep saying well whatif this or that. Letās go with what we know.
15:13:27.848552 IP 10.0.2.248.9999 > 255.255.255.255.9999: UDP, length 63
15:13:27.888424 IP 10.0.2.204.9999 > 10.0.2.248.9999: UDP, length 729
15:13:27.916211 IP 10.0.2.201.9999 > 10.0.2.248.51615: UDP, length 1145
15:13:27.877220 IP 10.0.2.204.9999 > 10.0.2.248.9999: UDP, length 729
15:13:27.905034 IP 10.0.2.201.9999 > 10.0.2.248.51615: UDP, length 1145
.248 = Sense
.206 = Device Iām having issues with (not on the list)
While Iām only showing a piece of a huge cap, itās repetitively the same.
1 Like