TPLink Data really randomly flaky

Technical Questions
1

Sense seems to be screwing up the TPLink plug data quite often. Right now I see one of my plugs drawing 42W but the TPLink app shows it drawing less than .25W. I’ve also see others where they report a single wattage for hours when the TPLink app shows the wattage varying. How to fix this?

1 Like
2

Have you tried rebooting your Sense monitor? Turn its breaker off briefly, see if that helps.

1 Like
3

Haven’t tried that. Not sure it would help as it comes and goes randomly(it fixes itself with no intervention). Sometimes it can go for a fairly long time with no issues and other times it just decides to lie about one devices usage for some period of time and even then its inconsistent about how it lies. Sometimes it just takes a random true reading and repeats it for hours other times it just makes up a value that the device has never reached(42W on a plug that didn’t have a draw of over 1W in the past day). Its also seems to usually just pick one random plug to make up data for I don’t think I’ve ever seen it do multiple plugs at the same time. You can tell if the issue is happening usually because the graph will be perfectly flat with no variations. I’ve used Sense for 2.25 years and it used to be better I think.This has only been occurring for about 6 months or so prior to that it appeared to be very reliable to me.

4

How many TPLink smart plugs are in your system? The intermittent performance you describe might be if there are too many. The recommended number is 20 before performance starts to degrade. Note that each HS-300 counts as six, since each of its ports are individually reported to Sense.

There are workarounds if you have more than 20, but you have to involve technical support to implement them.

5

I have two HS300s and two single plug ones.

6

I have many TPLink plugs and have not encountered that same issue yet. Based on my experience with them, Sense only reports what TP Link is reporting to it.

Are you sure that Sense (or you) has not somehow assigned other devices to that TPLink identified bubble. Sense will ask what is assigned to that plug, and if you have assigned other devices to it, even if you think it is the same device that it is plugged into it, it may be causing the problem.

Double click on the suspect bubble. When the detail screen shows up click on the gear icon in the upper right corner then on the device settings screen click on “Manage” then “What’s connected to this?”

7

Verified nothing like that is happening. Seeing it being weird again today. Where all of my TPLink devices are showing up in the other categories and the plugs all showing 0W or NA. When going into the TPLink app all are showing good data like always.

8

A couple of my plugs started showing up in Sense with data again. Still have a bunch that just show NA. Sense do you plan on fixing this at any point? I feel like Sense has really been going downhill for the past 12 months or so. Between this and it doing random other things like saying my espresso machine has been on for 6 hours when it was on for 15 minutes or popping up my alerts that say your garage lights have been on for 15 minutes when in reality they are off.

9

Kasa app is completely independent from Sense app - each issue their own requests for data, because the calls are slightly different, it’s not clear when smart plug, Sense or home network is the cause. Here’s what is known.

1 Like
10

Its not my LAN. I can say that for sure as my job is the lead data center architect for a Fortune 50 company. If they want to provide the API calls they are using I can set up monitoring with my home Telegraph/Influxdb/Grafana stack to rule that out. It’s had been really bad the past two weeks and wasn’t much of a problem a year ago. Most of the issues listed don’t apply to me as if the plug was offline the kasa app would not be able to get data or control the plugs in question either. I do also have a lot of Hue lights but that integration seems to always work with sense.

11

That’s great news. Sounds like you have the tooling to log the packets and detect the cause.

The protocol / API is something like this:

  • Sense sends the following UDP broadcast command to port 9999 on all the Kasa devices on the local subnet:
    {"emeter":{"get_realtime":{}}}

  • HS110s receiving that message will send a message back to the requesting IP address of the following form (with a little encryption in between)
    {"emeter":{"get_realtime":{"current":0.028915,"voltage":120.798141, "power":0.527563,"total":18.045000,"err_code":0}}}}

  • HS300s respond with the same data, with current and voltage reversed and in different units
    {"emeter":{"get_realtime":{"voltage_mv":121917,"current_ma":177, "power_mw":16194,"total_wh":90051,"err_code":0}}}}

  • KP115/125s respond in the same units as the HS300, but with current and voltage reversed. Note that the response below is for a situation where the KP115 is turned off via the Kasa/Sense On/Off switch. So there’s no specific data returned that tells that the smartplug is turned off - it just says 0 power. The On/Off is a separate command/status part of the protocol.
    {"emeter":{"get_realtime":{"current_ma":0,"voltage_mv":122525, "power_mw":0,"total_wh":107,"err_code":0}}}}

There’s also a WireShark detector that decodes the entire TP-Link command and debug protocol.

Full TP-Link command set here:

ps: the links give the sources of some of this info.

2 Likes
12

Thank you I’ll look into seeing if I can make a Telegraf plugin for this protocol(its a pretty ugly protocol though a sane developer would have used unicast or multicast). The only possible issue here is I’m not sure if Docker networking will work with the broadcasts(I’ve never had a need for it before I’ll have to look at the docker bridge driver code). In this day and age broadcast should only really be used for ARP/DHCP or things of that nature. Someone should give the guy at TPLink that wrote it a kick in the ass. The Wireshark plugin should give me enough to work with if Docker cooperates. Also since I currently have one of my HS300’s offline and one of my HS300’s reporting I should be able to do tcpdumps on my APs WLAN interfaces filtered to port 9999 to see whats going on. Will report back with findings when I have a chance to do some captures and analysis.

1 Like
13

I’m pretty sure the issue is just Sense ignoring the data it is getting but I need to verify. I did an sshdump capture on my AP hosting the Sense and I see both the working one and the broken one(HS300) replying to the Sense on the WLAN interface. I’ll go through the captures and see if I can scrub any private info out of it so I can post it here. I need to go through and verify all the JSON data to make sure its for each plug in the strip but I do see current, voltage and power fields with non zero data in the responses for both the working and non working strip. Its a bit complicated as its not just broadcast Sense also does a bunch of unicast requests with additional data looks like mostly adding context: {child-ids xxxx}.
One other note there is no encryption or authentication at all very badly designed insecure protocol. I have to do more research but it is likely an attacker on someones LAN could shut off devices or reset the devices to factory settings.

1 Like
14

So the capture shows its a sense problem. Working one Sense is doing a unicast request after the bcast with child-ids added to its JSON broken one its not. Sense also seems to be trying to open a TCP connection(only on broken one) on port 9999 which is ignored by the strip because there is no TCP listener on that port.

Good unicast request that sense makes to working but not broken strip. It enumerates each port with a different context-ID.
{
“system”: {
“get_sysinfo”: {}
},
“emeter”: {
“get_realtime”: {}
},
“context”: {
“child_ids”: [
“Long HEX STRING HERE last 2 digits are id # of port Sense is asking for power data on”
]
}
}

Reply from strip(good one only as Sense decides to not send this to the “broken” one although it does reply to its bcast request with the same general data as the working one).

{
“system”: {
“get_sysinfo”: {
“sw_ver”: “1.0.21 Build 210524 Rel.161309”,
“hw_ver”: “1.0”,
“model”: “HS300(US)”,
“deviceId”: “xxx”,
“oemId”: “xxx”,
“hwId”: “xxx”,
“rssi”: -61,
“longitude_i”: -1234,
“latitude_i”: 1234,
“alias”: “TP-LINK_Power Strip_DA56”,
“status”: “new”,
“mic_type”: “IOT.SMARTPLUGSWITCH”,
“feature”: “TIM:ENE”,
“mac”: “xx:xx:xx:xx”,
“updating”: 0,
“led_off”: 0,
“children”: [
{
“id”: “00”,
“state”: 1,
“alias”: “Phone charger”,
“on_time”: 4070638,
“next_action”: {
“type”: -1
}
},
{
“id”: “01”,
“state”: 0,
“alias”: “my laptop”,
“on_time”: 0,
“next_action”: {
“type”: -1
}
},
{
“id”: “02”,
“state”: 1,
“alias”: “Work laptop”,
“on_time”: 4070638,
“next_action”: {
“type”: -1
}
},
{
“id”: “03”,
“state”: 1,
“alias”: “couch 3”,
“on_time”: 4070638,
“next_action”: {
“type”: -1
}
},
{
“id”: “04”,
“state”: 1,
“alias”: “couch 2”,
“on_time”: 4070638,
“next_action”: {
“type”: -1
}
},
{
“id”: “05”,
“state”: 1,
“alias”: “couch 1”,
“on_time”: 4070638,
“next_action”: {
“type”: -1
}
}
],
“child_num”: 6,
“err_code”: 0
}
},
“emeter”: {
“get_realtime”: {
“voltage_mv”: 121901,
“current_ma”: 117,
“power_mw”: 11415,
“total_wh”: 20802,
“err_code”: 0
}
}
}

Here is the wierd TCP packet it sends to the broken one.

Frame 30: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface sshdump, id 0
Ethernet II, Src: MurataMa_xx:xx:xx (00:9d:6b:9xx:xx:xx), Dst: Tp-LinkT_xx:xx:Xx (e4:c3:2a:xx:xx:Xx)
Destination: Tp-LinkT_32:e1:70 (e4:c3:2a:xx:Xx:xx)
Source: MurataMa_9xx:Xx:Xx (00:9d:6b:9xx:xx:Xx)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 192.168.xx.xx, Dst: 192.168.xx.xx
0100 … = Version: 4
… 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
Total Length: 60
Identification: 0x57d3 (22483)
010. … = Flags: 0x2, Don’t fragment
…0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 64
Protocol: TCP (6)
Header Checksum: 0x5eac [validation disabled]
Source Address: 192.168.1.150
Destination Address: 192.168.1.86
Transmission Control Protocol, Src Port: 54222, Dst Port: 9999, Seq: 0, Len: 0
Source Port: 54222
Destination Port: 9999
[Stream index: 3]
[Conversation completeness: Incomplete, SYN_SENT (1)]
[TCP Segment Len: 0]
Sequence Number: 0 (relative sequence number)
Sequence Number (raw): 2877592655
Acknowledgment Number: 0
Acknowledgment number (raw): 0
1010 … = Header Length: 40 bytes (10)
Flags: 0x002 (SYN)
Window: 29200
Urgent Pointer: 0
Options: (20 bytes), Maximum segment size, SACK permitted, Timestamps, No-Operation (NOP), Window scale

1 Like
15

@chad.simmons, glad you’re making progress. Thanks again for investigating. A few thoughts.

  • When you started this thread I had 3 working HS300’s with all the plugs reporting correctly, but had a few HS110s not reporting (n/a). You inspired me to try one of the tricks that sometimes restores the smart-plug / Sense connection, turning NDI off, then on again. Unfortunately that killed off working operation of two of the HS300s. I have since tried a few more things (Sense reboot, one more off/on of NDI) and have now seen more Kasa devices, including the remaining HS300 plus some HS110s go n/a. So I agree, there is something very wrong with the current firmware. I’m going to file a ticket and would suggest you do too. I’m not in a position do debug as I’m getting set to do some traveling.

  • TP-Link protocol was state-of the-art IoT from 10 years ago. TP-Link is transitioning to Matter, but unfortunately TP-Link has a 10 year legacy and millions of non-upgradable smart devices out there that use the current “ancient” protocol.

  • Forgot about the child-ids - the HS300’s have to report back on multiple outlets per IP/MAC.

2 Likes
16

I’ll try opening a ticket but I get very frustrated with most non commercial vendors. Their 1st level support generally has no visibility to their codebase and they generally try to blame the LAN or the other vendors products and they don’t generally understand pcaps or json. What would really help would be for them to post their code to a GitHub repository so I could just see what they are doing wrong and send them a patch.

Ticket #378799 opened.

1 Like
17

I received a message back from Sense support. The response was the most professional I’ve ever encountered from a 1st level tech support with a company that provides non commercial products(honestly I’ve seen worse from enterprise vendors). They indicated that they would need some time to go through the data and would escalate to their engineering and didn’t ask me to perform any pointless troubleshooting. Big shout out to James from Sense support. If anyone from Sense management is reading the forum he definitely deserves a raise.

3 Likes