Smart plug power data glitches

Missing smart plug data samples need to be treated differently for the various features Sense provides. My suggestions:

Always On-ness:

  • Should only consider non-missing data samples.

"Ground truth" for aiding detection:
  • Should somehow only consider sets data samples that don't include or are near missing samples.

Merged discovered devices with smart plug:
  • (I'm not familiar with how Sense does this, as I only tried this for the first time yesterday, so I'm not going to suggest anything here.)

Device [smart plug] bubble:
  • Missing data to result in bubble disappearing, though perhaps with a low pass temporal filter so that the bubbles don't appear/disappear rapidly.
  • Don't think the bubble view should attempt to notify the user about the difference between missing smart plug data vs 0W.

Device power meter:
  • (I primarily use the iOS app and not the web app, but I just noticed that the web app doesn't even provide this feature. Is this a bug in the web app?)
  • Somehow clearly convey a difference between smart plug samples that are missing and which are truly 0W. (Maybe when a sample is missing, the plot should not have a red line at 0W; and to have the power value shown in the upper right as "NA" or "" when the vertical grey line is over a time where a missing sample occurred.)

Device [energy] usage:
  • If % of ~1s samples that are missing, within whatever duration {hour, day, month} the energy consumption has summed up, is greater than some threshold (maybe 5%), then somehow clearly convey that some of the energy data is missing. (Perhaps use a slightly lighter shade of green.)

Device [energy usage] data export:
  • Option 1: Add a new field (column) for each smart plug device that provides the % of ~1s samples that are missing, within whatever duration {hour, day, month} the energy consumption has summed up. I like this option, as it provides flexibility to offline processing of the exported data.
  • Option 2: If % of ~1s samples that are missing, within whatever duration {hour, day, month} the energy consumption has summed up, is greater than some threshold (perhaps 5%), rather than report a numeric energy consumption value, report either "NA" or "".

"Other" device:
  • (I'm going to ignore this for now; doesn't make sense to discuss how to handle the Other device when smart plug data is missing, before knowing how to handle the smart plug device itself when data is missing. Plus, the Other device already is weird, I think this thread is not the appropriate place to discuss the weirdness.)
1 Like

@james_reilley,
There’s a wishlist item for improved device management here, that kind of covers better handling of smart-plugs and their persistent histories.

@james_reilley,
All good suggestions. I tried to stay away from thoughts on how Sense should deal with NA in their data science / ML analysis, since I really have to trust them to experiment and get it right. That said, I agree that Always On should not consider missing data intervals as 0W !

As for the user viewable representations of the data:

  • I would like both the Device power meter and the main Power meter to show NA periods differently than 0’s, or else at least provide a log as @ixu suggests
  • I like your thoughts on thresholds for all the summary / rollup usage data
  • I like your ideas for Export
  • Agree on the bubble view

Thanks for putting deep thought into this semi-spec !
ps: Yes - The Web App lacks the Device power meters today. The major difference between iOS/Android and Web App.

Sense definitely has a bug in how it communicates to the Wemo and TP-link smart plugs. I have not contacted support about it, though I did start thread about this last month: Smart plug power data glitches

My reasons for calling this a Sense bug is that, just as @ian.heritage states above, that there are periods of time when Sense reports 0W from a smart plug when the smart plug apps (Wemo app or Kasa app) are able to successfully communicate with the smart plugs. While the LAN that Sense and the smart plugs are on may experience issues that cause 1 or 2 seconds of smart plug data to be missed, Sense should not loss contact with the smart plugs for hours.

1 Like

@james_reilley, not to be snippy, but Kasa communicating perfectly with smart plugs is NOT validation that your network is not the culprit in Sense to TP-110/300 communication. Sense and Kasa use different protocols. I have a multi-access point house where I recently trialed replacing my Apple network with an Orbi mesh. The Orbi network broke a bunch of things in my house, including the Sense to smart plug communications because its default was to block broadcasts between access points that Sense relies upon. My Kasa app was unphased by the change. Kasa also makes use of different calls to a the plugs. Instead of continuous sampling, it asks the plugs for their daily history instead of continuous measurement. Bottom line is that there are many possible reasons for the data dropouts but just because the Kasa app is working fine, it does NOT mean the Sense software / monitor is at fault.

ps: On the flip side, I do think Sense should do a better job flagging and diagnosing data dropouts, so the answer doesn’t have to be “contact support” as frequently.

3 Likes

Agreed, though it is a strong indication that Sense could do the communication differently (e.g. do it however the Kasa app does it, which seems to be more robust) to result in fewer dropouts (and hopefully no minute/hour/day long dropouts at all). Maybe “bug” is not the correct term, but I do believe Sense firmware could be updated to more robustly communicate with the smart plugs.

From the Kasa app, while it doesn’t support automatically polling the smart plug once every second or two, I can force it to give me a new power sample every second or two if I rapidly selected “energy” and then the back arrow and then “energy” again and so on; and the Kasa app can successfully do this when Sense is not reporting any smart plug data.

One of these days, I’ll hook wireshark up to my switch with port mirroring to see the network traffic to/from the smart plugs when:

  1. Sense is successfully communicating with the smart plug
  2. Sense is unsuccessful at communicating with the smart plug
  3. Kasa is communicating with the smart plug
I'd also want to see if there is any network shenanigans occurring right when Sense first starts to lose contact with the smart plugs.
1 Like

I would be really interested in seeing your Wireshark results. I did a little bit of playing around when I first experienced some dropouts. The Sense monitor broadcasts this TP-Link command every 2 seconds or so:

{“system”:{“get_sysinfo”:null},“emeter”:
{“get_realtime”:null}}

I can’t find my Wireshark info for the response, but I seem to remember each device broadcasting back.

So the failures can come in few forms:

  1. The Sense monitor stops broadcasting or slows down the rate of broadcast
  2. The smartplugs fail to respond
    • The initiating broadcast got lost on the way
    • The smart plug fails to respond
    • The response gets lost on the return
  3. The Sense monitor fails to register the response
    • The response arrives too late
    • The monitor is busy with something else

BTW - here’s the repertoire of TP-Link commands.

1 Like

@kevin1, I did get a wireshark capture that shows both how Sense device and the Kasa app communicates with the HS110. I installed the dissector (GitHub - softScheck/tplink-smartplug: TP-Link WiFi SmartPlug Client and Wireshark Dissector), which only worked for TCP packets, and I had to create a copy with some minor modifications to work with UDP packets.

As you mentioned, Sense sends a broadcast message every 2 seconds to port 9999 with a ‘{“emeter”: {“get_realtime”: {}}, “system”: {“get_sysinfo”: {}}}’ command, and then all the HS110’s respond with UDP packets with the data:


(My capture shows 2 broadcast commands from Sense every 2 seconds, but that is an artifact of me mirroring the 2 switch ports that are hardwired to 2 of my 4 wireless access points…the other 2 APs have a wireless back-haul to the 2 that are hard wired. The capture trace only shows responses from 2 of my 5 HS110s, because the other 3 HS110s are connected to the same half of my wireless network as the Sense device, and so their unicast traffic does not need to be sent through my wired switch; I hope to ARP-spoof either the Sense device or those other 3 HS110s, so that I can get their packet communication to flow through my switch to the mirror port to my next set of Wireshark captures.)

The Kasa app, upon loading issues a broadcast UDP to learn what HS110s exist on the network (using ‘{“system”:{“get_sysinfo”:{}}}’ command), but thereafter exclusively uses short/quick TCP sessions (<=50ms of TCP session handshake, command request and response, the ACKs and FIN) for each smart plug discovered for each request. The exact command it uses to request the real time power data is ‘{“context”:{“source”:“bced4ee3-cf54-4767-902a-45bab63136c7”},“emeter”:{get_realtime":{}}}’:

The Kasa app also wants to get energy stats for the current day and last 30 days and so issues the following extra commands to get the energy usage data for all the days of the current month and last month:
‘{“emeter”:{“get_daystat”{“month”:1,“year”:2020}},“context”:{“source”:“”}}’, and
‘{“emeter”:{“get_daystat”{“month”:12,“year”:2019}},“context”:{“source”:“”}}’:

I have not been able yet to capture a period of time when Sense is unable to get HS110 power data but when the Kasa app is. Hope to catch such an event over the weekend. (I also hope to catch my Wemo smart plug giving an erroneously high single power data samples, which I provided an example of what Sense sees at the beginning of this thread.)

My Wireshark capture also shows how the Sense device and the cloud communicate. Everything here is TLS sessions and is thus encrypted, but I can still see the various frequent DNS queries (to several different domain names; with Sense ignoring the DNS servers specified via my router in its DHCP lease, and rather using Cloudshare’s DNS…I’m OK with this…I may switch over to Cloudshare’s DNS rather than Comcast’s anyway) and TCP/TLS sessions, their cadences and durations and bandwidth usages. (BTW, I really hope that if Sense were to every close down or stop their cloud services, that they first create/distribute SW for the PC that can take over many of functions that the cloud services provide today (maybe not the device detection training) and update the Sense firmware to communicate to local network SW, so as to no have the sense device “bricked”.)

I have discovered, that my Asus RT-AC66U router will disconnect/reconnect from the WAN for several minutes roughly every 2 days and reports “Your ISP’s DHCP does not function properly”. I’ve seen some of the occurrences of Sense not being able to communicate with the HS110s as being caused by these events when the WAN connection is briefly down or just after the WAN connection is reestablished, though I’ve seen other occurrences when these WAN disconnects occur and Sense is able to continue to communicate with the HS110s (Sense device buffers the data internally and sends to cloud later when WAN is back up), and yet other cases where Sense is unable to communicate with the HS110s when there has been no WAN disconnects.

2 Likes

@james_reilley, great Wireshark sleuthing. Hoping you’ll catch a dropout in the act. I know some of my dropouts in the past have been caused by the Sense monitor being too busy to log the responses and others were caused by non-propagated broadcasts due to switch settings.

Here’s a discussion of the Sense DNS behavior:

They listened to user feedback in this thread to form the above policy:

I now have a Wireshark trace that shows my Wemo plug returning a single bad power sample to Sense. I see both Sense reporting, and the Wireshark trace showing the Wemo reporting, a relative constant 40W power consumption. Then for one 2s sample, Sense reports 5177W (but with a zero-wide spike up about 2x as high, so in the 10kW range), and I see from the Wireshark capture of the Wemo plug that it reports 10314W for one sample. (I enabled Wireshark TCP checksum validation, and it says the checksum is goo.)

Clearly my Wemo plug is sending out bad power data, erroneously high, several times a day on average. Do others see this behavior too with their Wemo smart plugs when viewed via Sense? Perhaps Wemo could fix the problem on their end with a firmware update…I’ll actually see if they have a support team like Sense does. That said, if this is a wide spread issue, then I could see Sense implementing some kind of work around to ignore samples that clearly erroneously high. I really want to avoid smart plug device power meter graphs like the following, which are useless with a huge erroneous spike in the middle of the data causing the Y-axis to be scaled all wrong:
wemo_glitch_24hr
Anyway, I’ll be swapping out this Wemo for a HS110 soon, since I want this switch to turn back on following a power loss.

1 Like

@james_reilley, FYI - I have my level 1 charging Ford Fusion Energi, on an HS110. Works well.

1 Like

Both smart plugs are only rated for up to 15A @ 120V, or in other words 1,800W… so I guess I’m struggling with why Sense would accept anything higher than 1,800W as a good, actual reading.

1 Like

Sense is measuring at a much higher frequency than spikes that would trip a typical 15A breaker. Many devices (big motors; capacitors) can easily present large short term loads that won’t trip a breaker and are well within the realm of “normal behavior”.

Look here for more info …

1 Like

@aj.vandenberghe,
Just an FYI - I don’t think you should question the accuracy of the HS110 at the top of it’s power rating. That’s more of a regulatory thing based on guardbanding and the current specs of the relay. The Euro version of the HS110 has the same electronic “guts” and measures up to 3.68kW. Same guts are capable of 16A at 240V.

1 Like

@kevin1, thanks for the info… I wasn’t so much questioning the accuracy, as I was misunderstanding their provided specs for actual hard physical limits. And yes, based on what @ixu brought up - what I would refer to as “inrush current” - this makes sense also.

So to clarify, the smart plugs are providing instantaneous data, that is not at all averaged over some sampling interval? If so, this could help explain one-off occurrences of astronomically high readings, i.e. if/when the smart plug is sampled during that inrush event.

Considering Sense samples millions of times a second and only returns to us averages of every 0.5 seconds, I guess I was wrong to think that the smart plugs would similarly return some type of average as well.

I’m going to make a couple comments on this one, because Sense and the whole discussion of AC power usage hides some critical details.

  • Very few devices are designed to reveal the instantaneous AC power, an exception being an oscilloscope with math functions, because the instantaneous AC power varies wildly depending on where in the 120Hz power cycle you actually sample. The diagram below shows AC power for a well-behaved resistive load. In actuality, the current waveform is typically going to be more distorted and perhaps slightly out of phase from the voltage, which pretty much is forced by your power company to be a perfect sine wave at 60Hz. The current waveform depends on the loads the voltage is driving. Instantaneous power is simply V x A (current) at that point in time.

5387D2F1-AE52-4A89-B233-D600B063956A

  • Almost all measurements in AC are done as Root mean Square (RMS), which applies the equation below to evenly sampled data throughout one or more AC cycles. It’s useful to know the period over which the RMS calculation is taking place, and whether it is a rolling calculation (with the additional of each new sample point) or a new calculation every new set of full set of data.

    image

  • Sense and the HS110 both do RMS calculations, though the actual sample windows and method of handling the windows (rolling or new windows each calculation) isn’t published. As some point in time, I looked up the hardware part inside the Wemo Insight and HS110. It has a native sample speed of 2520Hz, or 42 samples per AC 60Hz cycle, and calculates/presents new data every second. Sense actually only pulls data from the HS100s every 2 seconds.

  • From what I have seen, the astronomically high outbursts from HS110s are not related to inrush current, but rather some miscalibration issue, since it often shows up in both the Kasa App and Sense.

2 Likes

I’ve now had smart plugs (a handful of HS110s and one Wemo) for 2.5 months.

The wemo is still producing a huge 10kW spike several times a day. The Wemo discussion boards are not active; I don’t expect Wemo to fix the issue on their end. I do still hope that Sense can mask the erroneous readings on their end. I assume others with Wemo smart plugs have observed this behavior.

  • I don’t have any Wemo smart plugs integrated with Sense.
  • No, I’ve never seen a ~10kW datapoint plotted in Sense’s device power meter.
  • Yes, I’ve seen such ~10kW spikes, but very rarely.
  • Yes, I’ve seen such ~10kW spikes…several a day.

0 voters

It looks like Sense is not forgetting about my HS110’s any more (or broadcast UDP packets are not being dropped within my home network…which I don’t think is the issue, as everything else on my network appeared to be working find during these events). I can’t catch this bad behavior via nightly Wireshark captures. I’ve gone nearly a month now where I have no obvious gaps in my HS110 devices’ power meter data. Whereas for the first 1.5 months, I would get frequent hour to day long gaps in data, even though during these times I could access the HS110 power data from the Kasa app.

  • I don’t have any smart plugs integrated with Sense.
  • I have never noticed Sense loosing contact with any smart plugs.
  • I have noticed Sense loosing contact with my smart plugs for several hours, but not recently.
  • I continue to have Sense lose contact with my smart plugs for several hours.

0 voters

My router still reboots itself several times a day due to some DHCP issue with my ISP, but the network is only down for about a minute each time, and Sense must have a feature to deal with ~1 minute loss of response from smart plugs, as I see during these ~1 minute events a constant power consumption even on smart plugs that normally have a noisy power signature.

Even better would be for the device to recognize its out of range value and attempt some internal error recovery.
It would be better if the smart plugs sent in addition to the watt reading a joule (watt sec) value that resets to zero when device turns on or at local midnight to start a new integration. With a joule accumulation it would enable backfilling of missed readings or estimated power values by interpolation. As for error spikes in the data from the smart plug that has to be a concern for the smart plug device manufacturer.

Interesting thought @keithmclaurin… The TP-Link power monitoring devices log daily and monthly energy usage (in Wh, not joules), so there is the possibility of doing the kind of checking and recovery you suggest. But that solution would be proprietary to just TP-Link/Kasa energy monitoring smart plugs.

Get Daily Statistic for given Month
{"emeter":{"get_daystat":{"month":1,"year":2016}}}

Get Montly Statistic for given Year
{"emeter":{""get_monthstat":{"year":2016}}}
 

If I ever see a string of big dropouts again, I might just sample the daily stats via this protocol, on the side to see, just to see what got lost. But I would really appreciate Sense just giving us NAs when the the data is missing.