How to Tell When Your Sense is Lying to You

Chapter 1 - Introduction

I love my Sense ! In general it is a very accurate tool for looking at my energy consumption, especially when coupled with smart plugs, DCM, etc. But even though I trust it, the old saw goes, “Trust, But Verify”. I decided to look back over my almost 2 years (16,090 hourly observations) with my current Sense to see if I can find discernible patterns of when it led me astray, especially with the most basic of measurements.

The plot below highlights the comparison of Sense Net usage (Total Usage + Solar Production) vs. my utility’s net meter, based on hourly exported data. Perfect correlation between the two would result in all the points being along the perfect 45 degree “utility line”. Instead, there are bunch that a way off, and then another set that are long the “unity line”, but above or below it enough to represent a significant error. I have colorized the points based on the following conditions, to highlight some of the most common indicators that Sense isn’t likely giving you the correct result:

  1. Total Usage missing (NA) - there aren’t any points with this condition since I can’t really even chart. These probably occurred during power outages - 9 Hours
  2. Total Usage Negative - This has happened to me, mostly when my Sense Monitor has bugged out. A power cycle has typically fixed. - 18 Hours
  3. Solar Production missing (NA) - I’m not sure under which situations Sense fails to export a solar value, but I can investigate later - 247 Hours
  4. Solar Production Positive (and above 5W) - This would mean my inverter is ingesting power rather than producing it. This has happened to me, mostly when my Sense Monitor has bugged out. - 26 Hours
  5. Always On missing (NA) - Always On is a good indicator that something is wrong with Sense data collection - it is extremely responsive to short burst of bad data. - 850 Hours, though a few chunks of these (24 hours at a stretch), might be from Sense refilling it’s Always On measurement pipeline.
  6. Delta in Always On between consecutive hours greater than 100W. A sudden change in Always On, especially downwards, is a good hint that Sense might have lost some data readings during that hour. - 67 hours
  7. Nominal - When everything seemed to be working normally

It’s very easy to see that most of the very large errors occur during the top 5 conditions and that almost all “Nominal” points are pretty much on the unity line, thought there are a few that need further investigation.

If I look at just the points that have passed my high-level “lie-detector”, just the hours / points marked nominal, I still see a few points with substantial errors, though the “lie-detector” has screened many of the most egregiously bad readings out.

And if I look in greater detail at the hours with the largest absolute errors (absolute value of difference between Sense net and PG&E net), I see the largest errors are clustered in time in July, as the brownish yellow points also indicate.

It looks like there are a couple more indicators I can add to my “lie detector”.

  1. If the hourly Always On is negative, I really can’t trust the Sense result. That seems to be a reasonable assumption, and would remove most of the July problem points.
  2. I’m also speculating that if the Always On delta is NA (that means the previous Always On was missing - NA), that is a likely indication of erroneous data as well.

Now some of you out there might be saying, "how can you be so sure that these are reliable indicators ? Maybe many of the points with these attributes are actually the “truth”. Don’t worry - I’ll look at the reliability of each these detectors a little later. Right now, I’m going to see if removing those points improves the accuracy against the “unity line”.

Chapter 2 - Improving the “Lie Detector” and testing Reliability

I have now added the two new “lie detector” indicators to my list in the priority order below. Data points can only have one status and even if there are multiple triggers, data gets the status with the highest priority.

  1. Total Usage missing (NA) - there aren’t any points with this condition since I can’t really even chart. These probably occurred during power outages - 9 Hours
  2. Total Usage Negative - This has happened to me, mostly when my Sense Monitor has bugged out. A power cycle has typically fixed. - 18 Hours
  3. Solar Production missing (NA) - I’m not sure under which situations Sense fails to export a solar value, but I can investigate later - 247 Hours
  4. Solar Production Positive (and above 5W) - This would mean my inverter is ingesting power rather than producing it. This has happened to me, mostly when my Sense Monitor has bugged out. - 26 Hours
  5. Always On missing (NA) - Always On is a good indicator that something is wrong with Sense data collection - it is extremely responsive to short burst of bad data. - 850 Hours, though a few chunks of these (24 hours at a stretch), might be from Sense refilling it’s Always On measurement pipeline.
  6. Always On is negative, I really can’t trust the Sense result. That seems to be a reasonable assumption, and would remove most of the July problem points. - 198 Hours
  7. Always On delta is NA (that means the previous Always On was missing - NA), that is a likely indication of erroneous data as well. - 872 Hours
  8. Delta in Always On between consecutive hours greater than 100W. A sudden change in Always On, especially downwards, is a good hint that Sense might have lost some data readings during that hour. - 194 hours
  9. Nominal - When everything seemed to be working normally

If I look at the accuracy picture with the newest indicators, it sure looks like I have picked off most of the problem points and that the “Nominal” points all seem to live along the 45 degree unity line.

But the improvement is even more apparent if I look at only the Nominal point colorized by month. There are still some problem points, even within the Nominals, but far less than earlier.

To see how reliable an indicator of errors (lies) each of these is, I need to look at the distribution of the difference in kWh for each hour, between Sense and my “golden” PG&E meter, for each type of indicator. I could use histograms, but I’ll use something similar called a Box Plot.

image

The Box Plot reveals a wealth of information about the error distribution (PG&E Net - Sense Net). Consulting the handy reference on Box Plots, one can see that half of the population lives inside the the box around the median line, with the apple and lower ends of the box delineating quartiles of errors. And that the ends of the mark the end of normal errors and the start of statistical outliers. So the two Box Plot attributes that tell the severity of errors in particular category vs the golden measurement are the size of the box and how far the median (middle of the box) is offset from zero. Big boxes or a big offset or both, indicate big errors. An indicator where the offset drags the minimum or maximum marker past zero is an indicator that detects substantial Sense errors pretty much 100% of the time.


It’s clear from the chart above that if your Solar Production is positive more than some threshold amount, have you have hours with Missing Solar Production (NA), or you are seeing negative Total Consumption, Sense is giving you a wrong answer. No surprise in any of those. And the first two indicators are likely to give Sense a larger net reading than your net meter, while the latter will make your utility net greater than your Sense net reading. These indicators are 100% reliable since even the min/max for those distributions is greater/less than 0. You’ll also notice that “errors” tend to be large, plus or minus several kWh for an hour.

The other indicators look like they are only occasionally reliable. There are definitely outliers that can lead to large errors, but the outliers only occur some small percentage of the time. And the main body of the error distribution still looks very close to zero for all four. If I really zoom in on just those four indicators to see the main body of the distribution, maybe I’ll get a better picture of what is going on:

Here is zoomed in around 0 by 10x:

Here is zoomed in around 0 by 100x:

Zoom in close enough and indicators do show some more specific trends. Having a missing Always On the hour before typically will be associated with a small error where the Sense net is slightly smaller than my net meter. The same for situations where Always On is negative, but it also looks like that indicator also casts off a fair number of outliers in that same side. If situations where there is a large Alway On delta or where an hourly Always On reading is missing (NA) tend towards narrower and more symmetric error distributions, but it looks like the Always On mission case kicks out lots of outliers.

Chapter 3 - But How Different are Readings with “Lie Indicators” from Normal ?

Some of you are probably asking the same question I am at this stage in the game. How do the error distributions with “Lie Indicators” compare with the normal Nominal readings ? Are they really that different ? The following are the same Box Plots with Nominal included, but in reverse zoom order (close-up first).

100x

10x

1x

I draw two conclusions from these charts.

  1. Always On missing (NA) is Not a good indicator. Its error distribution looks virtually the same as Nominal. Previous Always on missing is a better indicator, because that distribution is skewed. It might also be that I should be looking at next Always On missing (NA) as na indicator. And Large Always On Delta is only a slight indicator of error bias.

  2. I should probably look closely at a the few far outliers in the bottom chart in the Large Always On Delta, Previous Always On NA, Always On Negative and even Nominal categories to see if there are any other clues.

Chapter 4 - Does Time Heal all Wounds ?
The last exercise got me digging into the details of hours with missing Always On, because they tended to happen in long strings, especially in the first few months of using my new Sense in April-May 2019. That inspired me to look at the count of hours in each status category over time. It’s hard to compare the exact numbers because only one year below is complete, but it is pretty clear that virtually all the “lie” indicators have fallen off over time. That’s great news for Sense users, BTW.

Screen Shot 2021-02-18 at 7.25.51 AM

And if I look more closely, month by month, it becomes apparent that the biggest bursts of “lie” indicators are clustered into just a few months. I can identify the root causes of a couple of those bursts. I had a second Sense installed on May 5th, 2020 and converted that second Sense from 2x200A operation to DCM during Oct 2020. Both entailed turning off the power for a stretch.

Looking at the unity line comparison for only the Always On indicators, down in the region where they seem to be indicating many errors, for all the data and just the data after the start of 2020, shows a marked improvement, especially if I rule out a couple problematic data points.

Not so great including 2019

Much better if I look beyond 2019

The difference that filtering out 2019 makes, is also apparent in Box Plots. When I filter out 2019, two of the categories drop out. 2020 and beyond, no longer contains any missing Solar Production (NA) or Negative Always Ons.

Including 2019 - 100x

Without 2019 -100x

One final graph illustrates what I believe to most of the signs that your Sense might be giving you bad data. It’s the same post 2019 Box Plot above but with the box width scaled to the size of the population within that box. That gives us a better idea proportionally of how many errors each indicate might be contributing.

As one would suspect, the Nominal box is by far the largest with Always On missing (NA) as the second biggest. Both of those show a tight error box with the median centered around 0, though with a substantial number of roughly symmetric (around 0) outliers. The outliers need to be investigated, but Nominal and the Always On NA indicator both look like non-indicators. The smaller Large Always On Delta, Next Always On NA, and Previous Always On NA have much wider boxes and skewed medians, though the max / min never extends beyond zero, telling me that this are probabilistic indicators of Sense errors, but never with 100% confidence. Solar Positive and Negative Consumption (negative Total Usage) are high confidence indicators that Sense is generating an error relative to your hourly utility reading.

Chapter 5 - The Importance of Being Anal
Some of these indicators are obvious (negative Total Usage, positive Solar Production above a threshold, negative Always On), but others require more work to detect. Here are some hints on how to dig out the trickier indicators:

  1. Absence of data - many of the indicators are linked to missing data (NA). One easy way to find NA’s is to use a Pivot Table in Excel (or cast in R/Python) to sum data by DateTime (hour) and by select device names (Total Usage, Solar Production, Always On). The Pivot Table or cast will use all the DateTime values in the dataset and will fill any empty DateTime / name pairs with either 0 or NA, depending on settings. I recommend NA since that is very different than zero. And don’t forget to look for any DateTime hours where there is absolutely no exported data, including Total Consumption - that can happen and that’s a big error, too. Just don’t let the missing hour from the “spring forward” switch from standard time (ST) to daylight savings time (DST) fool you, since it looks like a missing hour.

  2. Pay attention to duplicated DateTimes / name entries - there are a couple of reasons that you might get multiple DateTime / name pairs in your dataset and those will get summed together giving you 2 hour values, and will likely show falsely as having Large Always On Deltas. The two situations I discovered are:

  • Jan 1 at 00:00:00 for both 2020 and 2021 - These are all cases where I blame Sense export :wink: plus sloppiness on my part, when laminating multiple years because Sense includes that hour in both the prior and current export years. These two showed up initially as big errors because the Sense data is doubled while the utility net is not ! Since Sense associates data for an hour with the start time of that hour you need to remove the midnight (00:00:00) reading form the end of each year.
  • Nov 1, 2020 at 02:00:00 and Nov 3, 2019 at 02:00:00, where Sense exports a second 2AM hour that happens when we fall back from daylight savings time. One of those hours is actually 2AM DST and the next one is 2AM ST, but Sense doesn’t differentiate in export. In this case you spend a lot of time trying to separate all the data for the two hours, but it’s probably OK as long as you do the same with the utility data.
  1. Transition Delta / Detectors - You don’t have to get too fancy with Excel or R/Python to create flags for the Large Always On Delta or the Previous / Next Always on NA. you just need to compare Always On against the upshifted/downshifted and padded versions Always On.

  2. Prioritizing the indicators in order of the their likely error outcomes is important, since each hour can only have a single Status. I was lucky to stumble on a pretty good ordering.

  3. Making sure the utility is really “golden” - Sometimes the data from a utility can also be “wrong”, typically when the power goes out or the data network that passes meter readings back to the mothership goes out. The good news is that my utility, like most, will tell you when it does an estimated read vs. an actual read. In my case, PG&E will add an “estimated” in the notes column of the 15 min data (some PG&E customers get 30 min data instead). So I have high confidence that the utility data is very solid.

Chapter 6 - A Look at the Data After the Detected “Lies” are Gone
So I did all that meticulous work for two reasons:

  1. To pass on my understanding to help others - As I said, I love my Sense, but I really hate it when it gives me easy to detect “bad data”. So I decided to nail down some of the best indicators to help myself and others.
  2. So I can look more closely at the remaining errors and see if I can identify the original of those.

To reiterate, thanks to the the removal of hours with obvious error indicators and the data from 2019, I have been able to move from this:

To this !

When I look at the linear model and residuals for the remaining data, there are really only 3-10 points outside of what I would regard as the statistical limits of the error distribution, for me to poke into.

If I look at the biggest error/residual hours remaining, all but 3 (circles in green) of the top 10 are explainable due by the installation and playing around with my second Sense.

And the error on 1-20-2021 aligns with me rebooting my Sense because it “lost” a few of my Kasa smartplugs.