Making Sense of 'Always On' - Chapter 1 - Different 'Always On' Calculation over Time?

[kevin1]

Finally figured out a way to locate most Sense monitor errors without comparing with power company results by using hourly Always On here:

Making Sense of 'Always On' - Chapter 2 - 'Always On' Blips

A little more fun with Always On blips - Downward spikes in hourly Always On are definitely a good indicator of Sense monitor issues, either from a data transfer or a data read perspective. I very carefully analyzed my Power Meter waveforms from Jan to mid-Aug 18 for two types of monitor issues, data dropout and negative Total Usage. I then mapped those issues vs. downward spikes in Always On. The X’es below represent downward spikes from the 12 hour moving average that is greater than 80W. Th…

[Dcdyer]

@kevin1 I did a data review on my “Always On” values for 2018.

1. In the first graph, you can see that my “Always On” increased over the year. I attribute this to more Smart plugs and switches having been added to the house.

2. In June we took a vacation and powered off all the laptops and devices (external disk drives) that are normally running and you can see a drop in the “Always On” values. There was even another small drop in the June time period because an electronics cooling fan controller failed and I lost a power supply to a security camera at the same time.

3. It appeared that there were more values that were outside the base-line during the first 6-months of the year, so I created 2 separate histograms.

There were 538 outliers in the 1st-half of the year versus 348 outliers in the 2nd-half of the year. It appears that SENSE is doing a better job of calculating the “Always On” values.

I do not think that I made any major changes to my devices that were plugged in (and Always On) that would have created this difference. I do see a cycling pattern in the data. It is possible that a change in the “Always On” calculation occurred by the SENSE programmers. I expected to see a single bell curve in the histograms, but both charts show a double curve. I did not try to equate the ‘outlier data’ with missing downloads (or reject any data points).

You asked for some interesting data charts and I had not seen anyone publish their data in this type of presentation. Comments?

[kevin1]

@Dcdyer,

Thanks for publishing. A few thoughts from my perspective:

1. Take a look in the Power Meter at your outlying hours on the low end. I guarantee that many of them will have at least one data dropout in that hour.

2. Is see similar behavior in my chart - lower baseline before summer 2018, higher after summer 2018, but the similarities stop there.

• I have spikes for Halloween '17 (inflatable black cat and pumpkins on 24/7), plus winter break, spring break and summer break when my son was home from college
• Also have a weird period from Mid Aug through Mid Dec where I think Sense was playing with the Always On calculation. Starting in the beginning of Nov’18, Always On with smart plugs was added into the equation.
• Around Dec 20th, things wen crazy and I started seeing many data outages which affected Always On.

3. I’ll try to do histograms for my data, to see if my distributions are also bimodal, but I’m going not have to think about where to draw the cut lines. Any thoughts ??

My data below:

• Red dots are aggregated daily Always On totals
• Blue dots are hourly Always On x 24 (to normalize with daily)
• Green dots on the bottom indicate that there was some good data for that clock hour (still could be a data dropout for part of that hour).
• Orange dots at the top indicate data was missing for an entire clock hour.

What do we see ?

• A long stable period from Aug 17 until June 18, where Always On mostly stays in a stable zone, though the number of hourly dips in Always On increases after March 18.
• A wilder period of swings between June 18 and mid-Aug 18.
• A very flat, low period between mid-Aug and Mid-Sep 18
• A resumption of higher Always On with wider swings from mid-Sep to the end of Oct 18.
• The start of the smartplug beta at the start of Nov 18, with a falling Always On as smartplug always on data gets pulled out of Always On. More dropouts and outlier low Always On hours.
• The start of a new Always On calculation around Dec. 20. Lots of hourly dropouts and shorter dropouts as well.
• Starting Jan 1, 2019, I did several things to reduce dropouts and they seem to have partially worked.

[kevin1]

BTW - I’m still going to do the histograms, but I just put my nose back in my statistics books to try to figure out whether the central limit theorem is applicable to Always On. Also trying to remember the tools or find new ones that would enable me to separate the data into stuff that should fit a normal distribution and the part that shouldn’t, or would fit a different normal distribution. My current belief is that for populations (time periods) where the Sense Always On calculation remains stable, plus for the data points that don’t have data dropout, they should fit a normal distribution.

My histogram of all Always On hours is a mess…

[kevin1]

Since I have been on a bender removing hourly datapoints that have some dropout in them, I’ve been able to dig a little deeper into my weird Always On distributions.

Here’s the original data covering 17 months… Remember: blue is hourly data here, red is aggregated daily. The orange dots on the top represent hours where no data was available (a big data dropout).

Same chart with good hourly data points in green, data points subject to data dropout in orange:

Here’s the histogram of all data, including dropouts:

Here’s the histogram of all data, with the dropouts removed. Not much difference except at the low end.

Next I I broke up the timeline into the 5 domains I talked about earlier.

• Original - The original Always On calculation
• Crazy - A period when the Always On calculation appeared to go crazy
• Better - the subsequent period when things appeared to get better
• Smartplug1 - Sense’s first round of adding smart plug data into the Always On calculation. Started with the beta of SmartPlugs
• Smartplug2 - The approximate time Sense updated the Always On calculation with smart plugs

Each period seems to have its own distribution. Some might even have more !

And here are the histograms for each of those time periods:

Now I have to think about whether any of this has any meaning

ps: Looks like I may have entered a new domain after Smartplug2, Smartplug3 ! Starting to see smartplug results center around 209W.

[Dcdyer]

@kevin1 That is a lot analytical information on your “Always On” values!

1. After you excluded the ‘dropouts’, your data is fairly constant until it went “crazy” (as you described it.). Hopefully your values are returning to a more normal daily pattern.
2. What method did you use to determine when a ‘data dropout’ occurred? My Power company’s meter (which I used earlier to determine SENSE had bad data) only supplies me with daily information, so I don’t think I can determine bad data on the hourly basis that you are doing. You mentioned that you are using a secondary energy monitoring system to check your home usage.

I was not certain what my charts would reveal about my “Always On” values (hourly data).

1. I think that data dropouts early in 2018 were creating some bad data points for me. In March, I installed a delay-timer relay (set at 4-minutes) to assist with the SENSE reboot after a power outage. I lose 6 minutes of data (4-minutes for the intentional delay, 2-minutes for the SENSE unit to reboot/reconnect) after a power outage, but that is better than hours of lost data. (Or having to manually reset the breaker.)
2. There is a slight trend downward in the first half of 2018 on my data because SENSE was still identifying new devices. (My guess?) After 7/2018, I have not had many new devices identified.
3. Turning off un-need equipment while I was away from home in June showed an expected drop. It wasn’t much, but every little bit helps.
4. I am still trying to decide if this graph actually reveals any trends or problems.
5. I did develop an EXCEL spreadsheet for myself where I listed every item in my house that is always plugged in and using power, then assigned an estimated wattage. Some data I gathered using a ‘Kill-A-Watt’ device, some data was taken by looking up the manufacturer equipment specs and some was just an educated guess.
   My estimated “Always On” value is 330 watts.
   My SENSE “Always On” value is 348 watts.
   I think that my estimate is a close ‘ball park’ guess to what SENSE is calculating.
6. Special Note: I have 93 separate devices that are always pulling power. I counted every item that pulled even the smallest amount of power (even the ‘lighted doorbell buttons’).

The only reason I can think of for the fluctuating “Always On” pattern is there are days when we tend to use our computers more and access the external drives. Maybe the modem and routers consume more power when Web searches (or Netflix) are being used. I’m just guessing at reasons. I haven’t run any tests.

We do know that the “Always On” value is an averaging calculation so you would expect to see the data points being ‘smoothed’.

Thanks for providing us with a glimpse at your data and working to interpret what occurred during those times.

[kevin1]

Thanks as well for sharing yours. Your Always On appears much better behaved and predictable than mine during 2018, especially since you can correlate the what looks to be a downward slope in the beginning of the year with you actively doing Always On reductions. I’m guessing the bimodal pattern indicates hidden changes in the Always On algorithm. And that your low end outliers during the first 6 months (and second 6 months), are really data dropouts.

As for your questions about mine:

1. I do another export in a couple days - what I see from the bubbles is that my new normal is around 210W, but sometimes export reveals stuff the bubble does not.

2. As for finding dropouts, I cooked up a web app data scraper that looks at the Power Meter for every day, and counts the number of dropout events and negative usage events visible on the screen. Finally have a way to automatically find virtually all the dropouts. More here, including code:

Scraping the web app Power Meter for data analysis

One final chapter on this enterprise. I invested a bunch of time in “finding” data dropouts, because I kept seeing the impact of dropouts in Sense exported data. “Always On” data is exceptionally sensitive to dropouts, though I don’t know enough about how it is calculated to understand exactly why. But anomalies in “Always On” gave me a way to pretty much automatically locate 334 hours over the past 535 days, that included one or more data dropout events. In the chart below, I overlay the num…