Sense vs. Ecobee

[kevin1]

Now that I know that the correlation between Sense’s output and my utility’s net meter is quite good, on aggregate within 2%, I’m going to look at detection/identification using the same techniques.

https://community.sense.com/t/does-anybody-have-any-cool-graphs-or-analytics-using-sense-data-please-share/3302/8

I only have two sets of devices in my house that offer accurate logs of on/off behavior:

• 2 Ecobees, upstairs and downstairs that offer a full set of HVAC usage on a 5 min interval.
• Hue lighting - which is already integrated with Sense, so I’m not going to even do a comparison here.

I’m also going to give my conclusions up front as well.

• Sense has done quite a good job of identifying AC condensers in my household.
• But the data also exposes what I believe to be second order challenges of unsupervised machine learning.

So here’s the tale of a love triangle between AC Up, AC Down and AC 3, three devices identified in my house by Sense. AC Up and AC Down first showed up in Sept 17, after I did a Sense data reset in Aug 17. AC 3 showed up on the scene later, in Jan 18, and changed all the AC relationships in my household

Here’s what things looked like in the idyllic days of 2017. Most of the hours of AC runtime measured from my Ecobees correlated extremely well with the corresponding energy usage identified by Sense. AC Up energy from Sense correlated beautifully with CoolUp runtimes from my Ecobee. Ditto for AC Down and CoolDown.

Upstairs:

Downstairs:

So far so good. The correlation lines are clear, with very few missed identifications, the data points where Ecobee shows a non-zero value but Sense shows zero (the dots lying on top of the x axis). We can even discern the power usage of both AC condensers by looking at the y intercept at x equals 1 hour (3,600 seconds). The upstairs unit runs at about 3.5kW, the downstairs unit around 5kW. Everything is working like it is supposed to, though the number of points is really quite small since this is Sept - Dec is typically not AC season, though the upstairs does tend to run warmer than downstairs due rising warm air, and sunshine heating the stucco.

[kevin1]

Oops - Gonna have to tell this story a little differently. @AZAaron inspired me to look more closely at my AC devices, and I discovered that like any really interesting love triangle, there was actually a fourth party (device) involved as well. So first I need to introduce the players and their relationships.

It all started back in Aug 17 when I did a full data reset on my Sense, with the hope that identification would improve, especially in areas like AC. I should also mention that the folks at Sense gently discouraged me against the reset. I went ahead and was soon rewarded with reasonably correlating AC devices, AC and AC Down, which matched up nicely with thermostat commands CoolUp and CoolDown in earlier graphs.

Here’s what AC looks like today:

And AC Down (though I can’t guarantee that’s what they looked like when they were first detected).

A third device showed up in Sept 17, that mainly looked like a spike at the start of an AC cycle so I named it AC Spike:

When AC Spike showed up, it was almost always coincident with the running of the upstairs AC condenser and AC identification, so I merged AC and AC Spike into a composite device named AC Up in Oct 17. I have explicitly unmerged the two for purposes of this analysis, but they lived happily (or so I thought) merged together from Oct 17 until Sept 18.

Sense discovered a fourth device, AC 3, in Dec 18, one that looked like a typical AC (air conditioning) waveform

So which Sense AC devices represent real AC devices, and which ones go with which ? How do we figure out the correlations ?

[kevin1]

I would have been stuck sorting through reams of data and plots had not @duanetiemann had the bright suggestion to use Pearson product-moment correlation to look for the relationships. Pearson correlation coefficients can be calculated between pairs of data and the result will range between 1 and -1. A value close to 1 indicates that the two sets of data are strongly positively correlated - when one value is at the large end of its range, the same is true of the other value, and similarly for small values. A coefficient close to -1 indicates strong negative correlation where one set of data increases while the other corresponding data decreases. A value closer to zero indicates limited correlation and at zero indicates complete lack of correlation.

The good news is that R has some easy to use tools for doing precisely this kind of correlation analysis. I first decided to plot special correlation graphs, often called correlograms, to see how my 4 device identifications related to the upstairs and downstairs hourly runtimes.

First here’s CoolDown vs all the Sense-identified AC devices, for essentially the past year. The Sense device that correlates most closely with Ecobee CoolDown is AC 3. You can see that by scanning across the top row. Oddly, AC Down, the device that I named that way because it correlated with the running of my downstairs condenser back in Sept and Oct 2017, only loosely correlates with CoolDown in a year’s worth of hours.

Maybe it’s no surprise then, that AC Down and AC Spike best correlate with the CoolUp cycles of my upstairs AC condenser (below) over the past year, though the correlation isn’t nearly as strong as between CoolDown and AC 3.

Was I out of my mind when I merged AC and AC Spike and renamed them to AC Up ? I decided to look back on the same correlograms for the period from Sep 17 through Dec 17. So OK, I wasn’t crazy. Before AC3 came on the scene, CoolDown was indeed paired nicely with AC Down. There was also a looser correlation between CoolDown and AC, though this could be partially explained by the notion that when one AC unit is on, the other one is also likely to be on (people are at home and it is hot outside). AC Spike appears totally divorced from CoolDown.

And the same is true for CoolUp. AC appears fairly well correlated, but AC Spike seems a little more fickle. Even AC Down appears more correlated to CoolUp, than does AC Spike. So maybe I was a little hasty in deciding to merge those two.

What happened in the Jan through June to change things ? On the downstairs side, CoolDown, AC 3 clearly and quickly stole away the identification crown from AC Down.

On the upstairs side, CoolUp, AC Spike grew more attached to the actual cooling runtimes, while AC drifted away, even though they were merged. It also appears that AC Down began to substitute for AC.

Looking at the most recent data, Jun through Aug, shows a continuation of changing relationship dynamics. AC 3 is now solely wedded to downstairs Ecobee activity.

Upstairs not so clear, but AC is close to dropping out of sight and AC Down and AC Spike identifications correlate most strongly with upstairs Ecobee AC runtimes.

How and why AC 3 entered the scene when everything was so picture perfect in Dec 17 is a mystery, though those were months when the air conditioning never turned on. I’m also intrigued by the raison d’être for AC Spike, plus it’s role as a catalyst for change.

[duanetiemann]

I’m thinking there might have a better way to find devices that don’t have a consistent signature. Some things you can just turn off and on and see them pop up in Sense. Garage Door is like that.

But others are are murky, taking a variable amount through the day. Fridge, Dehumidfier, etc. You turn them on and see nothing in Sense.

I’m thinking I could get one of those smart plugs that record electricity consumption and correlate that data with Sense devices. So far, though, I’ve only found one smart plug that looks like it could do the proper recording, the Zooz Zen15. (Most?) All the others focus on turning off/on and do minimal historical recording. Looks like I have to also get some sort of Z Wave hub as well to get it connected up. It’s a little pricey, so I’m looking at just getting one and moving it around to interesting appliances every so often.

[kevin1]

All this discussion of correlation coefficients is a little abstract, though it provides a wealth of information. But now it’s time to get a little more concrete and look at a few comparison scatter plots where positive correlation is good, bad and ugly. A quick summary from ggplot2 (a package for R) can give us a little help. Here are plot summaries for every pairing of the AC devices with the upstairs (CoolUp) and downstairs (CoolDown) condenser runtimes, over the past year. We only need to look at the first column of plots for each, since that is the column that looks at Sense device energy results for each hour vs. Ecobee runtime. I’m using Ecobee runtimes as the golden reference since they are exact views on condenser activity. As usual, plots are somewhat colored by month to see correlations at different points in time.

Here’s the summary for the downstairs condenser - One can see the strong correlation line between CoolDown (Ecobee) and the Sense AC 3 device. Also visible is a slight line, in black between CoolDown and AC Down, representing the month or two of good correlation back in Sept 17. Points lying along the bottom of the plots in the first column of plots represent hours when the downstairs condenser was on, but the associated devices were not recognized.

The plot summary for the upstairs condenser shows a much less clear set of relationships, as expected. AC shows reasonable correlation with the upstairs condenser, CoolUp, for at least a couple of months (in black), but a lot of non-recognition as well (point on the bottom of the plot). And, as the correlograms indicated, there is also relatively strong correlation between the upstairs condenser runtimes and the AC Down device later the year being monitored. AC Spike is also well-correlated with the condenser, but shows a change in slope, really power usage over time.

What I want to do now is examine more detailed full year plots for the 5 pairings that showed strong correlation, at least in one point in time. First CoolDown and AC 3, no surprises. No identification in late 2017 through Mar 18. Great correlation after that. Looks like AC 3 runs at about 4.7kW.

CoolDown vs. AC Down is just the opposite. Good correlation in late 2017, but demising correlation thereafter, once AC 3 took control. Note all the 2018 vintage points on the bottom (missed identification of CoolDown) and on the left side (Sense identifying some other signature as AC Down). When it was recognized, AC Down looked like a 5kW device.

On the upstairs front, AC (part of the AC Up pairing or AC Up and AC Spike), correlated well from late 2017 into Apr or May 18, then pretty much gave up on any recognition (pink and purple dots on the bottom, missed identifications, but none on the left side, which would indicate identifying something else in those months). AC looks like a 3.5kW device.

Correlation between AC Down and the running of the upstairs condenser is much murkier. There’s a correlation line visible but an awful lot of missed identifications as well, all during the same time period. The correlation line points at a 3kW AC Down device, quite different than the original 5kW perspective on AC Down.

Final mystery - what’s happening with AC Spike and how does it relate to it’s original betrothed, AC and CoolUp, as part of AC Up ? Here’s CoolUp vs AC Spike. It’s the most mysterious since there is a visible correlation but the correlation is driven by a time-dependent set of lines (or one line of varying slope over time). When it first appeared in Sept-Oct 17, AC Spike appeared to be around 500W, but has diminished in consumption as time has gone on to about 250W.

I can see the drama over time now, but can’t really say why things happened. I do wonder if my AC condenser signature characteristics changed over time leading to a varying AC Spike and a swap of identifications. And did Sense introduce a smarter better model (AC 3) for my downstairs unit in Dec that eventually took over leaving my existing models to battle for “ownership” of the upstairs unit ?

Any ideas on how to probe more deeply ??

[kevin1]

I have tried out a couple of smartplugs that measure power and offer network connection for data access, both the TP-Link HS110 and the Elgato Eve Energy. I use them for monitoring my clusters of “Always On” consumers (modem, switch, TiVo, Hue bridge, etc.). Simple comparison below:

Biggest Eve Energy pro vs. TP-Link is that Eve Energy maintains a full energy history over time at 10 min intervals. The big con is that it must talk to the app via Bluetooth directly to update data logged by the device to either graphs or .csv, though you can control the plug remotely via Apple HomeKit. The biggest pro of the TP-Link is the WiFi connectivity and ability to report power remotely. Biggest con for my usage is lack of logging. You can only see 4 real unique values at any one time - Current Power, Total Consumption for the current day to date, Total Consumption for the past 7 days and Total Consumption for the past 30 days. Not so bad if things average out over time for what you are monitoring.

[rex]

Sense just won’t recognize my mini-split air conditioners. I have 25 devices found but not my three mini-splits.

[duanetiemann]

Thanks. The Eve sounds perfect. The Apple angle is a hurdle for me, though. Everything we have is Android or Windows.

I bit the bullet and ordered the Zooz and a Z-Wave hub. I’ll send them back if I can’t get them to work.

I have the HS110, but would need to dive into its interaction with Android to get the data logged as it doesn’t do it out of the box. I could do that on Windows, but haven’t gone there on Android.

[duanetiemann]

I’ve had Sense since April and it is still discovering devices. I don’t think it saw the air conditioners right away. But Sense support would know if that is suspicious or not.

[duanetiemann]

Returning the Zooz and Vera plus Z-Wave hub. You can’t export the data out of the box.

[kevin1]

OK, another month has gone by since I last looked at correlation between Sense AC detections and Ecobee cooling runtimes. And guess what ? I now have one additional Sense device to look at, AC4, that made its appearance in July in my export data, but wasn’t visible until Sep. via Sense.

Let’s start with the downstairs system, where Ecobee runtime is a variable called “CoolDown”

Starting in Sep’17, the Sense ACDown device is well-correlated with CoolDown (the first column of graphs is really the only one that matters), running at about 4.5kW. AC3 and AC4 devices won’t appear for a number of months so they show up as zeroed out lines.

Moving to Oct., ACDown continues to be correlated with CoolDown, but the number of AC runs have fallen off sharply. The number of AC days drops to near zero after Oct. and there won’t be enough points to chart again until May’18.

By May, AC3 has appeared and seems to be the main device associated with CoolDown. The AC3/CoolDown relationship seems to get further cemented in Jun-Jul’18, though AC4 starts to make its presence known in Jul.

Surprisingly, in Aug and Sep’18, correlation vs. AC3 and AC4, yields to a renewed association with ACDown, plus a new attachment to ACSpike. ACDown now appears to be a 3kW device with ACSpike coming in around 200-300W.

[kevin1]

And here is the same set of shifting correlations for the upstairs system, with the Ecobee runtime variable “CoolUp”

For Sep-Oct’17, CoolUp is well correlated with device AC, plus partially correlated with device ACSpike. AC runs just a little over 3kW, while ACSpike is about 400W.

The upstairs AC usage pretty much stops for Nob’17-Feb’18, but shows up enough in Mar’18 to look at the charts again (the upstairs gets warmer in the sun, much more quickly than downstairs).

With the start of the new cooling season, AC and ACSpike are still closely correlated with CoolUp.

But as we roll into May’18, device ACDown begins to correlate more closely with CoolUp. Is it just correlation because both AC units are running or is there real causation ??

That seems to get answered in Jun-Jul’18, when ACDown has much denser correlation with CoolUp than AC, which is zeroed out for much of the CoolUp runtime.

Oddly, as we get into Aug’18, AC3 and possibly AC4 take over association with CoolUp runtimes, as ACDown and ACSpike reconnect with CoolUp.

My take on all this correlation shifting is that my two real AC devices, which are both 20 year old single-stage Bryant Seer 11.5 units, must have varying signatures with the seasons.