Sense vs. Ecobee: Round II

The first thing I’m going to do on the Ecobee side is to look at the reliability of my data. Three things are different about Ecobee downloaded csv data vs. Sense

  1. The data comes with 5 minute granularity - good !
  2. The data also comes with better data integrity checking - if data is missing for a particular time interval, Ecobee still supplies the time / date with empty data for that 5 minute period, so you have to make fewer assumptions about that data - it really is NA, and not potentially 0. With Sense, a data dropout or an actual 0 usage (no detection) hour/date is skipped for that particular device - good !
  3. Unless you become a developer and use the API, data can only be retrieved a month at a time - not so great vs. Sense.

The chart below shows the number of Ecobee 5 minute dropouts per particular day. There are 72 upstairs and downstairs dropouts today, Aug 4th, because there were 6 hours of uncollected data left in the day when I did my download. But you can also see that both Ecobees have had data lapses, with many more occurring downstairs, and most before late May 2019. I know I have had dropouts when my downstairs system is in heating mode because my furnace was tripping the high-temp limit switch due to restricted airflow (hopefully fixed in July).

The next step in puzzling through the NAs is to try to figure out what the Ecobees were trying to do just prior to going NA. If I look at the table for each thermostat, the profile of states prior to NA (in red) are different in one respect, but similar in several others.

  • The downstairs Ecobee has a far higher percentage of heatOff to NA transitions (319 vs. 4). I believe I understand the cause of these - my furnace shutting down due to overtemping.
  • But both Ecobees have a significant number of NA to NA transitions, though downstairs has far more. That might be related to the same shutdown phenomenon lasting more than 5 minutes.
  • Most of the transitions are on the diagonal - staying in the same state, especially in the upper left (off states, plus cooling).

BTW - heatOff and compressorCoolOff are really similar off states, with the only key difference being whether the Ecobee is in heat mode or cooling mode based on the sensors temperatures WRT to the set points.

Next thing to do - look more closely at NA to NA patterns to see hoe they start and how repetitive they are.

It’s been a while because I’ve been trying to find ways to analyze the output of my Ecobee, including the periods when it has been disconnected for various reasons and output NA for those 5 minute periods. I didn’t know whether I could completely ignore the NAs or how I should treat them. Eventually I figured out a way to filter the Ecobee data by reconverting it to transition events (rather than 5 min polling data). I end up with four kinds of transitions for both heating and cooling transitions:

  • Turn-on - the heat/cooling turns on
  • Turn-off - the heat/cooling turns off
  • Spike - the heat/cooling goes on for period of time that is shorter than 10 minutes, and longer or equal to the time reported by the Ecobee, but unknown due to the Ecobee dropping out)
  • Dropout - the Ecobee drops out for less than 10mins between two known cycles (reverse of a spike)

First off is my downstairs heating cycles. I have lots of spikes because the thermal protection in my furnace was kicking off the furnace controller every 6 minus of runtime or so due to insufficient airflow. I finally fixed this problem this summer, when the HVAC guy installed new AC compressors. You can see the extend of my heating season from the time my new Sense was installed (early April) until no heating was needed (May 28th)

Here’s a more normal set of cycles from my upstairs heating. Only one legitimate spike.

And here is a view of the cooling cycles. You’ll notice the upstairs is more active than the downstairs because heat rises through our house and cold air sinks, especially via our two story entryway.

CoolUp

1 Like

Nice! So now if you super-impose the four and use color?

The “Down/Up” terminology threw me at first … that also implies a transition state :wink:

1 Like

Well, cooling season is almost over and I decided to do one more monthly correlation analysis between my Ecobee’s hourly cooling runtimes and the power consumed by various Sense AC detections, AC, AC1 and AC2. FurnaceDown and FurnaceUp are hourly power used by the smarplugs monitoring each of my furnaces (mostly the furnace blower used by AC during the summer). Remember that I reset my Sense unit to ground zero in April, then put in new AC compressor units in mid-July. It’s interesting how the detections sort themselves out at the end of the cooling season.

Here’s the story for downstairs…

  • No detections for AC in April
  • Two AC detection in May, AC and AC2 - “AC” is most closely associated with the downstairs compressor
  • New compressors in July upset the status quo. A new detection, AC3, aligns with the downstairs compressor and correlation continues to grow over time.
$`2019-04`
            Down_Cool
AC                 NA
AC2                NA
AC3                NA
FurnaceUp          NA
FurnaceDown        NA

$`2019-05`
              Down_Cool
FurnaceDown  0.96457804
AC           0.40228237
FurnaceUp    0.38490212
AC2         -0.02461567
AC3                  NA

$`2019-06`
            Down_Cool
FurnaceDown 0.9923947
FurnaceUp   0.6044697
AC          0.3183210
AC2         0.2872561
AC3                NA

$`2019-07`
             Down_Cool
FurnaceDown 0.98625399
AC3         0.56490245
FurnaceUp   0.51139072
AC2         0.32950875
AC          0.06935545

$`2019-08`
            Down_Cool
FurnaceDown 0.9939775
AC3         0.7953435
FurnaceUp   0.6062248
AC2         0.1969622
AC          0.1884249

$`2019-09`
             Down_Cool
FurnaceDown 0.98889843
AC3         0.95784466
FurnaceUp   0.39951440
AC2         0.36528948
AC          0.01010691

Here’s the story for upstairs…

  • No detections for AC in April
  • Two AC detections in May, AC and AC2 - “AC” correlates very closely with the upstairs compressor runtimes. Why was AC2 even created in the first place ???
  • New compressors in July upset the status quo. A new detection, AC3, aligns with the upstairs compressor initially, but eventually lonely AC2 muscles in and correlates best with upstairs.
$`2019-04`
               Up_Cool
AC           0.9791817
FurnaceUp    0.9769824
FurnaceDown -0.1541840
AC2                 NA
AC3                 NA

$`2019-05`
                Up_Cool
AC           0.99412114
FurnaceUp    0.97723727
FurnaceDown  0.14122794
AC2         -0.02226503
AC3                  NA

$`2019-06`
              Up_Cool
FurnaceUp   0.9892451
AC          0.6702095
FurnaceDown 0.5909811
AC2         0.1680601
AC3                NA

$`2019-07`
              Up_Cool
FurnaceUp   0.9850746
AC3         0.5987115
FurnaceDown 0.5123134
AC2         0.3918143
AC          0.1948437

$`2019-08`
              Up_Cool
FurnaceUp   0.9980361
FurnaceDown 0.6066122
AC3         0.5671459
AC2         0.4513146
AC          0.1011073

$`2019-09`
              Up_Cool
FurnaceUp   0.9990777
AC2         0.8910393
FurnaceDown 0.4199229
AC3         0.4070204
AC          0.2234853

I keep promising myself that I’ll delete AC to help improve things, but I kind of want to see if it goes away on it’s own. If I look at the amount of energy that was ascribed to each detection on a monthly basis, I can see AC is on a downward trend, but hasn’t found a route to disappear yet.

    Month      AC     AC2     AC3
1 2019-04  24.005   0.000   0.000
2 2019-05  84.903   0.596   0.000
3 2019-06 314.848  48.725   0.000
4 2019-07  44.516 151.708 260.085
5 2019-08  12.921 215.626 399.085
6 2019-09  16.744 125.794  61.367

3 Likes

One final readout at the end of Sept. I really can get rid of the AC device !

Downstairs correlation by month. AC3 eventually correlates well with Ecobee cooling - 0.94946833

$`2019-04`
            Down_Cool
AC                 NA
AC2                NA
AC3                NA
FurnaceUp          NA
FurnaceDown        NA

$`2019-05`
              Down_Cool
FurnaceDown  0.96457804
AC           0.40228237
FurnaceUp    0.38490212
AC2         -0.02461567
AC3                  NA

$`2019-06`
            Down_Cool
FurnaceDown 0.9923947
FurnaceUp   0.6044697
AC          0.3183210
AC2         0.2872561
AC3                NA

$`2019-07`
             Down_Cool
FurnaceDown 0.98573638
AC3         0.56490245
FurnaceUp   0.51124528
AC2         0.32950875
AC          0.06935545

$`2019-08`
            Down_Cool
FurnaceDown 0.9939775
AC3         0.7953435
FurnaceUp   0.6062248
AC2         0.1969622
AC          0.1884249

$`2019-09`
              Down_Cool
FurnaceDown  0.96806223
AC3          0.94946833
FurnaceUp    0.56990137
AC2          0.52039402
AC          -0.01887556

Upstairs correlation by month. AC2 eventually correlates fairly well with Ecobee cooling - 0.8995683

$`2019-04`
               Up_Cool
AC           0.9791817
FurnaceUp    0.9769824
FurnaceDown -0.1541840
AC2                 NA
AC3                 NA

$`2019-05`
                Up_Cool
AC           0.99412114
FurnaceUp    0.97723727
FurnaceDown  0.14122794
AC2         -0.02226503
AC3                  NA

$`2019-06`
              Up_Cool
FurnaceUp   0.9892451
AC          0.6702095
FurnaceDown 0.5909811
AC2         0.1680601
AC3                NA

$`2019-07`
              Up_Cool
FurnaceUp   0.9850562
AC3         0.5987115
FurnaceDown 0.5131142
AC2         0.3918143
AC          0.1948437

$`2019-08`
              Up_Cool
FurnaceUp   0.9980361
FurnaceDown 0.6066122
AC3         0.5671459
AC2         0.4513146
AC          0.1011073

$`2019-09`
              Up_Cool
FurnaceUp   0.9810785
AC2         0.8995683
FurnaceDown 0.5922095
AC3         0.5786095
AC          0.1425038

1 Like

Here’s a more graphical view on how Sense detections eventually hone in on the actual operation of my upstairs and downstairs compressor units as measured by the Ecobee thermostat commands. The number shown is the weekly Pearson correlation between hourly Sense detection energy usage and my upstairs and downstairs Ecobee cooling runtimes. A correlation of 1 means that there is a perfect linear correlation between energy usage and runtime. 0 means no correlation and a negative number means negatively correlated.

Upstairs Compressor
The upstairs compressor is on for more of the year since my upstairs heats up more (hot air rises, cool air falls and I have a big open entryway). Initially Sense detected my upstairs unit as device AC and had good correlation for a while until the unit was running pretty much all day long. Then, for some reason, a new AC detection from early May, AC2, that hadn’t correlated to anything particularly well, suddenly started matching more with my upstairs usage (and downstairs as well). Then, around July 10th I replaced both compressors and Sense was forced to sort thing out again. Eventually AC2 matched up with my upstairs compressor and AC3 matched up with my downstairs unit.

Downstairs Compressor
My downstairs compressor turns on much later in the season because my downstairs stays cooler.

The best part, is that I actually get a view that Sense is getting smarter as time goes by, despite my efforts to confuse it (new compressors).

EcobeeVsSense3.R (5.2 KB)

1 Like

Put differently, if I look at my results from Sept. 1, and onward, where Sense correlation with my Ecobee is relatively good, I can compute each AC units’ estimated hourly power usage by dividing hourly energy from Sense for AC2 and AC3 devices by the corresponding Ecobee runtimes (in hours). If I look at the distribution of results, I see very reasonable answers for each unit.

My upstairs 3 ton compressor uses about 2.1kW

My downstairs 4 ton compressor comes in at about 3kW

1 Like

One more way to view Sense vs. Ecobee. This one can be done in Excel with a little ingenuity - Detected hourly power usage vs. Ecobee runtime scatter plot. I’ve charted all the hours where the Ecobee reported non-zero cooling runtime. Dots on the power = x axis (power = 0) represent missed detections.

Upstairs - AC ran 360 hours - 30 worst mismatches in red (some red points overlap each other)

Downstairs - AC ran 211 hours - 5 worst mismatches in red

1 Like

Good news ! As the weather gets warmer my cooling system has started to kick in again. And Sense has managed to remember the separate upstairs and downstairs compressor units. The previous winter, Sense forgot all learning it had accumulated the previous cooling season, when it was fairly accurately detecting both units. But Sense seems to not to have drifted this year, at least since I put in the new compressors in July 19. When I look at the weekly correlation, it is very close to 1.0 between each unit on the Ecobee and the most closely associated device. And the mystery device, AC has gone away entirely (OK, I deleted it after it dropped to zero detections). Numbers on the bottom are weekly runtimes in hours.

Upstairs Compressor


Downstairs Compressor

3 Likes

What’s your takeaway? That Sense has gotten better, or that your nearly-failed compressors had enough of a change YoY that Sense had trouble seeing them as the same units? Or both?

Looking back on my old (summer of 2018) experiments, I think Sense is doing a better job. In the summer of 2018, Sense detected 5 different AC devices in a pretty short span and seemed to drift around between associations during different months.

I did a full factory reset in April ‘19 and after that, Sense detected fewer AC devices (2, same as number of compressors) with one being added after I installed the new compressors. Part of the better results might be due to the more stable operation of the new compressors, but the original experiment was done a whole season before the older compressor failed, and I replaced them both.

I think Mahesh has been making some progress, perhaps aided by better “ground truth” from the Ecobee Historic integration.

2 Likes

what a fascinating thread, thanks for downloading, analyzing, and sharing all of your data, I have what is in comparison a relatively simple system comprised of three major components, Heat Pump, Furnace, and Heat Pump electric assist. I find Sense has detected all three but only detects their operation a small percentage of the time. I am hopeful that my recently added EcoBee provides Sense the “ground truth” data needed to improve and that my recently added detail of equipment part numbers and details will help as well.

1 Like

Thanks ! I like having tangible feedback on how well Sense is tracking, and this is the rare case where I can make at least an hourly comparison. I have simple 240V single stage compressors that are easier to “see” than electronically controlled heat pumps of variable speed motors. But presumably Sense can build and test new models more easily with the 5 minute interval data that comes out of the Ecobee.

1 Like

Actually my heat pump is also a simple single stage unit, scroll compressor, fan motor, and reversing valve. Based on the increased cost of variable displacement units when I upgraded the system 5 years ago there was no payback in the higher complexity higher efficiency units for my use case.

All that to say it should be easy for Sense to detect the compressor and fan motor

1 Like

I’m jealous. It is still too early in the year for us so our AC systems have not yet run. The last two years our 2 air handlers and our 2 condensers have constantly been confused with each other. Our downstairs system is ~2.4 ton and the upstairs system is ~3.0 ton.

:crossed_fingers:t2: this year will be better.

1 Like

Wishing you luck. Mine are Bryant 3 and 4 ton SEER 16 systems.

Updated on 7/24 - added 3 more weeks of graphs

I decide to revisit my AC Sense vs. Ecobee correlation results with a little more of the cooling season under my belt. I had gotten a little concerned since Sense seemed to be falling off in it’s ability to correlate with my Ecobee, especially upstairs, where my AC unit was running far more than downstairs. You can see that Sense’s detection of my upstairs compressor (AC2) dipped for a few weeks, below. The number at the bottom indicates the number of hours that the AC had been running for that weeklong period.

I decided to push a little deeper into seeing what happened during the weeks of May 18 th through Jun 22nd, so I converted Ecobee runtime data into energy data based on my upstairs unit running at about 2.5kW (my unit’s rating - in blue), then plotted against the Sense hourly data for my AC2 device (in red). Sense nailed it during the first week and last week in that list, but had some correlation issues in the middle - what was different ?

In weeks 2-5, it looks like Sense has trouble keeping track of all day runs of the upstairs AC, while it seems very accurate on shorter runs or even all day runs with breaks in the middle. I’m guessing Sense might time out, thinking it missed an off signature after 4 hours of continuous running.

1 Like

I can see where this is going …

:wink:

1 Like

Yup,

  1. I could use the 5 min Ecobee data to generate 15 min synthesized energy data that I could employ to cancel out AC waveforms in my PGE data stream, as well as synthesizing the air handler as well. That could help simply EV detection, though I would still have to contend with the floor heaters, which have their own thermostat API similar to the Ecobees, but I haven’t cracked yet.

  2. I could also use Ecobee data to clean up my Sense AC detections, for purposes of my usage analysis. Looks like on a bad correlation day, I’m missing 10-15kWh of AC that shows up in Other instead. Maybe I should just be synthesizing the new data and replacing. Hmmm.

I have the same NuHeat thermostats I believe but haven’t bothered with the API yet … and now I’ve freed-up a Sense monitor to dedicate to my one floorheat loop. Ground-truth on the ground! Waiting for Winter.

Meanwhile, I wonder if the non-ramped resistive element in the floor heat (= predictable plateaued load) isn’t somewhat “easy” to synthesize and cull … at least statistically. I suppose you’ve pondered that. Maybe I missed that in your prior work. You’re good at histograms!