Back in early April I replaced my Sense monitor unit. Then, in July I replaced both my upstairs and downstairs AC systems, when the downstairs compressor died and the upstairs seemed to have given up the ghost. So I now get a chance to watch the evolution of Sense detection all over again, with a little more pre-knowledge of what to watch for.
A few key dates:
April 6th, 2019 - New Sense installed - it eventually detects AC, one of both of my old AC units.
July 9th, 2019 - The new higher efficiency AC units go in
After July 10th - Sense detects AC2 and AC3
Now comes the fun part - comparing stats from my upstairs and downstairs Ecobees against my Sense data. But itâs not as easy as it seems.
My Sense data has data dropouts - I can detect hour long dropouts and remove them from comparison, but I canât easily detect and fix hourly power usage that is missing some number of minutes of data missing from an hour.
Sense also misses some detections. And even though I have my both my furnaces on smart plugs, Sense has dropped some data from the smartplugs as well over time.
My downstairs Ecobee also seems to have had dropouts. Fortunately, I have been able to detect which 5 minute segments lost data and filled them with NA. The tough choice is whether to convert the runtime data for an entire hour to NA if only 5, 10, 15 minutes was missing.
Hereâs my first comparison - what the Ecobees commanded the furnace fans to do vs. the power used by each furnace. Colors represent the different months. Vertical axis is power consumption over an hour. Horizontal axis is runtime for the fan, in seconds.
So one is showing power being consumed with no runtime (vertical line of green circles on second pic) and the other is showing runtime with no power consumption (horizontal line of red and black circles on first pic).
Option 1: Are your monthly colors the same for both graphs? If not, and if Sense is/was incorrectly detecting that the âDown Fanâ is running when really itâs the âUp Fanâ I would expect data similar to what is shown.
Option 2: Your data drop outs are related, and there is a conflict between Sense and the Ecobees, causing only one or the other to be connected reliably. Therefore only one variable (power or runtime) to be available for portions of the data.
Option 3: Iâve completely missed the substance of your data and the question youâve posed.
Good questions and call @dylan.christopherson. The colors do represent the same months in each chart. The green vertical line mostly comes from a string of hours in June when the downstairs Ecobee says nothing was happening. After a little investigation, it looks like my first Ecobee .csv data dump for June was bad - it only went through 6/14 and was about 400KB in size. Just downloaded again and received a full month, 1MB in size.
Now itâs time to look at the case of zero power with positive fan runtime. Hint - the blacks represent April. My new Sense didnât go in until April 6th. If I look at my device power meter for FurnaceDown in April, I see this ugliness ! A huge data dropout for that smartplug that lasts about a month. This makes me wonder whether Sense sees these as NAs internally. If so, it would be nice if we had the option to reveal that in the UI and in data export.
Fortunately, I can do a quick fix by inserting NAs (not available) into the places where the dropouts occurred. My script was originally inserting zeros, during a âcastâ operation but I can change the fill option to NA. That cleans things up at the expense of giving up the compromised data.
A vertical line at 300 secs - thatâs the minimum fan runtime per hour setting for both my thermostats.
A vertical line at 3600 secs - thatâs the fan running for the full hour. Hint - Red is June.
If I look at the 3600 second runs that have hourly energy that drops below the regression line (FurnaceDown < 0.5, FurnaceUp < 0.35), most of the points are localized in time.
The points with FurnaceDown < 0.5kWh are mostly clustered from June 12th-26th, from noon into the evenings, during cooling cycles. The hours occur around the steep short drops in the middle of continuous run zones below. Coincidentally, a repair guy put additional freon into my old downstairs AC compressor (it wasnât cooling prior) on June 11, and the compressor blew out by June 14th.
The thing I canât really know is whether these are Sense dropouts, or my furnace inserting a break to give the blower motor a rest after a continuous long run. Sense dropouts longer than an hour show up as missing data, but dropouts shorter than an hour merely give a reduced energy value for that hour.
The points with FurnaceUp < 0.35kWh are mostly clustered from July 11th-current, mostly around noon. The hours in question again occur around the steep short drops in the middle of continuous run zones below. And this seems to be associated with my new upstairs compressor because that went online on July 10th.
The same thing goes for these dips. The Ecobee keeps asking for fan continuously during cooling, but the fan power measurement either goes to zero for a short period for one of two reasons: 1) the furnace inserts a break, or 2) Sense experiences a dropout.
Bottom line, I kind of know where the vertical line at 3600 seconds comes from, but donât know if it really needs fixing or not. Iâm going to check my furnace manuals.
My experience with furnace blowers has shown that if you are using the fan as a circulator (fan ON) and the the unit receives a call for either heating or cooling it will first stop the fan and the initiate either the heating or cooling sequence. If cooling, the fan may only stop for a couple seconds. For [gas] heating, the fan will remain off a bit longer until the main heat exchanger gets up to temp.
There is also an âanti-short cycleâ timer that prevents an AC compressor from restarting too soon after the previous run cycle. Example: unit is running in cooling mode when the space reaches set point. Compressor stops, fan continues to run for a short time to use up the âresidual coldâ in the coil and then stops. Thermostat calls for another cooling cycle before the timer had expired. Even though the thermostat is calling for cooling, the compressor and fan will not start until the anti- short cycle timer has expired and allows the cycle to begin. This can be exaggerated by âzonedâ systems where the various zones just happen to be âout of syncâ with each other.
Iâm guessing that the ecobee only reports what it is commanding the HVAC system to do, but has no actual feedback to know if/when that command gets executed.
The Ecobees only give data on their commands, not what the HVAC device actually does. In the case of the .csv output itâs a snapshot of the commands and previous runtime duration every 5 minutes.
Both the furnace controller and the compressor controller take an active role in their own self-preservation and perhaps efficiency. They will impose mandatory gaps and run overages beyond what the Ecobee requests. I think thatâs one of the reasons for some of the variability at some of the common runtime abscissa points (0, 300 and 3600). For instance, I see a small number of positive active power, zero runtime points where the Sense device power meter looks like the two waveforms below, with a small tail (less than 5 min) extending into the next hour (11pm and 10pm respectively), where the Ecobee reports requesting zero runtime, but the furnace control seems to keep things going for a bit longer in collaboration with the compressor.
I think Iâm ready to write off the 3 weirdities I described as ânoiseâ from the furnace and compressor controllers doing their own thing, and move on to looking at compressor power usage data.
So now that I have cleaned up my incoming data a little, Iâm going to bring my AC compressor detections into the mix while keeping my smartplug based blower fan detections for both furnaces. One of my goals is to see which of my Sense AC compressor detections reliably line up with corresponding Ecobee run time commands. But itâs a little tricky because:
I had one Sense runtime detection, AC, against my old compressors - Detections started showing up on April 7, one day after I installed the new Sense, but I believe that my data is also post-processed for detection so that an detection discovered later can be be run on earlier data to do additional identifications.
I had two more detections, AC2 and AC3, that showed up in May and July respectively. The July detection appears to be tightly linked to one of my new AC compressors since it only appeared on July 9th, after the new compressors were installed.
Looking just in the first correlation column, comparing the Ecobee cooling command times/hour, vs. the corresponding AC detectionsâ power usages for corresponding hours, the data is very clear.
AC correlates with the cooling of my old upstairs unit (May is red, green is June, Blue is July)
AC2 correlates with the cooling of my new upstairs unit.
And it appears the the new compressor, AC2, is more efficient than my 20 year old unit, AC, maybe 1.5kW, vs. 2.5kW in usage.
Iâll also make one more important point here - it looks like there are a bunch of points in the AC and AC2 vs Up_Cool chart where AC or AC2 look to be zero, but are really just very small. Digging deeper, those are mostly cases where both AC and AC2 were detected at different compressor startups during the same hour, but one of two predominated.
Thatâs exciting - Sense is making some solid identifications. But now comes the tricky part, looking for device usage that Sense missed. Remember that these graphs are done with any âmissingâ Sense device data marked as NA and removed from the graphs.
Because of the way that Sense shares exported hourly data, thereâs no way to differentiate between subtle differences in data composition. When Sense presents non-zero data for a particular hour, you can be sure that it âsawâ a detected device using power during that hour. But when Sense leaves out an hour of data from export, it could mean a number of things about the device:
The device hasnât been detected yet
The device has been detected, but wasnât spotted running during that hour
Sense encountered a data dropout
Or a mix of the latter two
So Iâm going to try to do two things for additional analysis
Replace the NAs with zeros again (treat all missing zeros) and see what happens
Try to intelligently convert NAs to zero, but only after Sense has started to reliably detect those devices, to see how much real runtime Sense might have ignored - hours where the Ecobee is telling the compressor to cool, but Sense doesnât give any data for that device hour.
Here are the same charts with option 2). I have converted all NAs from Sense to 0âs, and only selected data after July 9th, when my new AC compressors were installed. An interesting
pattern emerges:
Up doesnât look so good - looks lots of âzero powerâ in both AC and AC2. Poorly defined correlation âsignalâ. But just like before, when I push down into many of those zeros, they are really cases when Sense detected a little bit of power as AC and most of it as AC2, or vice versa. If I was to correct the graph by adding non-zero AC power to corresponding hourly AC2 power, I bet we would see a clearer signal (later).
AC3 gives a strong clear âcorrelation signalâ with respect to the Ecobee Down_Cool requests. There still look to be some missed detections, but not many as @ixu is seeing in his excellent correlation analysis here:
So one final experiment, adding AC and AC2 power to handle the case where Sense gets a little mixed up between the two, when identifying my upstairs AC unit during the same hour. I probably should have deleted AC once my old AC units were repalced, but I wanted to do this type of experiment first. Hereâs a more detailed vie of just Up_Cool (Ecobee command time) vs. ACcomb (AC+AC2 energy), with FurnaceDown smartplug detection as well.
There are now fewer missed identifications (near zero power for positive runtimes) in the ACcomb graph, but the âsignalâ is still a little fuzzy. Some of the points follow a trend line that indicates a 3kW device while other for a 2kW device line, but thatâs to be expected since weâre seeing a mix of identifications, when my older AC unit was less efficient.
Two final steps now that I have completed the month of July.
Iâm going to calculate the closest device correlations to both upstairs and downstairs Ecobee runtimes, by month.
Iâm going to look at the mix between good correlations, bad correlations and missing data for the Ecobee/device pairs that appear to be connected.
First the month by month correlations. NAs represent months where there was no valid data that the same time between the Ecobee and the device detections (one or both for that time period were NAs)
Upstairs
Initially AC and FurnaceUp devices correlate most closely with my Ecobee turning the AC on, but thatâs based on very little air conditioning usage in April. AC2 shows up in May and begins to take over. Things are very mixed in in June, but that might be due to us putting more freon into the unit around June 11th. AC2 rises to top in July, perhaps because it is closer to the profile of my new more efficient compressor. Iâm hoping that the signal / correlation gets stronger in August.
FurnaceUp stays closely aligned with Up_Cool through all the time because itâs using a smartplug. Thereâs bound to be a little mismatch because my fan/furnace is set to come on at least 5 minutes per hour, even when my Ecobee isnât calling for AC.
$`2019-04`
Up_Cool
AC 0.9848766
FurnaceUp 0.9769824
FurnaceDown -0.1541840
AC2 NA
AC3 NA
$`2019-05`
Up_Cool
AC 0.9851462
FurnaceUp 0.9772373
AC2 0.1661719
FurnaceDown 0.1412279
AC3 NA
$`2019-06`
Up_Cool
FurnaceUp 0.9892451
AC 0.7441818
AC2 0.7314602
FurnaceDown 0.5909811
AC3 NA
$`2019-07`
Up_Cool
FurnaceUp 0.9850746
AC2 0.6176325
FurnaceDown 0.5118411
AC3 0.4237348
AC 0.3627760
Downstairs
Downstairs is easier to figure out than upstairs. All correlation with AC devices in Sense-land was pretty much coincidental until Sense âdetectedâ AC3, which correlates very strongly with my Ecobeeâs Down_Cool command.
$`2019-04`
Down_Cool
AC NA
AC2 NA
AC3 NA
FurnaceUp NA
FurnaceDown NA
$`2019-05`
Down_Cool
FurnaceDown 0.9645780
FurnaceUp 0.3849021
AC 0.2430033
AC2 NA
AC3 NA
$`2019-06`
Down_Cool
FurnaceDown 0.9923947
AC2 0.7628676
FurnaceUp 0.6044697
AC 0.1213893
AC3 NA
$`2019-07`
Down_Cool
FurnaceDown 0.98625399
AC3 0.97380936
FurnaceUp 0.51139072
AC2 0.42605800
AC -0.08355149
NA Statistics for Downstairs
Hereâs a table of Down_Cool and AC3 hourly values for the two different states of the Downstairs Furnace Fan running (IDLE and ACTIVE), after AC3 was first detected. Looking at the number or hours of detected vs. missed by Sense, I need to look for the places where the Furnace Fan is ACTIVE and the Ecobee runtime is POSITIVE. 207 hours of good detections, 31 hours missed. There might also be 7 false detections where Ecobee didnât call for cooling but Sense saw some usage. I label this detection as âPretty Goodâ. Letâs hope it continues into August !
Iâve been watching my upstairs AC unit to see if Sense is likely to eventually detect it as reliably as the downstairs unit. If I look at whatâs happening with just AC2, things look iffy, with only about half (150 out of 296) of the Ecobee hours being detected.
If I push one level deeper to see if AC2 and AC detections are overlapping (ignoring when the Furnace Fan is idle), I can see that 34 of the correct hours of AC2 detections also have AC detections. Another 26 hours that the Ecobee says the unit is running, are detected solely as AC. So if you consider both AC2 and AC, 150+26 hours are detected out of 296 hours that Ecobee says the unit is running.
When I look at the running of my upstairs AC vs downstairs, using my furnace fans as proxies, I notice something that I had seen a while back - The upstairs AC stays on for much longer periods than the downstairs.
Just to confirm, I plotted the Ecobee cooling commands for the same period. Everything is the same as for the furnace fans, except for the the 5 min, every hour fan cycles, that I have programmed my Ecobees to add.
Longer on cycles upstairs has many consequences for the analysis I have been doing
Fewer on/off cycles for Sense to learn from
A missed on-detection leads to more hours of missed on data.
Sense might also prematurely timeout some valid on-detections, as it seeks to clean up any missed off-detections.
Iâm going to have to investigate more. Time to dig into my Ecobee data to find out why my upstairs is running for the much longer periods. @ixu, I guess Iâm seeing the same thing as you - good detection for short compressor runs, but less good detection on long compressor runs.
I started to assemble data yesterday from my AC and HW (hot water tank). Because I have the AC on the Wemo + Sense#2 Solar CT + Sense#1 native detection, I think it should be quite revealing. That along with the HW on Sense#1 native detection + Sense#2 Mains (dedicated to the HW circuit only) + Sense#2 native detection of the HW on the Mains.
Hopefully Iâll have time tomorrow to put things together.
Also thinking of @samwooly1âs âfalse notificationsâ. Though itâs a different issue, it made me think more about the on/off triggering and how that could be disrupted by other activity. It also has me wondering about the ML aspects, more of a raw calculation than raw patterns. e.g. If a 4kW HW tank is on while, say, thereâs 500W of âknown AOâ running, then a 4kW drop in power would be a very strong indication of HW OFF whereas, say, that same 4kW drop to near zero overall load is more unlikely (the AO has had no sign of OFF) ⌠a strong indication that the HW OFF was missed.
The first thing Iâm going to do on the Ecobee side is to look at the reliability of my data. Three things are different about Ecobee downloaded csv data vs. Sense
The data comes with 5 minute granularity - good !
The data also comes with better data integrity checking - if data is missing for a particular time interval, Ecobee still supplies the time / date with empty data for that 5 minute period, so you have to make fewer assumptions about that data - it really is NA, and not potentially 0. With Sense, a data dropout or an actual 0 usage (no detection) hour/date is skipped for that particular device - good !
Unless you become a developer and use the API, data can only be retrieved a month at a time - not so great vs. Sense.
The chart below shows the number of Ecobee 5 minute dropouts per particular day. There are 72 upstairs and downstairs dropouts today, Aug 4th, because there were 6 hours of uncollected data left in the day when I did my download. But you can also see that both Ecobees have had data lapses, with many more occurring downstairs, and most before late May 2019. I know I have had dropouts when my downstairs system is in heating mode because my furnace was tripping the high-temp limit switch due to restricted airflow (hopefully fixed in July).
The next step in puzzling through the NAs is to try to figure out what the Ecobees were trying to do just prior to going NA. If I look at the table for each thermostat, the profile of states prior to NA (in red) are different in one respect, but similar in several others.
The downstairs Ecobee has a far higher percentage of heatOff to NA transitions (319 vs. 4). I believe I understand the cause of these - my furnace shutting down due to overtemping.
But both Ecobees have a significant number of NA to NA transitions, though downstairs has far more. That might be related to the same shutdown phenomenon lasting more than 5 minutes.
Most of the transitions are on the diagonal - staying in the same state, especially in the upper left (off states, plus cooling).
BTW - heatOff and compressorCoolOff are really similar off states, with the only key difference being whether the Ecobee is in heat mode or cooling mode based on the sensors temperatures WRT to the set points.
Itâs been a while because Iâve been trying to find ways to analyze the output of my Ecobee, including the periods when it has been disconnected for various reasons and output NA for those 5 minute periods. I didnât know whether I could completely ignore the NAs or how I should treat them. Eventually I figured out a way to filter the Ecobee data by reconverting it to transition events (rather than 5 min polling data). I end up with four kinds of transitions for both heating and cooling transitions:
Turn-on - the heat/cooling turns on
Turn-off - the heat/cooling turns off
Spike - the heat/cooling goes on for period of time that is shorter than 10 minutes, and longer or equal to the time reported by the Ecobee, but unknown due to the Ecobee dropping out)
Dropout - the Ecobee drops out for less than 10mins between two known cycles (reverse of a spike)
First off is my downstairs heating cycles. I have lots of spikes because the thermal protection in my furnace was kicking off the furnace controller every 6 minus of runtime or so due to insufficient airflow. I finally fixed this problem this summer, when the HVAC guy installed new AC compressors. You can see the extend of my heating season from the time my new Sense was installed (early April) until no heating was needed (May 28th)
And here is a view of the cooling cycles. Youâll notice the upstairs is more active than the downstairs because heat rises through our house and cold air sinks, especially via our two story entryway.
Well, cooling season is almost over and I decided to do one more monthly correlation analysis between my Ecobeeâs hourly cooling runtimes and the power consumed by various Sense AC detections, AC, AC1 and AC2. FurnaceDown and FurnaceUp are hourly power used by the smarplugs monitoring each of my furnaces (mostly the furnace blower used by AC during the summer). Remember that I reset my Sense unit to ground zero in April, then put in new AC compressor units in mid-July. Itâs interesting how the detections sort themselves out at the end of the cooling season.
Hereâs the story for downstairsâŚ
No detections for AC in April
Two AC detection in May, AC and AC2 - âACâ is most closely associated with the downstairs compressor
New compressors in July upset the status quo. A new detection, AC3, aligns with the downstairs compressor and correlation continues to grow over time.
$`2019-04`
Down_Cool
AC NA
AC2 NA
AC3 NA
FurnaceUp NA
FurnaceDown NA
$`2019-05`
Down_Cool
FurnaceDown 0.96457804
AC 0.40228237
FurnaceUp 0.38490212
AC2 -0.02461567
AC3 NA
$`2019-06`
Down_Cool
FurnaceDown 0.9923947
FurnaceUp 0.6044697
AC 0.3183210
AC2 0.2872561
AC3 NA
$`2019-07`
Down_Cool
FurnaceDown 0.98625399
AC3 0.56490245
FurnaceUp 0.51139072
AC2 0.32950875
AC 0.06935545
$`2019-08`
Down_Cool
FurnaceDown 0.9939775
AC3 0.7953435
FurnaceUp 0.6062248
AC2 0.1969622
AC 0.1884249
$`2019-09`
Down_Cool
FurnaceDown 0.98889843
AC3 0.95784466
FurnaceUp 0.39951440
AC2 0.36528948
AC 0.01010691
Hereâs the story for upstairsâŚ
No detections for AC in April
Two AC detections in May, AC and AC2 - âACâ correlates very closely with the upstairs compressor runtimes. Why was AC2 even created in the first place ???
New compressors in July upset the status quo. A new detection, AC3, aligns with the upstairs compressor initially, but eventually lonely AC2 muscles in and correlates best with upstairs.
$`2019-04`
Up_Cool
AC 0.9791817
FurnaceUp 0.9769824
FurnaceDown -0.1541840
AC2 NA
AC3 NA
$`2019-05`
Up_Cool
AC 0.99412114
FurnaceUp 0.97723727
FurnaceDown 0.14122794
AC2 -0.02226503
AC3 NA
$`2019-06`
Up_Cool
FurnaceUp 0.9892451
AC 0.6702095
FurnaceDown 0.5909811
AC2 0.1680601
AC3 NA
$`2019-07`
Up_Cool
FurnaceUp 0.9850746
AC3 0.5987115
FurnaceDown 0.5123134
AC2 0.3918143
AC 0.1948437
$`2019-08`
Up_Cool
FurnaceUp 0.9980361
FurnaceDown 0.6066122
AC3 0.5671459
AC2 0.4513146
AC 0.1011073
$`2019-09`
Up_Cool
FurnaceUp 0.9990777
AC2 0.8910393
FurnaceDown 0.4199229
AC3 0.4070204
AC 0.2234853
I keep promising myself that Iâll delete AC to help improve things, but I kind of want to see if it goes away on itâs own. If I look at the amount of energy that was ascribed to each detection on a monthly basis, I can see AC is on a downward trend, but hasnât found a route to disappear yet.
Hereâs a more graphical view on how Sense detections eventually hone in on the actual operation of my upstairs and downstairs compressor units as measured by the Ecobee thermostat commands. The number shown is the weekly Pearson correlation between hourly Sense detection energy usage and my upstairs and downstairs Ecobee cooling runtimes. A correlation of 1 means that there is a perfect linear correlation between energy usage and runtime. 0 means no correlation and a negative number means negatively correlated.
Upstairs Compressor
The upstairs compressor is on for more of the year since my upstairs heats up more (hot air rises, cool air falls and I have a big open entryway). Initially Sense detected my upstairs unit as device AC and had good correlation for a while until the unit was running pretty much all day long. Then, for some reason, a new AC detection from early May, AC2, that hadnât correlated to anything particularly well, suddenly started matching more with my upstairs usage (and downstairs as well). Then, around July 10th I replaced both compressors and Sense was forced to sort thing out again. Eventually AC2 matched up with my upstairs compressor and AC3 matched up with my downstairs unit.
Put differently, if I look at my results from Sept. 1, and onward, where Sense correlation with my Ecobee is relatively good, I can compute each AC unitsâ estimated hourly power usage by dividing hourly energy from Sense for AC2 and AC3 devices by the corresponding Ecobee runtimes (in hours). If I look at the distribution of results, I see very reasonable answers for each unit.
One more way to view Sense vs. Ecobee. This one can be done in Excel with a little ingenuity - Detected hourly power usage vs. Ecobee runtime scatter plot. Iâve charted all the hours where the Ecobee reported non-zero cooling runtime. Dots on the power = x axis (power = 0) represent missed detections.
Upstairs - AC ran 360 hours - 30 worst mismatches in red (some red points overlap each other)
Good news ! As the weather gets warmer my cooling system has started to kick in again. And Sense has managed to remember the separate upstairs and downstairs compressor units. The previous winter, Sense forgot all learning it had accumulated the previous cooling season, when it was fairly accurately detecting both units. But Sense seems to not to have drifted this year, at least since I put in the new compressors in July 19. When I look at the weekly correlation, it is very close to 1.0 between each unit on the Ecobee and the most closely associated device. And the mystery device, AC has gone away entirely (OK, I deleted it after it dropped to zero detections). Numbers on the bottom are weekly runtimes in hours.
Whatâs your takeaway? That Sense has gotten better, or that your nearly-failed compressors had enough of a change YoY that Sense had trouble seeing them as the same units? Or both?
Looking back on my old (summer of 2018) experiments, I think Sense is doing a better job. In the summer of 2018, Sense detected 5 different AC devices in a pretty short span and seemed to drift around between associations during different months.
I did a full factory reset in April â19 and after that, Sense detected fewer AC devices (2, same as number of compressors) with one being added after I installed the new compressors. Part of the better results might be due to the more stable operation of the new compressors, but the original experiment was done a whole season before the older compressor failed, and I replaced them both.
I think Mahesh has been making some progress, perhaps aided by better âground truthâ from the Ecobee Historic integration.
