As most know, the Sense data servers went down several days ago. As a result i went outside and took a reading of my Power Meter early Saturday, Sunday, Monday and early Tuesday.
To recap:
As of ~3AM ET Sunday Morning the Sense data was only showing through 3AM Saturday - missing 24 hours
As of ~3AM ET Monday Morning the Sense data was beginning to fill in and actually about 28 hours behind real time.
As of ~2AM ET Tuesday Morning the Sense data appears to be backfilled in and up to date.
However, the data especially on my fixed speed 5 ton Heat Pump looked impossible as did the power usage Sense reported I used on a daily level compared to my actual readings. I should also note that the air handler in the attic is tracked by clamps and not included in my Heat Pump numbers, but as a different entry.
So I’ve spent the last 2 hours going through the last 4 months of my 5 ton single speed Heat Pump data to determine the daily total kw / minutes the unit uses to come up with a daily kwh per minute of operation since February which was generally in the 0.064 area +/- 4%
Saturday backfill show the number at .10925 and .10973 on Sunday. Even Monday’s number is off at 0.07257
So the backfill numbers while nice, are not accurate at all.
As an overall update about the outage that began on June 3, 2023, it should now be resolved. Our engineering team has processed millions of backlogged events during the outage, and all reports in your app should now be repopulated and appear as expected.
I added in pics to the post above as you were typing.
See if they answer your questions
Total Time on is from Sense. Converted to minutes from hours/minutes.
Also notice the low amount of “times on” and total time on in the backfill data compared to Friday and Monday when compared to the kWh used, which was my first tip off the data was bad
@Beachcomber, yup. Seems like the the backfill aggregation missed some time or double-counted some usage for those two days. Thanks for the warning. Let us know if support sorts out what happened or it mysteriously fixes itself.
Beachcomber… Just a note that mine was down too and back filled with with data I have no idea where it came from. Not even close… It was mostly under what was used…Gerry
@Beachcomber ,
I did a quick check on the top-line Total Usage / Solar vs. my PG&E utility data and it looks like the hourly exported Sense data (Trend data) is spot-on with my utility data. I compare them via Net Usage (SenseNet = Total Usage + Solar) in orange, in the chart below and PG&E USAGE, in cyan, from my utility.
So the overall numbers look good, at least for my accounts. But it sounds like we should be somewhat suspect of the device data for those two days. I might try to pull my HomeAssistant samples of various devices of comparison, but those comparisons are harder than it might seem to get right (sampled power in HomeAssistant, vs aggregated hourly energy in Sense).
@Beachcomber, I did another quick check against my Ecobee data - the Ecobee gives me a second view on how long the AC has been running. When I compare hourly energy usage of my upstairs AC 2 (Sense native detection) vs. AC runtime (Ecobee) for each hour, it looks like the Sense hourly data is in line with runtimes. The red dots are for hours in the last 2 1/2 weeks that were not during the outage. The blue dots represent AC 2 runs during the outage. There is a little scatter and there were only a small number of hours of AC runtime during the weekend, but all the hours look reasonable.
Based on what I’m seeing, I suspect the energy is right, but the backfilled runtimes are incorrect. I haven’t done an exhaustive spreadsheet, but here’s a spot sample:
Date
Ecobee Runtime
Sense Runtime
June 1
2.725h
2h 23m
June 2
1.233h
1h 4m
June 3
1.879h
18m
June 4
2.063h
2h 4m
June 5
2.004h
2h 20m
Most are in the ballpark, but apparent that my Sense runtime for June 3 is way too low.
One more view on the weekend meltdown via HomeAssistant which was collecting real-time and trend info from Sense all through the process.
Top chart is real-time Total Usage and Solar Production. Bottom chart is cumulative daily total for Total Usage and Production as it builds through the day.
It looks like something went out with the hourly data aggregation first, right at the stroke of midnight on Jun 3. Later the real-time data updates started failing from 7:41AM to 12:44PM Pacific Time on June 3. It also looks like the trend data started to ramp up later on June 6th after the real-time was fixed, but of course was wrong, because it had missed many previous hours.
I looks like trend calculations didn’t catch up again until late on June 5th, though I suspect that Sense had been working on previous backfilling prior to that.
It also looks like they didn’t lose any raw monitor data or at most just a very little. When I look at same time period in the Sense Power Meter, the “lost” region has been backfilled except for a few possible dropouts.
Hi @Beachcomber, I decided to take one more look at the history of one of my simple 3 ton AC units to get a better feel of whether I am seeing the same. I charted Sense daily runtime and energy consumption vs Ecobee runtime (the accurate runtime). My goal was to see whether accuracy issues were due to Sense producing bad runtimes or messing up both runtime and energy (likely due to missed detections). Simply put, if Sense whiffed on the runtime, you’ll see a blue dot below the unity line. If Sense whiffed on the detection, you’ll see both the blue and orange dots below the the unity line. And if the orange dot, is below the blue dot, and that’s a case where Sense missed some of the energy used.
One big Sense runtime “miss” on June 3rd (during the outage), but the energy usage matches up with the Ecobee runtimes.
One huge detection loss on June 12th - both Sense energy and runtime are coming up way off, but commensurately way off. I’ll need to look more closely at that.
A few minor detection miss days like June 2 and June 16th where both the blue and the orange fall off the unity line.
A few days, mostly toward the upper end of runtimes where Sense energy (orange) falls below runtime. The Sense runtime is the same as Ecobee, but the energy usage falls below the line.
@Beachcomber, you can pull Ecobee data via the Ecobee website - You can find the Download option under Home IQ > System Monitor > Download Data for each thermostat in the house. Download seems to work more reliably with Chrome vs. Safari. The download produces reams of data:
A CSV file with data & operating mode sampled every 5 minutes
Many columns - DateTime, temperature and humidity for every sensor the thermostat is attached to, plus separate runtimes for heating, cooling and fan.
Since the data is for every 5 minutes, you’ll need to use something like an Excel Pivot Table to aggregate into hourly or daily data to match up with Sense data.
I’ll look little more closely at my issue on the 2nd. That was the weekend where Sense had data issues. You’re 3 hours ahead of me, so an issue on the 2nd for me could be reflected mainly on the 3rd for you, assume the problem started at the same time for the both of us.
Support says they have redone the data again for June 3rd and 4th now. No change in my Sense data.
I pulled all the Ecobee data. While I do not have the tools you have and had to count the cycles manually and add the daily minutes with discrete formulas, some interesting numbers are shown in the “strange days”.
Would also note that I wonder if some minutes from Sunday June 4th were thrown into Monday June 5th in the data backfill?
BTW, would love clarification if the compressor turns on at 11:55pm and runs to 12:05am, I a counting that as an on cycle that day. I do not know how Sense handles that. In SOME instances, that could account for 1 cycle a day difference.
Also calculated the kw/min using the Ecobee data, and it is certainly more stable and what I expect.
So, have you seen weird numbers over the past several days?
I have. Again, these numbers were steady as a rock the entire year prior to June 3rd.
So my question is still, what happened after June 2nd to throw the numbers off?
I did a quick and dirty hour-by-hour comparison for each day in June below to resolve. I chose to do a few things to make trends more obvious.
Green is Ecobee runtime in seconds, per hour (so 3600 is max)
Orange is Sense kWh * 1500 (the 1500 is a number that converts power back to seconds - 3600 seconds / 2.5 kW per second when AC is on). You could also do this by using a primary and secondary axis in Excel.
I removed hours from midnight to 10AM because my AC almost never runs then.
The three things I notice vs the analysis I did the other day.
June 2nd shows a small problem for just one hour (in red).
June 3rd, which showed a big issue in terms of Sense runtime vs. Ecobee, but showed a Sense energy usage total that matched up with the Ecobee runtime, matches up as the earlier analysis suggests. I’ll double check the runtime to see if Sense did any further cleanup for me.
June 12th, that showed the biggest deviation for me, kind of reveals what the Sense issue might be - Sense loses two back-to-back hours of runtime (two hours of 3600 seconds, followed by another 1000 seconds, was lost - in red). I saw this same issue in earlier experiments. I think what happens is that Sense has timeouts when it thinks it might have seen an ‘on’ but missed an ‘off’.
