Just wanted to offer another view on Sense vs other power datasources. I have recently been working with Sense data under Home Assistant, since Home Assistant offers a great way to pull together time series power and energy data for my entire house, even in areas where Sense already does a good job. It turns out Home Assistant has an integration to pull in data from my smart meter, through my previously used Rainforest Automation Eagle bridge. Since I have a PG&E net meter, I was never able to look at the big picture including solar, like I am with Sense, but with Home Assistant, I have a chance to do real-time comparisons between Sense and my meter. Here’s a view with Sense Total Usage in orange, Sense Solar in blue and my net meter in green. Not surprising, at night, the orange (Sense) overlays the green from my meter. The big plateau on the right is the charging of our Model S.
Here’s why accurate samplings of my net meter are pretty useless - There’s no way to see Total Usage. And even though the brilliant guys at Rainforest compute an “Always On” value, it is useless because it is not based on Total Usage.
One final note - here’s why it CAN be very hard to compare power usage to determine accuracy - sampling intervals. Here’s a snippet of data for my first chart. As you can see, the Total Usage and Solar are sampled on one timebase and interval. My Eagle net reading is sampled on another timebase and interval. But how does one line them up to compare ? The real answer is that we really need to do numerical comparison using energy using fixed time points, unless I can exactly line up the sampling (and rule out any latency in the measurements).
All the blank cells in the data set are spiking my anxiety @kevin1. You probably know best here on a feasible methodology, but my gut reaction would be to group time intervals together for several minutes and compare the accuracy at a 5 or 10-minute interval. After re-reading, I think you’re suggesting a slightly different approach.
I am also well aware that there’s a reason i’m not a data scientist
No need to be concerned about blank cells. That’s just what happens when measurements are being taken using different sampling intervals and offsets - very typical in the IoT (internet of things) world because there is no global synchronizer telling all the devices to sample at the same time (plus there are lots of complications with that approach as well). Many ways for data science to handle the blanks - a couple
intelligent fill / padding, hysteresis, and/or resampling to fill in the blanks with what value “should” be there.
aligning sampling more closely. Not clear I could ever get sampling on the exact same time steps, but might be able to get them within a few seconds of one another if I dug deep.
Using the intelligence of the devices - Sense and my power meter are really good at accurately integrating energy usage. The energy integral is a much better comparison than instantaneous power, because we really mostly care about energy in the end, and the energy interval is not subject to the wild fluctuations of instantaneous power (fluctuations are small in relation to the magnitude of energy usage). That’s going to be my approach.
@kevin1 Do you have one meter (net) and another on the solar output from the utility or just the net meter? Im just curious if in the utility portal it shows the reverse energy back to the grid hourly.
I’m starting to move some of my data analysis in R to run directly off my Home Assistant Influxdb of power, HVAC and car charging data. Lots of good things come from minute-by-minute samples vs. using hourly energy exports from Sense and 15 min exports from PG&E. But there is a downside as well - instantaneous power varies wildly in many households, even within a minute or two. Below, I’m comparing the every minute power data coming from my Rainforest Eagle (from my PG&E net meter), against my every 2 minute calculated net data from Sense (Total Usage - Solar). When usage and solar production is somewhat stable, the curves overlay completely. But there are many points in time when instantaneous power readings within a minute or so of each other are vastly different as well (orange is eagle, blue is Sense net)
And if I view the power difference between my net meter and the Sense net for these sampled power values in a histogram, I can see that most of the big errors happen mid-day, not at night. Probably due to both varying solar and use of our floor heaters and dryer which are big users that go on and off regularly from minute to minute (hint - one of the floor loops and the dryer use about 5500W in action)
