Device Learning metrics have been revised down 10%-20% for my home

Technical Questions
,
1

Looking back at some device learning numbers in the Sense app, I noticed that the numbers that I pulled on January 1st 2017 seemed to have changed. After noticing this, I pulled all the numbers again to try and gauge the magnitude of the change over September 20th 2016 to December 31st 2017.

These are the rolling 7-day averages for learned devices from my manual extracts on January 1st and January 13th. Note that numbers appear to have been “restated” across the entire time Sense has been running.

Screen Shot 2017-01-13 at 3.13.25 PM.png1055×490 37.7 KB

Where I had originally seen Learned devices peak at just under 60% in Nov 2016 and decline to under 40% in Dec 2016, the restated numbers show that the peak was under 40% and dropped almost to 20% around Dec 25th before edging to above 30% at the end of December. Here are the last set of Device Learning metrics I shared at the end of the year:
Screen Shot 2017-01-13 at 3.38.52 PM.png1169×538 58 KB

And here is what they look like now that they appear to have been “restated” by Sense:
Screen Shot 2017-01-13 at 3.36.44 PM.png1167×535 61.8 KB

Device learning performance is measurably worse than I’d previously reported. What caused the “restatement” and the changes in these metrics? How often does Sense make these kinds of “restatements”? How can users discover that their metrics have been updated and know to re-examine them?

2

If a device was previously learnt and is now forgotten. How does I affect your graph?

3

I think the change in learned devices (going away) likely accounts entirely for the significant drop that is illustrated in the first chart.

One of the devices that Sense removed was my electric hot water heater, I don’t have it’s power usage data anymore to scale it, but I expect this is why the revision is so noticeable. If it was some LED lighting it would have been much less so.

1 Like
4

Hi, thank you for addressing this topic. Just to briefly clarify, the Sense model building process does update its estimate of how close it is to a confident model – it is not strictly the amount of time. This means that yes, these can get revised downwards if Sense becomes less confident in what it is seeing.

5

I’m less interested in the fact that the numbers can change than I am in being able to accurately track the learning progress over time. As Sense removes, forgets, and conflates devices it means I have to pull all data every time I want to get a idea of how Sense is doing.