So there is no way for Sense to be placed in “learn mode”(or whatever it would be called). People just need to give up on this idea. It’s not how Sense’s machine learning works.
But from my understanding, Sense has to reach a certain confidence level with a detection before it’ll present/label it to the user.
However, an option that may be helpful would be to allow Sense to provide the users some information concerning “low confidence” devices. An example would be it’ll show that there may 3 unknown motors; Motor A is 15%; Motor B is 20% and Motor C is 45%. It may display put some information that Motor A may merged with Motor C. This range may change daily.
Showing this changing range and how Sense does some basic background work without giving away any proprietary tech would at least allow the control freaks using these devices to feel as if the system is making progress…or not.
a “placebo train mode” is perhaps a way of placating the tweakers but Sense’s current approach, imho, is more reliable in the longer term. Bigger data = bigger confidence. Statistics. Go figure!
I don’t disagree that feedback (to the user) of a “confidence level” would be a nice feature … it’s something I have suggested for the UI. It shouldn’t, however, be conflated with or necessarily relied upon for “training”.
Consider, there are a couple of “confidences”:
A (partial) signature is related to a particular device
A signature represents the COMPLETE device disaggregation
I would break that down as being the confidence in detecting (1) on/off vs the confidence in (2) the complete tracking. AO devices with low DC power (via a transformer) are hard for (1) and easy for (2). Dynamic devices like an OLED TV may be easier for (1) but almost impossible for (2). What I’m getting at, and the problem with “confidence”, is that to a certain extent the confidences are built-in and are more like 0 or 1 than 1-to-100.
That said, I think there is room for confidence feedback somewhere in the system.
O You are asking for something interesting here, but it’s a hard problem to address. I’m going to greatly simplify what I conjecture is going in Sense-land WRT to machine learning, then ask you what you might like to see as far as confidence is concerned.
Sense really does three things to sort through the mains waveforms and convert into identifications and classifications of device types.
Filtering and condensing the raw data. The Sense monitor can collect data up to 4M samples of 2 byte (16 bit) data per second. This is way too much data to send back to the mothership so Sense uses a few methods to condense and filter the data. It condenses the data into a 1/2 second RMS power stream that you see in the power meter, plus a stream of data on “interesting” 1 second or so power transitions that might be on and off signatures.
Sense then runs the “features” of these interesting transitions into an initial categorization predictor that tries to detect the source of these transitions based on a set of learned models that fundamentally depend on the physics of the hardware plugged directly into the wall (motor, heating, microwave, etc.). Based on what I have seen, these models need to be associated with or even downloaded into your monitor for Sense to be able to do detection for that particular device class. I also think that some device classes, like EVs and AC mini-splits have a different set of models that work off of the 1/2 second data stream, not the incoming transitions, because those transitions are too slow to meet the monitor’s “interesting” criteria.
Once new fundamental devices are detected, Sense tries to classify them as a specific device type, using a logistic regression classifier and crowd-sourced identification data. In this step you will actually see a % for “confidence” about what Sense thinks the device is based on it’s statistical database (so you already see some confidence data).
Of course there are further steps that happen as Sense continues to update their models via new training, plus as Sense makes further connections between detections (combining data, getting rid of false detections), plus taking in human feedback (this device is not on).
What you seem to be asking for is a way to know how close some of those “interesting” detections are to hitting the point where they are deemed to be of high enough confidence to match a specific model type and be considered a detection. My belief, based on my experiences with different types machine learning, is that those numbers move around considerably, thanks to model training updates, interactions with other models, plus changes in your household. Sense used to have a feature that showed some of the detections “in the wings” but that only raised more questions than answers since some of the detections in that list never came to fruition, while others detections showed up without ever being in that list. Not so sure that that would instill confidence in control freaks.
Here’s one more thought on confidence measurements. Usually, with machine learning, there are some quantitative measures used to compare the effectiveness of different learning strategies and prediction techniques. The data science folks will use these metrics to decide which models to use for both the training (if supervised learning) and testing the models/features vs. the good dataset. Training and testing require ground truth so one can do error measurements between the predicted vs. actual results at the Evaluate stage.
Typically there are two different ways of looking at the results.
An aggregated error metric of some type, that machine learning tries to minimize
A “confusion matrix” that puts the results into a more human friendly picture of how the new models did. A confusion matrix for a Sense refrigerator on-model, for example, would give you the number of positive-positives (correct on-detections for true refrigerator on-transitions), negative-negatives (correctly not detecting a transition as the refrigerator when the refrigerator does not turn on), plus false positives (refrigerator detected as on, but not really) and false negative (refrigerator on, but Sense doesn’t detect).
All this is done a dataset that doesn’t likely have all of our devices in it (dataset can only contain devices for which ground truth exists)
Thanks for the great reply. I only have a general knowledge of machine learning and understand the difficulty in attempting to put the process in some form of easily understood information for the consumer to understand.
However, I understand some consumer psychology and the lack of feedback and lack of control are 2 major components that leads to stress and dissatisfaction with any process or product. Hence the constant complaints about lack of a learning mode or complaints that the system is not doing anything.
The power meter is cool but it does not show what the Sense machine learning is doing.
Some form of graphic of numerical feedback on what the system is doing in the background would at least satiate some of that feedback deficiency we have with the product.
With the “device inventory” part; this could be used to graphically show the “top 5 devices” that you check off as the most important to you. It can be something simple as “Refrigerator-<25% confidence, multiple similar signatures. Working on improving detection” or “AC-Very low Confidence, No distinct signature. No detection at this time.” or “Water Heater >50% confidence. Near detection threshold. Working on improving detection”.
Also throw in a random background work graphic-“Unknown motor detected at X time. Unclear device vs noise detected at this time” etc.
I also ponder the way, say, image recognition algorithms do their thing. Years ago if you did a search through iPhoto for “toaster” you’d come up empty unless you’d named an image with the word toaster. Now you may get some hits (I do!) but then the question is would you expect the algorithm to list the confidence? The fact that an image is listed as containing a “toaster” is the confidence. All the other images, one assumes are not exactly zero confidence but do I need to know what their confidences are as well?
Imagine a slider for Confidence 0% thru Confidence 100%. All the way at 100% you get just the Power Meter and no Devices as such, or no toaster images in my example. As you slide to 0% what do you see? Will it actually be useful?
Your point is well taken and I agree - customer psychology demands feedback…
I’m going to point you and @ixu to a little insight on what goes on under the hood in classification - I’m guessing that Sense uses some kind of softmax function to determine what a transition is and when to start treating as detected.
