How Sense does “Native Detection” Work ? - User Perspective

I’m just a user figuring these things out. I posted a few replies over the past couple days to community questions, then realized together, they tell a fuller story.

Device Detection Taxonomy

There are four different general technical types of device identifications in Sense-land. And there could be more in the future.

  • Native Detections - for devices that have “clean” on and off transitions in the timeframe Sense is looking for (1/2 second transitions). Sense learns these on/off pairs as detections using machine learning - more on this later.
  • Special Native Detections - for devices that use lots of power, but don’t have on and off transitions that fit Sense’s 1/2 second detection windows, mostly HVAC and EV charging. Sense has developed special models for these, but they are very dependent on picking up the complex power ramps and waveforms that specific electric vehicle / AC / Heat pump models kick out. My Tesla detections have been found, then lost numerous times because Tesla software often changes charge ramps.
  • Smart Devices that assist with detection - Sense uses readings from a smart device to assist with learning. The actual device still needs to have clear on and off transitions, but Sense gets clues from data coming from the device. Two flavors of this that I am aware of:
    • NDI (network device identification) - Sense monitor uses network traffic to spot ons and off of some very specific smart TV models.
    • Ecobee Historic - Uses Ecobee 5 minute updates to better refine HVAC detection. The data from the Ecobee comes too infrequently to be monitored on Sense directly (bubble would be off even though AC has kicked in), but is very useful in training Sense and improving Sense HVAC models.
  • Smart Devices that measure “detection” - where the smart device regularly updates Sense with the actual usage based on a measurement or calculation from the device. This data can be used as “ground truth” for training Sense, but isn’t directly used to “learn” the specific waveforms of the connected devices. Measurement comes in a few flavors today:
    • Smart plugs like HS300 and KP115 - report back to Sense monitor every 2 seconds or so
    • Hue lights - the Hue hub sends calculated usage info to the Sense monitor.
    • DCM (dedicated circuit monitoring) - additional Sense CTs that report power usage on 1-2 specific 120V/240V circuits in your house.
  • Sense is also working on an improvement called Progressive Device Detection - more on it here:
    [Video] 2021 Data Science Device Detection Updates

Because they are doing actual measurements, the second to last category (with smart plugs and DCM) include the Always On for a device, the rest likely don’t. Example - I have Ecobee Historic and Sense now does a darn good job of identifying my AC units, but it still misses the 6W or so that are always flowing to the electronic control board and thermostat.

What Type of Transitions Qualify as “Clean” for Native Detection ?

Sense is looking for specific on and off transitions for native detection - ones that are sharp, and last less than 1/2 a second or so. Here’s one way to “see” roughly what Sense is seeing, at least as far as traditional detection is concerned - let your Power Meter go for a while in iOS/Android app (specifically NOT the web app). The app Power Meter will tag “transitions of interest” using an heuristic algorithm with the power change of that transition - these are the ones that are most likely to meet Sense’s criteria for traditional detection. Here’s a view of some of the tagged transitions for my house.

If you see your device turn on and Sense doesn’t tag the on transition, or breaks it up into several tags, in the Power Meter, then the transition doesn’t fit the model for traditional detection.

But and even if Sense “sees” a transition in the Power Meter, that doesn’t mean that
it is ready to detect it as a device - the transitions must pass three other tests:

  • There must be enough transitions that “match” so that Sense can actually determine that there is a cluster of similar transitions, under a range of house conditions. More on what clustering looks like later.

  • There has to be a similar off-cluster because Sense can’t do a detection with our a matching off, and vice versa.

  • The transition has to be unique. If your device has both on and off transitions being tagged in the Power Meter, then Sense either doesn’t have enough repeats, or the on/off data doesn’t pass the uniqueness criteria - that would mean that it’s on and off signatures smear together with what seem to be other devices, or are so broad in variation that it might not be one device.

  • I also believe that the “cluster” also has to match a Sense model. I get the feeling that some electronic devices either look vastly different for different ons and off, or that the cluster of feature are so far outside the normal physics-based clusters that Sense has to treat each cluster one as a special case.

How Does Sense Combine Those “Transitions of Interest” and Learn from Them ?

Here’s a partial view of what happens when those “transitions of interest” reach Sense’s “brain” back at the Sense mothership. Each transition has around 20 measured parameters or features associated with it, and Sense drops each transition point into a corresponding 20 dimensional space - here we’re seeing only 2 of those dimensions, some measure of power on the y-axis, and some kind of phase measurement on the x-axis. (you can see that there are at least 17 dimensions or “features” from the screenshot labeling).

Over time, the accumulation of transition points create clusters. If the cluster is confined enough and dense enough, and if it has a matching “off cluster”, it graduates into a detection. I think colors here reflect different clusters, though you have to remember that we’re only seeing 2 dimensions of a 20 dimensional cluster, so the clusters would be much more apparent in 20-D space.

Google does something similar in this experiment- they categorize thousands of sounds by many features, then distill down to a 3D representation. You can roam around the neighborhood of a particular sound and hear similar sounds.

The Problem With Clusters
Clustering works well when clusters are truly distinct and algorithms can cleanly separate them. But unfortunately the real world isn’t usually that tidy. Clusters typically overlap in the dimension space and algorithms have sort out which overlapping cluster a “detection” belongs to. The example on the right shows an example of 3 slightly overlapping clusters. How should an algorithm slice up the boundary regions ? No matter how it is done, there are going to be some conflated detections.

How do Sense “Native Detection” Models Improve Over Time ?

As shown a little earlier, identifying devices is all about finding and divvying up the clusters of transitions in the many dimensional chart above. For purposes of this discussion, let’s call this the transition “dimension space”. So improving detection requires doing a “better” job of picking out clusters within the dimension space without upsetting existing working clusters, as much as possible. I can’t speak to the algorithms used, but can give a little insight into when the various clustering algorithms are invoked - they probably use several:

  • Sense model adds or refines a new model - Sense probably searches every house for likely clusters within the dimension space “neighborhood” where those model types live, and then sets about trying to cleanly cluster any unidentified groups of similar points. Sense might also use the Home Inventory to help, or at least test new model before more broadly releasing.
  • Sense support helps you with a problematic model in your house - in this case, Sense probably runs a localized re-clustering in the neighborhood where the problem detection is occurring. I don’t know what kind of controls they have, but there might be things like “split this cluster into two” or “combine these two clusters”.
  • A user deletes a device - this user action probably invokes re-clustering around the neighborhood where the deleted device lived. I’m not sure how far Sense looks back in time, but the neighborhood is likely to look a little different from when the models were first created, so the re-clustering will likely end up with a different, hopefully better, result.
  • Regular model tuning - Sense recently shared that they do readapt the models for your house every 60 days of so to handle changes due to time of year, etc. This probably results in a “tuning” of the clusters based on the past 60 days of data. Tuning would mean readjusting the boundaries without destroying existing cluster. I suspect that this is also share Sense would detect a new cluster from a device that a user had recently commissioned.

Any questions so far ? This is what learning looks like !

Picture excerpted from video, here:

4 Likes

This is great @kevin1 - I’m going to take a longer read through this today and add it to the New User Guide if everything looks alright.

2 Likes

AOK with me ! Feel free to correct or refine as needed. And feel free to rename things using Sense’s internal vernacular as you see fit - would be great to have users talking the same language as Sense developers. And of course, position correctly, in the light of ongoing development efforts :wink:

1 Like

@kevin1 @JustinAtSense does having a smart plug on something actually help Sense detect the underlying device? Does it help refine the signature for already detected devices? Both? Neither?

I’m a bit of a “seeing is believing” guy and I’ve got 15 or so smart plugs connected. Mix of detected devices, undetected - big, undetected - small. Mostly stuff I’d never expect Sense to find, but some exceptions Washer (detected pre-plug), Fridge (1 of 4, other 3 were detected but not this one), Fridge x3 (detected pre-plug).

Would really just love the factual/definitive answer on if having the plugs does anything for native detection.

My experience is that smart plugs are used instead of detection, but I have read that occasionally Sense does detect a device thats on a smart plug. My understanding is that smart plugs don’t have any benefit (or loss for that matter) for Sense detection. So, no, a smart plug doesn’t help Sense with detection or refinin.

If a device is both on a smart plug and gets detected, you get double counting and have to adjust the settings.

This is my thinking, not based on Sense input.

  • smart plugs supply ground truth on a 2sec (or 6 sec on my part because I have too many smart plugs) update, far different than the Sense monitor. So smart plug data is somewhat useful for telling Sense when devices are on and off, plus seeing the power usage patterns, but not useful for pulling out most of the parameters/features Sense uses to actually do detection or classification. That informs what Sense can and can’t do using the data - so no direct characterization of devices using smartplugs.
  • I would suspect that Sense uses the envelope supplied by smartplugs for specific device types to study the characteristics of the mains during those periods to develop new detection algorithms.
  • Sense could do machine learning by “training” on the standard features Sense extracts against the major transitions ground truth detected by the smart plug. That would enable them to build cluster maps for the specific devices. Those maps could help refine the clustering algorithms Sense uses as well as helping Sense assemble multi-component detections (washer, dryers and the devices that have multiple steps in an operation cycle).
  • There might also be useful applications for progressive detection. TBD.

BTW - one of the things that confuses people about detection, is that they see the time domain waveform as the device “signature”, without really comprehending that the real signature is really a many dimensional cluster of points.

Just my thoughts.

1 Like

@kevin1 response above is correct. But, to answer your question directly, no, the smart plug does not automatically feed that signal back into the detection algorithm for a variety of reasons. The posts stating that someone added a smart plug and then saw a native detection of the device come through are coincidental.