I like @MachoDrone 's idea of isolating specific loads by putting the cts directly on them. Overall use/production has been my primary goal, device detection was a bonus, but it seems like it just might work.
1 Like
Unfortunately, the Sense hardware is limited to two sets of CT’s, one for Solar and one for total consumption. So, without new hardware, I don’t see how they could do this.
@RyanAtSense - feel free to take a look at my images above, &/or my account and let me know if you have any practical tips for me.
I get that my (& other people’s) suggestion to consider other machine learning algorithms &/or training approaches is a much larger discussion at Sense, but one very practical suggestion I would like to see implemented - would be to allow your users who are interested, to upload a list of known devices on our power grids to Sense just to let you know that they are present.
For example, if I could tell Sense that I have a Whirlpool ABC123 refrigerator - it doesn’t matter if 6 months later that device is still undetected on my network, no harm there. But if @NJHaley has the same Whirlpool ABC123 refrigerator and Sense has correctly identified his, then perhaps Sense would have a better shot a possibily identifying mine someday. I do understand, that at a million times per second the signal-to-noise ratio and the uniqueness of two power grids could completely void the marginal benefits of knowing which devices are where, but I also can logic through examples when have a ‘list of known devices and model numbers in their homes’ would help.
Sampling a million times per second would also seem to create a lot of possible noise. Disamgibuation of devices with this volume of data is likely necessary when trying to differentiate between running devices, but seems like it could over complicate general identification when Sense is trying to figure out if a whole house fan is running, or if it isn’t… perhaps a sampling strategy that only models a user’s power consumption 1 time, or 10 times per second could assist generic device detection.
Lastly, a ‘list of known impossible devices’ could also prove to be valuable. If I told Sense that I don’t have a single large pump on my power grid, then a few false positives could be avoided.
Just a few musings to make me feel like I have any influence to improve my consumer experience with a device that I had very high hopes for when I became a customer. Thanks for reading.
2 Likes
I like @MachoDrone 's idea of isolating specific loads by putting the cts directly on them.
1 Like
Kinnwey.dan are users then expected to update this list of known devices as electronics are added/removed/ and replaced in their homes? The ideas that are presented for manual identification sound unscientific and error prone. Introducing this extra noise is far more likely to hurt than to help device detection.
1 Like
I disagree @senseinaz, it’s a population issue. If I want an algorithm to learn the 25 primary electrical devices in my home, there’s no reason to ask the algorithm to identify the device power signatures for my 25 devices out of ALL 10,000(?) possible device power signatures that Sense has identified across ALL its installs everywhere.
Said another way, if I were to ask you to go find 25 blue marbles that were hidden in a skyscraper, and I told you that these blue marbles were located on floors 17-27, then there is absolutely no way that you would start looking for blue marbles on the 80th floor and work your way down, right?
I get that uploading the list of your appliance makes & model numbers could be a pain in the rear - but 1) it would be completely optional and 2) how often do most people change out their biggest/most-used appliances? It’s not a list that one would have to micromanage due to frequent swapping our of your double wall oven.
Sampling a million times per second would also seem to create a lot of possible noise. Disamgibuation of devices with this volume of data is likely necessary when trying to differentiate between running devices, but seems like it could over complicate general identification when Sense is trying to figure out if a whole house fan is running, or if it isn’t… perhaps a sampling strategy that only models a user’s power consumption 1 time, or 10 times per second could assist generic device detection.
MachoDrone cannot answer on the behalf of the Sense data scientists team, but from decades of experience as a data analyst, MachoDrone is sure the over collection of 1M X’s per second is fine for the resolution. They may be reducing that resolution at the learning level as needed per algorithm. Also, another possibility, the hardware may be sampling at 1M times, but the data sent to Sense cloud may possibly be a lower resolution today, but may send 1M times in the future.
… … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … … V heart V
1 Like
I think the difference, to use another analogy shows in using water vs marbles. Marbles are a unique defined object. Though the algorithm banks on a unique electrical signatures, it is never separated from the body of water. This is why the support team never wants you to turn off other devices to recognize a signature, because when you turn the full volume of water back on it will be different and unrecognizable. The full body of water is never uniform. It is always changing, therefore the greater sampling and the greater database will always improve the algorithm. We sometimes imagine that once we find the electrical signature, it now has its own colored marble identity and in a way it does, but never separated from the full body of water. The only real measurement is the volume of the water tank which is determined by all the possible pumps. You can’t tag water molecules in the tank. This is also why water pump (electrical) signatures that have a quick ramp are easier to detect because slow pumping gets lost in all the other pumps.
The option that I would like to see in future models is the ability to add one or two additional CT clamps for big devices like EVSEs which they are having trouble with due to their slow ramp to full power. I think this must feel like a cop-out to their data scientist but would help control (not find) some pesky devices.
2 Likes
Hey Dan. I gave things a look and I’m not seeing any glaring issues. My first goto is Wi-Fi and you see to have a solid, stable connection. You could try reaching out to support@sense.com to see if they have an idea what’s blocking detection for you. My immediate guess would just be that you have a very noisy home, as that’s pretty subpar detection for two years.
I think the device list idea is solid and we’re looking into implementing something like that, but as @senseinaz and @markhovis73 note, it’s much more complicated than simply giving Sense a list of devices and associated signatures. Mark really hit the nail on the head…think water molecules instead of marbles, always changing in relation to the rest of the body of water. It’s a tough problem but there are possible solutions and we’re working hard on figuring out the best option.
1 Like
I just mean over-reliance on ML period. It seems like the whole sense thing is predicated on one particular unsupervised algorithm which clearly doesn’t work that well. And use a human guided learning mode.
As a basic point, suppose this magical algorithm does what seems at this point impossible and eventually learn all the signatures of every single device in my house (forgetting the fact that it can’t learn constant loads). It asks you to enter in the make/model of each of these units.
Why not at least ask for this information up front? Then you at least know what your target is.
Then it wouldn’t be too difficult to ask it for an option where it “listens” while you turn the device on and off repeatedly. This strategy would presumably also reduce the “always on” constant load bit as well.
Let’s say you do can effectively learn X% of your devices this way. Then you can residualize your power usage functions to those guys and then simply target the remaining devices. Modifying the training problem in this way is bound to make the problem significantly easier.
@ryanlewis,
I agree that things would go faster if Sense has some supervised learning in the loop in individual households, especially for devices that are viewable via smart controllers. But I’m loosing you on the “single algorithm” complaint. I’m fairly certain that Sense is utilizing hundreds of ML factors for identification and that they have done factor selection for different classes of device types so they really use a number of different “algorithms”. And it’s clear that they will be adding more with their updated focus on car chargers and perhaps other devices that don’t have readily identifiable millisecond on and off signatures.
The idea of providing a device list and some manual on/off training seems attractive, but given what I know from image recognition work, the benefits are illusionary. Manual training, without Sense pre-identification of the “bounding box”, seems pointless. And for many devices, flicking the switch on and off is NOT representative of the true on off cycle (heating, cooling, refrigeration), plus many devices have more complex behaviors that are only visible by letting the device go through it’s whole cycle (washers, driers, ovens). It might feel better to have some knows you can turn, but I believe only automated tagging from smart devices has the time resolution accuracy and consistency required for long-term learning improvement.
2 Likes
The fact that there are many features to the input data has nothing to do with the algorithm that it uses for identifying devices. Also – the problem clearly is some kind of a predictive one, where the training set is labelled device signatures, the goal is to classify other signals to the label. Unsupervised learning, typically some form of clustering, is not by itself sufficient without some knowledge of the predictive function. Also, who cares what type of ML is being used? I certainly could care less as long as it works with high accuracy on test data. Finally, I am not aware of any documentation of the specific algorithms being used. I would be willing to review that.
All I am saying is it’s all too common for a company to say, “we have some ML secret sauce algorithm for doing XYZ” and then it turns out the algorithms either over or under fit in ways that are addressable in a straightforward matter. In this case we can see that the algorithm is doing both, that is failing to fit the space of device signatures (noted by the large collection of Other and “Always On” devices), and also overfitting because it sometimes fails to characterize the ways in which I turn devices on and off. Finally, the correct algorithm shouldn’t care how I turn devices on and off. In particular, if I add new devices, or if ones die or get replaced, or removed. It should not affect the classification of other devices.
That they fail at this is a red flag.
1 Like
@ryanlewis,
Let me just put it this way - Sense had briefing for customers under NDA a while back to share more about what goes on on the data science side. You are pre-supposing a number of wrong things. I highly recommend you attend if they have another one. You won’t be able to review the documentation, but you’ll have a few of your blinders removed.
1 Like
Well, fine if they invite me ill go. in the meantime I paid $400 for a device which does a pretty terrible job and they have hyperbolic marketing about how it will save me money. So far all it’s good for is instantaneous loads.
I can’t see how the information of how many amps and volts (or watts) a device uses and the list of devices couldn’t make the problem easier.
2 Likes
Many of us having lots of trouble were (carefully) not invited. Wonder why?
2 Likes
@andy,
The invite went out over the users forum that you’re using today:
Maybe you missed it or couldn’t make the date ?
I found the session a real plus for users that want a deeper picture of the challenges and possible solutions for energy disaggregation.
2 Likes
I have been running for a little over three months now. I started weeks in with an Always On in the 180s. I am now holding in the 50s which I consider very accurate to my list of devices. I see this number holding constant when my “Other” is quiet even completely off at times. I am still having problems with devices mentioned above including LEDs and my EVSEs, but I’m pretty impressed with my Always On bucket.
1 Like
ok, that may be, but in my house the device cant find many devices, and it regularly conflates one device with another. It tends to discover things like combinations of devices. It’s clearly an overfit ML model. Glad to hear it works better at your house. I still think I overpaid for a device that doesn’t work that well.
1 Like
Ryan, I am not a data scientist so I am going to bring up some of your points with our data science team and get back to you.
But as @kevin1 mentioned, there is some proprietary stuff happening that we cannot share. We’ll hopefully do another conference call soon and hopefully you’ll be able to join us (and @andy, that call was open to the public, it just required advance registration).
In any case, I’ll extend the offer I extended above: if you’re absolutely unsatisfied with Sense, shoot me a PM and we’ll figure out a way to correct it.
2 Likes
I just searched my Sense folder (which has hundreds of messages, all the way back to Feb 17 when I bought Sense), my personal folder, and my SPAM folder. No mention what-so-ever of that conference call. I’d wonder how many others didn’t get the invitation.