What's new in v23 and Web App v5.5: Smarts Plugs and Themes

[RyanAtSense]

We are very excited to announce two major updates to the mobile and Web apps: Themes and Smart Plugs.

Themes

Themes give you the ability to customize the look and feel of the Sense app. In addition to the traditional Sense orange theme, there is also a theme which should be particularly helpful for users that have red-green color blindness. Each Theme also has an optional Dark mode, which darkens the entire app experience, reducing the brightness of the screen. Note that we do not have an automatic switching feature for Dark mode at this time, but hope to include that in a future update.

You can change Themes and enable Dark mode under Settings > General > Themes.

Smart Plugs

We’re very excited to roll out some more integrations. Sense is now compatible with select smart plugs: the TP-Link Kasa HS110 and the Belkin Wemo Insight Plug. These smart plugs report both on/off events to Sense, as well as energy consumption. This means that you can ensure Sense properly tracks certain devices you particularly care about, as well as providing valuable ground truth data to help Sense get better for everyone. Here’s how to set it up:

1. Before turning on the integration, you should make sure your smart plugs are on the latest firmware. You can do this via the Kasa and Wemo apps.
2. Go to Settings > My Home > Connected Devices, and enable the integration for your smart plug(s).
3. Once enabled, it will take a few minutes for Sense to add your new smart plug devices to your device list

And that’s it! Once added, these devices will function like any other device in Sense, albeit with a couple key differences:

• Control: The added bonus is that you will also be able to control the smart plug device (turning it on/off) right from the Sense app, by tapping the on/off button, shown on both the Devices screen and in the Device Details view.
• What’s plugged in?: For any smart plug device, you will also see a "What’s plugged in?" option in Device Details. We encourage you to answer this, so that you can better categorize the device, help Sense learn, and also ensure that you do not have any duplicate devices (in cases where you have a device on a smart plug that Sense has already detected).
• Idle: As time goes on, Sense will even learn if a device on a smart plug has clear differences between its "off,” "idle," and "on" states. For example, if you have a TV that always draws a few watts even when it’s "off,” and then shoots up in energy when you actually turn it "on," then Sense will actually display an "idle" state instead of “off.”

For more in-depth setup details, please see the associated knowledge base article.

We know that these are not the only smart plugs on the market, but we chose to focus on them for a two main reasons: (1) they are both popular models in the market, and (2) they report not only on/off data but also wattage. If other smart plugs meet these criteria, we will consider adding support for them as well.

In addition, we are not yet training your Sense monitor to recognize the plugged-in device in the absence of the smart plug. Therefore, for the best experience, we recommend that you set up your smart plug to monitor a device or set of devices, and then leaving it in place. That said, the new data collected by smart plugs will be of great benefit to our Data Science team in improving our device models and detection. With this in mind, we hope to utilize them towards implementing such a training feature in the future

For some more technical details on the smart plug integration, check out the help articles on the Sense Knowledge Base.

While most of the above features are also included in Web App v5.5, there are a couple of caveats:

• At this time, theme settings will not sync between mobile and web apps, so you’ll need to change it in the Web App separately from the mobile app.
• The smart plug integration can only be enabled and disabled from the mobile app.
• The “What’s plugged in?” option for smart plugs is only available from the mobile app.

As always, let us know what you think in the comments!

UPDATE: v23.1 (iOS) and v23.2 (Android) have been released. This patch brings two major changes to v23:

• Ability to disable control for smart plug devices
• Updated (and more aesthetically pleasing) teal and purple themes

[frankwin.hooglander]

Awesome new additions My TP-Links are working fine with this (I was on the beta). I have 4 more TP-Links on the way now.

[frankwin.hooglander]

For those looking to add some TP-Links HS-110 switches. Amazon has them on sale right now for $15.99 each

Kasa Smart WiFi Plug w/Energy Monitoring by TP-Link - Reliable WiFi Connection,...

[sean11allen]

LOL. You support the smart plugs that we shouldn’t need because we have Sense. I use several WeMo devices and plugs around the house. I saw the Insight and said to myself “Self, you don’t need to spend the extra on that just get the regular plugs. If you want to see usage info on that plugged-in device, you’ve already invested in Sense - it’ll do it!”

Sorry to be cranky, but man. SMH.

Sean

[kevin1]

The reality is that there are lots of device types, or sub-components of devices, that Sense isn’t able to identify today, marketing videos notwithstanding. The good news is that those devices can be “found” or more completely fleshed out by smartplugs. I’ve been testing with the HS110’s and seen some really interesting improvements…

Sense was able to identify my furnace fan and my AC compressor, but never found the 2-3 other components that fire up when my furnace is heating. To Sense, my furnace looked the same as my furnace fan, just like this (minus the 6 W baseline) …

Now, when my furnace is running with the smartplug integration, it looks like this:

You can see all the other components that Sense hasn’t been able to perceive just yet, plus Sense extracts the associated ground truth data to sort things out better in the future. Here’s heating and the standalone furnace fan, both in action (heating cycles on the left, fan cycles on the right).

And how was Sense ever going be able to able to detect these things that my smartplugs were able to measure and tag ?

Our Sonos amplifiers both in idle and playing music - no signatures to detect.

Our home networking service closet - where are the signatures here ?

Or 2 different washing machine cycles:

The beauty is that these difficult-to-detect items look like any other Sense device, and automatically get stripped away from Other and Always On as appropriate. Maybe in the future Sense will be able to figure them out given all the additional ground truth data,

I see this integration as a super addition to Sense’s capability.

[rex]

just curious why you flipped the Solar and real time Watts display?

[kevin1]

That’s how they display in the Power Meter for an individual device (on top). Different than the Total Usage (main) Power Meter (on bottom)

[sean11allen]

Great reply, thanks kevin1. I totally understand where you’re coming from, however, I have always assumed that the graph we see is an averaging and smoothing of what Sense sees. Based on the number of samples per second we’ve been told are being collected, it would have to be. Therefore, I also have to assume that the wizardry ML models behind the scenes here are spotting the correlations and consistent inconsistencies ;^) and pulling them into a device. At least that’s the hype.

I guess this is just their way of quietly saying “we give up - if you want better detection for complex 120V devices, buy a smart plug.” That just isn’t how the release notes read, because marketing… Anyone know whether detection will still be as good once a device is detected, then pulled off the smart plug? Or are we just investing $15-30 per device we really want tracked?

One of my furnaces is hardwired (which feels “not to code”), so no joy there.

Thanks again for the awesome reply.

Sean

[RyanAtSense]

We’re definitely not giving up. The truth is actually quite the opposite: These devices give us better data to improve device detection. Finally, we can fulfill our data science team’s dreams of real, crowd-sourced ground truth data. This will make device detection better for everybody, and those improvements require no smart plugs at all. Of course, this will take time, but this integration will help us make major improvements to our machine learning algorithms. In the short term, yes, they are also very useful for users to get immediate detection on particularly tough-to-track devices, but moreover, we see their value in control. Sense has been pretty good at giving users actionable insights, but action has fallen into the hands of users. Now we can give you the tools to take action right in the Sense app.

As for your 2nd question, the unfortunate answer is that any devices you hook up to your smart plugs will require those smart plugs to maintain detection. We believe this is ultimately solvable as we pull in better ground truth data, but for now, detection requires a constant smart plug connection.

Thanks @kevin1 for the thorough and insightful analysis.

[kevin1]

@sean11allen,
I don’t see the smartplug integration as a retreat. More of way to move more quickly ahead, but probably because I have some background in machine learning. It’s great for Sense to collect deep microsecond-level data (features) from the house mains, but the fastest and most accurate way to learn, at least at this stage of power disaggregation, is via ground truth feedback (supervised learning). The biggest breakthroughs in more mature areas of machine learning, like image recognition, all came with advent of an enormous, well-populated, TAGGED data sets. Tagging means the addition of extremely accurate “ground truth” data to the dataset.

Here’s a great article on how and why machine learning has gotten so much better at identifying faces, and then just about every other “noun”.

The data that transformed AI research—and possibly the world

And here’s a more detailed retrospective on what a breakthrough the right, tagged dataset can be:

http://image-net.org/challenges/talks_2017/imagenet_ilsvrc2017_v1.0.pdf

The part that’s especially interesting is the narrative about the 3 attempts and strategies it took to finally assemble a big enough and accurate enough dataset to hit critical mass. That was the real breakthrough - essentially a “crowd-sourced” dataset.

[sean11allen]

@RyanAtSense, thanks for your reply and thoughts. I get what y’all are going after here. It is pure and objective data versus all of us saying “we’re SURE that’s the fridge! (we think)” as we find patterns in our data. Weeding through the varying levels of diligence in us coming to those conclusions would make everything a mess - especially if you start treating that data as gospel (labelled data serving as input to a supervised learning ML model). It would be a way of distributing out the data collection and labelling in a much more controlled manner. It’s frustrating to see that once the pattern has been identified, I can’t pull the device off the smart plug and move it to the next tough-to-track device in the house. I need to go buy one for all those devices.

Question - what happens if I plug my furnace/blower into a smart plug and the pattern that comes back overlaps with what Sense already identified as the furnace? Do I need to delete the old furnace device and lose all the (less accurate) historical data?

@kevin1 - I’ve done a fair amount of reading on ML. I’ve read the ImageNet backstory before. I’ve also read about more recent approaches like GANs that require far less training data (and labelling). I had assumed that was what was being used here. I am nowhere near the smartest guy in the room on the topic, so I probably vastly oversimplified. I know there is no magic fairy dust. I do know, however, that several data scientists and engineers employed by Sense have been working on this problem for years.

Yes, this allows Sense to lean into collecting more ground-truth feedback to improve the overall detection capabilities of the system. Positioned as that, I can certainly wrap my head around it. It is a way to tell all customers that if they’re frustrated by the lack of detection on some devices in their house, there is an answer that is more deterministic and reliable than “yup, that’s my furnace.” For that, it’s great news. I just wish I’d know this was going to be the chosen path before I bought all my dumb smart plugs. ;^)

Thanks for all the thoughtful replies!

Sean

[kevin1]

Bummer that you bought the plugs too early… A few of us in the beta had been strongly hinting at the smartplug models to buy on the forum over the past month or so, but couldn’t share specifically.

As for GANs, they are great and amazing to extend machine learning, but they almost always require an even more robust dataset with ground truth, than basic machine learning.

Here’s a super simple explanation of how a GAN for image recognition works. Both the generator and discriminator use the same core dataset of real, tagged images. And, mind you, the discriminator is essentially the same neural network one would use for image recognition, and the generator gets smarter and smarter about creating “fake versions” of images of things based on feedback from the discriminator.

Here are the steps a GAN takes:

• The generator takes in random numbers and returns an image.
• This generated image is fed into the discriminator alongside a stream of images taken from the actual dataset.
• The discriminator takes in both real and fake images and returns probabilities, a number between 0 and 1, with 1 representing a prediction of authenticity and 0 representing fake.

So you have a double feedback loop:

• The discriminator is in a feedback loop with the ground truth of the images, which we know.
• The generator is in a feedback loop with the discriminator.

Credit: O’Reilly

https://skymind.ai/wiki/generative-adversarial-network-gan

I wouldn’t be surprised if Sense is already using GANs, but it’s tough without a critical mass of devices. Kind of like trying to recognize all the animals in the world after only being exposed to the creatures of North America.

[sean11allen]

Another great reply, @kevin1. I’ve been making a few, likely flawed, assumptions.

1. There is a large enough install base of Sense customers.
2. There are enough data scientists and engineers at Sense.
3. The combination of those across the amount of time the problem has been attacked would’ve created a dataset large enough to be more accurate.

I don’t say that to minimize the magic that has already been created. The number of things that can pull power in a house is daunting, no doubt - multiplied by the near-infinite variety of ways each can consume it. Just had in my mind that we’d be farther along at the recent pace of technology innovation (especially in ML). I totally get that the hardest thing in ML is data.

Have a good one.

Sean

[mattlebaugh]

@sean11allen

Question - what happens if I plug my furnace/blower into a smart plug and the pattern that comes back overlaps with what Sense already identified as the furnace? Do I need to delete the old furnace device and lose all the (less accurate) historical data?

The integration merges existing devices. You tell it “what’s plugged in”, and if that is a device sense already knows about.

[kevin1]

Thanks @sean11allen.

I peg the electrical disaggregation (ED) problem as a harder one than image recognition problem for a few reasons:

• time domain / resolution - ED has to look at patterns over a large and variable range of time. Some on and off signatures only last a few 60Hz cycles. Other on and off patterns are much larger, like the ramp of a EV charger which can sometimes take minutes. Virtually impossible to build a single RNN/LSTM that can work across that breadth.
• tagged data set - Sense has built a substantial dataset, but it’s far from completely and accurately tagged. I suspect they instrumented a few homes nearly completely in the beginning to bootstrap their dataset (and to do their promo video :), but most customer data is likely far from complete. There’s no way for customers to completely and accurately confirm each device event that Sense detects 24/7 for weeks on end, nor is there a way for customers to append accurate tags to missed events. Sense does make the best of what humans are able to do - provide device names, select crowd-sourced type, and additional data when it does find patterns it recognizes, plus confirm one-off detection mistakes. Smartplugs should help immensely with tagging, when used with single devices.
• composition - devices can be composed of many varying pattern generators as you can see in my furnace and washing machine traces. There are really two levels of learning here - picking up all the basics patterns, then learning how to compose them together, into a single device.
• noise - house wiring is a noisy environment from an electrical perspective. Lots of devices all changing their current draw, at all times. Some devices like plasma TVs and computers can vary quite a bit in power usage in a very irregular way depending on what you are watching/doing with them. I think Sense’s founders had a lot of experience in digital signal processing (DSP), so they have brought their knowledge to bear on this, especially WRT what features to observe for machine learning to pull out the “signal” from the noise, but it’s a hard problem.

Machine learning has come a long way, plus there are many more people working on RNN/LSTM type problems today. But it took a lot more experts than Sense has, plus a more complete dataset, plus 8 or so years to mature image recognition to where it is today.

[sean11allen]

@kevin1 Understood!

@mattlebaugh Thanks! Any idea what it’ll do when you tell it that the smart plug has a power strip plugged into it that has a series of signatures that currently fall into Always On? I was thinking about quantifying my “network closet” at the house (networking devices, HT laptop, several external HDDs, etc)… Do I get to cheat and call my Always On number lower than it is? ;^)

Sean

[kevin1]

Once you classify a smartplug, be it connected to an individual device or an outlet strip with many devices connected, Sense reworks the forward-going accounting for both Other and Always On to reflect the reallocation of power. Both should shrink commensurately, though the Always On is a little tricky since it is a low-watermark calculation over 24 hours.

[MachoDrone]

There’s a lot to read here. But for everyone reading, I chose TP-Link HS110 only because if the power is lost, the Wemo smart plug does not resume it’s last status… that is, when power is restored after an outage, the Wemo stays off, TP-Link will resume it’s last status of on/off.

[mattlebaugh]

@sean11allen it does subtract from your always on. Additionally it will attempt to determine an “idle” state for the plug. I think some work still needs to be done on the idle. Some devices don’t get an idle state, and you do end up with a bubble for each device in an idle state.

Personally I think the idle states should be merged with always on, where you can drill into the “always on” to see what devices use the most at idle or is still unknown. Maybe a user selectable “show idle device bubbles” option. I have quite a few plugs at this point so my bubbles look like a hot mess. It’s hard to determine what is actually “on”.

Also, that way there is no cheating

[iberthiaume]

Thank you for adding a TP Link Kasa smart plug. I have one of that version HS110 smart plug in use. Looking forward to adding their electrical switches HS200 (I have 13 installed in my house) and HS210 (9 of those switches installed) and HS105 smart plugs ( 5 of those installed.) Thanks again for starting to add TP Link / Kasa smart devices.