RESOLVED: 1/10 Second Power Outage = Sense Offline

A UPS with an AVR transformer attempts to keep voltage at 120V so Sense would calculate Watts based on 120V always which will throw off your numbers a bit.

I don’t want to open up my sense monitor to check how it’s engineered but i’m going to guess that it’s stepping down voltage to power itself off of 120v. If that’s the case, I don’t see how being out of sync would cause an issue. If the regulator is powered off V1 and V2 at 240V then it would be an issue.

Don, these are great ideas. It’s been requested before to have more user-facing diagnostic data like wifi strength available to users, so I’ll definitely pass that on.

And I love the idea about crowd sourcing documentation. I was brought onboard here to help build up the community and its resources, so I am all for that. You’re totally right that there’s some brilliant people here who could really help out. Let me do some brainstorming on it and then let’s definitely talk further.

@pswired @KHouse75 let me ask one of the EEs here about the viability of UPSs.

2 Likes

I had heard earlier from the support team that there is a watchdog timer, and have seen it work during power outages when I’m away from home. But I think the timer period is fairly long, like 4-8 hours ? That’s beyond my “just wait” period when I am home…

1 Like

That’s correct but when we see the issue where sense goes offline after a quick power outage, it never restarts and reconnects. It’s like it’s in some type of hung state where the watchdog doesn’t even run.

1 Like

I just talked to one of our EEs about UPSs. Sense gets powered off of one breaker leg and steps down to about 4V. A UPS wouldn’t affect powering the monitor, as @KHouse75 guessed. However, it may negatively affect measurement accuracy. If a line goes out of sync it could cause a phase mismatch and this would greatly throw off the accuracy of device detection. In short: not recommended.

1 Like

Thanks for clarifying. I guess I haven’t experienced the brownout style hang. Some interesting stuff at the bottom in the link below on possible brownout issues that could corrupt TI microcontrollers - supply voltage raises too slowly for reset when restarting, or a “Lazarus effect” restart that occurs too early. The challenge is that in this situation, a hardware watchdog timer could also get corrupted.

Brownout issues with TI micro controllers.

I second Khouse’s view that a UPS could help in the longer term outage situation. In my mind, you would only need one on the 120V power leg that supplies the Sense (I don’t think that’s documented anywhere in Senseland). Your Sense would then collect data with near zero current on both legs, 120V on one leg and 0V on the other resulting in near zero power/energy until the power comes back on. I’m betting that the Sense would not be sensitive to a 4-6msec switchover transition of the UPS.

This thread has a lot of comments. What I am seeing are numerous opinions, speculations and theories. What I need to see is data, testing and documentation (with diagrams). I am not saying anyone is incorrect with their statements, just that I would prefer for all this discussion to be backed-up with hard data results and testing by the SENSE support team.

@kevin1 makes a good point about brown-outs. What we typically experience is a power outage that is followed up with several other brief power outages. I’m guessing that the power company is rerouting power when all this occurs. By delaying the SENSE monitor reboot, I avoid all the additional power surges that occur when the power is restored. If the problem is a brown-out, then my relay will drop the power input connection until full voltage is restored.

I also have SmartThings installed at my home. I don’t have to manually reset any breakers or manually intervene on SmartThings behalf. It always successfully reboots and connects to the internet without my help. I live in Texas near the coast where hurricanes are always interrupting power. We have short outages and long outages. A UPS may solve the problem of a brief power outage, but that still doesn’t address long power outages.

What I have asked the SENSE software team to develop is a feature that allows each customer to set their own value that delays the WiFi reconnect process (‘no delay’ to 99-minutes). That allows the internet to be fully functional before SENSE attempts to reconnect. If the SENSE data is being stored for 6 hours, then a delay of 10-minutes before restarting the WiFi connection should not be a problem.

@RyanAtSense - This is the reply I received from ‘Support’. None of this information is documented on your website.

Ticket # 66962
Julia (Sense)
Feb 14, 2018 16:10 EST
Good Afternoon,

Thank you for your patience. Unfortunately, this issue you have been experiencing is a known issue that we are currently working hard to address. Due to this issue, Sense is unable to reconnect to a network when it loses connection due to a power failure. At this time, only a reboot of the Sense monitor will allow it to reconnect to the network.

Regarding the use of phones to reconnect Sense, Sense requires a Bluetooth connection to tether to the phone. Sense only enables its Bluetooth connection during the first 15 minutes of being powered on. As it should be connected to WiFi after that point, Sense shuts the Bluetooth connection completely off. Pressing the WiFi button from the Settings Menu sends a ping to the monitor to enable the Bluetooth, but if the monitor is offline at the time, it cannot receive this ping. As this is the case, you would need to manually reboot the monitor to bring Sense back online.

To answer your specific questions, when Sense drops offline, it will automatically try to reconnect around every 10 seconds, increasing in time by double of the previous attempt, until it reaches 5 minutes per attempt. There is an internal log file in the Sense monitor that records these attempts, but these are unavailable to the user. While I am unable to provide an exact number for the amount of attempts the Monitor will make in each time interval, or the total number of attempts, I hope that this additional information helps in understanding Sense’s reconnection process. We are currently working on a reliable solution for these issues, and will be sure to let you know when this is available.

About the offline monitor notification, the Sense servers perform the check for offline monitors every hour for monitors that have been offline for at least 30 minutes. Unfortunately, it is not possible for the individual Sense user to set custom time intervals to check the status of their monitor. Theoretically, it can take up to 1 hour and 29 minutes before an alert is triggered, but it will more often then not be much sooner than this.

As the issue you are experiencing is an due to an error in Sense’s network chip itself, rather than the speed of your network, the two suggestions you have provided will not resolve the issue or allow Sense to reconnect without manually rebooting the monitor. One possible way to avoid issues with your network is to try attaching an uninterruptible power supply to Sense, but even with this we cannot guarantee that Sense will be able to reconnect without manual intervention following a power outage.

2 Likes

So, since it appears that the sense is internally powered from one 120v leg, then a single UPS does appear that it will do the trick, albeit with a risk of inaccuracy if there is an AVR in the mix, or if the UPS transfers to battery too often due to PQ issues. This is all assuming a standard cheap line interactive UPS, not an online UPS. Keep in mind that sense is not just measuring an absolute voltage value with the AC input lines, it’s using the phase angle difference with the CTs to get power factor data, etc.

Another flicker last night took it down. No response from support yet, but I expect something similar to the response posted above, which was appreciated.

@Dcdyer and @pswired

I will definitely make it a point to get some of this info into our knowledge base, as it’d be useful to have in there for troubleshoot purposes.

I’ve gone back and forth with Support and one of our EEs on this and this seems strange enough that we want to look further at your monitors. Support should be getting in touch with both of you to find out exactly what’s going on.

Thank you, @RyanAtSense . I have a dialogue open with your support team now. Hopefully we can make some progress.

Also, here is a $5 device that I believe will solve the problem:

http://www.supco.com/web/supco_live/products/TD69.html

This timer will interrupt power to a connected device when a drop >=70ms occurs. The time delay is adjustable. This could be wired inline with the 120v leg that supplies the Sense monitor internal power supply and it should fix the problem.

1 Like

@pswired Did you see the picture and my write-up that I posted 5 days earlier in this thread?

If you want a better “On-Delay” timer with all the features built into a single device, then check out this idea. It’s a Single-Pole, Double-Throw On-Delay Timer with a variable timer adjustment. ABB Timing Relays, 1SVR500 Series Item# 1SVR500020R0000 (0.055 sec to 100 hr range for the delay time.)
My idea used spare parts from my brother’s garage.

http://www.galco.com/scripts/cgiip.exe/wa/wcat/catalog.htm?searchbox=1SVR500020R0000
http://www.galco.com/images/abbg/large/1svr500020r0000_p.jpg

There are many other companies that make similar relays.

@Dcdyer yes, I did see your solution and it seems to work well. I was looking for something that was smaller and could mount in the panel next to the sense monitor (I realize the code implications for this are questionable) and that was low cost. I have one of the $5 time delay relays in the mail, so I’ll report back when it arrives and I get it installed.

1 Like

@Dcdyer I just re-read your earlier post and have to apologize for glossing over the details the first time. Didn’t realize you were using the exact same part as I suggested earlier.

I’m planning to omit the double pole relay that you have in your design based on the input provided by Sense staff earlier in this thread that only one of the two AC legs is used for the monitor’s internal power supply. As long as that’s the case, I should be able to just put the TD-69 inline with that AC leg, set the time delay to 5 seconds, and call it a day. We’ll see if that works as planned.

1 Like

Please let me know which phase (color wire) that you installed it on to delay the ‘SENSE monitor’ restart. I’m interested in hearing about your results. Don’t forget to clip the jumper wire that comes with TD-69 relay.

1 Like

Ok, TD69 has arrived and is installed. Confirmed that the black lead to the sense is the one that powers the unit. Disconnecting red leaf halves the power reported but the unit stays online.

Very simple install, will see how well it works:

As long as sense can survive a <70ms interruption, this should do the trick.

Also, I didn’t notice any voltage drop across the TD69, so unless it has some weird nonlinear stuff going on internally, there shouldn’t be a measurement error problem due to its presence.

2 Likes

Well, I got a notification earlier today from my network UPS that the power was out for 5 seconds. Sense is offline and hasn’t come back after a few hours. Dang. I will investigate when I go back home tomorrow.

1 Like

Update: this started working at 5:30 AM this morning with no action from me. I wasn’t home, but noticed that the data started flowing again. Period of data between when the monitor went offline and 5:30 was lost. I was not able to detect any power fluctuations that could have triggered the monitor into coming back to life when it did. No logs in my network UPS, and I watched security camera recordings of some lamps that were on at the time and didn’t see any flickers. Very strange.

1 Like