OK, I finally broke down and wrote some simple code to scrape the Power Meter waveforms from the web app. Not incredibly helpful if you want to pull second-by-second power data because all one can really pull out is graphical path data. But I really wanted to write some code that counts the number of zero data events (dropouts) occurring over a period of time. That seems possible via web scraping.
I have included two R programs here, the first (CollectData) that uses the RSelenium package to login, traverse and pull data from the web app on a week-by-week basis. The second, DropDetect, analyzes the weekly “waveforms” to detect events where the Total Usage power drops to zeros, and to also to detect events where the Total Usage goes below zero. BTW - I don’t know if it is the new RSelenium 1.75, or better coding and luck on my part, but my mean time between web crashes has vastly improved over my experience about 1 year ago.
Not too pretty here. The Sense web app offers up a huge set of path coordinates for displaying both the Total Usage and Solar waveforms. With a little bit of parsing and reformatting, they can be converted to a .csv format, and saved away along with additional meta data about the data (Range and zero location information). The meta info is needed since all the path geometry data generated by Sense is web-window-size specific. To get the greatest range of data and the best resolution, I ran the web scraping with a full monitor Chrome window. Even with that, I still might be missing some extremely short zero/dropout events that weren’t visible in the 1 week window, but I’m not sure I care about those. I do care about the dropouts visible at the 1 week level though.
I can also plot the waveforms, though Sense, for some reason, presents the Total Usage and Solar y-data inverted in the web app, then flips via a transform, to display. I decided to do all my work on the “raw” display data, which is simple to reformat for use by ggplot.
Here are 3 weekly plots replayed. Red dots are zero-events, blue dots are negative-events, though they are both inverted in my plots. Just ignore the axis values because those are all just relative values used for displaying the data in the web app.
My “bad” end of Dec. - 140 dropouts in the week.
A better new year - only 18 dropouts the next week. I stopped leaving my iPad on realtime display. But also, a big self inflicted network outage on Dec 9th that wiped out many hours of data.
Here’s what negative Total Usage looked like back ins Sep 18. Glad that’s over with !
OK - You have read this far… Here’s the code. LMK if you have any questions… It’s still pretty raw.
CollectData.R (3.2 KB)
DropDetect.R (1.4 KB)