More data !
I reworked my program this aft to liberate the daily (vs. weekly) Power Meter waveforms plus managed to get it to scrape all 535 days of data in a single go. Trickier than it looks, since the Sense web app doesn’t like snapping to exact day boundaries. I had to pre-compute the URLs for the windows I wanted to look at, then ask for them individually, rather than using the day-to-day traversal in the web app. Runs more slowly that my weekly scraper since Sense does caching when using their menu traversal arrows (plus many more days). I have to build in delays to allow Sense to update the display before scraping…
Thanks to this daily approach, I now have a fairly accurate census of my data dropouts over time. I say “fairly accurate” because I did notice some data overspraying in the time dimension between the the data available via scraping vs. what is visible on the web app. Scraping gives a little extra data outside the viewing window, so my results may have a small number of double counted dropouts on the borders between days.
Here’s a column chart of dropouts over time, with color indicating the other monitor issue I experienced, events where total usage actually went negative due to a reversed CT.
The spike around Sept 13, 2018 was mainly a self imposed “negative total usage” event. My electrician reversed a CT when we added extension cables, and I couldn’t get him back out for several days…
Here’s a view from my “dropout detector” for one of those days - the blue dots are negative crossovers, the red dots are places where total usage “touches” zero. Many of the red dots are occluded by the blue dots because they are same event.
Here’s the same thing in the Sense web app:
What about the bad dropout days, without any negative going Total Usage? Consider Dec. 28, 2018, below:
And the corresponding Sense web app picture:
Daily scraping works, at least for my limited purposes !
Oops - here are the new day-by-day collection and detection scripts. Remember to customize using your login and password before you try.
CollectDataDays.R (3.7 KB)
DropDetectDays.R (1.9 KB)
Enjoy !