Converting Web App Data Back to Real Data

I’m on a mission now to “scrape” and recover data displayed in the Sense web app, back into relatively higher resolution data than we can get out of export. The good news is that I have been able to successfully recover data using just the web vector graphics data. But it’s still challenging:

  • Data is displayed by Sense as a HTML vector path construct - pretty easy to extract
  • The current path data for a day on the screen actually consists of about 3 days worth of data - has to be clipped to a single day window
  • The data is all in terms of the on-screen coordinates - data has to be rescaled to a 24hr period and to the vertical scale maximum for that day

At this point in time, I have 4288 data points per day all properly scaled, with a few more challenges to unpack.

  • All the data is in a path format so I still need to reduce to a single point per time format - simple reduction
  • But before that, I have to make sense of the data which has 2 (or really 4 until I do the path to point reduction) power values for each time point. The way Sense displays the data, it does a right-ward sweep of data, then comes back with a left-going sweep via the path construct where the right going and left going sweeps, put down different power values for the same point in time.

For well behaved days, the numbers aren’t all so different and the left looks like a shadow of the right trace.

But other days, where there has been a data dropout or 3, I get less correspondence between the right and the left:

Any creative thoughts on the best way to resolve the two lines ??

Matching Sense web app view below.

1 Like

This is great and something that I was starting to explore but knew you were in deeper …

Thought: Abandon the left sweep?

looking (i.e. not at the underlying path data) at full resolution you always see spurious verticals that don’t correlate to displayed wattage when dragging left/right.

Have you tried smartplug data at your 6s sample? [Edit: I mean looking at it in app vs web … realizing you can’t get device data on web]

1 Like

Good suggestion. I’m actually going to try separating the right scan from the left, then aggregating both into hourly energy/power. Then I’ll compare against the Sense hourly exports and see which one comes closest. First I have a little code cleanup to do and then I need to move back to my highest res monitor because the amount of data delivers in the HTML paths increases with browser window resolution and width, as far as I can tell. More later !

BTW - with my 5120 x 2880 monitor, I can extract a data point every 35 sec or so. Still not every 1/2 sec or every 2 secs, but much better than once per hour. Let’s hope the Power Meter graphs are accurate.

2 Likes

Here’s the test result - a moderate success. Tested on 5 days worth of data and the results look linear, though there is a bit of variation depending on whether I use the right sweep or the left sweep, and whether I use the vertical or horizontal moves of the path as my data points. It’s also clear that I need to add a fudge factor into my power/energy calibration to get matching results.

This is a comparison between Sense supplied energy data vs. the computed aggregated energy data from my graphics grabs of each day.

Next step is to do for a much larger number of days, then:

  • Figure out the best fit combination
  • Compute the “fudge factor”
1 Like

Finally managed to work on this a little more and discovered another issue. Sometimes the data scraped from the page doesn’t really match the display window size and doesn’t scale appropriately. Not sure why just yet and it is tough to track down since things are fairly dynamic when I’m actually scraping the data.

This is what my first “big data” test looked like… Crazy data scattered throughout. Many power values way too large. All the data that shows reasonable correlation is down on the bottom in the line that looks like it is zeroed out. Typical data science challenge - finding the signal in the noise.

The good news is that it is easy to detect and remove days where the data exceeds the window size. Once I remove those days I still have 76 days of data to work with. Plus I get a much more linear plot:

No matter how I slice the path data, all four combinations have R2 fits of around 0.9955 vs. the Sense data. Using all the data, my computed energy (from data scraping) = 0.8345 * Sense Energy + 18.9893W. So my fudge factor = 1/0.8345 or basically 1.2.

So hurray - I now have a way of extracting 35 sec resolution data if I really, really want it.