Good questions - I’ll attempt to answer as a user who has exploited exported data for a lot of different projects in the Data Analysis Category of the community.
-
Export is found in the upper right of the Usage (Sense) Dashboard of the Web App (not the iOS or Android app). Right now there is a known bug with the Web App and Safari, so you’ll want to use Chrome or FireFox to view the Usage summary charts.
-
Export Data is the Same as in the Summary Graph - Export outputs the same data that is shown in the summary graph for whichever time period (Day/Week/Month/Year/Billing) you are looking at, though you can choose to export with either Hourly or Daily resolution. So the graph shows you the extent of what data you should expect to see in Export.
-
New Data is Aggregated Hourly - Usage bar charts in the Dashboard are aggregated and updated with new data every hour, usually a few minutes after the hour. So the 10AM to 11AM data will be aggregated and added to the Day/Week/Month/Year/Billing charts at about 11:10PM. Aggregation is the process of summing up the new data incoming data stored at the Sense servers for the new period. Aggregation takes time and compute power, especially for many thousands of Sense accounts that are all updating each hour, so Sense probably staggers the start times of all the aggregations over maybe 15 minutes, so not all of them hit the servers at the same time.
-
Sense Stores Much More Detailed Data in the Cloud (Sense mothership) - Your Sense monitor gathers data at a very accurate level, with the ability to capture millions of samples per second, but then processes the data so that is can send a better organized and far more compact set of data to the Sense servers hosted at Amazon AWS. You can see some of the more detailed granularity data in the Sense Power Meter, which contains half-second resolution data. If you want more info, read this.
-
Underlying Sense Data stored at the Mothership Can be Messy - We’re used to seeing nice, simple Sense data the apps, but the underlying data can be messy, between data dropouts (numerous causes), CT/sensor issues (which garble the data), etc, so Sense often does a fair amount of extra work at the Mothership to clean up the data before we see it in the Power Meter or subsequent displays. In some cases missing data gets filled or garbled data gets reconstituted. Occasionally Sense will offer views of their internal data viewer that includes greater resolution of some regions (not all) of the data, plus separated out by leg/phase.
-
Export uses Pre-Computed Aggregated Data - Even though I wish I could get more detailed data directly from Sense, I think there are two good reasons why they only export data that is already used in the Day/Week/Month/Year/Billing summary. 1) It is already there, pre-cleaned, computed and ready - doing other ranges, would require significant additional overhead that Sense has to pay for. 2) The higher resolution could be messy, with holes, and potentially missing values.
-
You can access higher resolution data form Sense, but it comes at a cost - I use an open source home automation hub called Home Assistant. It has a very nice Sense integration that samples data from the Sense mothership every 2 minutes (configurable). But this approach requires extra hardware and expertise, plus exposes you to the underlying messiness of rawer data than your you see in the Sense apps or Export…
-
50 Users - 50 Different Preferred Formats - The problem with formats is each person seems to require something slightly different, and there will never be a perfect format for each person’s usage model. The current Sense data format is a very complete and simple “long format” that is very amenable to easy reformatting on the user end. That’s really what export is for - so you can reformat and aggregate to the format you want. I think you’ll be much happier with Sense and other home devices that provide detailed data if you learn how to use the relatively complex, but powerful Pivot Table feature in Excel, or the equivalent in most other spreadsheets. You’re using Excel, and the reshaping can be done in just one command.
- Import your Sense .csv into Excel, and select the whole data long data table. Here, I’m looking at my Hourly Export for yesterday.
- Invoke Pivot Table, under the Data menu and click through until you get the Pivot Table Field configuration tab that looks like below. Drag the 3 items with checks into the corresponding Rows, Columns and Summation boxes.
- You’ll get a wide version of the same data that looks like this. Each column shows the hourly usage for each device. There will be one or two special columns, Total Usage and Solar Production, that represent house level data. You won’t see an Other, because that’s just the energy that isn’t accounted for when you sum up Always On and all the devices. You’ll also notice blanks - those are hours where usage is zero.
