I’m the one who originally posted this request and still want it badly. At the same time, I have come to the conclusion that it is structurally hard for Sense to provide the collected data to us in intervals that don’t fit their internal time-steps and aggregation intervals. Right now, those intervals are:
1/2 second - update interval for the Power Meter
1 hour - the fundamental aggregation interval for all the other charts beyond the Power Meter
Days, Months, Billing Periods - used in the Trends.
Thanks for bumping and adding your like to the original request… We still need more (hint to everyone reading this - like the initial request).
I was originally going to post this in response to another topic but my thoughts went orthogonal so here goes …
Not so easy.
A few thoughts:
The historical Power meter rendering using the meta-data would be made more difficult and necessarily slower. You need to ponder the lowest-common-denominator.
There would be (significant?) data storage overhead = cost (with no revenue motivation).
The tagging may help some humans some of the time but it doesn’t necessarily help the mothership.
In order to be exploited by the mothership, the Device model or otherwise notable “tag model” would need some data upload to the mother (AWS) at a higher resolution than what we see. This may well be possible, and even happening selectively, but at scale “for all tags” it seems like a taxing process.
Here’s something to ponder in terms of data resolution vs cost:
1s vs 1hr aggregations would translate to pretty much 3,600 times the data. “Sense” (below) has 20 smart plugs, 3 Hues and 38 found devices and is at the sensible limit of its processing capacity due to smart plug load. “Sense2” is not linked to any smart plugs and has not detected any devices because it’s currently dedicated to my hot water tank circuit.
So the question is, doing the math, could Sense deal, technically and financially, with 29MBps … or even 9MBps … if there is no revenue increase?
Thanks for jogging my memory on something @ixu. I need to reframe of the datapoints, since we’re talking about long-term history. Our current data is available to us in 1/2 second Power Meter resolution, but if you look at the Power Meter back in time, it jumps to viewing in 1 min resolution. So Sense has some kind of storage hierarchy for what’s kept around for customer viewing, probably to help manage data storage and communications costs with AWS. I would guess that they don’t throw the “original” data away since they do backfills when new models are introduced. But we can’t assume that all the high res power data is visible all of the time for customers due to economics and technical constraints.
True. I was going to be explicit and post something like what you did above. Thanks for simplifying (compressing!) my thoughts.
I use Apple’s pedometer “Health Data” as my model for the way Sense likely confronts the data stream and the storage/UI costs. The pedometer is streaming “high frequency” variable bit rate data to the iPhone which is then aggregated and uploaded to the cloud which is then re-aggregated periodically. The process is inherently lossy but matched to the likely need for historical access.
What’s most intriguing to me is the possibility of using local (Device) pattern matching in the DSP to create a lossy compression of the higher resolution data.
We get hints of this when we see artificially flat plateaus in the rendering of historical data. The question is: “How confident is the plateau and how consistent is the underlying high-res signal matching?”
Put simply, is the ultimate compression something like:
“Device X was on between time A and time B and used W watts with 99.99% confidence” or is it
“Device X was on between time A and time B and matched device pattern P with 95% confidence”
#1 is currently what a smart plug can deliver and #2 is what Sense native detection strives to achieve. Both methods have advantages and disadvantages in terms of the backend data aggregation and “compression”; the technical and financial costs.
This is obviously a popular topic. The data is IN Sense (because it plots it). So why are PAYING CUSTOMERS having to create their own solutions? This is ridiculous - it seems that Sense created a great idea, put some flashy marketing around it, and now are sitting back and doing minimal additional investment, but watching the revenue flow in. If I had done more thorough research of the rate of significant improvements to Sense before I plonked down my $300, I would not have done so. Sadly, this seems to be the modern business model - create a “Minimum Viable Product”, sell it for as much as possible, minimize your support and development costs, and leave it to your customers to help each other. I’m really very disappointed - I thought that Sense was different, but more fool me for believing that.
Compared to the top posts in the product wishlist, this isn’t even one of the top 10 most popular product wishlist requests on these forums. We’ve actually wound up developing and already released some of the top requested features from this forum, so this post is somewhat vexing to me…
I think there are a lot of folks here that would disagree with the sentiment that we don’t release significant updates frequently enough, but I could be wrong.
We’re a small company, but I can assure you that a majority of our resources go towards development and data science. I’m genuinely sorry that you’re not satisfied with Sense so far - if you want to message me more specifics about why and what you’re disappointed in, specifically, I would love to help.
@frank_kyne, I would be interested in what you are trying to do with sub-hourly export. I’m the original requestor for this one, but this enhancement doesn’t feel as urgent as it once was because I have access to sampled every two minute Sense data using Home Assistant. New-ish Sense enhancements to improve the underlying data, like DCM, improved detection, and smart plug / Hue / Ecobee integrations, plus cost-oriented features like TOU seem to have bigger payoffs, at least for me.
Maybe I can introduce you to the benefits of using free Home Assistant, on top of Sense:
Pull 2 minute sampled power data out of Sense for all devices.
Acquire and integrate data from multiple Sense accounts
Merge with data coming from other sources (i.e. Ecobee, Tesla charge data)
Use Sense data with other data to build home automations
Any updates on this topic of allowing downloads of higher time resolution data (e.g. 5 minutes or 1 min)?
I am doing analyses cross-correlating data from multiple Smart Home sensors, such as Ecobee (5 min), Aranet CO2/T/RH (1 min), and others, and higher time resolution would help a lot. Understood that this was not a priority in mid-2021, but can @JustinAtSense clarify if it is coming closer to the top of the priority list?
Well, it looks like I will be leaving Sense. I have kept it only because of hopes that I would be able to export existing and future data from my account so I could analyze power usage and coordination of some hardware. The visible graphs reveal the information is available, and I just wanted to be able to export meaningful data.
I’ll sell my Sense and move to another platform that will allow me to export and process the data. This means I will lose all of the data I collected with Sense in the past, but I’m finally ready to let go and find real data export capability.
@cgray, have you tried Sense with HomeAssistant / InfluxDB / Grafana ? If you haven’t, it’s a good way to get all the data you are crying out for as well as a way to control equipment in your house. If you have, it would be interested to hear your issues.
Offers:
Active sensors (all overall and device power measurements): Updated every 60 seconds.
Usage: Current active power usage in Watts.
Production: Current active power production in Watts.
Trend sensors: Data provided for daily, weekly, monthly and yearly scales. Updated every 5 minutes.
Usage: Energy usage in kWh.
Production: Energy production in kWh.
To Grid: Energy exported to the grid in kWh.
From Grid: Energy imported from the grid in kWh.
Net Production: Measures total solar production against usage.
Net Production Percentage: Net production as a percentage value.
Solar Powered Percentage: Percentage of power used directly from solar.
@kevin1, I pulled my Sense unit before I recently started working with HomeAssistant. I’m under the impression the integration cannot be used to pull old data. The old data is what is most important to me right now so I haven’t tried to integrate.
@cgray, you’re correct - you can’t Sense sample data in the past using the Home Assistant integration, because it collects real-time data from Sense every minute. There are ways to dump Sense hourly resolution into Home Assistant / InfluxDB for analysis.
I don’t think going to find another solution that collects and stores high resolution house / device power data on their dime for you and then makes it available to you on-demand. And once you need to collect data on your own hardware, I don’t see a better combo than Sense / Home Assistant / InfluxDB / Grafana.
@kevin1, I appreciate your work at defense of Sense. Hourly data has almost no value when looking at the whole home; especially when Sense hasn’t identified your devices completely or correctly. I have been able to browse the data and see the image with the necessary resolution, so they already have the data available and stored; but I cannot export the data.
One of my primary motivations to export higher resolution data was to calculate some usages on my own where I can see what was using energy but where Sense didn’t have it sorted.
It was my mistake buying into Sense when they talk about how they collect so much data; without first confirming that I could actually get some useful variation of that data as the customer. At no point did I think I would get data points “over 1 million times each second”, but I imagined a company so data-oriented would recognize the value of reasonable-resolution data for customers.
I would even pay for the related historical data, but that isn’t an option.
I will be storing data on my own hardware going forward. Since I know Sense will not successfully identify many of my devices, I don’t consider Sense + HASS + etc. as my preferred solution since I will need many individual sensors.
Good luck ! I’m trying to build out a Sense-like system based on Home Assistant and other sensor types. It’s a WIP, and I’m discovering that mating sensors with different sample periods and behaviors is harder than it looks.
