I was trying to setup Pi with small monitor display current solar production/consumption at the kitchen, and it seams like it’s only possible to pull data from sense servers or stream it over the web socket which seams like overkill for some use cases when people just try to use it as current meters for other projects. It would be nice to be able to pull data locally from the device.
I understand it require some work to implement a web server on the device and implement API for it. But create UDP broadcast with on some port, with basic info about our solar production, consumption voltages for each line should be simple enough to implement. If people able to pull data locally it would reduce load on sense servers , which are not cheap I imagine.
I think you’re missing one of the biggest pieces of the puzzle here. Doing all the data collection and access locally ignores the whole signature recognition, identification and machine learning that requires extremely detailed data back at the Sense mothership. The quantity of data that Sense presents back to the user is relatively small compared to the amount data required for training Sense machine learning algorithms.
If your are really just looking for local access to your net usage data, and have a smartmeter, you might be better off with something like this.
I have one, and it has the kind of API you describe, but I find the data very incomplete compared to Sense.
Of if you want to see your just your solar production, your inverter is likely to have a realtime web interface. Mine does, but once again, it is grossly inferior to Sense’s UI.
I understand that Sense feeds a lot of data to Sense’s servers, and I don’t have any problems with that. My suggestion was about simple local data feed people can utilize for home automation projects and why not , without poking Sense server in order to get this basic data. It’s an addition not a replacement for sense web interface or mobile app
I get where you’re coming from. But I think you’re making one more assumption that’s likely not true - assuming that the Sense monitor is transmitting data similar to what you see in the Sense UI. But that’s not really the case - it’s spending all of it’s CPU power collecting current and voltage data at up to 4M samples/sec, performing math and DSP function to calculate power and phase, as well as filtering out any unnecessary raw data, then compressing, packaging and sending. Not the nice, simple 2x / sec combined phase power output you see in the UI.
It seems like there should be plenty of horsepower left in the monitor to take an instantaneous real power reading from the main CTs, package the value into a UDP datagram, and blast it out onto the network via broadcast (or multicast?) once every second or so. This is how the WeatherFlow product works. There is a separate local UDP broadcast mechanism that sends all the observation values out for local processing by third party systems, while all of the fancy cloud-connected stuff is still going on. They also have a cloud API if you’d like to get data that way, but for many applications the UDP stream is the way to go.
@pswired,
I’m not so sure there is enough horsepower in the current monitor for even current activities given the occasional dropouts I see - I have some 30 devices on smart plugs. I think my current Sense misses some deadlines, and would be even worse if it had yet another set of reporting/communication/broadcast deadlines to hit.
Interesting. Well, I certainly can’t offer any decisive information about the platform, but it seems that if the known model processing is being done locally, there should be a lot of CPU available. But maybe that’s being done in a separate FPGA or DSP?
@pswired,
I think most of the realtime math (P= V * I, phase calculations for both legs) and DSP functions (filtering, who knows what else) are done in the FPGA. But that still leaves a lot of tasking to be done in the ARM processor - data collection from multiple sources (4 CTs, 2 voltages, outputs of the FPGA, data from all the integrations), communication with and local processing of all the integrations (NDI, Wemo, Kasa/TP-Link, Hue, other), compression and communication with the Sense servers.
I’m not sure what you mean by “misses some deadlines” given the hardware they have it should have plenty of firepower to issue UDP broadcast since the device has that data readily available when it uploads it to a cloud. So instead of sending it only over the socket/REST you would just dump part of that data over UDP broadcast (you don’t care of it was received or not) it’s literally 6-8 integers. Besides it can be easily configurable with a simple flag and disable by default if you don’t need it.
Deadlines are time targets set by a tasks running on a realtime OS, for completion of prioritized tasks. The idea is that the OS gives priority to the most essential (highest priority) tasks accomplishing their objectives within a deadline. There are probably 20-30 realtime tasks going on within the Sense monitor all at the same time, all with different degrees of urgency and timing deadlines. In a realtime environment like this, you don’t just add some code to poll and broadcast every second. You add a task with a deadline of broadcasting every 1 sec, plus an associated priority. The realtime OS scheduler then dynamically works out how fit the code in (or not if higher priority tasks are taking up the CPU).
My supposition is that I am already seeing cases where lower priority tasks are getting dropped. Adding yet more tasks to the mix, is a formula for even more dropped data sampling from the smartplugs. One other possibility that might explain my dropped data could be broadcast collisions between requests being sent to the smartplugs and the return replies, but that again would speak to likely issues in adding yet another UDP broadcast on the Sense subnet.
@den_by, I think you are also missing another key point here - those 6-8 integers come from calculations that may not be entirely done on the Sense monitor. Remember that the numbers you see are aggregated 1/2 second power numbers that come from microsecond level data. There’s a whole hierarchy of data managed by the Sense probe out to the Sense servers and back again - looks kind of like this…
Raw data generated at the probe/monitor (1 microsecond resolution for some of the measurements)
Specially processed data that is sent back to the mothership that is stored and used for identification and training (who knows how they compress and extract machine learning features - resolution somewhere between 1 microsecond and 1/2 second). But this includes separate power data for each house leg, plus realtime phase information.
Realtime data that is sent to our apps from the mothership when we see the moving waveforms and bubbles - 1/2 sec resolution aggregated from rawish data, some computed realtime (Always On, Other bubbles, some identification bubbles), though some people are also reporting data delays in receiving this data.
Historic display data that is sent to our apps from the mothership when we either:
are displaying realtime data, but need some additional history to fill out time window
go back in time and/or change resolution.
Identification and summary data - rolled up from the historic display data AFAIK. Used for device displays, trends, usage and export.
I think you keep assuming that what you see in the UI is all queued up and ready in the monitor, and just needs to be broadcast, but most likely not. And broadcasting instantaneous readings every second or so isn’t useful in seeing the real picture of power usage.
I don’t think getting historical data or any other data from “mothership” was part of the feature request, there is an API to pull that data directly from sense servers if required. Having a raw data from probes is good enough in most use cases, if UDP broadcast gets drop once an a while it still good enough. I think you’ve overblown the complexity of the feature.
OK, precisely where on the 60Hz waveform do you want to sample ? And do you want to hit the same exact point on the waveform every second or move the sampling around a little randomly ?
I don’t think you really want instantaneous samples of an AC waveform. What you really want is an RMS (root mean square) average, which requires a many datapoint calculation. I suspect that is indeed done on the monitor since those values can be seen in the monitor pane under settings, but again I’ll make the case that adding another another monitor task, might not be a good idea.
ps: There is an informal API on github that will give you this data today, but not via local access.
I’d really like a local API of some kind as well. An option to enable a broadcast/multicast UDP packet at whatever interval the Sense could spare the processing power would be fine by me.
As an aside, I’d argue that it has at least some amount of processing power to spare most of the time. Otherwise it would have a great deal of difficulty handling any broadcast packets coming in over the wifi. Devices such as PCs or smart phones will send broadcast packets for various kinds of device discovery.
Before I bought a Sense, I had a ZWave home energy monitor that would send updates to OpenHAB every 15 minutes - something like that from Sense would be great, in my book.
