Web app / data download feature


#21

Sorry about that! Just sent you a message.


What other 'smart' tech are you using in your homes?
#22

For any of you who are intrepid and know the R language, here’s a rudimentary script that will scrape usage and solar data from the web app, until the export capabilities arrive. It makes use of the RSelenium package to open up a Chrome session on the Sense web app, then logs in and navigates to the usage page.

A few details:

  • You need to fill in your own email and password in the script as indicated.
  • I run the script in fragments since it doesn’t navigate through all the months of data I have. To do so run the login part of the script, navigate to where you want to start scraping data, then run the rest.
  • The scraper actually obtains data for one period before and one period after the one you are showing on the screen. For example, running on the month of December will deliver data for Nov-Jan, if full data is available for all those months.
  • The script has problems with months with incomplete data at the start. So if you installed Sense in mid-Aug, Aug energy and solar data will be offset and padded with NA’s (not available) in the wrong direction. You’ll need to move the data start from the beginning of the month to the day that you actually started collecting data, overwriting the Gas at the end of the month. The script works fine on the current incomplete month which has NAs at the end.
  • The script dumps the complete table to Energy.csv. Not protection against overwriting with the next run of the script, so rename if you are scraping more than one period.
  • Script runs in either the free pure R language environment or in R-Studio
  • You may need to play with the Sys.sleep() timing depending on the response of your system and Sense response times.

library(RSelenium)
rD <- rsDriver (port = 4445L)
remDr <- rD$client
# Login to the website
remDr$navigate ("https://home.sense.com/login")
Sys.sleep(10)
webElem <- remDr$findElement(using = "xpath", '//*[@id="application__main"]/div/div[1]/form/div[1]/input')
webElem$sendKeysToElement(list("your@email.com\n"))
Sys.sleep(1)
webElem <- remDr$findElement(using = "xpath", '//*[@id="application__main"]/div/div[1]/form/div[2]/input')
webElem$sendKeysToElement(list("yourpassword\n"))
Sys.sleep(5)

# Go to usage page for current month.  
# Can add navigational steps to do multiple months
remDr$navigate ("https://home.sense.com/trends/usage")
Sys.sleep(5)
# Extract the date and kWh data on that page...
# Data will include current month (to date), plus previous/next month
webElem <- remDr$findElements(using = "class", 'bar-graph__hover-info')
date.list <- sapply(webElem, function(x){x$getElementAttribute("outerHTML")})
webElem <- remDr$findElements(using = "class", 'consumed')
energy.list <- sapply(webElem, function(x){x$getElementAttribute("outerHTML")})
# Get rid of all the extra HTML gleet (smarter grep minds could probably do in a single line)
date.list <- gsub ('"'," ", date.list)
date.list <- gsub (""," ", date.list)
date.list <- gsub ("<.*/div>","", date.list)
energy.list <- gsub ('', "", energy.list)
energy.list <- gsub ('', "", energy.list)
energy.list <- gsub (' kWh', "", energy.list)
# Pad energy.list to be the same length as date.list
length (energy.list) <- length(date.list)
# Convert to dataframe
energy.df <- data.frame(matrix(unlist(date.list), ncol=1))
energy.df$Energy <- as.numeric(energy.list)
colnames(energy.df) <- c("Date", "Energy")

remDr$navigate ("https://home.sense.com/trends/solar")
Sys.sleep(5)
# Extract the solar kWh data on that page...
webElem <- remDr$findElements(using = "class", 'solar')
senergy.list <- sapply(webElem, function(x){x$getElementAttribute("outerHTML")})
# Get rid of all the extra HTML gleet (smarter grep minds could probably do in a single line)
senergy.list <- gsub ('', "", senergy.list)
senergy.list <- gsub ('', "", senergy.list)
senergy.list <- gsub (' kWh', "", senergy.list)
# Pad senergy.list to be the same length as date.list
length (senergy.list) <- length(sdate.list)
# Add to dataframe
energy.df$Solar <- as.numeric(senergy.list)
energy.df$Net <- energy.df$Energy - energy.df$Solar
# Write to CSV
write.csv(energy.df, "./Energy.csv")