{"id":928,"date":"2022-02-28T06:55:37","date_gmt":"2022-02-28T06:55:37","guid":{"rendered":"https:\/\/www.crccheck.com\/blog\/?p=928"},"modified":"2022-02-28T06:57:12","modified_gmt":"2022-02-28T06:57:12","slug":"tracking-my-electricity-usage","status":"publish","type":"post","link":"https:\/\/www.crccheck.com\/blog\/tracking-my-electricity-usage\/","title":{"rendered":"Tracking my electricity usage"},"content":{"rendered":"\n

My electric utility gives me nice a good amount of data, including a download button! This made me want to get this data into Grafana so I could explore it better.<\/p>\n\n\n\n

\"\"<\/a>
Screenshot of electricity usage in 15-minute increments contrast with outdoor temperature<\/figcaption><\/figure>\n\n\n\n

Getting the data<\/h2>\n\n\n\n

That “Download my data” button is tempting. The data ends up looking like:<\/p>\n\n\n\n

TYPE,DATE,START TIME,END TIME,USAGE,UNITS,COST,NOTES\nElectric usage,2022-01-01,00:00,00:14,0.14,kWh,$0.01,<\/code><\/pre>\n\n\n\n
and I noticed the API returned a little bit more like:<\/p>\n\n\n\n
    {\n      \"startTime\": \"2022-01-29T00:00:00.000-06:00\",\n      \"endTime\": \"2022-01-29T00:15:00.000-06:00\",\n      \"value\": 0.552,\n      \"readType\": \"ACTUAL\",\n      \"providedCost\": 0.03219264,\n      \"readComponents\": [\n        {\n          \"tierType\": \"ORDINAL\",\n          \"tierNumber\": 2,\n          \"season\": null,\n          \"dayPart\": null,\n          \"cost\": 0.03219264,\n          \"value\": 0.552\n        }\n      ],\n      \"isPeakPeriod\": false,\n      \"rebateAmount\": 0,\n      \"milesDriven\": 0\n    },<\/code><\/pre>\n\n\n\n
Now not all that JSON is useful, but it I also wanted that temperature data too, so I wanted to scrape the JSON instead going after the CSV.<\/p>\n\n\n\n
Scraping Attempt 1: Can I just curl<\/code> it?<\/h3>\n\n\n\n
I use my inspector’s “copy as curl”, made some tweaks, and it worked! That is until my session timed out. I tried keeping it alive, but it became clear that wasn’t going to be sustainable. Searching around, I found one other person attempted this same problem…<\/p>\n\n\n\n
Scraping Attempt 2: Puppeteer<\/h3>\n\n\n\n
When I search for other attempted scrapers, I search for pieces of the URLs in Github. Doing that, I found https:\/\/github.com\/mhoran\/coned-rtu<\/a>. That project uses Mechanize<\/a> to do browser automation. I just had some PTSD dealing with migrating Python projects to a M1 Mac so I chose to use NodeJS instead. I had good luck writing scrapers with Puppeteer<\/a> before so I chose that. After a snafu with setRequestInterception<\/code>, my scraper was working.<\/p>\n\n\n\n
Metrics Attempt 1: Prometheus<\/h3>\n\n\n\n
Prometheus is my go-to for metrics. It’s CNCF approved<\/a>. The query language is amazing. What’s not to like? Well apparently two things:<\/p>\n\n\n\n
  1. Generally, Prometheus doesn’t like gauges<\/strong>. Electrical consumption works well as a counter, but the data format is as a gauge<\/li>
  2. You can’t backdate data.<\/strong> This data is historical, not real-time. Prometheus is designed to follow real-time data<\/li><\/ol>\n\n\n\n
    Metrics Attempt 2: InfluxDB<\/h3>\n\n\n\n
    I’ve known about InfluxDB and I’ve been wanting to try it. To do some due diligence, I compared it to other Grafana data sources<\/a> like OpenTSDB<\/a> and InfluxDB<\/a> looked like the one for me. Even better, they had a hosted solution with a free tier so I could skip setting it up myself.<\/p>\n\n\n\n
    Final-ish product<\/h2>\n\n\n\n
    After a few hours, I was able to figure out infuxdata.com’s dashboard interface enough to get this:<\/p>\n\n\n\n
    \"\"<\/a>
    InfuxDB dashboard of the data I was able to build<\/figcaption><\/figure>\n\n\n\n
    There’s a lot of oddities, like why is the x-axis of the last chart off? Why is it splitting tags into different series? There are lots of features I miss from Grafana too: right-side Y-axis, shared cursors.<\/p>\n\n\n\n
    I pushed my code here: https:\/\/github.com\/crccheck\/opower-exporter<\/a><\/p>\n\n\n\n
    What’s next?<\/h2>\n\n\n\n
    1. Host my own InfluxDB + Grafana<\/li><\/ol>\n","protected":false},"excerpt":{"rendered":"
      My electric utility gives me nice a good amount of data, including a download button! This made me want to get this data into Grafana so I could explore it better. Getting the data That “Download my data” button is tempting. The data ends up looking like: and I noticed the API returned a little…<\/p>\n Read More<\/span> Tracking my electricity usage<\/span><\/a>","protected":false},"author":2,"featured_media":931,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"Layout":"","footnotes":""},"categories":[4],"tags":[12],"class_list":["entry","author-showmewhatyougot","post-928","post","type-post","status-publish","format-standard","has-post-thumbnail","category-technical","tag-js"],"_links":{"self":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts\/928","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/comments?post=928"}],"version-history":[{"count":2,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts\/928\/revisions"}],"predecessor-version":[{"id":934,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/posts\/928\/revisions\/934"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/media\/931"}],"wp:attachment":[{"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/media?parent=928"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/categories?post=928"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.crccheck.com\/blog\/wp-json\/wp\/v2\/tags?post=928"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}