Parse "GET" JSON data into a variable?

#20

Small interjection here…

I would not trigger this at “0” minutes past any hour. My logic being that pistons often fire a couple of seconds early. IE: If the trigger is at 4PM, it might actually fire at 3:59:58pm, so the hour would record incorrectly.

If you do not want to use a different minute for the trigger, an alternative to this would be to add a 5 second WAIT at the top of the piston, and THEN process the rest of the code. This will ensure that you are fully into the new hour before calculating variables, or sending web requests.

#21

I thought of that, so set the times to be 1 min & 31 past the hour.

I’d also use a global variable to hold the current cost, so other pistons could access this. This piston would just run at the specified times to fetch the current value.

#22

Good point! also noted that @Paul1964 has a solution to that.

The only problem is if we trigger it a minute late is that we need to ask the API for the exact half an hour time slot - eg it must be 15:00 to 15:30. If we ask for 15:01 to 15:31 it’ll actually report back the full hours pricing (which includes 15:00 to 15:30 and 15:30 to 16:00) and the formula won’t work.

Perhaps we could somehow round the time up or down? So if it’s 15:27 it gets rounded down to 15:00 and if its past half the an hour mark it’s rounded down to 15:30 - and the 2nd part is requested in sync.

Any input on that? I think actually the waiting 5 seconds might be the most accurate.

#23

I think my code does exactly what you need. I’ve been monitoring for a few hours and it seems to be working.

The trigger is at 1 min & 31 mins past the hour, but the requested time from the api is in line with your requirements. This is the latest version I’ve been running. If you click test & trace, check the variables after each run (hh:01 & hh:31) and you will see all values are correct.

image
image.png1178×692 49.7 KB

This is the command (from the log) sent from the 19:00 run
+637ms ║║Sending external web request to: api.octopus.energy/v1/products/AGILE-18-02-21/electricity-tariffs/E-1R-AGILE-18-02-21-D/standard-unit-rates/?period_from=2020-3-24T19:00:00Z&period_to=2020-3-24T19:30:00Z

#24

Thanks for the help Paul - I ended up with the follow piston which could be useful if anyone is Googling Agile Octopus for Webcore or SmartThings (there were no results when I tried) as a starting block.

It might not be the most efficient way but it is working so far.

I run every half an hour and then tackle the problem we’d get at midnight (where 23 would become 24 not 00) it will just return an error from the API so it the “value_inc_vat” wouldn’t be replaced it’d stay the same so I run a midnight specific one with the midnight hour hard programmed which should get that last update.

I’ve now started logging them to a global variable and I think i’ll delete the piston variable, it makes more sense to use global as now I can use it as a trigger for other things (might add those pistons here as examples)

@Paul1964 - do you know of a way i can log how long a switch has been on, within a 24 hour period to a variable? What i’m thinking is, whilst at the moment I can get the hot water boiler to turn on/off if the price drops below or above a certain threshold - I don’t want it to run for more than 6 hours a day. If it was continuously on, that’d be no problem as you can use the “turn off when switch has been on for X amount of time” but I suspect on many days it’ll be on/off a few times through the day as the price changes.

Unknown
Unknown.png1178×1206 271 KB

#25

I like your solution to the midnight issue, not the most elegant programming wise, but a good practical solution. I was thinking how to handle it, but I’m not sure webcore has enough flexibility with date manipulation to make a simple programmed solution possible. If things get too complicated for webcore I’ve used a node.js programme running on a raspberry Pi.

I’m guessing you don’t want your boiler to run for more than 6 hours in a 24 hour period. I think you could accomplish this by updating a variable within the boiler control piston. This variable could be reset to 0 at midnight. Each time the boiler starts, record the “on” time, then when it turns off, add the time on (now - “on” time) to the total. Next time its scheduled to come on, check it hasn’t already been on over 6 hours. This could work quite well, if you check every 30 mins to determine if you want to run the boiler, and then just run it for 30 mins before checking again.

How do you determine the boiler should be on based on the current cost? I think this could be very complex, even requiring AI type features. Ideally, the costs for the day & predicted weather would need analysing to optimise on times to use the cheapest rates while maintaining the required temperatures.

#26

If it’s on or not is pretty easy - basically I set it so if the rate is less than 5.1p it turns on.

When we were on economy 7 and a fixed over night rate it would just auto turn on at midnight and turn off at 7am. So about 7 hours was enough to heat the tank up. It has a thermostat in it so once it reaches the full temperate it turns off, but I did find that 7 hours way too long and by 5am it had full charged - so i’d put a smart swtich on it anyway and started it later. I found that the thermostat would cut out and as it would cool off a little bit switch back on- and would carry on flicking on and off for two hours until the circuit was cut - so at 3.5kw it did save a bit of power to have it switch on at more or less the right time.

With this method it could come on at 2o’clock in the afternoon the the power is cheaper (in fact looking at Agile, 12-2pm can be really cheap, 10pm till midnight and then around 4am in the morning.

It’s just if it’s done 6 hours in a day, I don’t really want it running for 14 hours if there are 14 hours less than 5.1p.

#27

Unknown
Unknown.png1178×994 150 KB

I’ve got for this to turn the hot water tank on and off.

Basically using the now created global variable “value_inc_vat” which is the current price of electricity when it rises above 6.51 it turns off and when it drops below it turns on.

At the same time when it turns on it records the time it turned on and when it turns off it records the time it was off - then I can run an expression to give us the time it was on. Of course this doesn’t work properly yet as each time it turns on it’ll reset the start time whatever time it is when it turns on.

I’ve got that variable to be reset at midnight, but I obviously need a way to record all the ones and offs and calculate the time between them for the full 24 hour period. Someone might suggest a loop, i’m not sure how to implement that.

On top of this it might be best to put some more logic there for different price points. Like only turn on at less than 7p IF its later than 5am and it’s not been running for at least 2 hours that day - then you could ramp this up to 6p if it’s later than 3am and it’s not run for 4 hours etc etc. So basically it would only run at 5-7p per kWh for a couple of hours if it’s not already fully “charged” previously at a cheaper rate. This way you can ensure you have some hot water without having it fill the tank at 7p per kWh when there might be 6 hours of 3p or less available in the night. Clear as mud?

#28

I think I get the idea, but its getting quite complex. A concern I’d have with the approach is that its relying on fixed prices. When Octopus change these, you could end up with no hot water!

I’d start building a simple prototype piston. The first step is to ensure the piston is triggered correctly. I’ve not tried using the change of a global variable as a trigger, so I’d confirm that works first. I’m sure others on here could confirm.

#29

The Octopus prices change every 30 minutes - the price i’ve chosen is my level where i’m happy to run the hot water element at a price point. I’ve yet to see a day where the electricity doesn’t drop below 6.1p

Though of course if most of the day is below 6.1p and in the night you’ve got 1p per kWh you want to take advantage of the 1p time instead. So some logic needs to be built up where I only use more expensive electricity if it hasn’t been charging for X amount of time already. Same goes for car (and I can use it’s charge % to calculate that) and for the storage heaters.

The global variable as a trigger is a bit weird - i’ve had to set “when it goes above” to actually trigger anything but if I don put “is above” as well it doesn’t work as I assume once it’s gone above it never matches that rule again as it’s already above, not “going” above.

#30

The global variable makes sense. If it changes above or below, that is an event that can cause a trigger and therefore wake the piston (which then runs from the top) Comparing it to a value is a condition. As you say, once its gone above a value and triggered, it will not trigger again until it has dropped below and gone above again.

My point about the cost per unit is that this could change over the months/years. the 6.1p & 1p rates at a particular time may change as wholesale energy costs to Octopus change. So you will need to keep an eye on these. I wonder if these could be more variable than the times when they charge less? i.e if 1am to 2am is currently the cheapest time, will this always be the case, irrespective of the overall wholesale electricity cost? It would be useful to see some graphs over a few years to see further details.

#31

I see what you’re saying Paul - over time I guess the price would go up.

Generally from looking back at the historic data, the graph shows that 10pm till midnight, 5am-ish and usually early afternoon are the cheapest areas - but it really is totally fluid. It depends on the weather too and how much spare wind energy or solar there is. At the moment its quite expensive overall, but some days there’s quite a few hours less than 3p - and if there is a surplus of wind you get half an hour periods where they pay you to use the power! (At this point I want to make sure everything in the house is switched on!)

4pm till 7pm are the most expensive hours every single day that’s the only thing set in stone. What I suppose you could do is load up the following 24 hour period and get the average price of all the hours outside of this period, then set your maximum at less than the average. This would then make the 6.1p an evolving number day by day depending on what the average is.

If you’re going to process future figures though this is also where you could make sure it cames on for the cheapest 6 hours of the day too. So maybe it is worth coming up with something that gets the new figures at 4pm and processes them ahead of time.

Processing JSON data feed from API - possible for in Webcore or too much? (Octopus Agile API)
#32

I agree with you both here… I would consider making each day dynamic, so no matter how much the rates (economy) changes, you will always get accurate percentages for every hour within the day. Something like:

  • Grab all price points for the day
  • Determine the lowest and the highest dataPoints
  • Convert the lowest to 0, and the highest to 100
  • All other time slots use the same formula to determine the cost as a percentage

This way, a time period showing 20% would always be 20% of the max for that day.
(IE: It automatically recalculates and “scales” itself each day)

Pro Tip:

For high load appliances / vehicles that may only need power a few times a week, I might add extra complexity to account for averages changing by day of week.

Processing JSON data feed from API - possible for in Webcore or too much? (Octopus Agile API)
#33

Yes it’s a bit of a complex one, my original idea isn’t very efficient.

6.1p is now far too high. Tomorrow for instance there are a few hours where its less than a penny and this is when you want your hot water and space heaters to take advantage, but if it’s already fully charged by then you can’t.

The idea solution really would be, you enter how many hours you need a storage heater or the water tank to run a day. So say 6 hours for the water is plenty, it then works out the cheapest 12 half an hour periods the next day and turns on only for those. Even if the cheapest is 8p that day at least it’ll turn on.

Programming this logic, whilst possible because the API gives you the next 24 hours results is pretty far out of my depth unfortunately.

For things like washers, dryers, dish washer etc, there’s an ios app where you tell it how long you need to use the device (say two hours for the wash) and it’ll show you the cheapest two hour period for the day. It of course works on consecutive hours it assumes it’s something which can’t stop until it’s finished, unlike the items above which could be switched on and off more or less whenever you wanted

#34

Popping back on this Paul, it’s 1 year later and $response.results[0].value_inc_vat is just giving me an Error retrieving JSON data part null

I haven’t changed the piston and the API still returns the same data from what I can see, so not sure what’s happened here - has something changed in WebCore?

Edit: Just tested your piston and it doesn’t work there either Paul, so must be a change somewhere? I tried rolling back to 2020 Webcore version but it makes no difference for me.

#35

I don’t actually use this piston, it was just to help resolve your query.

I’ve just tried it and get the same result as you. One quick thought, has the amount of data returned by the api call increased? or has it changed to a multi part message? I know webcore can not handle these.

#36

Not that I know of. Although here is the raw out put now…

{"count":1,"next":null,"previous":null,"results":[{"value_exc_vat":9.79,"value_inc_vat":10.2795,"valid_from":"2021-03-05T03:00:00Z","valid_to":"2021-03-05T03:30:00Z"}]}

#37

You could do with some data before it started to fail.

You may find this useful to look at the JSON data more easily.

#38

I’m just in the process of writing my own external php script to do it from my webserver. This is how I ended up handling the daily amounts sorted by cheapest first and it’s been rock solid so it probably makes sense for webcore to do as little complex stuff as possible and just take the web result and put it in the global variable so other scripts can use it.

#39

Thats what I do with my plex media server, which sends JSON data to an external URL when media playback starts/stops etc. I have a nodes.js app on a raspberry PI that pulls out the data I need in webcore. The main problem is Plex sends a jpeg image of the media icon as a multipart message. I don’t find it too reliable though, I’d be interested in seeing your php script.