Need your feedback on a new key-value piston input type in webCoRE


#1

There are many feature requests out there so I attempted to find a common thread that connects a number of them, a feature that could be developed as an important building block for other improvements. The best balance between difficulty of implementation and applicability to feature requests seemed to be a new input type that would allow building out a key => value mapping, I’ll refer to it as “KV Input” in this post.

I have some of this feature implemented but need to gather requirements to ensure that it is flexible enough for all the situations where we would need to use it. This will not be finished soon, but I want to gauge interest and flesh out additional requirements.

Preview

The KV input accepts any number of key and value pairs, currently it looks like this:

When evaluated this produces a data structure like the following:

{
    "time": 1523633104179, 
    "power": 5339.0, 
    "request_id": "something_2"
}

Applications of key-value input

This is a building block for several features; please reply if you think of any other places where this would be useful.

Setting the initial items in a list variable

  • Currently:
    • the initial value of a list variable cannot be set when defining that variable
    • requires one Set Variable statement in the piston for each index that needs to be set
  • With KV input:
    • set the initial value of a list variable in the define block as you can with other variables

Set Variable action with list variables

  • Currently:
    • Index field uses “magic” to determine whether the value you type is a string or a variable name, probably does not support expressions
    • Can only set one index at a time
    • No way to clear out the list
  • With KV input:
    • Index can be specified with the same type of field we use elsewhere, making it obvious that you can choose a variable or a text value and avoiding any potential conflict there
    • Could set multiple indexes in one Set Variable statement
    • Could add a toggle to specify whether the KV input should add to or clear and replace the
      list

Passing arguments to a piston

  • Currently:
    • variable list works fine but requires the calling piston to define and populate variable names identical to arguments expected by the callee
    • can get messy with complex pistons reusing the same variables or with multiple callees
  • With KV Input:
    • pass arguments by value or expression to avoid unnecessary variables

HTTP request body

  • Currently:
    • accepts a list of variables, variable name is used as the request parameter name
  • With KV input:
    • support for parameter names that are invalid as webCoRE variable names (e.g. if you try to name a variable $loc-coords you’ll get _loc_coords)
    • expressions and logic in the request statement rather than polluting the piston with variables and set variable statements
    • dynamic parameter names

Any changes to the request body should be approached in a way that does not preclude the ability to support a custom body (i.e. body as freeform text rather than key-value pairs).

HTTP request headers

  • Currently:
    • only the Authorization header can be set
    • variable input would not work to set others since most headers include the - character which is invalid in webCoRE variables (e.g. Content-Type)
  • With KV input:
    • could allow any headers to be set

Related input types

I expect any field that accepts KV input to offer a choice of three input types: Key-value mapping, Variables by name, and List variable.

Variables by name

webCoRE already has the “variables” operand that allows you to select multiple variables where the variable name is used as the key to map each variable value. That may be somewhat obsolete but I don’t think it needs to go away. It is easier to keep it around for backwards compatibility than to automatically migrate those to the KV input and in some cases it may be more convenient. I will probably rename it to “Variables by name” to better confer that it generates a mapping.

List variable

List variables, while a bit tough to work with currently, can represent the same data that could be added in a KV input. It should be possible to select a single list variable as an alternative to the KV input.

Related topics

KV input is a building block for solutions to the following:

Discussion Questions

  • Is this flexible enough? Sometimes I feel like we need support for deep objects (e.g. build a multi-level object for a JSON request) but a UI for that gets incredibly complicated. If it is (or can be) possible to make lists of lists that might be good enough.
  • Does the field UI convey the key/value association well enough? There did not seem to be enough space to put the key and value side-by-side.
  • Should the operand types (Value, Expression, Device, etc) for keys be restricted? For example, I think that anything requiring the value of [Energy meter : power] as a key would be strange… but I don’t know that we really need to prevent that. It could be less confusing to only show Value, Variable, Expression, and Argument especially since device values would still be available in expressions. Restrict the dropdown or no?

HTTP Web Requests
List variable question
webCoRE Update v0.3.105.20180628: improvements to web requests, reorder variables, collapse fuel streams
Help with JSON POST request
#2

A lot of this is over my head, and I have a hard time seeing where I would use it. But one weakness that I think this could address is the ability to make better JSON calls - specifically where there is a subvalue. Like the “params” value in this post:

Understanding this is building block functionality and might not actually solve that problem on its own…

milhouse


#3

Once I understood (thanks to your assistance) of how to enter the list variables in a single line array, most of my issue was solved. What I really wanted was a multi-dimensional array, but, now that you’ve shown me the way to use the single-line array methods SmartThings webCoRE doesn’t really require that due to the normally limited volume of data used.

With that said, if your KV input method can make the setting initial array variables easier, I’m happy.


#4

I am not sure if this is related… but I would love to be able to pass a string (short text) along with the number to my fuel streams… Right now, it automatically records the date & time, the canister, fuel stream name, and a number. It would be infinitely helpful if I could pass a string along with each “Write data point”

For example, in the pic below, 15 is actually ‘Overcast’, but there is no legend visible, so it relies on my memory of what each number means.

When hovering, I would like for it to display something like this:

temp

Here is a little snippet of code that I am using to draw this out:

(contains(wuCurrent,'Snow') ? '90' : 
(contains(wuCurrent,'Hail') ? '80' : 
(contains(wuCurrent,'Thunderstorm') ? '70' : 
(contains(wuCurrent,'Rain') ? '60' : 
(contains(wuCurrent,'Drizzle') ? '50' : 
(contains(wuCurrent,'Fog') ? '35' : 
(contains(wuCurrent,'Overcast') ? '15' : 
(contains(wuCurrent,'Mostly Cloudy') ? '10' : 
(contains(wuCurrent,'Cloud') ? '5' : '0')))))))))

#5

Definitely I think there is benefit to a key-value input type – particularly with web requests which can require headers beyond the one “Authorization” header that is set by default. I have several APIs I’d like to leverage, but without passing my API key I am unable to use them.


#6

I don’t have any additional use cases to add, but this would be very helpful for me!


#7

I believe this would help me in getting information on a specific call where there are numerous Request Headers and Body. At the moment, i can use Postman to get it but don’t have much luck using WC. This is the cURL code:

curl -X POST
https://api.onegov.nsw.gov.au/FuelPriceCheck/v1/fuel/prices/nearby
-H ‘Authorization: Bearer myAccessToken’
-H ‘Cache-Control: no-cache’
-H ‘Content-Type: application/x-www-form-urlencoded’
-H ‘Postman-Token: 3b1e4b26-0957-4519-9db9-75f50879ecf6’
-H ‘apikey: myAPIkey’
-H ‘requesttimestamp: 05/11/2018 3:30 PM’
-H ‘transactionid: 0001’
-d ‘fueltype=E10&latitude=-33.879605&longitude=151.155464&radius=2&sortby=price’

This is for a fuel price check.


#8

I would definitely like to add custom headers. I am currently unable to utilize a external API because of this.

Thank you.


#9

Just adding my voice to others in support of custom headers for http requests. In my current use case, I need to set a specific custom header to authenticate against a BOND home remote transmitter to use its local API.


#10

I don’t think I’ll be able to work on this but if anyone else wants to take a stab at this or something similar for headers… contributions are welcome =)


#11

Just thought I’d add another voice to the rest in regards to custom headers. Would be nice to be able to make requests directly to my AC unit rather than going through tasker.


#12

Another hand in the air for customer headers. Specifically looking for ‘x-api-key’ to use with AWS Gateway which does not support ‘Authorization’ header. I assume with the popularity of AWS this would be on others radar/wishlists.

curl -X POST -H "x-api-key: theKey" -H "Content-Type: application/json" -d '{"key":"val"}' https://[api-id].execute-api.[region].amazonaws.com

#13

I made a hack to allow for adding a custom header tag if you in a ‘:’ in the string. Update the ‘webCoRE Piston’ application at line 3282 to the following:

headers: (auth ? (auth.contains(':') ? [(auth.split(':')[0]): (auth.split(':')[1])] : [Authorization : auth]) : [:]),

Works fine with AWS API Gateway with API key enabled. Screen shot of my POST function below.


API header in web reqest
#14

Sorry for multiple ‘to me’ replies, but after my initial headers hack determined that having more than one header entry addition would be beneficial in the future. This change will look for the colon ‘:’ and will then parse on a JSON object which requires double quotes to be used around keys and values to build a header map.

Change again is only at line 3282 in the ‘webCoRE Piston’ and is backward compatible:

headers: (auth ? (auth.contains(':') ? ( new groovy.json.JsonSlurper().parseText("{"+auth+"}") ) : [Authorization : auth]) : [:]),

Here is a reasonably complicated header solution tested:

"x-api-key" : "{apikey}", "authorization" : "{apikey}", "Public-Key-Pins": "max-age=2592000; pin-sha256=\\"E9CZ9INDbd+2eRQozYqqbQ2yXLVKB9+xcprMF+44U1g=\\";"

Webcore screenshot:

Which results in the following posted properly as shown:

POST / HTTP/1.1
Host: testit.requestcatcher.com
Accept: */*
Accept-Encoding: gzip,deflate
Authorization: abcefH9rjkl67854rhP082zxcvbS1zbcdewjjNLY
Connection: Keep-Alive
Content-Length: 161
Content-Type: application/json
Public-Key-Pins: max-age=2592000; pin-sha256="E9CZ9INDbd+2eRQozYqqbQ2yXLVKB9+xcprMF+44U1g=";
User-Agent: Apache-HttpClient/4.5.2 (Java/1.7.0_201)
X-Api-Key: abcefH9rjkl67854rhP082zxcvbS1zbcdewjjNLY

{"lastSpokenToTime":"1564146881965","lastSpeakCmd":"Test this function ran at 8:27:27 AM","lastVoiceActivity":"stop","volume":"25","device":"My Test Echo"}

#15

Hello @Bloodtick_Jones
Just wanted to check if this hack will affect existing/working Webcore web-requests that already have an authorization header or would I need to re-write those pistons?
Thanks
Tim


#16

It should work as-is without re-write unless the existing authorization header has a colon “:” which I do not believe is possible since the authorization syntax uses base64 encoding for the (username:password) combination and the “:” is not available.

I check for the colon with “auth.contains(’:’)” statement and then look for the JSON object in the user field; otherwise it will use the original logic that just copies the information from the user field as “Authorization: {your input copied here}”; or if empty does nothing.


#17

@Bloodtick_Jones
Thanks very much for this. Hack works for me. I finally have control of my ac unit directly from webcore so no need to go through tasker anymore. I can also confirm that this had no affect on existing webcore requests.
I’m new to hacking the changes in the webcore code so when it comes to updating webcore to a new version in the future how will I know which line of code to change?
Thanks again
Tim


#18

If you look at the SmartApps you will see now a magnifier has appeared. Click that and scan down and you will see the diff between the GitHub master branch and your change. I doubt if ady624 is in any hurry to refactor his code so the area should not change much. You can just left push this change and then overwrite the local version, save and then go back and publish.

51%20AM

I will create a branch and submit a pull request to the master and see if ady264 will accept it; so this would be part of the solution and a hack isn’t required. Great to hear it worked with your original authorization header pistons.


#19

@Bloodtick_Jones
Yep all still working great. Thanks for the reply. The update seems like it will be straight forward enough.


#20

I thought I replied to your pull request last week but I don’t see the reply on GitHub. Just a few tweaks then this can be pulled into webCoRE so that manual patching is not necessary.