Web requests may fail with 406 status code due to incorrect Accept header

[ipaterson]

The Accept header tells the server what type of response it should send back. It should not be set to match the request type, but that is the default behavior. Fortunately we can override SmartThings’ default.

Think about a fast food drive-thru – your expectation is to give money and receive food, right? SmartThings wants to receive money because money was given. The Accept header tells the server that we want food, rather than money or a less preferable thing like… hugs I guess?

Of course rather than burgers, webCoRE actually prefers JSON responses - but it will take anything you give it. Responsible servers will respect the Accept header and throw an error rather than trying to give you something you don’t expect. This affects GET requests and any other methods sending a request formatted as FORM data.

There are a few other fixes in the works for web requests, so if you need this fix quickly and want to apply a code patch, line 3301 in the webCoRE Piston smart app sets the requestContentType and needs the following contentType line added (this code is not specific to you, JSON is the only response that webCoRE understands and “prefers” but anything is acceptable)

requestContentType: requestContentType,
contentType: 'application/json, */*;q=0.9',  // add this line

Thanks to @leanbarton for reporting a piston that was having this issue

GET weather API data in a web request, for color changing bulb?

Oh nope, this is something new. webCoRE is sending a very wrong request header: Accept: application/x-www-form-urlencoded That doesn’t make sense, the Accept header tells the server what type of response it should send back. It should not be set to match the request. I don’t see any code in webCoRE doing this, SmartThings is defaulting that to the request type but I can override it. That’s an illogical default If you need this quickly and want to apply a code patch, line 3301 in th…

[ipaterson]

I need to go back to version 0.3.100 to see what Accept header the old version actually sent. There may be cases where specifying JSON as the preferred format causes responses to come back in a different format (like maybe your piston slogs through an XML response as text and now suddenly it gets nice though unexpected $response.foo.bar compatible JSON data).

This may just end up being a */* to avoid backwards compatibility issues, but hopefully json was already marked as preferred.

[orangebucket]

I don’t see anything inherently wrong with SmartThings setting the Accept header to the request content type by default, especially as it defaults the content type to application/json. It isn’t forcing the content type on anyone. It just means if you are throwing JSON around you don’t need to add the parameters.

What is curious is that webCoRE just seems to have gone along with this without offering any way to set the Accept header.

What is even more curious is that webCoRE sets the request content type to application/x-www-form-urlencoded on GET requests as well as for forms (line 3235). Why on earth would it want to do that? GET requests don’t even need content types as they don’t have content, but if it makes the code more elegant to set it then the SmartThings default of application/json seems more sensible.

The triple whammy is that it seems to have worked. I wonder if, along with seemingly managing to handle redirects by itself, the synchronous HTTP implementation also applies a sanity check to the Accept header and refuses to set it to application/x-www-form-urlencoded.

[ipaterson]

I don’t see anything inherently wrong with SmartThings setting the Accept header to the request content type by default, especially as it defaults the content type to application/json . It isn’t forcing the content type on anyone. It just means if you are throwing JSON around you don’t need to add the parameters.

I believe the problem is the exclusive nature of the Accept header. It may be a great convenience to prefer the same response type (e.g. if it defaulted to responseContentType with a lower quality */*) but requiring it bit us. It may not be a poor choice for the default in limited circumstances but the real problem was the subtle change from the sync implementation (more on that below).

What is curious is that webCoRE just seems to have gone along with this without offering any way to set the Accept header.

This was requested once and can be achieved by sending custom headers. I’m glad you mentioned this though because in the new networking SmartThings appends the Accept header to the default rather than replacing it which could be a regression for anyone relying on custom headers. For example, set the custom header with value \{"Accept": "text/csv"\} and after the change proposed in this thread ST would send the header Accept: application/json, */*;q=0.9, text/csv which if you’re setting a custom Accept header is not what you want.

Why on earth would it want to do that? GET requests don’t even need content types as they don’t have content, but if it makes the code more elegant

Correct, that is exactly the reason. Sending a content type with GET requests requires less code than omitting that header.

The triple whammy is that it seems to have worked. I wonder if, along with seemingly managing to handle redirects by itself, the synchronous HTTP implementation also applies a sanity check to the Accept header and refuses to set it to application/x-www-form-urlencoded .

The sync HTTP implementation had the reverse default logic. Instead of setting the Accept header (contentType) to match the Content-Type header (requestContentType) like the async implementation does it defaulted the requestContentType to the value of contentType if provided. Since we only provided the requestContentType for sync HTTP there was no value specified for contentType and requests were sent using the default Accept: */* header. This logical flip-flop made for an easy mistake in refactoring the web request code.

[orangebucket]

This logical flip-flop made for an easy mistake in refactoring the web request code.

Ah yes, I have to agree as I spent ages staring at it last night and I still didn’t spot that.

[leanbarton]

@ipaterson I’ve since discovered that the temporary fix only works some of the time. Since I pull data from my weather station, sometimes webcore will show variables with “not set” or default to a number (32 degrees for example). Looks like there’s quite a few http get request issues currently?