XML response handling from web request

alexjhart · 2017-10-05 14:57:10 UTC

Yes, I am attempting to make internal calls. Many things in SmartThings execute locally from the hub. In fact, this is executing locally. Not only can I see the 200 HTTP status code, but I can also see the GET request from this piston in my web server access logs, so I know that part of the request is working locally. I see others have contributed to the code base to allow this and have had success, for example https://community.smartthings.com/t/google-home-notifier/71237/68. I’m not interested in doing port forwarding. I can go back to device handlers and smart apps if webcore isn’t going to make this happen.

bangali · 2017-10-05 16:07:58 UTC

i do local http get requests with webcore. that works for the header part of the response well and is all i need. i have never had a need to retrieve content from the local ip with webcore. i will also look in to this one a bit and see if i can also retrieve content locally.

bangali · 2017-10-05 16:40:55 UTC

have you already checked $json … are the contents of it empty after the get request?

xml response from a local http call may not be getting returned in the code unless its json.

alexjhart · 2017-10-05 17:26:16 UTC

I have (contents empty as seen above). I’ll try putting json in a test page and hit that.

mtuckman · 2017-10-06 00:38:55 UTC

I just wrote a quick piston to hit my local server. If the response isn’t well-formed JSON, then $json doesn’t work. But I did still get the value in the $response variable.

alexjhart · 2017-10-06 03:15:08 UTC

So the question remains what I’m doing wrong. Do you have screenshots of piston execute and results? Also, what started this ultimately is that I want to do logic based on the response from a device that only returns xml. So does it work with that as well?

alexjhart · 2017-10-06 04:24:22 UTC

Well formatted JSON has the same result for me. Both $json and $response are “empty” ([:]).

ipaterson · 2017-10-06 16:12:41 UTC

@alexjhart I have two more things to check here. First, the 20170906 version of webCoRE (from your earlier screenshot) concerns me. There was a critical bug in one of the earlier commits on that day, it’s possible that you’re running a version with a number of glitches resulting from that. Please update to the latest webCoRE and try this again.

If it’s still not working for you, can you post the exact headers that are being returned when you test with curl? Use the -v flag to show the request and response headers and response body then redact whatever you consider to be sensitive. At this point my greatest concern is that $httpContentType seems to be blank from your earlier log screenshot.

alexjhart · 2017-10-12 02:56:53 UTC

I did a pull from Github. Code in SmartThings is now showing v0.2.0fa.20171011. $httpContentType is still blank. I’m getting 200 $httpStatusCode and true for $httpStatusOk. $response is still “[:]”. curl shows:

$ curl -v http://192.168.1.100/test.html
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.100...
* Connected to 192.168.1.100 (192.168.1.100) port 80 (#0)
> GET /test.html HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.100
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 12 Oct 2017 02:32:26 GMT
* Server Apache/2.4.7 (Ubuntu) is not blacklisted
< Server: Apache/2.4.7 (Ubuntu)
< Last-Modified: Thu, 12 Oct 2017 02:32:10 GMT
< ETag: "11-55b5058df72a2"
< Accept-Ranges: bytes
< Content-Length: 17
< Content-Type: text/html
<
{"test": "test"}
* Connection #0 to host 192.168.1.100 left intact

ipaterson · 2017-10-12 16:08:20 UTC

@ady624 there is a lot of code around web requests but I decided to jump in to this since there are a few bugs specific to local requests. I was going to write them out here but a pull request seemed like a more useful way to show the changes.

@alexjhart I was able to get the same behavior that you noticed testing with local requests to JSON and JSON-like files. The problems were rooted in the separation of code for local and remote web requests. The various fixes result in $response and $json (along with $response.test and $json.test) working for your example, and $httpContentType being set correctly.

ipaterson · 2017-10-12 16:29:46 UTC

Using the data {"test": "test"} in a test.json file and the following piston (code kpyh)

the log goes from

10/12/2017, 12:20:55 PM +103ms
+0ms	╔Received event [Home].wc_async_reply = httpRequest with a delay of 186ms
+239ms	║[:]
+251ms	║null
+258ms	║[:]
+277ms	╚Event processed successfully (278ms)

before the fixes are applied to the following after:

10/12/2017, 1:08:36 PM +819ms
+4ms	╔Received event [Home].wc_async_reply = httpRequest with a delay of 13ms
+295ms	║{"test": "test"}
+307ms	║test
+317ms	║[test:test]
+323ms	║application/json
+365ms	╚Event processed successfully (366ms)

@alexjhart, the webCoRE Piston SmartApp code with this set of changes is available here in case you want to try it out before ady gets time to check and apply the updates.

Edit: Revised output above after code changes due to feedback from @mtuckman

mtuckman · 2017-10-12 16:59:39 UTC

@ipaterson - Shouldn’t $response be the actual response and not parsed and then the $json be the parsed version?

bangali · 2017-10-12 17:01:12 UTC

this seems like a patch but not fixing the root cause?

for(variable in variables.findAll{ !!it }

ipaterson · 2017-10-12 17:07:01 UTC

Yes you’re right, I misread the code for remote requests, $response should always have the raw data. The code is updated now and I’ll edit the example output above.

Not related as this request is not using variables. I was a bit surprised that the request worked at all when Alex said he updated to v0.2.0fa.20171011 yesterday, then realized that exception was only thrown for requests that use variables.

ipaterson · 2017-10-13 02:07:49 UTC

@alexjhart, the fixes for this were pushed out in tonight’s update to v0.2.0fa.20171010. Please update when you have a chance to test again.

alexjhart · 2017-10-13 03:34:11 UTC

Would you look at that!

Thank you

So now we can return to my original request? I’d like to be able to POST XML to a local device AND process the $response (as xml). I know Smart Apps can parse XML, so this is just a matter of adding the support to the piston code, right?

ipaterson · 2017-10-13 03:57:23 UTC

Oh, you need to post XML formatted data as well? We’re back to the point now of being able to do a GET request and act on the unparsed XML string in $response.

I know that the hub response includes automatic XML parsing but I’m not sure what format that is in or how difficult it would be to expose the parsed response for use in expressions. I won’t be able to add that but perhaps someone else could. I think that being able to POST XML (probably just freeform text inputs for the body and content type) would be even more complicated to add.

In the meantime you would probably have to set up a local server to transform and forward requests in a format that WC can produce to the XML that your device requires.

alexjhart · 2017-10-14 03:37:00 UTC

I was really hoping with Webcore that I could get all the logic in one place. I can certainly continue using my custom smartapps and local server code, but was hoping to get away from that. At least you’ve helped me move some things into Webcore while avoiding firewall exceptions. Is it unreasonable to think Webcore could someday have XML posting and parsing?

ipaterson · 2017-10-14 12:10:56 UTC

It’s certainly possible! Please post two feature suggestions with specific details on your use cases, one for providing access to the parsed XML for HTTP responses and one for allowing a variable or freeform value as the request body with custom content type. You may also consider what functions would be needed for properly formatting and escaping XML data.