Large web request responses error out if response is slow

#1

@E_Sch any ideas? It seems like asynchttp_v1 may be failing on large responses. Here is a sample piston that compares large text and JSON responses:

exs3p
exs3p.png1178Ɨ594 149 KB

In this configuration, the full text of Moby D fails to load with the same type of error (DataCharacterLimitExceededException: Handler data length (526075) cannot be greater than 2500 characters.) but the JSON which is well over the 2500 limit works fine.

Sometimes the JSON errors out and sometimes it does not, so the data type is probably irrelevant. Seems like if Piston waited at a semaphore for 9780ms is logged the piston has to schedule a wakeup in which case the response data, if too large, fails to process.

webCoRE Optional Update v0.3.111.20210130
Smartthings Status Tile
#2

so the issue I saw in HE was in regards to method

It tries to save some variables for the wakeup, and response, or json may be too big (so I truncated them)

private void requestWakeUp(Map rtD,Map statement, Map task, Long timeOrDelay, String data=sNULL){
    Long time=timeOrDelay>9999999999L ? timeOrDelay:now()+timeOrDelay
    List<Integer> cs=[]+ ((String)statement.tcp==sB || (String)statement.tcp==sC ? (rtD.stack?.cs!=null ? (List<Integer>)rtD.stack.cs:[]):[]) // task cancelation policy
    Integer ps= (String)statement.tcp==sB || (String)statement.tcp==sP ? 1:0
    Boolean a=cs.removeAll{ it==0 }
// state to save across a sleep
    Boolean fnd=false
    def myResp=rtD.response
    if(myResp.toString().size() > 10000) { myResp=[:]; fnd=true } // state can only be total 100KB
    def myJson=rtD.json
    if(myJson.toString().size() > 10000) { myJson=[:]; fnd=true }
    if(fnd) debug 'trimming saved $response and/or $json for scheduled wakeup due to large size' , rtD
    Map mmschedule=[
#3

I also seem to be having problems with get requests. I can write data to a google sheet but whenever I try to read data, I just get an empty response of ā€˜[:]ā€™. Works fine in a browser window.

#4

Are there errors shown at account.smartthings.com > Live Logging?

#5

Interesting log:

Error executing external web request: Status 302 raw response: &lt;HTML&gt;
&lt;HEAD&gt;
&lt;TITLE&gt;Moved Temporarily&lt;/TITLE&gt;
&lt;/HEAD&gt;
&lt;BODY BGCOLOR="#FFFFFF" TEXT="#000000"&gt;
&lt;H1&gt;Moved Temporarily&lt;/H1&gt;
The document has moved...

There is a new location I think in the ā€˜ā€¦ā€™. I donā€™t get the same response, though, in a web browser window. Do you think it is a permission issue?

Browser window response:

[[ā€œFebruaryā€,13,62,43,79,1957,53,2017,42,1949,20,1949]]

webcore piston response:

2/6/2021, 8:27:08 AM +20ms
+141ms	ā•‘null
+146ms	ā•‘[:]

with the log above.

#6

So I took the URL from the live log and it also worked in a browser window. When I tried that string in a piston, I get the same null response and now the log says:

8:40:44 AM: error ā•‘ Error executing external web request: Status 302 raw response: &lt;HTML&gt;
&lt;HEAD&gt;
&lt;TITLE&gt;Moved Temporarily&lt;/TITLE&gt;
&lt;/HEAD&gt;
&lt;BODY BGCOLOR="#FFFFFF" TEXT="#000000"&gt;
&lt;H1&gt;Moved Temporarily&lt;/H1&gt;
The document has moved &lt;A HREF="https://script.google.com/macros/s/<MY CODE>/exec?XtabX=NWS+Temp+Records&amp;amp;startRow=37&amp;amp;endRow=37"&gt;here&lt;/A&gt;.
&lt;/BODY&gt;

Which is my original request string. Strange loopā€¦

#7

@E_Sch do you know how to make asynchttp_v1 follow redirects? None of the 3xx redirect codes are working, i.e. in 0.3.110 a web request to https://httpstat.us/301 or https://httpstat.us/302 etc would follow the redirect leaving use with an HTML $response and 200 $httpStatusCode. Now the request ends at 301 $httpStatusCode and since we donā€™t have a way to get headers thereā€™s really nothing you can do with that in a piston.

#8

No, I donā€™t know a lot about web requests. Just have tried to stick to whatever works in a browser window. Usually once I can get it working there, it works in webcore but now they are not acting the same.

#9

Hmmm

I would expect we have all the headers with getHeaders - is that not true on ST?

Assuming it is and if they donā€™t follow redirects for async (but do for sync), then the code could try to re-issue to the redirect location.

May have to ask Adrian if he can get an answer inside ST. I wonder if this is a recent change on the platform?

#10

I assume so. My piston has worked fine for months and now after the recent update it no longer works. Been looking for a work around but donā€™t see how to force webcore to follow the redirects.

#11

I tested on HE, and all I get back is 200, so I assume the redirect was followed using the URL @ipaterson suggested.

I did not see any 301 response that made it to webcore.

#12

HE might be working. Not sure of the differences. I didnā€™t see an actual URL suggested by @ipaterson, just some checking he did for various status codes.

If you look at $httpStatusCode, it returns 301. I made a slight modification to the webcore piston app to get the error data out. I inserted line 3373:

42%20PM
Screen Shot 2021-02-06 at 3.03.42 PM.png2364Ɨ264 47.4 KB

This along with the status code above produced the following log:

2/6/2021, 3:02:50 PM +187ms
+1034ms ā•‘https://script.googleusercontent.com/macros/echo?user_content_key=ICjyfLDZlYD8CPerbXGAlp5q3enZRYp_4z-AcdGdsbBeEnir8OvoLEnLW1ocDUcsBJIraXgO31ltJJMvRxyAK3Yey85Rpr9EOJmA1Yb3SEsKFZqtv3DaNYcMrmhZHmUMWojr9NvTBuBLhyHCd5hHa1g9G7SXu46cSMRTgdoHlZNAhLBJTtGUbp20VC0wTp-o9T4T0bKse72v-_7jaQIGTYFF0yTcMrEs1wqIsZqjPCCad1JgEkknfsmTKExbncrkgOnt3gl3fI9XEVXdxkJq2lWDMtRtYB9retbuCvcbZHA&amp;lib=MrhiMYjacqkkub_4Z1M4FdHTIkpzRVIDu
+1044ms ā•‘Response is &lt;HTML&gt;
+1045ms ā•‘&lt;HEAD&gt;
+1045ms ā•‘&lt;TITLE&gt;Moved Temporarily&lt;/TITLE&gt;
+1045ms ā•‘&lt;/HEAD&gt;
+1045ms ā•‘&lt;BODY BGCOLOR=" #FFFFFF" TEXT=" #000000"&gt;
+1045ms ā•‘&lt;H1&gt;Moved Temporarily&lt;/H1&gt;
+1045ms ā•‘The document has moved &lt;A HREF="https://script.googleusercontent.com/macros/echo?user_content_key=ICjyfLDZlYD8CPerbXGAlp5q3enZRYp_4z-AcdGdsbBeEnir8OvoLEnLW1ocDUcsBJIraXgO31ltJJMvRxyAK3Yey85Rpr9EOJmA1Yb3SEsKFZqtv3DaNYcMrmhZHmUMWojr9NvTBuBLhyHCd5hHa1g9G7SXu46cSMRTgdoHlZNAhLBJTtGUbp20VC0wTp-o9T4T0bKse72v-_7jaQIGTYFF0yTcMrEs1wqIsZqjPCCad1JgEkknfsmTKExbncrkgOnt3gl3fI9XEVXdxkJq2lWDMtRtYB9retbuCvcbZHA&amp;lib=MrhiMYjacqkkub_4Z1M4FdHTIkpzRVIDu"&gt;here&lt;/A&gt;.
+1045ms ā•‘&lt;/BODY&gt;
+1045ms ā•‘&lt;/HTML&gt;
+1066ms ā•‘status is 302
+1066ms ā•‘ content type is
+1066ms ā•‘ status OK is false

I also extracted the URL and tried to run a GET request on that but then for a 402 error so still didnā€™t work.

#13

Iā€™m not comfortable handling things like relative URLs in the Location header, 303 (forces request method to GET), and redirect loops. Combined with the other issues it has caused, if ST does not support following redirection the asynchttp_v1 changes will be rolled back for ST.

#14

your call. Should see what they say. this rollback will cost them a lot of $$ is AWS.

#15

My main concern is that reverting doesnā€™t make future changes unmaintainable for you in the HE fork. I think it is divergent enough that most upstream changes already need at least a bit of massaging for HE, so correct me if Iā€™m wrong but it should be OK to have HE only on async. Choosing between something that has mostly worked in the past and something that has caused numerous conflicts just in the first week, I donā€™t feel particularly charitable toward SmartThingsā€™ bottom line.

#16

correct no issue for them to be different

I only mention ST may appreciate the cost savings, but if they donā€™t, that is fine too.

I know they are dealing with a lot of cloud cost issues (hence they shutdown echo speaks for example)

#17

So is there a fix or workaround for this issue on ST? thanks.

#18

I asked ST and smart app devs yesterday but still waiting on a response. If there is no workaround I plan to revert to the style of web requests used in version 0.3.100.

#19

Anyone who would like to test this out once I have some code ready later this week please drop a :heart: on this reply.

#20

@fieldsjm is there a public URL that you can share that is reliably erroring out with DataCharacterLimitExceededException? I am setting up test cases but having a hard time reproducing this one reliably.