Large web request responses error out if response is slow

guxdude · 2021-02-06 18:42:26 UTC

No, I don’t know a lot about web requests. Just have tried to stick to whatever works in a browser window. Usually once I can get it working there, it works in webcore but now they are not acting the same.

E_Sch · 2021-02-06 19:23:00 UTC

Hmmm

I would expect we have all the headers with getHeaders - is that not true on ST?

Assuming it is and if they don’t follow redirects for async (but do for sync), then the code could try to re-issue to the redirect location.

May have to ask Adrian if he can get an answer inside ST. I wonder if this is a recent change on the platform?

guxdude · 2021-02-06 19:53:40 UTC

I assume so. My piston has worked fine for months and now after the recent update it no longer works. Been looking for a work around but don’t see how to force webcore to follow the redirects.

E_Sch · 2021-02-06 21:49:44 UTC

I tested on HE, and all I get back is 200, so I assume the redirect was followed using the URL @ipaterson suggested.

I did not see any 301 response that made it to webcore.

guxdude · 2021-02-06 23:09:00 UTC

HE might be working. Not sure of the differences. I didn’t see an actual URL suggested by @ipaterson, just some checking he did for various status codes.

If you look at $httpStatusCode, it returns 301. I made a slight modification to the webcore piston app to get the error data out. I inserted line 3373:

This along with the status code above produced the following log:

2/6/2021, 3:02:50 PM +187ms
+1034ms ║https://script.googleusercontent.com/macros/echo?user_content_key=ICjyfLDZlYD8CPerbXGAlp5q3enZRYp_4z-AcdGdsbBeEnir8OvoLEnLW1ocDUcsBJIraXgO31ltJJMvRxyAK3Yey85Rpr9EOJmA1Yb3SEsKFZqtv3DaNYcMrmhZHmUMWojr9NvTBuBLhyHCd5hHa1g9G7SXu46cSMRTgdoHlZNAhLBJTtGUbp20VC0wTp-o9T4T0bKse72v-_7jaQIGTYFF0yTcMrEs1wqIsZqjPCCad1JgEkknfsmTKExbncrkgOnt3gl3fI9XEVXdxkJq2lWDMtRtYB9retbuCvcbZHA&amp;lib=MrhiMYjacqkkub_4Z1M4FdHTIkpzRVIDu
+1044ms ║Response is &lt;HTML&gt;
+1045ms ║&lt;HEAD&gt;
+1045ms ║&lt;TITLE&gt;Moved Temporarily&lt;/TITLE&gt;
+1045ms ║&lt;/HEAD&gt;
+1045ms ║&lt;BODY BGCOLOR=" #FFFFFF" TEXT=" #000000"&gt;
+1045ms ║&lt;H1&gt;Moved Temporarily&lt;/H1&gt;
+1045ms ║The document has moved &lt;A HREF="https://script.googleusercontent.com/macros/echo?user_content_key=ICjyfLDZlYD8CPerbXGAlp5q3enZRYp_4z-AcdGdsbBeEnir8OvoLEnLW1ocDUcsBJIraXgO31ltJJMvRxyAK3Yey85Rpr9EOJmA1Yb3SEsKFZqtv3DaNYcMrmhZHmUMWojr9NvTBuBLhyHCd5hHa1g9G7SXu46cSMRTgdoHlZNAhLBJTtGUbp20VC0wTp-o9T4T0bKse72v-_7jaQIGTYFF0yTcMrEs1wqIsZqjPCCad1JgEkknfsmTKExbncrkgOnt3gl3fI9XEVXdxkJq2lWDMtRtYB9retbuCvcbZHA&amp;lib=MrhiMYjacqkkub_4Z1M4FdHTIkpzRVIDu"&gt;here&lt;/A&gt;.
+1045ms ║&lt;/BODY&gt;
+1045ms ║&lt;/HTML&gt;
+1066ms ║status is 302
+1066ms ║ content type is
+1066ms ║ status OK is false

I also extracted the URL and tried to run a GET request on that but then for a 402 error so still didn’t work.

ipaterson · 2021-02-07 14:06:57 UTC

I’m not comfortable handling things like relative URLs in the Location header, 303 (forces request method to GET), and redirect loops. Combined with the other issues it has caused, if ST does not support following redirection the asynchttp_v1 changes will be rolled back for ST.

E_Sch · 2021-02-07 14:17:46 UTC

your call. Should see what they say. this rollback will cost them a lot of $$ is AWS.

ipaterson · 2021-02-07 14:57:39 UTC

My main concern is that reverting doesn’t make future changes unmaintainable for you in the HE fork. I think it is divergent enough that most upstream changes already need at least a bit of massaging for HE, so correct me if I’m wrong but it should be OK to have HE only on async. Choosing between something that has mostly worked in the past and something that has caused numerous conflicts just in the first week, I don’t feel particularly charitable toward SmartThings’ bottom line.

E_Sch · 2021-02-07 15:17:00 UTC

correct no issue for them to be different

I only mention ST may appreciate the cost savings, but if they don’t, that is fine too.

I know they are dealing with a lot of cloud cost issues (hence they shutdown echo speaks for example)

guxdude · 2021-02-09 07:20:32 UTC

So is there a fix or workaround for this issue on ST? thanks.

ipaterson · 2021-02-09 14:44:19 UTC

I asked ST and smart app devs yesterday but still waiting on a response. If there is no workaround I plan to revert to the style of web requests used in version 0.3.100.

ipaterson · 2021-02-10 00:27:31 UTC

Anyone who would like to test this out once I have some code ready later this week please drop a on this reply.

ipaterson · 2021-02-13 14:45:13 UTC

@fieldsjm is there a public URL that you can share that is reliably erroring out with DataCharacterLimitExceededException? I am setting up test cases but having a hard time reproducing this one reliably.

fieldsjm · 2021-02-13 17:53:55 UTC

Thus one is from my score alert piston, just tested the piston and got the error

http://site.api.espn.com/apis/site/v2/sports/baseball/mlb/summary?event=401288363

Update: just tested in a standalone piston, able to make the request and set a variable to the response without errors, however not parsing of the data

fieldsjm · 2021-02-15 16:15:22 UTC

@ipaterson - I’ve decide to see what would happen if I limited the processing in my piston. I started disabling a block here and there and turns out that I was just a little over zealous. I split out parts of the piston, specifically a section that schedules times of future runs, and have not received the error since.

ipaterson · 2021-02-16 01:48:33 UTC

Thanks for following up. From what I’ve seen it can depend on how long the piston has been running when the request starts and how long it takes to respond.

Next step for me is to finish test cases for local network web requests. If this error is not affecting anyone else I may stick with the new asynchttp_v1 and implement redirect following rather than reverting the networking changes.

guxdude · 2021-02-20 08:43:16 UTC

I’m still getting the status 302 error with my google sheet web request. Another person is using my same script, however, and it is working fine for him. For some reason, my webcore is not following the redirects. Any ideas? thanks,

ipaterson · 2021-02-22 14:27:06 UTC

Versions 0.3.111 through 0.3.113 will not follow redirects. Still working through a lot of surprises like that for the next release but for now you would need to use v0.3.110.20191009. You can grab the code for each of the four smart apps here just click “Raw” to get an easy-to-copy version of each file.

guxdude · 2021-02-25 23:52:57 UTC

Successfully went back to 110 and working good again. Will keep my eye out for 114. Thanks!

guxdude · 2022-03-29 17:40:29 UTC

I just tried up-dating to 114 and it still has the same problem with web requests. Will go back to 110 again