Get requests failing/timing out

#1

I have an intermittent problem with my FTTC internet connection, where it occasionally failed and needs a reboot of the modem. I wrote a piston which performs a get request every minute on a random website from a list of popular sites.
This code has been working perfectly for months, only restarting the modem maybe once every few weeks. However, in the last week or so, it keeps getting failures from the get requests. A later request to the same site will often work. Due to the height number of errors, the reset is being triggered unnecessarily, so I’ve had to pause the piston for now.
Are there any reason why I’m suddenly getting these errors?

The log below shows $response, $httpStatus and the website called
22/05/2024, 13:51:37 +112ms
+75ms â•‘response = 408 and false -www.sky.com
22/05/2024, 13:50:37 +116ms
+83ms â•‘response = 408 and false -www.sky.com
22/05/2024, 13:49:37 +78ms
+39ms â•‘response = 200 and true -www.sky.com
22/05/2024, 13:48:37 +163ms
+77ms â•‘response = 200 and true -www.microsoft.com

#2

I do a GET to 8.8.8.8 (Google public DNS) and it almost never fails.

#3

Thanks, I’ll give that a go and see if it works any better. It’s just strange that it’s been working fine for over 6 months and suddenly started to play up.

#4

Is 1.1.1.1 or 8.8 8.8 faster?

While Google’s 8.8.8.8 DNS server is more popular, according to DNSPerf, Cloudflare’s 1.1.1.1 DNS server is faster . In addition, the 1.1.1.1 DNS server does not store user data, offering more privacy.
Aug 28, 2023

#5

My pistons been running all evening using 8.8.8.8 and not picked up a single issue (timeout).
I hadn’t thought of using a DNS server to check internet connectivity. I’m not sure what a call to 8.8.8.8 returns, it doesn’t work on a browser URL. I thought of using 1.1.1.1 also, but in a browser, it returns a large web page.

I wonder if the problem I was having maybe caused by the volume of data returned, but not sure why its only just started.

Edit… Just checked and this is what’s returned from 8.8.8.8
length=1381 response = <!DOCTYPE html>
+39ms ║<html lang=“en”> <head> <title>Google Public DNS</title> <meta charset=“UTF-8”> <link href="/static/93dd5954/favicon.png" rel=“shortcut icon” type=“image/png”> <link href="/static/e6eca759/matter.min.css" rel=“stylesheet”> <link href="/static/b8536c37/shared.css" rel=“stylesheet”> <meta name=“viewport” content=“width=device-width, initial-scale=1”> <link href="/static/d05cd6ba/root.css" rel=“stylesheet”> </head> <body> <span class=“filler top”></span> <div class=“logo” title=“Google Public DNS”> <div class=“logo-text”><span>Public DNS</span></div> </div> <form action="/query" method=“GET”> <div class=“row”> <label class=“matter-textfield-outlined”> <input type=“text” name=“name” placeholder=" "> <span>DNS Name</span> <p class=“help”> Enter a domain (like example.com) or IP address (like 8.8.8.8 or 2001:4860:4860::8844) here. </p> </label> <button class=“matter-button-contained matter-primary” type=“submit”>Resolve</button> </div> </form> <span class="filler b…[TRUNCATED]

I wanted to use more than one url as wondered if calls to one every minute might be detected and blocked, but guess this is much les likely calling a DNS server.

#6

Theres something strange going on with the get requests. I left the piston running all night, and had no issues doing a get on 8.8.8.8, each time the length of the returned message was the same, this is one of the log messages.
23/05/2024, 13:42:00 +377ms
+129ms â•‘response = 200 and true len=1381 - 8.8.8.8

However, when randomly using other urls, the length of the returned messages are inconsistent. Calls to the same url can be different (which may be understandable with changing content) but the length of the 8.8.8.8 response is now different and exactly the same length as the previous message.
Also there is a failure to 8.8.8.8 within minutes, yet it worked fine for over 12 hours when it was the only url being called.

23/05/2024, 14:57:10 +527ms
+47ms â•‘response = 200 and true len=1381 - 8.8.8.8 (expected len)
23/05/2024, 14:56:10 +477ms
+50ms â•‘response = 200 and true len=21298 - www.microsoft.com
23/05/2024, 14:55:10 +544ms
+41ms â•‘response = 408 and false len=0 - www.microsoft.com
23/05/2024, 14:54:11 +370ms
+43ms â•‘response = 200 and true len=178223 - www.microsoft.com
23/05/2024, 14:53:10 +557ms
+43ms â•‘response = 200 and true len=611172 - 8.8.8.8 (len too long?)
23/05/2024, 14:52:10 +455ms
+45ms â•‘response = 200 and true len=21242 - www.microsoft.com
23/05/2024, 14:51:10 +406ms
+87ms â•‘response = 503 and false len=0 - 8.8.8.8 (failed)
23/05/2024, 14:50:11 +51ms
+40ms â•‘response = 200 and true len=204324 - www.sky.com
23/05/2024, 14:49:10 +628ms
+144ms â•‘response = 200 and true len=611172 - www.microsoft.com
23/05/2024, 14:48:11 +147ms
+39ms â•‘response = 200 and true len=204324 - 8.8.8.8 (len too long = len of previous reply)
23/05/2024, 14:47:11 +34ms
+38ms â•‘response = 200 and true len=204324 - www.google.com
23/05/2024, 14:46:10 +350ms
+123ms â•‘response = 503 and false len=0 - www.sky.com
23/05/2024, 14:45:10 +525ms
+115ms â•‘response = 200 and true len=295677 - www.oracle.com
23/05/2024, 14:44:10 +456ms
+51ms â•‘response = 200 and true len=21289 - www.sky.com
23/05/2024, 14:43:11 +110ms
+37ms â•‘response = 200 and true len=178214 - www.bbc.co.uk
23/05/2024, 14:42:10 +439ms
+40ms â•‘response = 408 and false len=0 - www.google.com

This makes me wonder if WC is correctly handling the reply’s from the get command? I vaguely remember from some node.js code I wrote a while ago, that the reply could be split, and you needed to wait for each segment and join them to the final reply

I wonder if @E_Sch could take a look?

#7

Do you have a small reproducer piston?

#8

This is the core of the piston, which demonstrates the problem

image
image.png1178Ă—510 26.7 KB

and a few log entrys from it
24/05/2024, 09:43:20 +250ms
+70ms â•‘response = 200 and true len=175914 - www.bt.co.uk
24/05/2024, 09:42:19 +992ms
+43ms â•‘response = 408 and false len=0 - www.amazon.co.uk
24/05/2024, 09:41:20 +90ms
+37ms â•‘response = 200 and true len=611008 - www.sky.com
24/05/2024, 09:40:20 +732ms
+34ms â•‘response = 200 and true len=175914 - www.microsoft.com
24/05/2024, 09:39:19 +925ms
+35ms â•‘response = 408 and false len=0 - www.bbc.co.uk
24/05/2024, 09:38:20 +556ms
+41ms â•‘response = 200 and true len=175903 - www.amazon.co.uk
24/05/2024, 09:37:19 +955ms
+33ms â•‘response = 408 and false len=0 - www.google.com
24/05/2024, 09:36:19 +961ms
+138ms â•‘response = 200 and true len=611008 - www.google.com
24/05/2024, 09:35:19 +912ms
+33ms â•‘response = 408 and false len=0 - www.google.com