How do I code an HTTPS request with header data

#1

1) Give a description of the problem
I use https://api.weather.gov/gridpoints/AFG/382,360/forecast to get forecast data. I am part of an ongoing discussion about data errors provided by the National Weather Service. The data is often several days off. Someone suggested using a random number in the header, as that seemed to work for them. He suggested the following:

The Feature_Flags workaround seems to work … the first time. Then that gets cached and you have the same problem. So then if I change the Feature-Flag value each time, would that work? Hmmm … seems to. I simply input a random number for the Feature-Flag:

pp = requests.get(url,headers={‘Feature-Flags’: str(random.randint(1,1000))})

2) What is the expected behaviour?
How do I incorporate the “Feature-Flag” into a webcore GET request?
This works in the piston below https://api.weather.gov/gridpoints/AFG/382,360/forecast
I tried something like https://api.weather.gov/gridpoints/AFG/382,360/forecast,headers={‘Feature-Flag’: Random_Integer_String} but that produced an error.

**4) Post a Green Snapshot of the piston![image|45x37]

image
image.png1178×518 15.1 KB

#2

From here webCoRE Update v0.3.10f.20190822 - Custom headers on web requests by @Bloodtick_Jones, capability selection fix I was able to glean this piston:

image
image.png1178×556 16.4 KB

It SEEMS to work. Do I have the syntax correct?

#3

It is always nice to see a free weather API… but I am curious…

When I test it, I see much less data that what’s available to us using TWC.

pic

(Although I do like the icons)

sct . few

Is there value in this API that I am missing?

With regards to your question, I found this snippet on their webpage:

Known Issues

Before contacting us, please review the following list of issues that have been identified for a future update.

  • Radar station information (/radar endpoints) performance is unreliable.
  • The gridpoints endpoint serves stale data or returns a 500 error. Retrying the request generally returns valid data.
  • Observations are not being sorted in chronological order.

Updated 10/21/2021

#4

I submitted a ticket about that time so that may be because of my observation.

#5

I stumbled upon this and I have not really compared the two. I have just been playing around with it. But yes, TWC does offer more information.

#6

Do you have a link to available API queries? I am especially curious what we can use at the end of the URL, besides: /forecast

I saw vague references to /alerts (untested), but I’d love to see what dataPoints they list for current conditions… There are quite a few missing (or inaccurate) dataPoints in TWC I wouldn’t mind replacing…

#7

Unfortunately, I have not been able to confirm much.
https://weather-gov.github.io/api/gridpoints
https://weather-gov.github.io/api/general-faqs
https://www.weather.gov/documentation/services-web-api

#8
#9

I’ve been playing around a bit…

You’ve probably noticed that adding: “/forecast/hourly” to the end of the URL returns nearly 7 days into the future… (much more time slots than TWC is offering)

I found a slew of dataPoints by removing: “/forecast” from the link. (under “properties”)

pic
pic.png502×698 12.4 KB

This also looks like approx 7 days into the future, although not every hour is stored.
(IE: heatIndex only at certain hours)

It’s also worth noting the (lame) shortcuts they took with this API:
(IE:maxTemperature is taken from 2pm, and minTemperature from 2am)

This method practically guarantees false data…
(High / low temperatures do not occur on a precise 24 hour cycle)

I am going to guess that this API is fairly new, and unpolished…

Note to self:

I wonder if we can pull Imperial data instead of Metric from that last link…

#10

It appears that you are correct. This API is nearly useless for me. The data is often incorrect and usually appears to be off by several days. And that seems to be completely random. That is why I thought that webcore might be caching the old data. That is the reason for this post…to attempt header manipulation to see if I can force up-to-date information (as suggested by another non-webcore API user).

12-07-21%20Data%20Errors
12-07-21 Data Errors.png585×923 39.6 KB

I have been trying to use this API to push up-to-date forecasts to all of the dashboards in my house. Err…WAF is poor.
Clearly this API needs some work.

#11

Yes, and I did try to use that for a while. But again, the data appears to be inconsistent. What is really strange is that I can poll the data from my browser and at the same time poll the same data with webcore, and they are completely different. The browser poll SEEMS to be more consistently accurate.

#12

I wonder if it needs some nocache type header on the request?

#13

I hate to say… but I suspect that even if we can force “fresh” data, the current API layout might frustrate more than illuminate.

For example, the numbers for each dataPoint do NOT line up. Currently, here is what I see:

112 slots in temperature (hourly for approx 4.6 days)
118 slots in dewpoint (4.9 days)
..8 slots in maxTemperature (approx a week)
..9 slots in minTemperature (approx a week)
.81 slots in relativeHumidity
..1 slots in heatIndex
133 slots in windChill
.34 slots in skyCover (every 3+ hours?)
.24 slots in windDirection (every 3 hours?)

Wow. This is a coding nightmare. You would have to pull (and convert) the validTime for every single data point needed…

IE: No more simple coding like:
IF temp[1] is greater than temp[2]
(because 1 or 2 could mean anything)

#14

Clearly you have been digging into this more than I have. You are shedding some light on some things that I suspect very few people realize. It is clear I’m not going to be able to use this for my purposes around home. Thanks for taking the time to look into it. It certainly appears that this API needs lots of work.

#15

A bit. I got excited seeing somewhat familiar dataPoints that I hadn’t seen since IBM pulled the plug on WUnderground…

I really wanted to like this free API, but at it’s current state, the most I could see is plugging an empty hole in TWC’s data. (with a butt-load of coding to get a single dataPoint)

On the other hand…

It is possible that a different link would return more consistent data, but I still suspect it would pull from the same data pool.

#16

Were you looking for something like this : https://api.weather.gov/openapi.json
Specifically, Paths?

#17

I was, in fact… I also found more detailed info here…

Previously, I found a page to determine my local
{wfo}/{x},{y}
(which worked for the forecast mentioned in the beginning of this thread)

… but to really test these new paths, we’d also have to find out our local:

  • {area}
  • {id}
  • {locationId}
  • {officeId}
  • {point}
  • {productId}
  • {region}
  • {stationId}
  • {typeId}
  • {zoneId}

Unfortunately, I have not found a list or command to determine these yet…
(and my free time is a bit scarce at the moment)