TWC - Sunrise/Set - Forecast & conditions slightly different

guxdude · 2020-07-29 23:33:59 UTC

Could just be the short period I tested (~2 hrs). I set up a better piston to log for a longer period of time. Will let you know what I find…

WCmore · 2020-07-29 23:36:20 UTC

Slightly off topic, but I just realized that I need to go back in and edit some pistons…
(now that we know $twcweather.forecast.___[0] is only accurate ~20.5 hours a day)

Which unfortunately, likely includes all of these:

$twcweather.forecast.dayOfWeek
$twcweather.forecast.expirationTimeUtc
$twcweather.forecast.moonPhase
$twcweather.forecast.moonPhaseCode
$twcweather.forecast.moonPhaseDay
$twcweather.forecast.moonriseTimeLocal
$twcweather.forecast.moonriseTimeUtc
$twcweather.forecast.moonsetTimeLocal
$twcweather.forecast.moonsetTimeUtc
$twcweather.forecast.narrative
$twcweather.forecast.qpf
$twcweather.forecast.qpfSnow
$twcweather.forecast.sunriseTimeLocal
$twcweather.forecast.sunriseTimeUtc
$twcweather.forecast.sunsetTimeLocal
$twcweather.forecast.sunsetTimeUtc
$twcweather.forecast.temperatureMax
$twcweather.forecast.temperatureMin
$twcweather.forecast.validTimeLocal
$twcweather.forecast.validTimeUtc

guxdude · 2020-07-29 23:49:56 UTC

many of those have ‘conditions’ values which are more reliable. we usually use those.

WCmore · 2020-07-29 23:50:46 UTC

Note to self:

Perhaps an easy way to check to see if the data is current is by this little snippet:

IF $dayOfWeekName is equal to $twcweather.forecast.dayOfWeek[0]
    THEN Grab [0] data & assume it is today
    ELSE Grab [1] data & assume it is today
END IF

Hmmm… Perhaps even storing the 0 or 1 as a global… Which can automatically be referenced by all other weather pistons trying to grab the most current data…

Edit after deep thought:

I cannot give myself a , but this idea is definitely worthy.
It could literally solve TWC’s “outdated data” issue 100% across the board.

IE: This same logic should also be able to be applied to forecast.daypart[0] (which I use a lot).

WCmore · 2020-07-29 23:52:00 UTC

Which is likely why this 3+ hour boo-boo was over looked for so long.

guxdude · 2020-07-30 01:42:05 UTC

OK, my new piston has been running for a couple hours now and over the 2 hours, the values have changed 13 times. Definitely not as benign as this morning.

orangebucket · 2020-07-30 15:38:56 UTC

Interesting discussion going on here. However could you perhaps be overlooking the fact that the actual sunrise time varies depending on weather conditions?

So the figure for ‘conditions’ might perhaps be calculated assuming 34’ of refraction, or whatever the standard number used for predictions is, and the ‘forecast’ could be adjusting for the latest observed or forecast temperature and humidity and anything else that affects refraction.

I’m not necessarily suggesting that is what is happening, only that it is perhaps worth considering.

WCmore · 2020-07-30 17:41:14 UTC

Interesting point of view…

I know, from the point of view of the observer, the sunrise/set may change a few seconds due to weather… but so far, I have never seen any math formula to adjust that.
(and knowing how consistently unreliable TWC data changes are, I suspect they have not implemented that either)

My hypothesis is the following:
conditions.sunX are somewhat “fixed” times. (either based on zip code, city or county)
forecast.sunX seems to be calculated internally by TWC, and then the results are rounded.

When it comes to estimated math based time formulas, it is easy to be a few seconds off every time it is calculated.

guxdude · 2020-07-30 18:42:26 UTC

The confusing thing is why it would change so often. Sometimes, it goes slower but I have seen it change every 5 minutes for half an hour. Do they really update the forecast that often? I wouldn’t think so.

WCmore · 2020-07-30 19:52:08 UTC

From what I have seen, all indications point towards a rounding math formula.
(likely because they are trying to “guesstimate” times for those living between stations)

So my hunch is, the data they are pulling from has not changed, but because each pull uses slightly different math (the current time is always changing), then the output will consistently be a smidgen off.

I see this kind of variance a lot when doing math with time.

Here is a quick visual… Two formulas, based on the same data, plus the ever changing time:

One pair matches, one pair is slightly off, due to rounding 21 decimals.
(the variance would be much greater if I only used 3 decimals)

One other thought just popped into my head…

I wonder if $twc.forecast is based on our GPS location… Meaning, GPS jumps a bit throughout the day, and would account for the slight changes.

Wait a second, this can’t be…
My cell does not report GPS to webCoRE, so from webCoRE’s point of view, my GPS should never change…

orangebucket · 2020-07-30 21:59:38 UTC

They might perhaps base sunrise/sunset predictions on the current conditions. Just as likely to be a different explanation though.

A document I was reading, that was produced by HM Nautical Almanac Office, suggested that the height relative to the horizon and weather conditions could make a difference of a minute or two in the observed sunrise and sunset times compared to the standard predictions.

It is still curious, though.

WCmore · 2020-07-30 22:42:59 UTC

This is 100% true. I have watched the sunset from sealevel, then immediately got into an elevator to the 15th floor… Where I watched the sunset a second time!

Note: This works best if both you and the elevator move swiftly.

I would be sincerely impressed with TWC if they implemented that… but for the benefit of the doubt, I am now considering logging these in my test piston to better understand.

$twcweather.conditions.cloudCeiling » null, 100, 2100, 8500
$twcweather.conditions.cloudCoverPhrase » "Clear", "Partly Cloudy", "Mostly Cloudy", "Cloudy"
$twcweather.conditions.visibility » 16.09

… but in all honesty, there are many other dataPoints that affect our perception as well… This one may also be tricky to ‘nail down’.

orangebucket · 2020-07-30 23:40:15 UTC

I think it is a little unlikely too. However it seems temperature, humidity and pressure make a difference.

guxdude · 2020-07-31 00:44:30 UTC

So I recorded the forecast narrative along with the sunrise/set times. It often changed at the same time the times changed but not always. It does appear there are frequent updates but the frequency is not consistent. Here is a plot of the delta time for condition vs forecast over the last 24+ hours:

Neglecting the midnight to 3 am, you can see a little clearer:

EDIT: Should have mentioned the data is every 5 minutes.

WCmore · 2020-08-01 01:16:42 UTC

Thanks for this. I appreciate the visual.

I think I can better elucidate my thoughts now…

I suspect somewhere in the backend (the weather source? IBM? SmartThings?):

a time is being converted into a decimal
math is being done with that decimal
the results are converted back into time (a few secs off)

To clarify, I have no hard proof of this.
This hunch is based on my past experiences with time and calculations.
(depending on the complexity of math, it is rare those 3 steps return a perfect answer)

What baffles me though is, why would the sunrise math at 2pm be any different than the sunrise math done at 3pm? Mathematically speaking, the current time should not be in the equation when calculating the sunrise or sunset for the day.

Now, if I was defending TWC for a moment… Most of the data in forecast can benefit from math based on the current time. For example, if I was calculating the forecast wind between two points, doing math based on the current time makes sense.
► 2pm forecast of 4mph
► 3pm forecast of 8mph
… then we can use math to “guesstimate” 2:30 to be 6mph

This strategy of estimating the “gradient spectrum” seems to be a common thing with TWC, especially noticeable on fast changing data like wind speed. (IE: Nowhere near the accuracy levels of WUnderground)

If my ramblings above are accurate, then someone from above just needs to stop using a time in the formula when estimating the sunrise/sunset…

(or the lazy man’s way, and just let the forecast reference the condition)

However it is done, logically, the sunrise & sunset times should only change once per day. (per location)

However, I am grateful for this quirk, because it helps to showcase some of the convoluted logic in TWC’s weather data.

It’s not just forecast though. I have actually seen this multiple times using the condition:
$twcweather.conditions.sunriseTimeLocal
Here it switched 3 times in 2m 12s:

guxdude · 2020-08-01 02:06:17 UTC

I believe the math can cause issues but I wonder if it is just a reporting issue. Say there are 10 weather reporting stations within 100 miles of me (maybe more if they include any crowdsourcing data?). If they take into account the position of all the stations reporting at a given update and do some sort of averaging, that could shift the “location” of the forecast around enough to make a difference depending on which stations reported or didn’t. Or not. Probably never know unless someone at TWC chimed in.

WCmore · 2020-08-01 04:43:19 UTC

Good point.

I think the “gradient spectrum” includes both time and distance, as well as averages from multiple sensors. (which would explain why all my numbers look “washed out”, with no real spikes ever occurring)

Edit: Added pic to show typical example ($twcweather.conditions.windSpeed)

In real life, the speed varied much more than this “flat-line” number.
(IE: the more they try to ‘smarten’ up their algorithm, the ‘dumber’ the results)

*sighs*

I really miss the raw data from WUnderground…

Speaking of which… Is anybody willing to share a private API key with me?
(I promise to stay within agreed upon limits)

WCmore · 2020-09-20 00:08:19 UTC

… A few months later…

After analyzing this giant log, I have a few more observations:

$twcweather.conditions.sunriseTimeLocal
$twcweather.conditions.sunsetTimeLocal

These two stays consistent all day long…
Conditions update to the new day sometime between 12 am and 12:35 am.

$twcweather.forecast.sunriseTimeLocal[0]
$twcweather.forecast.sunsetTimeLocal[0]

These two change slightly all day long, but generally stays within a 7 sec window.
Forecasts update to the new day sometime between 3 am and 3:35 am.

Here is an 8 hour snippet showing minutes and seconds…
The consistent “Conditions” are in yellow…
The fickle “Forecasts” are in blue.

Notice how often Forecasts jump around, and how it took over three hours to update to the new day.

TL;DR:

I would not recommend using either of these lines for current queries.

$twcweather.forecast.sunriseTimeLocal
$twcweather.forecast.sunsetTimeLocal

Although in all fairness, if you need to “predict” the sunset two weeks in the future, I guess 7 sec off is no big deal.:
$twcweather.forecast.sunsetTimeLocal[13]

guxdude · 2020-09-20 05:12:37 UTC

Agree. I only started looking at these because I wanted moon rise/set data. While sun rise/set is available in both conditions and forecast, moon data is only available in forecast for some reason. Would love to have more reliable ‘condition’ data for the moon rise/set. For now I am stuck using the forecast and make sure to update after 3:30 am.

WCmore · 2020-09-20 05:21:28 UTC

My apologies… That wasn’t meant to be directed at you specifically…
I was just tidying up, and did one more analysis before pausing that test piston.

I had forgot I had left it running…