JSON parsing Astronomical data from Naval Observatory

Gbwilson1 · 2018-09-19 15:38:19 UTC

1) Give a description of the problem
I am using the API from the Naval Observatory to retrieve astronomical data about a specific location.

http://aa.usno.navy.mil/data/docs/api.php

When I get the data using a web request, the $response will not parse and returns an error.

2) What is the expected behavior?
When I submit the following query, I expect that I would retrieve astronomical data related to the location as JSON that I can parse to retrieve specific data about the location.

http://api.usno.navy.mil/rstt/oneday?date=9/18/2018&loc=Louisville,%20KY

{
“error”:false,
“apiversion”:“2.1.0”,
“year”:2018,
“month”:9,
“day”:18,
“dayofweek”:“Tuesday”,
“datechanged”:false,
“state”:“KY”,
“city”:“Louisville”,
“lon”:-85.75,
“lat”:38.22,
“county”:“Jefferson”,
“tz”:-5,
“isdst”:“yes”,

“sundata”:[
{“phen”:“BC”, “time”:“7:01 a.m. DT”},
{“phen”:“R”, “time”:“7:27 a.m. DT”},
{“phen”:“U”, “time”:“1:37 p.m. DT”},
{“phen”:“S”, “time”:“7:46 p.m. DT”},
{“phen”:“EC”, “time”:“8:13 p.m. DT”}],

“moondata”:[
{“phen”:“S”, “time”:“1:23 a.m. DT”},
{“phen”:“R”, “time”:“4:13 p.m. DT”},
{“phen”:“U”, “time”:“9:12 p.m. DT”}],

“nextmoondata”:[
{“phen”:“S”,“time”:“2:13 a.m. DT”}],

“closestphase”:{“phase”:“First Quarter”,“date”:“September 16, 2018”,“time”:“7:15 p.m. DT”},
“fracillum”:“66%”,
“curphase”:“Waxing Gibbous”
}

3) What is happening/not happening?
When I submit the query, I retrieve the response but the data appears to be altered. Although the return does appear to be a valid JSON when I look at it outside of WebCore. When I attempt to parse the data, an error is returned.

This is the value of $response stored in a variable and the elements appear to be out of order from the JSON data retrieved outside of WebCore. Because of the comma’s inside of data elements, parsing using ArrayItem is troublesome.

[apiversion:2.1.0,
city:Louisville,
closestphase:,
county:Jefferson,
curphase:Waxing Gibbous,
datechanged:false,
day:18,
dayofweek:Tuesday,
error:false,
fracillum:66%,
isdst:yes,
lat:38.22,
lon:-85.75,
month:9,
moondata:[[phen:S, time:1:23 a.m. DT],
[phen:R, time:4:13 p.m. DT],
[phen:U, time:9:12 p.m. DT]],
nextmoondata:[[phen:S, time:2:13 a.m. DT]],
state:KY,
sundata:[[phen:BC, time:7:01 a.m. DT],
[phen:R, time:7:27 a.m. DT],
[phen:U, time:1:37 p.m. DT],
[phen:S, time:7:46 p.m. DT],
[phen:EC, time:8:13 p.m. DT]],
tz:-5, year:2018]

**4) Post a Green Snapshot of the piston

5) Attach any logs (From ST IDE and by turning logging level to Full)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: info ╔ Event processed successfully (2954ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: trace ║║ Executed virtual command log (2ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: trace ║╔ Execution stage complete. (2829ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: trace ║║ Executed virtual command parseJson (3ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: trace ║║ Executed virtual command log (3ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: info ║║ null
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: error ║║ Error parsing JSON data [apiversion:2.1.0, city:Louisville, closestphase:, county:Jefferson, curphase:Waxing Gibbous, datechanged:false, day:19, dayofweek:Wednesday, error:false, fracillum:75%, isdst:yes, lat:38.22, lon:-85.75, month:9, moondata:[[phen:S, time:2:13 a.m. DT], [phen:R, time:4:57 p.m. DT], [phen:U, time:10:00 p.m. DT]], nextmoondata:[[phen:S, time:3:05 a.m. DT]], state:KY, sundata:[[phen:BC, time:7:02 a.m. DT], [phen:R, time:7:28 a.m. DT], [phen:U, time:1:37 p.m. DT], [phen:S, time:7:45 p.m. DT], [phen:EC, time:8:11 p.m. DT]], tz:-5, year:2018]
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: info ║║ [apiversion:2.1.0, city:Louisville, closestphase:, county:Jefferson, curphase:Waxing Gibbous, datechanged:false, day:19, dayofweek:Wednesday, error:false, fracillum:75%, isdst:yes, lat:38.22, lon:-85.75, month:9, moondata:[[phen:S, time:2:13 a.m. DT], [phen:R, time:4:57 p.m. DT], [phen:U, time:10:00 p.m. DT]], nextmoondata:[[phen:S, time:3:05 a.m. DT]], state:KY, sundata:[[phen:BC, time:7:02 a.m. DT], [phen:R, time:7:28 a.m. DT], [phen:U, time:1:37 p.m. DT], [phen:S, time:7:45 p.m. DT], [phen:EC, time:8:11 p.m. DT]], tz:-5, year:2018]
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: trace ║║ Executed virtual command setVariable (3ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: info ║║ Retrieved data from api.usno.navy.mil
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: trace ║║ Executed virtual command log (4ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: trace ║║ Executed virtual command log (2ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: debug ║║ Calculating (string) Http content type: application/json + (string) >> (string) Http content type: application/json
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: debug ║║ Calculating (string) Retrieved data from + (string) api.usno.navy.mil >> (string) Retrieved data from api.usno.navy.mil
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: info ║║ Http content type: application/json
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: debug ║║ Cancelling statement #7’s schedules…
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: debug ║║ Calculating (string) Http content type: + (string) application/json >> (string) Http content type: application/json
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: debug ║║ Condition #10 evaluated true (7ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: debug ║║ Condition group #6 evaluated true (state did not change) (8ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: trace ║║ Executed virtual command log (2ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: debug ║║ Comparison (boolean) true is (boolean) true = true (2ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: trace ║║ Executed virtual command log (2ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: info ║║ true
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: trace ║║ Executed virtual command httpRequest (651ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:48 AM: info ║║ 200
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:47 AM: debug ║║ Calculating (string) http://api.usno.navy.mil/rstt/oneday?date=09/19/2018&loc= + (string) Louisville,%20KY >> (string) http://api.usno.navy.mil/rstt/oneday?date=09/19/2018&loc=Louisville,%20KY
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:47 AM: debug ║║ Sending external web request to: api.usno.navy.mil/rstt/oneday?date=09/19/2018&loc=Louisville,%20KY
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:47 AM: debug ║║ Calculating (string) http://api.usno.navy.mil/rstt/oneday?date= + (string) 09/19/2018 >> (string) http://api.usno.navy.mil/rstt/oneday?date=09/19/2018
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:47 AM: debug ║║ Calculating (string) http://api.usno.navy.mil/rstt/oneday?date=09/19/2018 + (string) &loc= >> (string) http://api.usno.navy.mil/rstt/oneday?date=09/19/2018&loc=
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:47 AM: debug ║║ Calculating (string) http:// + (string) api.usno.navy.mil >> (string) http://api.usno.navy.mil
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:47 AM: debug ║║ Calculating (string) http://api.usno.navy.mil + (string) /rstt/oneday?date= >> (string) http://api.usno.navy.mil/rstt/oneday?date=
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:45 AM: info smartapp_pause called by smartAppName=webCoRE Piston, value=2000, smartAppVersionId=67c98787-3bac-483d-baf4-88a6c324e0d9
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:45 AM: trace ║║ Waiting for 2000ms
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:45 AM: trace ║║ Executed virtual command wait (1ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:45 AM: debug ║║ Cancelling statement #1’s schedules…
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:45 AM: trace ║ Runtime (44139 bytes) successfully initialized in 70ms (v0.3.108.20180906) (120ms)
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:45 AM: trace ║╚ Execution stage started
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:45 AM: debug ║ RunTime Analysis CS > 23ms > PS > 70ms > PE > 26ms > CE
6bfed639-327d-4b9f-8bed-0f872117f12c 11:21:45 AM: info ╚ Received event [Home].test = 1537370505232 with a delay of 0ms

WCmore · 2018-09-19 15:59:33 UTC

Your green snapshot didn’t go thru…
Can you post it via this method please?
temp

ipaterson · 2018-09-19 17:17:42 UTC

The data is already parsed, you can use expressions like $response.closestphase.phase and $response.sundata[0].time to access it.

WCmore · 2018-09-19 19:55:56 UTC

I always put this command directly after the GET request:
temp2

Like this:
temp

it helps me to see the data the way that webCoRE see it.
I then call the specific data the way @ipaterson mentions above.

Gbwilson1 · 2018-09-20 15:31:02 UTC

Thanks. I was making it harder than it was.
Is there a way to address individual elements of the multi-valued elements such as SunData.

Sundata contains
BC (Begin Civil Twilight)
R (Sunrise)
U (Upper Transit - apogee)
S (Sunset)
EC (End Civil Twilight)

I know I can step through the array using Arrayitem($x,{$response.sundata.phen}) and Arrayitem($x,{$response.sundata.time}) to get the solar phases, but is there a way to address the specific phen, i.e. “BC” to get the time of Begin Civil Twilight?

Thanks

WCmore · 2018-09-20 16:05:17 UTC

For the SunData BC time, you can use this:

which returns:

You can apply that to scrape any of the data points

For complex JSON, sometimes I use a site like https://jsoneditoronline.org/ to see the array numbers a bit easier. Just paste a sample output on the left, and then use the display on the right.

You can also quickly test data points from that page by clicking on the filter icon

temp

(just remember to add the $response. to the beginning when back in webCoRE)

NOTE
All these tests were done using today’s date

Gbwilson1 · 2018-09-21 02:13:54 UTC

Thanks all for the assist.

Here is a snip on how I obtained the Begin Civil Twilight and End Civil Twilight from the response.

I run this every morning around 0400 to get the Twilight times from the observatory. I use these instead of Sunrise and Sunset.