Limit on amount of data sent to a piston from an external request

#1

I’m calling a simple piston from an app running on my raspberry pi. The data is sent as JSON, and is accessed from the $args variable within the piston.

Everything is working fine, until a larger amount of data is sent, I then receive the following error from the app sending the data.
statusCode: 500
{“error”:true,“type”:“org.codehaus.groovy.grails.web.converters.exceptions.ConverterException”,“message”:“An unexpected error occurred.”}

The message is probably about 3k chars, I’m wondering if there’s a limit at 2048 or 1024?

#3

Sorry, I did some testing by extracting data from a google spreadsheet. At first I thought the data got truncated but I had an error in my calculation. I successfully extracted the full spreadsheet into WebCoRE with 4777 characters. Seems like maybe there is an error in your data? Have you tried running the URL in a browser to see what is returned?

#4

Also worth noting that an illegal character can return “An unexpected error occurred”

#5

Thanks for your suggestions, I’m a bit stumped by this now.

I’ve updated the code to delete JSON items I’m not interested in within webcore. This proved rather troublesome as the string was about 4k, and the node.js code kept failing. I did track it down to some illegal characters e.g f:\\tv shows so that could have been causing problems.

However, its still failing, the JSON items with illegal characters is now stripped and the message sent is no longer than others that work. I can’t see anything wrong with the JSON data below. The data is received as a string, converted into JSON, edited, then turned back into a string, so I’m pretty sure the JSON formatting is ok.

{“payload”:{“event”:“library.new”,“user”:true,“owner”:true,“Account”:{“id”:1,“thumb”:“https://plex.tv/users/f2b7e33f0f2066a7/avatar?c=1557876154",“title”:“PAUL”},“Server”:{“title”:“i5-7500-W10-64”,“uuid”:“c6d50ea329a958c7915728fc5e6c99b2d6d405fd”},“Metadata”:{“librarySectionType”:“show”,“ratingKey”:“83454”,“key”:"/library/metadata/83454/children",“guid”:“com.plexapp.agents.thetvdb://277928?lang=en”,“studio”:“Amazon”,“type”:“show”,“title”:“Bosch”,“contentRating”:“TV-MA”,“summary”:"Meet LAPD homicide cop Harry Bosch — hero, maverick, nighthawk. From a dangerous maze of blind alleys to a daring criminal heist beneath the city to the tortuous link that must be uncovered, his survival instincts will be tested to their limit. Joining with an enigmatic and seductive female FBI agent, pitted against enemies inside his own department, Bosch must make the agonizing choice between justice and vengeance, as he tracks down a killer whose true face will shock him.”,“index”:1,“rating”:8.2,“viewCount”:20,“lastViewedAt”:1556537215,“year”:2014,“thumb”:"/library/metadata/83454/thumb/1587139081",“art”:"/library/metadata/83454/art/1587139081",“banner”:"/library/metadata/83454/banner/1587139081",“theme”:"/library/metadata/83454/theme/1587139081",“duration”:2700000,“originallyAvailableAt”:“2014-02-06”,“leafCount”:60,“viewedLeafCount”:0,“childCount”:6,“addedAt”:1478888673,“updatedAt”:1587139081,“Genre”:[{“id”:6545,“tag”:“Crime”},{“id”:288,“tag”:“Drama”},{“id”:450,“tag”:“Suspense”}]}}}

#6

Sometimes, I dump the output on a page like this to better see the structure.
(it also validates the output if you press Process)

Edit:
It currently shows 56 errors

#7

Also note that in your payload above, the double quotes are not uniform… It is opening and closing quotes, instead of standard double quotes. (not sure if this matters, but it has given me issues in the past)

#8

Thanks, thats very useful.

I’ve pasted the code in and it says its valid (VALID RFC8259)

The overall message is shorter than other messages that work, so its not a length issue. My other thoughts are the length of an individual item, the summary field seems long.

{
“payload”:{
“event”:“library.new”,
“user”:true,
“owner”:true,
“Account”:{
“id”:1,
“thumb”:“https://plex.tv/users/f2b7e33f0f2066a7/avatar?c=1557876154”,
“title”:“PAUL_THOMAS1964”
},
“Server”:{
“title”:“i5-7500-W10-64”,
“uuid”:“c6d50ea329a958c7915728fc5e6c99b2d6d405fd”
},
“Metadata”:{
“librarySectionType”:“show”,
“ratingKey”:“83454”,
“key”:"/library/metadata/83454/children",
“guid”:“com.plexapp.agents.thetvdb://277928?lang=en”,
“studio”:“Amazon”,
“type”:“show”,
“title”:“Bosch”,
“contentRating”:“TV-MA”,
“summary”:“Meet LAPD homicide cop Harry Bosch — hero, maverick, nighthawk. From a dangerous maze of blind alleys to a daring criminal heist beneath the city to the tortuous link that must be uncovered, his survival instincts will be tested to their limit. Joining with an enigmatic and seductive female FBI agent, pitted against enemies inside his own department, Bosch must make the agonizing choice between justice and vengeance, as he tracks down a killer whose true face will shock him.”,
“index”:1,
“rating”:8.2,
“viewCount”:20,
“lastViewedAt”:1556537215,
“year”:2014,
“thumb”:"/library/metadata/83454/thumb/1587139081",
“art”:"/library/metadata/83454/art/1587139081",
“banner”:"/library/metadata/83454/banner/1587139081",
“theme”:"/library/metadata/83454/theme/1587139081",
“duration”:2700000,
“originallyAvailableAt”:“2014-02-06”,
“leafCount”:60,
“viewedLeafCount”:0,
“childCount”:6,
“addedAt”:1478888673,
“updatedAt”:1587139081,
“Genre”:[
{
“id”:6545,
“tag”:“Crime”
},
{
“id”:288,
“tag”:“Drama”
},
{
“id”:450,
“tag”:“Suspense”
}
]
}
}
}

#9

Strange. When I checked, your post above returned:

pic

#10

Thats strange, I pasted the code from terminal session on the rasppi, and its ok. I also pasted the same into the post. I’ve tried pasting from the forum and it seems to add quote for some reason.

Which double quotes are you referring to, they all look the same to me!

The only part of the message I’ve added to from the original from plex is to add the “payload{“plex string”]” this keeps it the same as the original message plex would have sent to webcore.

Incidentally, what I’m trying to achieve is to fix a problem when the plex webhook calls webcore. Certain events such as media.play & media.add send a multipart message with a jpeg attached. Webcore can not handle this, so my code just strips this off and relays the remaining data.

Its working fine for most events (media.play, media.pause, media.resume, media.add)
Media.add events for a single show work fine, this one is for the addition of a series, but it doesn’t look much different (it just lackes series & episode details)

I’ll post full details in a new post, including the node.js code once I get it fully working. There are quite a few interesting options you could so with the integration.

#11

Kind of hard to see unless the font is increased a bit:

pic

Notice they are not identical.

#12

Thanks, I can see it now, my eyes were not up to the job!

I think the quotes have been formatted when posted into the forum. In the node.js code they are standard quotes (shift 2).

I am wondering about codesets though, I’ve not specified a codeset and wonder if I should & what it should be?
hostname: ‘graph-eu01-euwest1.api.smartthings.com’,
path: ‘/api/token/*******:’,
method: ‘GET’,
headers: {
‘Content-Type’: ‘application/json’,
‘Content-Length’: data2.length
}

#13

I’ve made some progress.

Without making any other changes, I’ve removed the summary item from the JSON data and its worked. So there must either be something hiding in there causing a problem or its the length of that data item (approx 500 chars). I’m sure its not the total message length as other longer messages work.

#14

Maybe it’s the — ???

#15

Thanks, thats my suspicion too! I’ll try some tests tomorrow.

#16

Just one observation. There is often subtle differences between what you see in terminal, and what webCoRE sees in the response.

I normally recommend programming a piston entirely based on what webCoRE actually sees.
(ignoring the results from elsewhere)

#17

Excellent advice, I’d normally do that. The challenge in this case is that the piston does not even execute, so there’s nothing you can look at. The error code 500 is returned within the node.js code from the request to call the piston. I’m not sure if theres anywhere else to look for any further errors.

A big problem with resolving this is actually analysing exactly what the data is. I’m calling log.console in node.js to examine the data, and its this I’m pasting into the forum and setting to string values to test, however, I don’t think this will correctly show any control chars. e.g at the end it prints “f:\tv shows” If I paste this into a string to test, the string assign fails.

I think I’m going to have to write some code to print each byte in hex. I wish I had an IDE for node.js!

#18

Thanks @WCmore, spot on! It was the underscore. I’m not sure which character it was, but I added a regular expression to remove all unprintables and it was taken out and the piston ran successfully.

#19

Would you be able to share this with the community? I think it would be helpful to many people.

#20

I’m happy to share,however I’m not sure how useful it is. The processing is not done in webcore. With the non printable chars in the http request to the piston, the piston never executes, so you can not correct the data in webcore. The regular expression is in my node.js programme, which acts as a gateway between Plex & webcore, as webcore can not process all the Plex webhook calls.

The node.js code below converts the string containing json data into a JSON object. This allows you to optionally delete items (currently commented out). It then converts it back to a string and the “replace” method removes any non printable chars.

var jsondata2 = JSON.parse(data2);
// remove unwanted JSON items
// delete jsondata2.payload.Metadata[‘Role’];
// delete jsondata2.payload.Metadata[‘Similar’];
// delete jsondata2.payload.Metadata[‘Location’];
// delete jsondata2.payload.Metadata[‘summary’];
data2 = JSON.stringify(jsondata2);
data2 = data2.replace(/[^\x20-\x7E]/g, ‘’); // Remove non printable chars

I’ve applied this to all data, not just the summary field where the issue was originally identified. I still wonder whether there’s a way to get webcore to accept the non standard chars. They don’t cause any issues converting the string to JSON & back to a string, so node.js seems to be able to handle them ok.

I’ll put a new post together with the full code once I’ve tested it further.

#21

Man, I must be a geek… I’ve got a craving to turn this into a T-shirt:

pic