Cannot save piston editing session, State Character Limit Exceeded Exception

#6

OK, but first an apology: Sorry, but because I have found success in doing so, the code is trimmed of comments and variable names are very short. Here is how it works …

My “Perimeter Lighting” piston is in charge of the state of all my perimeter lights based on the state of all my perimeter sensors. As sensor state changes, related lights are affected. The effect desired is that nearby (with regard to the sensors) lights are brightest, and the next nearest are not so bright. This results in the lights following sensed events around the perimeter and anticipating the next move. Some of the lights are also used as dim “landscape lighting” and, so, are also affected by time of day events and sunrise/sunset. All lights are affected by “darkness” as measured by nearby illumination sensors so that lights are not turned on unless it is dark.

Each light (e.g. DW for driveway) has a set of bit flags (e.g. dw) where each bit is controlled independently by sensor(s). The bits are for darkness (DK), requests for high brightness (H, from motion (HM) or open (HO) events), medium brightness (M, from motion (MM) or open (MO) events) or low brightness (L, from time (LT) and sun-based (LS) events).

The first part of “Perimeter Lighting” handles the sensed events. As they occur, the bits of state variables are flipped for their related lights. The last part checks to see if these state variables have changed and, if they have, they set the brightness of the light based on the latest state.

My “Set Light Level” piston, is given the light device, its state and, optionally, overriding high, medium and low levels for this light. The light will always be turned off unless it is dark (DK). Otherwise, if anyone wants it to be high (H), it will be set on at the high level (h). Otherwise, if anyone wants it to be medium (M), it will be set on at the medium level (m). Otherwise, if everyone wants it to be low (L), it will be set on at the low level (l). Otherwise, it will be turned off.

Perimeter%20Lighting
Perimeter Lighting.png1179×4728 349 KB

Set%20Light%20Level
Set Light Level.png1179×992 50.2 KB

#7

That piston looks like a lot of fun!

As far as running out of space… You have already done the tricks that I use to conserve space as much as possible… (no comments, short variables, etc)

One thing I have noticed with large (complex) pistons is the chunk size…

21 chunks always saves for me
22 chunks sometimes saves…
23 chunks rarely saves for me…
24 chunks have never been succesfully saved for me.

In general, I try to stay at 21 or below, but like you noticed, the more chunks you have, your log will get shorter and shorter. (since they share the same memory allocation)

I am not sure of your future plans, but I would either (A) be happy with the piston as it is (since there’s no room left for any new code)… or what I would do is (B) start brainstorming on how to split it up as @jkp suggests.

#8

The last change made took it from 24 to 26 chunks. I was able to make that change and lose the StateCharacterLimitExceededException after I duplicated the piston. This brought Memory Used down from 82% to 54%. I noticed it starting to creep up again with logging so I turned it off. Now, without any action on my part besides letting it run and respond to events, it is at 76%.

How did that happen? It seems that such growth ultimately results in bad behavior (StateCharacterLimitExceededException) and that I will have to continually babysit and remediate as described. Why? How can this be avoided?

#9

I am unfamiliar with the StateCharacterLimitExceededException… but unless
you want to limit yourself in the future, I would aim for 21 chunks or less.

(visualize 10 big guys playing basketball :basketball: in a closet…
No matter how good the team is, you have given them no elbow room!)

#10

I am not sure if/how StateCharacterLimitExceededException relates to Memory Used relates to Chunks (relates to elbows :wink:). Borrowing your metaphor, it seems my team starts playing in a gym (54% memory used) and some time later find themselves in that closet (76% memory used). I thought I reserved the gym.

#11

I guess I will add to the metaphor then…

Maybe the 54% memory used is like when they first walk into the closet… and 76% is when they actually start playing the game. If they all stood still (doing nothing) there is enough room for them… but when the game starts, they keep tripping over each other.

The way I understand it, the (closet size) limitation is based on the SmartThings HUB / ecosystem. As far as I know, Samsung has no plans to make the ‘closet’ size any bigger. We webCoRE ‘programmers’ are simply using that small amount of space to do things that they never intended.

Personally, I hate limitations. (pistons are meant to grow and evolve) My strategy would be to copy that piston into a new piston, and let each piston control about half of the events. This gives you the best of both worlds. Lots of logging space, lots of room for growth & new devices, and no need to micromanage the available space.

#12

I just don’t see what is growing. If I could get a clue then I might be able to nip it in the bud. Potentially, no matter what the size of the embryo, if it grows into Andre the Giant, I am screwed.

Also, if there is some economy in an alternate way of expressing things that would be great to know. For example, are my constant expressions reduced/evaluated statically by a compiler or would it be thriftier, from this perspective, to evaluate them before writing the code? It’s hard to measure such things myself.

The design of my Perimeter Lighting has its lights and sensors circularly linked. There are some places in the circle that are better to cut than others but that might change. Freedom for such change is good. In any case, stitching together overlapping “arcs” to model a single continuous “circle” is rather crude and I would like to avoid it, if possible.

#13

Someone please correct me if I am wrong, but I am under the assumption that the 100K limitation is based on bytes in the code. If that is true, then less code means more “elbow” room.

Be that as it may, if the piston saved correctly, and logs are unimportant to you, then just roll with it as it is. If logs are important to you and/or you get more devices, you will have to either simplify your code somehow, or split it up into two pistons.

Honestly, I don’t see how you could reduce your code except maybe by a hundred characters. Although, I guess you could group certain lights/sensors into one cluster, and thereby remove the extra block of code… but that method would be dumbing down what looks like a beautiful piston.

I can speak firsthand of issues I have had with any code 23 chunks or larger.

#14

Logs are of less importance than it working. It has been working without logs but it has still grown (in terms of Memory Used). This seems to conflict with your understanding. I know that I don’t understand it.

Why does it grow? Will I really have to duplicate it (to reduce its size back to ~50%) every time I want to make a change? Even without change, is it doomed to grow and eventually start misbehaving (StateCharacterLimitExceededException)?

#15

If you want me to be blunt, then I think you are working outside the safe size. (I think you got extremely lucky even saving it once!) Not saying that you can’t get it working, but I know that I would not want to be working that cramped.

My advice? Get it down to 21 chunks or less… No matter how you achieve this.
(I will be honest, I do have one piston at 22 chunks that has been running smoothly)
Although my advice to myself (and others) is to keep each piston at 21 chunks or less.

#16

One more answer to your question…
Each variable, as it gets written, will take up space.
This would explain why it grows AFTER you create it.

Now that I think about it, my 22 chunk piston has very few variables in the code

#17

54% seems safe. Even 76% seems safe. But I don’t know what this even measures. I don’t know how much headroom is needed or how/why it is consumed. I am sure that is a function of my code but what?

AFAIK (I’ve read), a “chunk” is just a safe unit of transfer to ST for the whole and my assembled whole uses only about half of its memory resources, to begin with. So, in some way, # chunks is a good resource measure but, I would think, a measure of the assembled whole (memory used?) is much better.

I only have 8 integer “state” variables. What is that, 32 bytes/characters? 64? With ST overhead of storing state in a map, 1K, at the most? That should be insignificant. Furthermore, these variables do not grow.

I agree, empirically, that it seems that I am dancing close to the edge. I don’t know enough to question your advice (which, BTW, I very much appreciate). I would like know more. I just doesn’t make sense to me.

You mention your experience/sensitivity with chunks. Do you have any idea how that correlates with memory used? Have you noticed the magnitude of growth in your pistons as I have?

#18

tagging @Robin

#19

The biggest observation I have noticed is the more chunks I have, the less likely it will save properly, and the smaller my log file gets. If I am happy with both, then I don’t worry about the percents you are focused on.

Without having hard data at my disposal, the best analogy would be something like if I made a text file that contains only one word, “hello”… Even though the file would be 5 bytes in size, it would take up a lot more space on my device. (typically between 2,000 and 32,000 bytes)

The real question I have is:
Does the ST hub split up our piston into individual chunks, and if so, where does it decide on the cutoff point between chunks.

  • Can it split in between any two characters? (optimal choice)
  • Must it split between lines of code? (fairly decent choice)
  • Must it split between devices? (crappy choice)
  • Must it split between blocks of code? (worst choice)

This single answer could easily explain hundreds (if not thousands) of bytes of wasted space.

I am going to flat out make an educated guess now:
If 100,000 bytes is our limit, and 25 chunks is the biggest I have ever heard about being saved correctly, then perhaps each ‘chunk’ could be 4KB or 4,000 bytes. Depending on how it is split up into chunks, quite a lot of space might be lost due to the allocation units.

Moral of the story, stay at 21 chunks or less :grinning:

I wonder if, in the future, we’ll be able to compress our code somehow so we can get 1000% more code in each piston. (text is typically compressed at 90% or better) Of course, then we’d likely have delays as it uncompressed at time of action… but a fellow can dream, right?

#20

Yeah, I had read the @Robin post. If that is correct, the sensitivity does not seem to be directly related to the number of chunks. Yes, I can imagine some waste in fragmentation might cause some problems but once the piston is assembled a “chunk” is a lost artifact – their number has no significance other than a loose correlation with the assembled size. It is this size (Memory Used?) that is significant at run time.

Similarly, I would expect chunk compression would not result in any runtime gain as, I would guess, the code needs to be uncompressed to run.

I understand that logging increases Memory Used. That’s why I have turned it off and cleared the logs. I also have turned off Tracing. Unlike my other pistons, however, the space that was used for such in my “Perimeter Lighting” piston is not returned. It seems that there is a garbage collection problem with my “Perimeter Lighting” piston that is only cleaned up when I duplicate it.

#21

I am running out of things to say, other than this simple analogy:

If 10 basketball players are crammed into a closet, even if one left, you would likely STILL feel crowded.

I know you don’t want to hear this, but you are way too close to the maximum piston size. The more complex the piston is, the more ‘elbow room’ I would recommend. Yes, I believe you could get this piston running, but the moment you needed to tweak your code, or add another device, you would be back in the same boat you are in now.

The only two choices I see is to (A) simplify the code, or (B) split it into two pistons.

Personally I like (B) better because that gives you more options and more room for growth… but that is just me. Either way, keep us posted.

#22

I notice that lines 54-93 in the long example above could be moved to a separate piston if you used global variables… That would likely drop a chunk or two… Although, I am not sure how well webCoRE can track changes to global variables. I know they cannot be written to and then have the new data read back during the same execution. but if one piston is writing the globals, and shortly thereafter, another piston is reading the globals, then you should be fine.

#23

WebCoRE autonatically reduces memory usage from logs etc as required so you really don’t need to worry about the value returned by the browser.

Using the rebuild data cache function you can force it to reduce immediately but really not necessary for that purpose.

#24

About chunks:

Because ST does not allow CORS, webCoRE uses jsonp from the browser. The problem with jsonp is it uses the query string, which means you have a size limit of about 3kB. If the piston is larger than that, It needs to be split into chunks. But each chunk counts as a request to ST (there is a limit per minute).

If you are getting errors, try waiting one minute before hitting that Save button, see if it helps.

#25

Thanks for all the replies. It has allowed me to decide that webCoRE is not the best solution for my problem. Instead, this is what I have come up with.

While developing this alternative, I stumbled on a way to investigate state (and other) information of your SmartApps in the ST IDE: My Locations: <location>: List SmartApps: <smartapp>. Not only was this very helpful in developing my own SmartApp, it is enlightening (genearally and with regard to the subject of this thread) when looking at webCoRE written smartapps.