Cannot save piston editing session, State Character Limit Exceeded Exception

#1

I have a “Perimeter Lighting” piston that I have been incrementally growing by adding sensors and lights. Generally, I am very happy with what I have been able to do with WebCoRE. However, …

Several weeks ago, after adding a few more lines, I noticed that I could not successfully Save my editing session. I factored “Set Light Level” logic out into its own piston and this seemed to clear the problem up for a while. Unfortunately, as I continue to grow the “Perimeter Lighting” piston, I have reached the same impasse. I have had to remove the latest offensive addition.

I don’t know if it is related but this problem seems to have occurred when “Perimeter Lighting” has grown over 25 “chunks”. I have found that reducing name sizes and removing comments has helped but there is not much more that I can do on that front.

I have lost all logging capability in the webCoRE dashboard since my first experience with this issue.

“Perimeter Lighting” Quick Facts tells me
Subscriptions: 10 events, 8 controls
Devices used: 15
Memory used: 82% (82104 bytes)

Performing a “Test” of the “Perimeter Lighting” piston results in the following message in the ST IDE live log:
error physicalgraph.exception.StateCharacterLimitExceededException: State cannot be greater than 100000.0 characters @line 1291 (api_intf_dashboard_piston_test)

I have tried a “Clean up and rebuild data cache” operation from the WebCoRE SmartApp in the ST mobile app.

Pausing and resuming does not help.

It does not seem to me that I should be using much “state”. I have only 8 integer variables that, I think, would contribute to “state”. All other variables are initialized on each execution.

#2

Have you considered breaking it down to multiple smaller pistons?

#3

Have you considered breaking it down to multiple smaller pistons

Yes, and I have. It is not obvious how I could do more. It is certainly not optimal from a design standpoint. I was hoping that there was something that I could reset to regain “state” space. Again, I don’t see how I am making much demand on such.

#4

I tried duplicating the piston, removing the original and keeping the copy.

Quick Facts of the copy now says:
Memory used: 54% (53588 bytes)

No more StateCharacterLimitExceededException.
Logging in the webCoRE dashboard is working again.
However, it appears that logging eats up “state” space.
Quick Facts now says:
Memory used: 56% (55724 bytes)

It doesn’t seem right that I need to maintain my code this way (periodically duplicate pistons to remove log state). Do I understand things right? Am I doing something wrong? I think I’ll turn logging off for now.

#5

Post a copy of your piston, would be interesting to see what it looks like :slight_smile:

#6

OK, but first an apology: Sorry, but because I have found success in doing so, the code is trimmed of comments and variable names are very short. Here is how it works …

My “Perimeter Lighting” piston is in charge of the state of all my perimeter lights based on the state of all my perimeter sensors. As sensor state changes, related lights are affected. The effect desired is that nearby (with regard to the sensors) lights are brightest, and the next nearest are not so bright. This results in the lights following sensed events around the perimeter and anticipating the next move. Some of the lights are also used as dim “landscape lighting” and, so, are also affected by time of day events and sunrise/sunset. All lights are affected by “darkness” as measured by nearby illumination sensors so that lights are not turned on unless it is dark.

Each light (e.g. DW for driveway) has a set of bit flags (e.g. dw) where each bit is controlled independently by sensor(s). The bits are for darkness (DK), requests for high brightness (H, from motion (HM) or open (HO) events), medium brightness (M, from motion (MM) or open (MO) events) or low brightness (L, from time (LT) and sun-based (LS) events).

The first part of “Perimeter Lighting” handles the sensed events. As they occur, the bits of state variables are flipped for their related lights. The last part checks to see if these state variables have changed and, if they have, they set the brightness of the light based on the latest state.

My “Set Light Level” piston, is given the light device, its state and, optionally, overriding high, medium and low levels for this light. The light will always be turned off unless it is dark (DK). Otherwise, if anyone wants it to be high (H), it will be set on at the high level (h). Otherwise, if anyone wants it to be medium (M), it will be set on at the medium level (m). Otherwise, if everyone wants it to be low (L), it will be set on at the low level (l). Otherwise, it will be turned off.

Perimeter%20Lighting
Perimeter Lighting.png1179×4728 349 KB

Set%20Light%20Level
Set Light Level.png1179×992 50.2 KB

#7

That piston looks like a lot of fun!

As far as running out of space… You have already done the tricks that I use to conserve space as much as possible… (no comments, short variables, etc)

One thing I have noticed with large (complex) pistons is the chunk size…

21 chunks always saves for me
22 chunks sometimes saves…
23 chunks rarely saves for me…
24 chunks have never been succesfully saved for me.

In general, I try to stay at 21 or below, but like you noticed, the more chunks you have, your log will get shorter and shorter. (since they share the same memory allocation)

I am not sure of your future plans, but I would either (A) be happy with the piston as it is (since there’s no room left for any new code)… or what I would do is (B) start brainstorming on how to split it up as @jkp suggests.

#8

The last change made took it from 24 to 26 chunks. I was able to make that change and lose the StateCharacterLimitExceededException after I duplicated the piston. This brought Memory Used down from 82% to 54%. I noticed it starting to creep up again with logging so I turned it off. Now, without any action on my part besides letting it run and respond to events, it is at 76%.

How did that happen? It seems that such growth ultimately results in bad behavior (StateCharacterLimitExceededException) and that I will have to continually babysit and remediate as described. Why? How can this be avoided?

#9

I am unfamiliar with the StateCharacterLimitExceededException… but unless
you want to limit yourself in the future, I would aim for 21 chunks or less.

(visualize 10 big guys playing basketball :basketball: in a closet…
No matter how good the team is, you have given them no elbow room!)

#10

I am not sure if/how StateCharacterLimitExceededException relates to Memory Used relates to Chunks (relates to elbows :wink:). Borrowing your metaphor, it seems my team starts playing in a gym (54% memory used) and some time later find themselves in that closet (76% memory used). I thought I reserved the gym.

#11

I guess I will add to the metaphor then…

Maybe the 54% memory used is like when they first walk into the closet… and 76% is when they actually start playing the game. If they all stood still (doing nothing) there is enough room for them… but when the game starts, they keep tripping over each other.

The way I understand it, the (closet size) limitation is based on the SmartThings HUB / ecosystem. As far as I know, Samsung has no plans to make the ‘closet’ size any bigger. We webCoRE ‘programmers’ are simply using that small amount of space to do things that they never intended.

Personally, I hate limitations. (pistons are meant to grow and evolve) My strategy would be to copy that piston into a new piston, and let each piston control about half of the events. This gives you the best of both worlds. Lots of logging space, lots of room for growth & new devices, and no need to micromanage the available space.

#12

I just don’t see what is growing. If I could get a clue then I might be able to nip it in the bud. Potentially, no matter what the size of the embryo, if it grows into Andre the Giant, I am screwed.

Also, if there is some economy in an alternate way of expressing things that would be great to know. For example, are my constant expressions reduced/evaluated statically by a compiler or would it be thriftier, from this perspective, to evaluate them before writing the code? It’s hard to measure such things myself.

The design of my Perimeter Lighting has its lights and sensors circularly linked. There are some places in the circle that are better to cut than others but that might change. Freedom for such change is good. In any case, stitching together overlapping “arcs” to model a single continuous “circle” is rather crude and I would like to avoid it, if possible.

#13

Someone please correct me if I am wrong, but I am under the assumption that the 100K limitation is based on bytes in the code. If that is true, then less code means more “elbow” room.

Be that as it may, if the piston saved correctly, and logs are unimportant to you, then just roll with it as it is. If logs are important to you and/or you get more devices, you will have to either simplify your code somehow, or split it up into two pistons.

Honestly, I don’t see how you could reduce your code except maybe by a hundred characters. Although, I guess you could group certain lights/sensors into one cluster, and thereby remove the extra block of code… but that method would be dumbing down what looks like a beautiful piston.

I can speak firsthand of issues I have had with any code 23 chunks or larger.

#14

Logs are of less importance than it working. It has been working without logs but it has still grown (in terms of Memory Used). This seems to conflict with your understanding. I know that I don’t understand it.

Why does it grow? Will I really have to duplicate it (to reduce its size back to ~50%) every time I want to make a change? Even without change, is it doomed to grow and eventually start misbehaving (StateCharacterLimitExceededException)?

#15

If you want me to be blunt, then I think you are working outside the safe size. (I think you got extremely lucky even saving it once!) Not saying that you can’t get it working, but I know that I would not want to be working that cramped.

My advice? Get it down to 21 chunks or less… No matter how you achieve this.
(I will be honest, I do have one piston at 22 chunks that has been running smoothly)
Although my advice to myself (and others) is to keep each piston at 21 chunks or less.

#16

One more answer to your question…
Each variable, as it gets written, will take up space.
This would explain why it grows AFTER you create it.

Now that I think about it, my 22 chunk piston has very few variables in the code

#17

54% seems safe. Even 76% seems safe. But I don’t know what this even measures. I don’t know how much headroom is needed or how/why it is consumed. I am sure that is a function of my code but what?

AFAIK (I’ve read), a “chunk” is just a safe unit of transfer to ST for the whole and my assembled whole uses only about half of its memory resources, to begin with. So, in some way, # chunks is a good resource measure but, I would think, a measure of the assembled whole (memory used?) is much better.

I only have 8 integer “state” variables. What is that, 32 bytes/characters? 64? With ST overhead of storing state in a map, 1K, at the most? That should be insignificant. Furthermore, these variables do not grow.

I agree, empirically, that it seems that I am dancing close to the edge. I don’t know enough to question your advice (which, BTW, I very much appreciate). I would like know more. I just doesn’t make sense to me.

You mention your experience/sensitivity with chunks. Do you have any idea how that correlates with memory used? Have you noticed the magnitude of growth in your pistons as I have?

#18

tagging @Robin

#19

The biggest observation I have noticed is the more chunks I have, the less likely it will save properly, and the smaller my log file gets. If I am happy with both, then I don’t worry about the percents you are focused on.

Without having hard data at my disposal, the best analogy would be something like if I made a text file that contains only one word, “hello”… Even though the file would be 5 bytes in size, it would take up a lot more space on my device. (typically between 2,000 and 32,000 bytes)

The real question I have is:
Does the ST hub split up our piston into individual chunks, and if so, where does it decide on the cutoff point between chunks.

  • Can it split in between any two characters? (optimal choice)
  • Must it split between lines of code? (fairly decent choice)
  • Must it split between devices? (crappy choice)
  • Must it split between blocks of code? (worst choice)

This single answer could easily explain hundreds (if not thousands) of bytes of wasted space.

I am going to flat out make an educated guess now:
If 100,000 bytes is our limit, and 25 chunks is the biggest I have ever heard about being saved correctly, then perhaps each ‘chunk’ could be 4KB or 4,000 bytes. Depending on how it is split up into chunks, quite a lot of space might be lost due to the allocation units.

Moral of the story, stay at 21 chunks or less :grinning:

I wonder if, in the future, we’ll be able to compress our code somehow so we can get 1000% more code in each piston. (text is typically compressed at 90% or better) Of course, then we’d likely have delays as it uncompressed at time of action… but a fellow can dream, right?

#20

Yeah, I had read the @Robin post. If that is correct, the sensitivity does not seem to be directly related to the number of chunks. Yes, I can imagine some waste in fragmentation might cause some problems but once the piston is assembled a “chunk” is a lost artifact – their number has no significance other than a loose correlation with the assembled size. It is this size (Memory Used?) that is significant at run time.

Similarly, I would expect chunk compression would not result in any runtime gain as, I would guess, the code needs to be uncompressed to run.

I understand that logging increases Memory Used. That’s why I have turned it off and cleared the logs. I also have turned off Tracing. Unlike my other pistons, however, the space that was used for such in my “Perimeter Lighting” piston is not returned. It seems that there is a garbage collection problem with my “Perimeter Lighting” piston that is only cleaned up when I duplicate it.