KeyError: 'label'

[proctorgeorge]

Hello All,

Today when I logged in to my Splunk Search app I was greeted by a not-so-friendly page stating:

500 Internal Server Error

KeyError: 'label'

Now from what I can tell, everything was working fine yesterday when I left at 5, and it does not seem like anything was changed since then. I am running a Indexer/Deployment Server that is indexing data from about 100 Light Weight Forwarders. Everything is running Splunk 4.1.5. I am not using a Search Head.

Searching through Splunk Answers and the forums has yielded few results that all seem to be unrelated to anything I have been doing.

This problem is only effecting my Search App and one other App I have been using to test new dashboards, but other apps are unaffected.

Can anyone give me any clues as to what exactly a KeyError is?

EDIT x1: Also I have tried copying "virgin" untouched versions of the Search app into the app folders and those do not work either, this implies that thew problem is larger then the specific apps but is only targeting those 2 Apps and not the other ones.

EDIT x2: There is an error that shows up in the web_service log which looks like this:

2011-02-22 14:39:59,299 WARNING [4d63cabf145378438] viewstate:133 - Found invalid keyname in viewstate stanza: disabled

Though this error seems to have been showing up regularly since the Log first started recording, would this have anything to do with the problem? I am unsure how to error check viewstates, anyone want to chime in on how viewstates.conf creates its name/key pairs or why it would be off?

EDITx3: The root cause of the problem seems to be the flashtimeline view, all my apps are not working when they use the flastimeline view I just did not notice this common problem because only the two Apps were using the flashtimeline as their default view. Would viewstates be causing the flastimeline to have problems using label?

My flashtmeline is still stock, no changes were made to it.

[carasso]

I had a similar problem and found the problem, after a couple of hours wasted. The docs said to "Create a block in setup.xml". It failed to mention that the "block" must be wrapped in " ... "

http://docs.splunk.com/Documentation/Splunk/5.0.4/AdvancedDev/SetupExampleCredentials

[proctorgeorge · Accepted solution]

You could use a backup you handily made (4 months ago) to revert to an old version that was working and backup the current (broken) splunk install so that you can slowly add in the broken versions files until you isolate the the exactly file that was making the problem.

You may find that it was "times.conf" which was somehow setup wrong and making the flashtimeline break even though there was nothing label related in it and no one had changed it in weeks (but maybe was referencing a index that got depricated? Who cares it's 5 o'clock!).

Why was the Error message cryptic and totally useless? No Idea! Why was it breaking the flashtimeline? No Idea! Why were none of the log files showing anything useful? No Idea!

Lessons Learned: BACKUP EVERYTHING, EVERY DAY.

[rroberts]

I also get this error if I dont restart Splunk after creating an App.

[proctorgeorge]

Any fix? I have tried restarting multiple times and none of the apps that are effected were newly created ones.

[sideview]

It sounds like a missing 'label' param in a module.

If so then I think someone edited that view in that app, and put in some module that has a required <param> called 'label', but neglected to fill in the label, and they never tested the change.

There should be a little 'debugging' link right next to that stack trace. If you paste in some details from that we should be able to narrow down what module is throwing the error. Lots of modules have 'label' as a required param.

[proctorgeorge]

Unfortunately there is also no Stack trace or debugging link.
I did not edit the two apps or add any new modules, though I WAS editing the apps that are unaffected by the problem. I will check for missing params but as I added in an edit above, even a totally clean Search app is not working which leads me to believe it is a more global problem.