I've found that my calculated fields are not behaving as expected.
I have a search that uses a combination of fields to create its value from a subset of another field.
The conventional way of doing this inline might be :
index=blah sourcetype=blah mydevice|eval EQUIP=case((TitleCode=="AA" AND TitleIndex=="0120" OR ( TitleCode=="GS" AND TitleIndex=="0116") OR (TitleCode=="GS" AND TitleIndex=="0118")), substr(PointDescription, 8, 8))
For a specific time frame, this results in a new field with 3 matching results.
This eval can be stored in a calculated field thusly
props.conf [blah] EVAL-EQUIP = case((TitleCode=="AA" AND TitleIndex=="0120" OR ( TitleCode=="GS" AND TitleIndex=="0116") OR (TitleCode=="GS" AND TitleIndex=="0118")), substr(PointDescription, 8, 8))
The problem is that this calculated field results in a new field with only 1 matching result. I'm doing this in verbose mode also.
Re-running the original search with a renamed inline eval will show the inline eval with 3 and the calculated with 1.
I don't understand what is going on.
I thought it might be an issue with a large number of field extractions being stopped by limits.conf so I tried.
limits.conf [kv] maxcols = 2000 limit = 100
But its not this and I can't find any other limits that refer to calculating fields.
The problem may be related to configuration file precedence:
When you run it on the search bar, it is the last thing being run, but when you run it automatically in a configuration file, the order may change such that something that this depends on to (fully) work may not exist (yet) when it is slotted to execute in the chain.
I figured it out.
I checked out all of our search heads in the cluster and none reported any problems.
I then looked inside the bundles that were transferred to the indexers and found that the calculated fields were missing.
Looked like the indexers weren't getting the proper extractions as such they failed. The only thing it was extracting were older fields.
Looking deeper into it I found that a savedsearch was creating a self updating lookup file that was slowly growing. This was resulting in a ridiculously large csv file that was causing captain to index replication time to go from seconds to minutes (multi GB csv file). As it was eventually successful and not failing there were no other errors.