Splunk support asked that we remove phased_execution_mode=singlethreaded from the ES cluster, which fixed the timechart issue. I believe this is specified in the known issues under SPL-164718 and SPL-165363. Unfortunately without this setting our CPU spikes constantly, so although we can't fix it we do know the cause. ... View more

Update: I found the right way after realizing the error in my testing. So here it goes: The transforms.conf stanza for creating that subKey now looks like this: [subKey] REGEX = ^[a-zA-Z]*(?P<subKey>\d+)$ #FORMAT = subKey::$1 SOURCE_KEY = mainKey MV_ADD = true FORMAT is commented out because I have a named field extraction in the REGEX itself, but it's there as a reminder. Moreover, I found that if I need more than one subKey extraction (for example, if there are different formats of mainKey requiring different results), I can add another stanza and reference it from props.conf - even in the same REPORT property. For example, I also added the following extraction, which, when found a mainKey value starting with "!" (exclamation sign), strips it and saves the rest as a subKey (just to serve as an example): [subKey1] REGEX = (?m-s)^!(?P<subKey>.*)$ SOURCE_KEY = mainKey MV_ADD = true and referenced both from this props.conf property: REPORT-subKey = subKey, subKey1 . I needed that (?m-s) at the beginning of the second REGEX because otherwise the .* would consume all the subsequent mainKey values. $ matches both the end of the whole set of multiple values (which seems to be treated internally as multi-line) and the ends of each individual value with or without that flag, as evidenced by my first stanza. When there are multiple extracting stanzas matching one or more values in mainKey, the first one extracts its (potentially multitude of) values, then all those extracted by the second one are added, and so on. This might mess up the order of the mainKey and the corresponding extracted subKey values, but it's ok in my case. ... View more

After a lot of experiments, I found that we'll have to: a) tokenize the conditions field with TOKENIZER = ([^,]+) in fields.conf b) resort to have all known conditions converted to eventtypes individually like this: [WeightOK] search = conditions=WeightOK etc. We are still checking the performance, even though it must already be better than our former search = conditions=*WeightOK* ... View more

An update for the curious: I had a fields.conf file which had two stanzas: [barcodeKey] INDEXED = true [trackId] INDEXED = true Once I removed those INDEXED properties, a search (not exactly the same but extremely similar and suffering from exactly the same symptoms) started working! I then ran those exact searches and they started working as well. My problem with all this is that my fix runs contrary to the description of the INDEXED property in fields.conf . However, INDEXED_VALUE , if set to true , would have produced this exact effect in trackId searches, so I wonder if there is a bug with the processing of those two properties flipped. ... View more