Is Splunk extracting unnecessary fields?

[joshua_hart1]

I've noticed that my searches are taking a very long time to complete. For instance, a one-hour search for Bro IDS events with sourcetype of bro_http takes 127 seconds to complete. When looking through the search.log, I see Splunk identifying valid field extraction stanzas for sourcetypes of all kinds--not just sourcetype=bro_http. When I use fast mode, the search takes all of a few seconds. Is there any way I can restrict the field extraction to only those stanzas that match the search terms?

[clanier]

I am experiencing similar issues, was there any answer or solution which was decided? I also made a post here: [Splunk Answers Question][1]
Any Insight would help!

https://answers.splunk.com/answers/586903/searchlog-indicates-search-is-attempting-to-extrac.html

[mikaelbje]

Perhaps you have a props stanza for a wildcard source, sourcetype or host that gets applied to all sourcetypes? I.e.

[source::.*]
TRANSFORM-my_horrible_transform = my_horrible_transform

This stanza will also get executed together with any transforms you have for the bro_http sourcetype.

This may be an explanation if you have problems with slow searches for other sourcetypes too.

[joshua_hart1]

I'll check for that. Not sure I do, but I'll check. To be honest, I'm seeing stanza matches for Windows event logs when I search for Bro events or anything else. It's like Splunk loads every single props/transforms stanza for field extraction on every search regardless of the search criteria.

[lmyrefelt]

Since you say the sourcetype is bro_http i am guessing your data is full of key=value pairs? (or in json)
If so, splunk tries to extract all those per default.

you could in props.conf for sourcetype bro_http turn off automatic field extraction
[bro_http]
KV_MODE = none

However then you need to specify your own fields if you want to have fields searchable.

[lmyrefelt]

docs:

http://docs.splunk.com/Documentation/Splunk/6.2.1/Knowledge/Aboutfields

creating fields,
http://docs.splunk.com/Documentation/Splunk/6.2.1/Knowledge/Createandmaintainsearch-timefieldextract...

[thomrs]

Use fast mode and be explicit about the fields you want.

Mode docs are here, http://docs.splunk.com/Documentation/Splunk/6.2.1/Search/Changethesearchmode.

More about the fields command, http://docs.splunk.com/Documentation/Splunk/6.2.1/SearchReference/Fields.

[joshua_hart1]

I've read the mode documentation and I agree, fast mode is just as it's named: faster. However, when running inline searches for dashboard panels, searches often take a long time. Also, the root question about whether or not Splunk should attempt to extract fields based on field extraction stanzas that do not match the search terms remains to be answered.

For example, if my search is:

sourcetype=foo earliest=-60m@m

shouldn't Splunk only load the field extractions that match that source, sourcetype, or host? Currently, whenever I search for anything, Splunk loads the field extractions for everything under the sun. I believe this is leading to longer search times.

[chanfoli]

I too would like your original question addressed. I have extractions which are specified for a certain sourcetype, but see messages from splunk about them being slow, when my search is specific to a different sourcetype.