topic Re: Why am i unable to search a value in an extracted field? in Splunk Search

Why am i unable to search a value in an extracted field?

gtonti — Tue, 29 Sep 2020 19:30:32 GMT

I have a log file with three lines.

09-05-2018 10:12:15,123 ABC12I_AAA
09-05-2018 10:12:15,123 ABC12I_BBB
09-05-2018 10:12:15,123 ABC12O_CCC

In transforms.conf my REGEX is:
(?\d+-\d+-\d+\s+\d+:\d+:\d+,\d+)\s+(?\w{5})(?\w{1})_

If I make a simple search on the sourcetype I get the three events of the log. If I look to the fields, I see the field "Sp" with the value "ABC12" for each event

If I make a search with:

Sp=ABC12
I get no events

If I make a search with:

Sp=ABC12*
I get 3 events

If I make a search with:

Sp=*ABC12
I get 3 events

I am not able to understand why the search Sp=ABC12 gives no events.
Any suggestions?
Can somebody help me?

Thank you

Re: Why am i unable to search a value in an extracted field?

xpac — Wed, 09 May 2018 21:57:42 GMT

Did some parts of the regex go missing? It looks like it.

Re: Why am i unable to search a value in an extracted field?

maciep — Thu, 10 May 2018 00:46:06 GMT

I think it might be due to how splunk is searching for data. A splunk ninja can correct me if I'm wrong, but I think this is what's happening here....

Splunk needs to find potential events that should be returned in your search. To do that, it will find the field/value pairs in your search, e.g. Sp=ABC12. It will then grab the value and use that as a keyword to find related events. So in this case, it would be doing a search for the keyword ABC12.

However, when Splunk is indexing events, it finds keywords based on major/minor breakers (defined in segmenters.conf I think), e.g. space, dots, dashes, underscores, etc. So in your events, you'll have keywords for ABC12I, AAA, ABC12I, ABC12O, CCC etc. But there is no keyword for ABC12. So your search is not finding any results to return.

And that's why it does work when you specify the wildcard, because then it does match the keywords in your events. Also, if you made it an index-time extraction instead of a search-time extraction, then it would then as well. Because the entire Sp=ABC12 would get indexed with the data, so it would return those events in this case.

If I am correct here and you want more details, check out Martin Muller's session from .conf - both the slides and recording are available

Re: Why am i unable to search a value in an extracted field?

woodcock — Thu, 10 May 2018 04:28:45 GMT

You are probably running in to this well-known problem:

http://blogs.splunk.com/2011/10/07/cannot-search-based-on-an-extracted-field/

The solution is to put this into fields.conf in the same directory that you have your field extractions (where props.conf is):

[Sp]
INDEXED_VALUE = false

See this Q*A for a GREAT explanation from @cpride (you won't find many where the UNaccepted answer has more votes that the accepted one):

https://answers.splunk.com/answers/326291/why-am-i-getting-inconsistent-event-counts-when-us.html

Re: Why am i unable to search a value in an extracted field?

gtonti — Tue, 29 Sep 2020 19:30:45 GMT

Hello,

I try to explain better.
I want an index-time field called "Sp".
My inputs.conf is:

[monitor:///log/logServer/file.log]
disabled = false
index = test01
sourcetype = file-test

My props.conf is:

[file-test]
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE=false
pulldown_type = 1
TRANSFORMS-filter_logs = extract_fields-test

My transforms.conf is:

[extract_fields-test]
REGEX = (?\d+-\d+-\d+\s+\d+:\d+:\d+,\d+)\s+(?\w{5})(?\w{1})_
FORMAT = Ts::"$1" Sp::"$2" Tsp::"$3"
WRITE_META = true

My log File is:

09-05-2018 10:12:15,123 ABC12I_AAA
09-05-2018 10:12:15,123 ABC12I_BBB
09-05-2018 10:12:15,123 ABC12O_CCC

Index time extration creates the Key "Sp" (for the 3 events) but I am not able to search using Sp=ABC12 without wildcards.

I want to create an index-time field called Sp and I want to be able to search the field without wildcards

Thanks

Re: Why am i unable to search a value in an extracted field?

gtonti — Tue, 15 May 2018 06:22:38 GMT

UPDATE: the solution I adopted is the one shown over. To solve I added the fields.conf in the search head in the "local" folder of my app

Re: Why am i unable to search a value in an extracted field?

bontesl — Tue, 21 Apr 2020 12:36:59 GMT

Looks like the devs added a nice workaround that allows you to modify the lispy to search for both the indexed value in an event, as well as a fully indexed field. Can be very useful if you have cases where the same field name is used in different ways.
Add the following to fields.conf on the SH:

[myfield]
INDEXED=false
INDEXED_VALUE=[OR <VALUE> myfield::<VALUE>]

Re: Why am i unable to search a value in an extracted field?

efika — Thu, 07 May 2020 10:56:22 GMT

Thanks @woodcock !
Although they say in the blog post that the issue was resolved post version 4.3 I've just faced it with version 6.6.3.

Adding the fields.conf resolved the problem.