Splunk Search

Why am i unable to search a value in an extracted field?

gtonti
Explorer

I have a log file with three lines.

09-05-2018 10:12:15,123 ABC12I_AAA
09-05-2018 10:12:15,123 ABC12I_BBB
09-05-2018 10:12:15,123 ABC12O_CCC

In transforms.conf my REGEX is:
(?\d+-\d+-\d+\s+\d+:\d+:\d+,\d+)\s+(?\w{5})(?\w{1})_

If I make a simple search on the sourcetype I get the three events of the log. If I look to the fields, I see the field "Sp" with the value "ABC12" for each event

If I make a search with:

Sp=ABC12
I get no events

If I make a search with:

Sp=ABC12*
I get 3 events

If I make a search with:

Sp=*ABC12
I get 3 events

I am not able to understand why the search Sp=ABC12 gives no events.
Any suggestions?
Can somebody help me?

Thank you

1 Solution

woodcock
Esteemed Legend

You are probably running in to this well-known problem:

http://blogs.splunk.com/2011/10/07/cannot-search-based-on-an-extracted-field/

The solution is to put this into fields.conf in the same directory that you have your field extractions (where props.conf is):

[Sp]
INDEXED_VALUE = false

See this Q*A for a GREAT explanation from @cpride (you won't find many where the UNaccepted answer has more votes that the accepted one):

https://answers.splunk.com/answers/326291/why-am-i-getting-inconsistent-event-counts-when-us.html

View solution in original post

woodcock
Esteemed Legend

You are probably running in to this well-known problem:

http://blogs.splunk.com/2011/10/07/cannot-search-based-on-an-extracted-field/

The solution is to put this into fields.conf in the same directory that you have your field extractions (where props.conf is):

[Sp]
INDEXED_VALUE = false

See this Q*A for a GREAT explanation from @cpride (you won't find many where the UNaccepted answer has more votes that the accepted one):

https://answers.splunk.com/answers/326291/why-am-i-getting-inconsistent-event-counts-when-us.html

efika
Communicator

Thanks @woodcock !
Although they say in the blog post that the issue was resolved post version 4.3 I've just faced it with version 6.6.3.

Adding the fields.conf resolved the problem.

0 Karma

bontesl
Explorer

Looks like the devs added a nice workaround that allows you to modify the lispy to search for both the indexed value in an event, as well as a fully indexed field. Can be very useful if you have cases where the same field name is used in different ways.
Add the following to fields.conf on the SH:

[myfield]
INDEXED=false
INDEXED_VALUE=[OR <VALUE> myfield::<VALUE>]
0 Karma

gtonti
Explorer

UPDATE: the solution I adopted is the one shown over. To solve I added the fields.conf in the search head in the "local" folder of my app

0 Karma

maciep
Champion

I think it might be due to how splunk is searching for data. A splunk ninja can correct me if I'm wrong, but I think this is what's happening here....

Splunk needs to find potential events that should be returned in your search. To do that, it will find the field/value pairs in your search, e.g. Sp=ABC12. It will then grab the value and use that as a keyword to find related events. So in this case, it would be doing a search for the keyword ABC12.

However, when Splunk is indexing events, it finds keywords based on major/minor breakers (defined in segmenters.conf I think), e.g. space, dots, dashes, underscores, etc. So in your events, you'll have keywords for ABC12I, AAA, ABC12I, ABC12O, CCC etc. But there is no keyword for ABC12. So your search is not finding any results to return.

And that's why it does work when you specify the wildcard, because then it does match the keywords in your events. Also, if you made it an index-time extraction instead of a search-time extraction, then it would then as well. Because the entire Sp=ABC12 would get indexed with the data, so it would return those events in this case.

If I am correct here and you want more details, check out Martin Muller's session from .conf - both the slides and recording are available

0 Karma

gtonti
Explorer

Hello,

I try to explain better.
I want an index-time field called "Sp".
My inputs.conf is:

  1. [monitor:///log/logServer/file.log]
  2. disabled = false
  3. index = test01
  4. sourcetype = file-test

My props.conf is:

  1. [file-test]
  2. NO_BINARY_CHECK = 1
  3. SHOULD_LINEMERGE=false
  4. pulldown_type = 1
  5. TRANSFORMS-filter_logs = extract_fields-test

My transforms.conf is:

  1. [extract_fields-test]
  2. REGEX = (?\d+-\d+-\d+\s+\d+:\d+:\d+,\d+)\s+(?\w{5})(?\w{1})_
  3. FORMAT = Ts::"$1" Sp::"$2" Tsp::"$3"
  4. WRITE_META = true

My log File is:

  1. 09-05-2018 10:12:15,123 ABC12I_AAA
  2. 09-05-2018 10:12:15,123 ABC12I_BBB
  3. 09-05-2018 10:12:15,123 ABC12O_CCC

Index time extration creates the Key "Sp" (for the 3 events) but I am not able to search using Sp=ABC12 without wildcards.

I want to create an index-time field called Sp and I want to be able to search the field without wildcards

Thanks

0 Karma

xpac
SplunkTrust
SplunkTrust

Did some parts of the regex go missing? It looks like it.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...