Splunk Search

Why is the following search that contains "field=" not retuning results unless I use a wildcard?

cjmckenna
New Member

Running into a strange issue that I, nor my Splunk admins, can figure out. We have a filed extraction called "Service" that holds the name of our SOA services. When I do a search using:
index=blah Service="examplename" , i get no results.

If I do this, I get results:

index=blah Service="*examplename"

We can not figure out why we have to add the asterisk for it to work. Even if I do index=blah and select a value for Service from the interesting fields, and let Splunk pop that in the search, I get no results. As soon as I add the *, BAM there are the results.

I hope someone can help as we are stumped. This happens on several fields while other extracted fields work fine.

0 Karma

payl_chdhry
Explorer

We too are facing this issue and the only solution we have been able to come up change delimiter before upload.
We are using ~ as delimiter in csv file. Using FIELD_NAMES parameter to provide header field names. It is extracting fine but we are not able to search data using extracted field unless we use "*" in search.
Were you able to resolve it.

Adding INDEXED_VALUE = false did not work.

0 Karma

brannonrad
Explorer

I know what the issue is. It is explained here:

https://www.splunk.com/en_us/blog/tips-and-tricks/cannot-search-based-on-an-extracted-field.html

Here is the way I understand it. Splunk automatically indexes word tokens, which are detected using "standard" delimiters like spaces, tabs, commas, etc.

Here is what most people don't know or understand: even though you specify a search like so (in your example):

index=blah Service="examplename"

What Splunk actually does initially is a search like this:

index=blah "examplename"

Under normal circumstances, this works fine - and it normally returns a "super set" of the results you are looking for. Then, Splunk refines the results further with a search like this:

| search Service="examplename"

Which should reduce the data set down to the specific results you are looking for.

What went wrong here? Well, your data doesn't have the standard word boundary delimiters Splunk expects. In your case, there are tilde characters around the word you want to search on, so the "examplename" string doesn't match, but the "*examplename" does.

How can you fix this? I think you have two choices:

1) Don't use tildes as delimiters! Actually, don't use any non-standard delimiters. But we don't always have control over that, so...
2) If you know you have this issue with your data, create a fields.conf file in the same app that contains your props.conf, and set it like this (NOTE: replace "fieldname" below with the actual fieldname!):

[fieldname]
INDEXED_VALUE = false

But don't arbitrarily do this for all of your fields - as it can make your searches less efficient.

skoelpin
SplunkTrust
SplunkTrust

Post some sample data too. This will make it easy to recreate

0 Karma

cjmckenna
New Member

Oct 17 09:40:48 info <14>Oct 17 09:40:48 hostname123 SSG[2,651]: [L7Metrics-PRD] INFO com.l7tech.log.custom.si.metrics : -4: ~obfuscateddomainname~obfuscatedservicename~obfuscatedoperationname~http://obfuscatedWSDLURL/1/~obfuscatedWSSid~63~73~

obfuscatedservicename is where the service name would be and what the extraction is pulling out.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

That regex has a lot of steps and could be refactored..

Try this in your search. Field name will be called servicename. If this works then you could save this as a search time extraction

| rex ^(?:[^~\n]*~){2}(?<servicename>\w+)

0 Karma

cjmckenna
New Member

The extraction is working fine. Please see my original post. In "Interesting Fields" Service is there and has a list of values. The issue is whan we do Service="something" we get no results. Service="*something" does.

We will look at cleaning up the regex but for now its actually extracting data

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Yes, I'm thinking it has something to do with your extraction. Keep it simple

Use the extraction I provided you and save it as a search time extraction so it appears in interesting fields without having to explicitly call the rex command. Also, please accept/upvote the answer since it helped solved your problem

0 Karma

cjmckenna
New Member

it has not helped my issue. Cant upvote something that is not a solution

0 Karma

skoelpin
SplunkTrust
SplunkTrust

You just said this worked as expected... Why waste time fixing a poorly written regular expression when I just provided you with a working solution? Your logic doesn't make sense..

0 Karma

cjmckenna
New Member

No... the EXISTING extraction is working fine. We see the service names listed in "Service" the "Interesting Fields" with the existing extraction. The problem is when that field is used in a search we always have to add an *

0 Karma

skoelpin
SplunkTrust
SplunkTrust

If your existing solution is working fine, then why ask for help on Answers? Your regex is bad and you can prove it by using an inline regex like I provided above to test against..

0 Karma

cjmckenna
New Member

So help me understand exactly what is wrong in our existing regex when it finds the data and extracts it and it shows up in interesting fields. Please explain to me why using that data from the existing extraction will not allow us to use index=blah Service="servicename" in a search. You are saying do this, do that but not providing any insight as to why.

Maybe its me and I am not explaining that actual issue clear enough

0 Karma

skoelpin
SplunkTrust
SplunkTrust

I gave you a working solution to test against and told you your regex is bad which is causing the issue. I'm not going to troubleshoot your buggy regex and give you a play by play of why its bad. I gave you the solution and an easy way to test against it. You easily have enough to troubleshoot the problem on your own

0 Karma

DalJeanis
Legend

Please post the extraction code.

0 Karma

cjmckenna
New Member

[layer7:sisyslog]
EXTRACT-Service = ^(?:[^~\n]*~){2}(?P\w+)

0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...