Splunk Enterprise

Interesting behaviour

eduardo1989
Path Finder

I have faced a very interesting situation and have no clue what is going wrong.

I have a forwarded info from a particular host and if use a search like this I have all results.

index=win host=MYHOST

If I use this search it gives no results.

index=win host=MYHOST sourcetype=mysourcetype

BUT in realtime search it gives me the results!

The setup on the host looks like this for mysourcetype.

[mylogsource]
disabled = 0
index = win
sourcetype = mysourcetype
Labels (2)
0 Karma
1 Solution

eduardo1989
Path Finder

Yeah I found out it was a MetaData problem.

I checked the SourceTypes.data file  in the actual db folder and my sourcetype was not prefixed with sourcetype:: and after I changed it fixed my problem.

So somehow the data was incorrectly handled.

Thanks for the help!

 

View solution in original post

0 Karma

scelikok
SplunkTrust
SplunkTrust

Hi @eduardo1989,

Did you check Job Inspector? Maybe we can find something from job inspector logs.

Could you please share search.log for the below search?

index=win host=MYHOST sourcetype=mysourcetype
If this reply helps you an upvote and "Accept as Solution" is appreciated.
0 Karma

eduardo1989
Path Finder

Yeah I found out it was a MetaData problem.

I checked the SourceTypes.data file  in the actual db folder and my sourcetype was not prefixed with sourcetype:: and after I changed it fixed my problem.

So somehow the data was incorrectly handled.

Thanks for the help!

 

0 Karma

bowesmana
SplunkTrust
SplunkTrust

When you do the basic search and it gives you the list of results, what do you have in the left hand field list

bowesmana_0-1637095139973.png

and if you click on sourcetype does that show 'mysourcetype' and if  and then click on that do you no longer see any results?

0 Karma

eduardo1989
Path Finder

Yes exactly,

the field is there and the sourcetype is also but when I query it gives no results

 

0 Karma

bowesmana
SplunkTrust
SplunkTrust

Sounds like there are some strange characters in there...

Can you do each of these searches separately

index=win host=MYHOST sourcetype=mysourcetype*
index=win host=MYHOST sourcetype=*mysourcetype
index=win host=MYHOST sourcetype=*mysourcetype*
index=win host=MYHOST sourcetype=*

and then also do this

index=win host=MYHOST
| stats count by sourcetype
| eval st=":".sourcetype.":"
| eval st_len=len(sourcetype)

and ensure that they all make sense - i.e. no extra spaces. What it might be is a trailing space in your config - something rings a bell...

 

0 Karma

eduardo1989
Path Finder

I do not think so unfortunately,

index=win host=MYHOST sourcetype=*

This query gives me the results.

Your second query regarding the characters gives me perfect results, there is no space or anything else.

Moreover I checked with eval st=_sourcetype and it was perfectly fine.

0 Karma

bowesmana
SplunkTrust
SplunkTrust

What about the leading and trailing wildcards after your sourcetype - did they all yield results too, or did some of them not show results?

 

0 Karma

eduardo1989
Path Finder

Nothing else showed the results. Only the one I mentioned.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

If you run this, do you see your sourcetype "mysourcetype" listed? If yes, try clicking on it for drilldown and see how the query is formatted.

index=win host=MYHOST | stats count by sourcetype
0 Karma

eduardo1989
Path Finder

It is shown and if I drilldown the query is formatted the same as for the another sourcetype which is shown correctly with the same query.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...