topic Re: Interesting behaviour in Splunk Enterprise

Interesting behaviour

eduardo1989 — Tue, 16 Nov 2021 19:17:42 GMT

I have faced a very interesting situation and have no clue what is going wrong.

I have a forwarded info from a particular host and if use a search like this I have all results.

index=win host=MYHOST

If I use this search it gives no results.

index=win host=MYHOST sourcetype=mysourcetype

BUT in realtime search it gives me the results!

The setup on the host looks like this for mysourcetype.

[mylogsource]

disabled = 0

index = win

sourcetype = mysourcetype

Re: Interesting behaviour

somesoni2 — Tue, 16 Nov 2021 20:14:17 GMT

If you run this, do you see your sourcetype "mysourcetype" listed? If yes, try clicking on it for drilldown and see how the query is formatted.

index=win host=MYHOST | stats count by sourcetype

Re: Interesting behaviour

eduardo1989 — Tue, 16 Nov 2021 20:22:20 GMT

It is shown and if I drilldown the query is formatted the same as for the another sourcetype which is shown correctly with the same query.

Re: Interesting behaviour

bowesmana — Tue, 16 Nov 2021 20:40:17 GMT

When you do the basic search and it gives you the list of results, what do you have in the left hand field list

and if you click on sourcetype does that show 'mysourcetype' and if and then click on that do you no longer see any results?

Re: Interesting behaviour

eduardo1989 — Tue, 16 Nov 2021 22:52:17 GMT

Yes exactly,

the field is there and the sourcetype is also but when I query it gives no results

Re: Interesting behaviour

bowesmana — Wed, 17 Nov 2021 02:19:09 GMT

Sounds like there are some strange characters in there...

Can you do each of these searches separately

index=win host=MYHOST sourcetype=mysourcetype* index=win host=MYHOST sourcetype=*mysourcetype index=win host=MYHOST sourcetype=*mysourcetype* index=win host=MYHOST sourcetype=*

and then also do this

index=win host=MYHOST | stats count by sourcetype | eval st=":".sourcetype.":" | eval st_len=len(sourcetype)

and ensure that they all make sense - i.e. no extra spaces. What it might be is a trailing space in your config - something rings a bell...

Re: Interesting behaviour

eduardo1989 — Wed, 17 Nov 2021 07:10:01 GMT

I do not think so unfortunately,

index=win host=MYHOST sourcetype=*

This query gives me the results.

Your second query regarding the characters gives me perfect results, there is no space or anything else.

Moreover I checked with eval st=_sourcetype and it was perfectly fine.

Re: Interesting behaviour

bowesmana — Wed, 17 Nov 2021 22:05:30 GMT

What about the leading and trailing wildcards after your sourcetype - did they all yield results too, or did some of them not show results?

Re: Interesting behaviour

eduardo1989 — Thu, 18 Nov 2021 07:58:42 GMT

Nothing else showed the results. Only the one I mentioned.

Re: Interesting behaviour

scelikok — Thu, 18 Nov 2021 08:20:30 GMT

Hi @eduardo1989,

Did you check Job Inspector? Maybe we can find something from job inspector logs.

Could you please share search.log for the below search?

index=win host=MYHOST sourcetype=mysourcetype

Re: Interesting behaviour

eduardo1989 — Thu, 18 Nov 2021 09:56:01 GMT

Yeah I found out it was a MetaData problem.

I checked the SourceTypes.data file in the actual db folder and my sourcetype was not prefixed with sourcetype:: and after I changed it fixed my problem.

So somehow the data was incorrectly handled.

Thanks for the help!