topic Unsual behavior with search events in Splunk Enterprise

Unsual behavior with search events

akarivaratharaj — Thu, 02 Nov 2023 10:18:36 GMT

We have recently upgraded to Splunk Enterprise 9.0. When I try to run a search query without adding the index field into it, the event count are showing wrong. Also if I try to see the respective event logs, from Verbose mode they are weird and this is not usual format of logs.

In other case, if index is mentioned in the query, everything is working fine and asusual.

This issue occurs only when the search query have stats or chart commands to visualise the data. Below is the sample search query which I used

host=abc sourcetype=xyz |stats count

I am not sure whether it is a bug in Splunk 9.0 or any other issue from config side (like limitations in search head). Could anyone please help me on this.

Re: Unsual behavior with search events

isoutamo — Thu, 02 Nov 2023 10:21:53 GMT

Usually you should always add index=xyz on your query to avoid this situation. This is the best practices!

The reason for that behaviour is that every role has attribute srchIndexesDefault which are used if you don't add index=xyz on your query.

srchIndexesDefault = <semicolon-separated list>
* A list of indexes to search when no index is specified.
* These indexes can be wild-carded ("*"), with the exception that "*" does not
  match internal indexes.
* To match internal indexes, start with an underscore ("_"). All internal indexes are
  represented by "_*".
* The wildcard character "*" is limited to match either all the non-internal
  indexes or all the internal indexes, but not both at once.
* No default.

As users usually have different roles they have different combination of srchIndexesDefault and for that reason the real searches gives you to different results.

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Authorizeconf

r. Ismo

Re: Unsual behavior with search events

akarivaratharaj — Thu, 02 Nov 2023 10:41:23 GMT

With the same query, if I try to view the events from verbose mode, I get something like blank events. Please. refer the attached screenshot. But this was not occurring earlier. We used to see the respective log events for the host and sourcetype which are mentioned in the query (though index is not included.)

Re: Unsual behavior with search events

isoutamo — Thu, 02 Nov 2023 10:50:23 GMT

Which kind of environment you have (single node, distributed) and have all nodes updated to the same version Splunk + OS and are all nodes using same OS?

Re: Unsual behavior with search events

akarivaratharaj — Thu, 02 Nov 2023 11:29:01 GMT

We have distributed environment. The Splunk version is same. The OS version of indexer, search heads are same but for deployment server it is different.

Re: Unsual behavior with search events

isoutamo — Thu, 02 Nov 2023 11:32:03 GMT

Have you check that your OS is supported by splunk with your current Splunk version?

Re: Unsual behavior with search events

akarivaratharaj — Thu, 02 Nov 2023 13:17:08 GMT

Yes I have cross verified and all of the OS versions are supported for the Splunk version 9.0, as mentioned - here

Re: Unsual behavior with search events

akarivaratharaj — Tue, 07 Nov 2023 09:46:58 GMT

Could any help or suggest me on this? Why am I getting blank events in the verbose mode when I run the search query without index field?

Re: Unsual behavior with search events

akarivaratharaj — Tue, 07 Nov 2023 10:10:30 GMT

If the search indexes are based on roles, then the search query should behave in same way with or without any commands (like statistical command, chart commands or any other functions).

In my case, I am getting the empty logs whenever I run any of the below queries

host=abc sourcetype=xyz |stats count (or) host=abc sourcetype=xyz |timechart count

whereas, with the below query (without mentioning index) I am able to see the log events successfully.

host=abc sourcetype=xyz