Splunk Search

Number of events not stable in time

emallinger
Path Finder

Hello,

 

I wonder if you have any suggestion as to why, over time, results of a stats count may vary for a past time frame.

I have a planned report doint this search each week:

index=x1 OR index=x2 OR index=x3 OR index=x4
| eval tempo = strftime(_time,"%Y-%m")
| stats count by tempo,index
| sort by tempo, index

2 of the 4 indexes are closed (no new event since at least a year) => older events

Recent events are indexed on the other 2 indexes.

Over the past period in the report, I should have each week the same result (ie for 2016-04 index=x1 result_stats_count=125469522).

Except not.

Beginning 3 weeks ago, results changed over some period (2016-04 for example) even if there is no new data for that period of time (I checked : no new event has been indexed for these index this year)

In some cases, the number increases, in other it decreases, or both over 3 weeks. This data has not yet reach the retention limit.

The splunk platform is 2 SHC, 1 indexer cluster multisite and a few forwarders.

Operation  that has been done the last 3 weeks : new cluster bundle conf with rolling restart, some SHC rolling restart.

 

I didn't find anyting helpfull in _internal explain this behaviour.

Do you have any idea ? Pointers ?

 

Thanks a lot,

Ema

Labels (1)
Tags (1)
0 Karma
1 Solution

emallinger
Path Finder

Hi,

Issue solved with Splunk support : DMC was activated on search heads. (not a good idea !)

Deactivation of DMC on SHs and workload management fixed the problem.

View solution in original post

0 Karma

emallinger
Path Finder

Hi,

Issue solved with Splunk support : DMC was activated on search heads. (not a good idea !)

Deactivation of DMC on SHs and workload management fixed the problem.

View solution in original post

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!