Splunk Enterprise

How to resolve when the memory spike occurs in splunk indexer?

human96
Communicator

How to resolve memory spike ?

Labels (3)
Tags (2)
0 Karma

venky1544
Builder

Hey @human96 

you probably should throw more content to this context to get better answers.  what is generating the source data seems like its a scripted input you probably need to check the script 

as highlighted by @PickleRick  data never gets modified on the fly its always on the source or the dependencies which create the sources which brings the change in existing data ingestion flow 

Tags (1)
0 Karma

human96
Communicator

i'm getting a strange number of bytes (IF-MIB:: ifHCInOctets, IF-MIB:: ifHCOutOctets) received/sent on the device interface.

a value of nearly 90 times of the normal value is displaying. This is only happening when a memory spike in the  indexer.

 

can you tell me why it's happening ? and what could be the possible solution ?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Unless you hit a very strange bug, events shouldn't suddenly be indexed with "wrong contents".

They queues could get stuck and your data could be delayed or even lost but it shouldn't get modified for no reason.

What do you mean by wrong contents?

0 Karma

human96
Communicator

a value of nearly 90 times of the normal value is displaying. This is only happening when a memory spike in the  indexer.

 

can you tell me why it's happening ? and what could be the possible solution ?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Hard to diagnose anything based on such limited information but I'd hazard a guess that it's the other way around - you're getting sudden spike of either a huge load of data to be processed, huge amount of searches requested or even simply a huge number of general "random" requests from vulnerability manager or something like that. And the requests cause your splunk to build up resource usage. Not the other way around.

But it's up to you to find what this traffic is.

0 Karma

human96
Communicator

i investigated and found out no large amount of traffic inflow in that specific time range.

and one more thing those are Derive  type of data and was imported in HEC.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

OK. If your device counters show high volume of traffic but you don't see much traffic on the machine, there is definitely something wrong in your setup but it's way beyond scope of this forum.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...