How to resolve when the memory spike occurs in splunk indexer?

[human96]

How to resolve memory spike ?

[venky1544]

Hey @human96 

you probably should throw more content to this context to get better answers.  what is generating the source data seems like its a scripted input you probably need to check the script 

as highlighted by @PickleRick  data never gets modified on the fly its always on the source or the dependencies which create the sources which brings the change in existing data ingestion flow 

[human96]

i'm getting a strange number of bytes (IF-MIB:: ifHCInOctets, IF-MIB:: ifHCOutOctets) received/sent on the device interface.

a value of nearly 90 times of the normal value is displaying. This is only happening when a memory spike in the  indexer.

can you tell me why it's happening ? and what could be the possible solution ?

[PickleRick]

Unless you hit a very strange bug, events shouldn't suddenly be indexed with "wrong contents".

They queues could get stuck and your data could be delayed or even lost but it shouldn't get modified for no reason.

What do you mean by wrong contents?

[human96]

a value of nearly 90 times of the normal value is displaying. This is only happening when a memory spike in the  indexer.

can you tell me why it's happening ? and what could be the possible solution ?

[PickleRick]

Hard to diagnose anything based on such limited information but I'd hazard a guess that it's the other way around - you're getting sudden spike of either a huge load of data to be processed, huge amount of searches requested or even simply a huge number of general "random" requests from vulnerability manager or something like that. And the requests cause your splunk to build up resource usage. Not the other way around.

But it's up to you to find what this traffic is.

[human96]

i investigated and found out no large amount of traffic inflow in that specific time range.

and one more thing those are Derive  type of data and was imported in HEC.

[PickleRick]

OK. If your device counters show high volume of traffic but you don't see much traffic on the machine, there is definitely something wrong in your setup but it's way beyond scope of this forum.