topic Re: splunk enterprise in Splunk Enterprise

How to resolve when the memory spike occurs in splunk indexer?

human96 — Tue, 29 Mar 2022 11:39:00 GMT

How to resolve memory spike ?

Re: splunk enterprise

PickleRick — Mon, 28 Mar 2022 12:03:52 GMT

Unless you hit a very strange bug, events shouldn't suddenly be indexed with "wrong contents".

They queues could get stuck and your data could be delayed or even lost but it shouldn't get modified for no reason.

What do you mean by wrong contents?

Re: How to resolve when the memory spike occurs in splunk indexer, the data is registered with the wrong contents

venky1544 — Mon, 28 Mar 2022 15:42:56 GMT

Hey @human96

you probably should throw more content to this context to get better answers. what is generating the source data seems like its a scripted input you probably need to check the script

as highlighted by @PickleRick data never gets modified on the fly its always on the source or the dependencies which create the sources which brings the change in existing data ingestion flow

Re: splunk enterprise

human96 — Thu, 31 Mar 2022 05:28:01 GMT

a value of nearly 90 times of the normal value is displaying. This is only happening when a memory spike in the indexer.

can you tell me why it's happening ? and what could be the possible solution ?

Re: How to resolve when the memory spike occurs in splunk indexer, the data is registered with the wrong contents

human96 — Tue, 29 Mar 2022 04:56:34 GMT

i'm getting a strange number of bytes (IF-MIB:: ifHCInOctets, IF-MIB:: ifHCOutOctets) received/sent on the device interface.

a value of nearly 90 times of the normal value is displaying. This is only happening when a memory spike in the indexer.

can you tell me why it's happening ? and what could be the possible solution ?

Re: splunk enterprise

PickleRick — Tue, 29 Mar 2022 06:23:36 GMT

Hard to diagnose anything based on such limited information but I'd hazard a guess that it's the other way around - you're getting sudden spike of either a huge load of data to be processed, huge amount of searches requested or even simply a huge number of general "random" requests from vulnerability manager or something like that. And the requests cause your splunk to build up resource usage. Not the other way around.

But it's up to you to find what this traffic is.

Re: splunk enterprise

human96 — Tue, 29 Mar 2022 06:28:46 GMT

i investigated and found out no large amount of traffic inflow in that specific time range.

and one more thing those are Derive type of data and was imported in HEC.

Re: splunk enterprise

PickleRick — Tue, 29 Mar 2022 06:44:13 GMT

OK. If your device counters show high volume of traffic but you don't see much traffic on the machine, there is definitely something wrong in your setup but it's way beyond scope of this forum.