topic Re: Issues with Rollup Events in Splunk Search

Issues with Rollup Events

SplunkDash — Thu, 11 May 2023 21:48:42 GMT

Hello,

I have a Roll Up events. One file created every month and new events added up every day within that file. How would I avoid duplicate ingestion (or avoid same events to be indexed twice) for the same events as SPLUNK is using the same file to read and ingest? Any help will be highly appreciated. Thank you.

Re: Issues with Rollup Events

yeahnah — Fri, 12 May 2023 05:16:02 GMT

Hi @SplunkDash

I've read your question a couple of times but I still do not really understand what you are asking. Please expand on the situation you describe and provide examples.

If it is just that a file is renamed each month in a monitored folder, then this is OK, as Splunk does not track files by only their filename. It also check sums the first 256 bytes of the head of the file, so if it is renamed then it knows it has already ingested it.

Anyway, best to describe your concern better so the correct answer can be provided.

Re: Issues with Rollup Events

gcusello — Fri, 12 May 2023 06:10:41 GMT

Hi @SplunkDash ,

if you don't use crcSalt = <SOURCE> option, Splunk recognizes already indexed events and it doesn't index them twice. even if they come from files with different filenames.

The only situation where the same logs from different files are indexed is using the above option in inputs.conf.

Ciao.

Giuseppe

Re: Issues with Rollup Events

SplunkDash — Fri, 12 May 2023 06:11:47 GMT

Hello @yeahnah

Thank you so much for your quick response. My question was, new records/events added everyday within the same file (like at the end of the same file)), how SPLUNK will ingest those new events/records?

Re: Issues with Rollup Events

SplunkDash — Fri, 12 May 2023 06:27:57 GMT

Hello @gcusello,

Thank you so much for your respond, it's answered major part of my question. But, other part of my question, new records/events added everyday within the same file (like added at the end of the same file)), how SPLUNK will ingest/treat those new events/records as those new events will be within the same file?

Re: Issues with Rollup Events

SplunkDash — Fri, 26 May 2023 08:50:41 GMT

Hello @gcusello

I posted a new question here (in following link), would it be possible to have your recommendation when you get a chance, thank you so much.

Re: Field Extraction -Key/ Value Pairs with Specia... - Splunk Community