topic Re: Wait time before indexing data in Getting Data In

Wait time before indexing data

shayhibah — Tue, 03 Jul 2018 06:08:08 GMT

Hi,

I would like to know if there is an option to wait for/ minute X seconds before indexing the data.
The goal is to index only the last log in the time range (there is no way to recognize the last log).

Is it possible?

Thanks

Re: Wait time before indexing data

richgalloway — Tue, 03 Jul 2018 13:06:25 GMT

You didn't give us much to work with, but perhaps the inputs.conf settings time_before_close or multiline_event_extra_waittime will help.

Re: Wait time before indexing data

shayhibah — Tue, 03 Jul 2018 14:13:10 GMT

Hi richgalloway,

In my system, I export logs to splunk server.
each log may be updated by another log - it means that in case I export log with some data and now I have an update on this data, a log with the new information is sent.

I would like to ignore all logs with the same uid but only keep the last one.
Is it possible to do it?

Thank you!

Re: Wait time before indexing data

richgalloway — Tue, 03 Jul 2018 18:52:15 GMT

Are you replacing the entire log when an update is made or adding to the end of a log?
The normal process for Splunk is to read log files as they are written. That makes data available to searches about a minute after it is written to the log.

Re: Wait time before indexing data

shayhibah — Tue, 03 Jul 2018 20:03:09 GMT

I would like to replace the entire log.

Is it possible?

Re: Wait time before indexing data

richgalloway — Fri, 06 Jul 2018 12:35:48 GMT

I find Splunk works best when new events are added to the end of a log file. One can replace the entire file and Splunk will usually (but not always) recognize the change and re-index the entire file. However, that results in duplicate events which must be accommodated in your searches.