Splunk Search
Highlighted

How to avoid displaying duplicate logs in search results?

Communicator

I am trying to search /var/log/messages log with keywords like shutdown or Error and storing it in message.log

and doing index on message.log

I have a log around at 5 mins ago like 11:30 AM EST says system is shutdown and I displayed this error log in a table format.

Around at 11:40 AM EST, when I ran search command in a table format, I see SYSTEM is Shutdown, but we already knew that system is down at 11:30 AM EST and now I don't want to see this message again.

How can we achieve this?

0 Karma
Highlighted

Re: How to avoid displaying duplicate logs in search results?

Communicator

rajgowd1,

I'd like more information or perhaps a screenshot to help you to remove the duplicate logs. It sounds like you just need to change your search time window.

Adam

0 Karma
Highlighted

Re: How to avoid displaying duplicate logs in search results?

Legend

You can use timemodifiers to limit your logs to a specific timeframe. So for example, if you run you search every 5 mins, you can limit your search to prev 5 mins data using earliest=-5M@m in your base search.

https://docs.splunk.com/Documentation/Splunk/6.5.0/SearchReference/SearchTimeModifiers

0 Karma
Highlighted

Re: How to avoid displaying duplicate logs in search results?

Super Champion

It sounds like you should be extracting these fields at search time and using dedup, but it is difficult to be sure without seeing your search. Can you post your search?

0 Karma
Highlighted

Re: How to avoid displaying duplicate logs in search results?

Legend

I agree with the others that the best way is to limit your timerange.
Every way you could extract a field with error and dedup using this field and host.

your search | dedup host error | table ...

Bye,
Giuseppe

View solution in original post

0 Karma