How to avoid displaying duplicate logs in search results?

[rajgowd1]

I am trying to search /var/log/messages log with keywords like shutdown or Error and storing it in message.log

and doing index on message.log

I have a log around at 5 mins ago like 11:30 AM EST says system is shutdown and I displayed this error log in a table format.

Around at 11:40 AM EST, when I ran search command in a table format, I see SYSTEM is Shutdown, but we already knew that system is down at 11:30 AM EST and now I don't want to see this message again.

How can we achieve this?

[gcusello · Accepted solution]

I agree with the others that the best way is to limit your timerange.
Every way you could extract a field with error and dedup using this field and host.

your search | dedup host error | table ...

Bye,
Giuseppe

[lukejadamec]

It sounds like you should be extracting these fields at search time and using dedup, but it is difficult to be sure without seeing your search. Can you post your search?

[adamsaul]

rajgowd1,

I'd like more information or perhaps a screenshot to help you to remove the duplicate logs. It sounds like you just need to change your search time window.

Adam

[sundareshr]

You can use timemodifiers to limit your logs to a specific timeframe. So for example, if you run you search every 5 mins, you can limit your search to prev 5 mins data using earliest=-5M@m in your base search.

https://docs.splunk.com/Documentation/Splunk/6.5.0/SearchReference/SearchTimeModifiers