Splunk Search

How to avoid displaying duplicate logs in search results?

rajgowd1
Communicator

I am trying to search /var/log/messages log with keywords like shutdown or Error and storing it in message.log

and doing index on message.log

I have a log around at 5 mins ago like 11:30 AM EST says system is shutdown and I displayed this error log in a table format.

Around at 11:40 AM EST, when I ran search command in a table format, I see SYSTEM is Shutdown, but we already knew that system is down at 11:30 AM EST and now I don't want to see this message again.

How can we achieve this?

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

I agree with the others that the best way is to limit your timerange.
Every way you could extract a field with error and dedup using this field and host.

your search | dedup host error | table ...

Bye,
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

I agree with the others that the best way is to limit your timerange.
Every way you could extract a field with error and dedup using this field and host.

your search | dedup host error | table ...

Bye,
Giuseppe

0 Karma

lukejadamec
Super Champion

It sounds like you should be extracting these fields at search time and using dedup, but it is difficult to be sure without seeing your search. Can you post your search?

0 Karma

adamsaul
Communicator

rajgowd1,

I'd like more information or perhaps a screenshot to help you to remove the duplicate logs. It sounds like you just need to change your search time window.

Adam

0 Karma

sundareshr
Legend

You can use timemodifiers to limit your logs to a specific timeframe. So for example, if you run you search every 5 mins, you can limit your search to prev 5 mins data using earliest=-5M@m in your base search.

https://docs.splunk.com/Documentation/Splunk/6.5.0/SearchReference/SearchTimeModifiers

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...