I am trying to search /var/log/messages log with keywords like shutdown or Error and storing it in message.log
and doing index on message.log
I have a log around at 5 mins ago like 11:30 AM EST says system is shutdown and I displayed this error log in a table format.
Around at 11:40 AM EST, when I ran search command in a table format, I see SYSTEM is Shutdown, but we already knew that system is down at 11:30 AM EST and now I don't want to see this message again.
How can we achieve this?
I'd like more information or perhaps a screenshot to help you to remove the duplicate logs. It sounds like you just need to change your search time window.
You can use timemodifiers to limit your logs to a specific timeframe. So for example, if you run you search every 5 mins, you can limit your search to prev 5 mins data using
earliest=-5M@m in your base search.
It sounds like you should be extracting these fields at search time and using dedup, but it is difficult to be sure without seeing your search. Can you post your search?
I agree with the others that the best way is to limit your timerange.
Every way you could extract a field with error and dedup using this field and host.
your search | dedup host error | table ...