- Splunk Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- How to remove the duplicate logs or events?
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How to remove the duplicate logs or events?
I have this serch string
source=/xxxx/log/xxxx/server.log ERROR
and i got this:
2014-01-06 13:28:33,828 ERROR xxx.xxx.xxxUsersRolesLoginModule Failed to load users/passwords/role files
2014-01-06 13:28:32,878 ERROR xxx.xxx.xxxUsersRolesLoginModule Failed to load users/passwords/role files
2014-01-06 13:28:32,814 ERROR xxx.xxx.xxxUsersRolesLoginModule Failed to load users/passwords/role files
and so on..
Question: How to remove the duplicate?I tried some of the suggestions I found on this site but to no avail. I used dedup and uniq but it returned the same number of duplicate logs. Please advise what is the best way to remove duplicates and retain just one instance of logs.Thanks in advance
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As suggested by ShaneNewman/Iguinn, all these entries are unique as they have different timestamp/workerthread/IP. The only thing duplicate thing is the error message among these events. To filter the duplicate events based on the error message only, you can do field extraction or use rex command, something like below: (rex example)
source=/xxxx/log/xxxx/server.log ERROR | rex "\]\)\s(?<ErrorMessage>.*)" | dedup ErrorMessage
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello- Thanks for your suggestion however it did not seem to work... so for the time being i just used NOT command to atleast remove all the duplicate logs. Thank you.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
These are not duplicate logs. If you notice, the IP and port are different for each entry above. If you want to remove these, you will have to do field extraction and leave out the WorkerThread field from the dedup list.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks Iguinn.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The timestamps and worker thread numbers are different as well.
It may be that all these entries represent the same problem or incident - but these are definitely unique events.
As Shane points out, you will need to create the fields - and decide how you define "duplicate" in this context. I'd be interested in seeing your definition of "duplicate".
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Not in the eyes of Splunk. Because the port numbers are different, they are unique. That is why you will have to do field extraction to get them to be seen as duplicates.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Shane, thanks for the prompts response. These are actually a duplicate logs.
Introducing the 2024 Splunk MVPs!
Splunk Custom Visualizations App End of Life
Introducing Splunk Enterprise 9.2
