As I am indexing the data, I notice that apart from the 'sources' that are appearing correctly (/var/log/filename.gz | 38,219 ), there is one source coming up as 'unknown' (unknown | 109,368,099). There is not any file or directory called unknown. So we are surprised by this. This is important as sometimes we search by 'source'.
On further investigation of the source=unknown, I noticed that the timestamps are actually not picked correctly from the events for all the events that are showing source=unknown. And at this particular time (wronlgy picked time) there are lots of events (as the wrongly picked events are timestamped at one particular time).
Would you have an idea of why this is happening. And a procedure to resolve this.
Thanks,
HB.
----------------------------------*-------------------------------------
Thanks Guys for the answer. I could give hardcoding the timestamp a try, although its surprising that the sourcetype as standard as cisco syslog with clear timestamps is not getting timestamped properly (the prblem is evident only with events that is displaying source=unknown). As Lowell indicated, i am a bit doubtful this would solve the problem.
I tried the search that you indicated : source=unknown | stats count by host, sourcetype, index | sort -count
however this search doesn't run because of the following error: Error in 'IndexScopedSearch': The search failed. More than 500000 events found at time 1281228609.
As i said, the wrongly timestampled events are appearing at a single time.
In terms of sourcetype, the unknown source has various sourcetypes, for instance syslog, cisco_firewall. In order to actually see the events that were coming up as source=unknown, i had to be selective and thus ran the following search:
sourcetype="syslog" IOS_Messages="%CDP-4-DUPLEX_MISMATCH"
A snippent of the results of this search which resulted in 822,882 matching events is pasted below (the first 2 are correctly matched source, and the rest are source=unkown):
#
7 24/07/2010 00:50:05.000
Jul 24 00:50:05 host1.com.au 44796: Jul 24 00:50:04.172 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0 (not half duplex), with Router Ethernet0 (half duplex).
* sourcetype=syslog Options|
* source=/var/log/splunk/logsw1/syslog/syslog.log.20100725-0005.gz Options
#
8 24/07/2010 00:50:05.000
Jul 24 00:50:05 host2.com.au 363791: *Jul 24 00:47:56.237 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0/1 (not half duplex), with eduPaSS_wap.eduPaSS_wap FastEthernet0 (half duplex).
* sourcetype=syslog Options|
* source=/var/log/splunk/logsw1/syslog/syslog.log.20100725-0005.gz Options
#
9 24/07/2010 00:50:05.000
Jul 24 00:15:32 host3.com.au 4484: Jul 24 00:15:30.952 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet1 (not full duplex), with BSLMELSWFO01.cscnms.bsl.net GigabitEthernet0/1 (full duplex).
* sourcetype=syslog Options|
* source=unknown Options
#
10 24/07/2010 00:50:05.000
Jul 24 00:15:31 host4.com.au 504361: Jul 24 00:15:30.878 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet1 (not half duplex), with AHM-NSWHUR-R1 Ethernet0/0 (half duplex).
* sourcetype=syslog Options|
* source=unknown Options
#
11 24/07/2010 00:50:05.000
Jul 24 00:15:31 host5.au 54387: *Jul 24 00:30:17.126 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/1 (not half duplex), with NTSW-LAKE1 FastEthernet0/1 (half duplex).
* sourcetype=syslog Options|
* source=unknown Options
#
12 24/07/2010 00:50:05.000
Jul 24 00:15:31 host6.com.au 375229: 375237: Jul 24 00:15:29.993 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on GigabitEthernet0/0 (not half duplex), with host7.com.au GigabitEthernet0/1 (half duplex).
* sourcetype=syslog Options|
* source=unknown Options
... View more