Archive

what is the reason for getting events with source=unknown

Explorer

As I am indexing the data, I notice that apart from the 'sources' that are appearing correctly (/var/log/filename.gz | 38,219 ), there is one source coming up as 'unknown' (unknown | 109,368,099). There is not any file or directory called unknown. So we are surprised by this. This is important as sometimes we search by 'source'.

On further investigation of the source=unknown, I noticed that the timestamps are actually not picked correctly from the events for all the events that are showing source=unknown. And at this particular time (wronlgy picked time) there are lots of events (as the wrongly picked events are timestamped at one particular time).

Would you have an idea of why this is happening. And a procedure to resolve this. Thanks, HB.

----------------------------------*-------------------------------------

Thanks Guys for the answer. I could give hardcoding the timestamp a try, although its surprising that the sourcetype as standard as cisco syslog with clear timestamps is not getting timestamped properly (the prblem is evident only with events that is displaying source=unknown). As Lowell indicated, i am a bit doubtful this would solve the problem.

I tried the search that you indicated : source=unknown | stats count by host, sourcetype, index | sort -count however this search doesn't run because of the following error: Error in 'IndexScopedSearch': The search failed. More than 500000 events found at time 1281228609. As i said, the wrongly timestampled events are appearing at a single time.

In terms of sourcetype, the unknown source has various sourcetypes, for instance syslog, cisco_firewall. In order to actually see the events that were coming up as source=unknown, i had to be selective and thus ran the following search: sourcetype="syslog" IOS_Messages="%CDP-4-DUPLEX_MISMATCH" A snippent of the results of this search which resulted in 822,882 matching events is pasted below (the first 2 are correctly matched source, and the rest are source=unkown):

# 7 24/07/2010 00:50:05.000

Jul 24 00:50:05 host1.com.au 44796: Jul 24 00:50:04.172 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0 (not half duplex), with Router Ethernet0 (half duplex).

* sourcetype=syslog   Options|  
* source=/var/log/splunk/logsw1/syslog/syslog.log.20100725-0005.gz   Options

# 8 24/07/2010 00:50:05.000

Jul 24 00:50:05 host2.com.au 363791: *Jul 24 00:47:56.237 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0/1 (not half duplex), with eduPaSS_wap.eduPaSS_wap FastEthernet0 (half duplex).

* sourcetype=syslog   Options|  
* source=/var/log/splunk/logsw1/syslog/syslog.log.20100725-0005.gz   Options

# 9 24/07/2010 00:50:05.000

Jul 24 00:15:32 host3.com.au 4484: Jul 24 00:15:30.952 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet1 (not full duplex), with BSLMELSWFO01.cscnms.bsl.net GigabitEthernet0/1 (full duplex).

* sourcetype=syslog   Options|  
* source=unknown   Options

# 10 24/07/2010 00:50:05.000

Jul 24 00:15:31 host4.com.au 504361: Jul 24 00:15:30.878 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet1 (not half duplex), with AHM-NSWHUR-R1 Ethernet0/0 (half duplex).

* sourcetype=syslog   Options|  
* source=unknown   Options

# 11 24/07/2010 00:50:05.000

Jul 24 00:15:31 host5.au 54387: *Jul 24 00:30:17.126 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/1 (not half duplex), with NTSW-LAKE1 FastEthernet0/1 (half duplex).

* sourcetype=syslog   Options|  
* source=unknown   Options

# 12 24/07/2010 00:50:05.000

Jul 24 00:15:31 host6.com.au 375229: 375237: Jul 24 00:15:29.993 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on GigabitEthernet0/0 (not half duplex), with host7.com.au GigabitEthernet0/1 (half duplex).

* sourcetype=syslog   Options|  
* source=unknown   Options
Tags (1)
0 Karma

New Member

This is not a timestamp problem. It is a log rotation problem. Don't use /var/log/syslog* or /var/log/* and or rotate away from that directory. Or maybe use the TCP/UDP receiver that splunk has.

0 Karma

Super Champion

So you have no idea where these events are coming from based on the event content?

Are the host/sourcetype of these events of any help and are the always the same?

Try a simple search like:

source=unknown | stats count by host, sourcetype, index | sort -count

And see if that gives you a better idea of what's going on.

Also make sure you don't have a "source=unknown" line in any of your inputs.conf files. That would certainly cause this.

0 Karma

Path Finder

My recommendation would be to fix timestamp recognition on the originating data, there are lots of option, but the one that works best for me is explicitely setting TIME_FORMAT and TIME_PREFIX in props.conf.

Please see: http://www.splunk.com/base/Documentation/4.1.4/admin/Configuretimestamprecognition

If you can post a sanitized sample of the data we can make some further recommendations.

Super Champion

Even if the timestamps are not setup properly that wouldn't change the source to "unknown".

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!