Splunk Search

what is the reason for getting events with source=unknown

hemantbhatta
Explorer

As I am indexing the data, I notice that apart from the 'sources' that are appearing correctly (/var/log/filename.gz | 38,219 ), there is one source coming up as 'unknown' (unknown | 109,368,099). There is not any file or directory called unknown. So we are surprised by this. This is important as sometimes we search by 'source'.

On further investigation of the source=unknown, I noticed that the timestamps are actually not picked correctly from the events for all the events that are showing source=unknown. And at this particular time (wronlgy picked time) there are lots of events (as the wrongly picked events are timestamped at one particular time).

Would you have an idea of why this is happening. And a procedure to resolve this. Thanks, HB.

----------------------------------*-------------------------------------

Thanks Guys for the answer. I could give hardcoding the timestamp a try, although its surprising that the sourcetype as standard as cisco syslog with clear timestamps is not getting timestamped properly (the prblem is evident only with events that is displaying source=unknown). As Lowell indicated, i am a bit doubtful this would solve the problem.

I tried the search that you indicated : source=unknown | stats count by host, sourcetype, index | sort -count however this search doesn't run because of the following error: Error in 'IndexScopedSearch': The search failed. More than 500000 events found at time 1281228609. As i said, the wrongly timestampled events are appearing at a single time.

In terms of sourcetype, the unknown source has various sourcetypes, for instance syslog, cisco_firewall. In order to actually see the events that were coming up as source=unknown, i had to be selective and thus ran the following search: sourcetype="syslog" IOS_Messages="%CDP-4-DUPLEX_MISMATCH" A snippent of the results of this search which resulted in 822,882 matching events is pasted below (the first 2 are correctly matched source, and the rest are source=unkown):

# 7 24/07/2010 00:50:05.000

Jul 24 00:50:05 host1.com.au 44796: Jul 24 00:50:04.172 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0 (not half duplex), with Router Ethernet0 (half duplex).

* sourcetype=syslog   Options|  
* source=/var/log/splunk/logsw1/syslog/syslog.log.20100725-0005.gz   Options

# 8 24/07/2010 00:50:05.000

Jul 24 00:50:05 host2.com.au 363791: *Jul 24 00:47:56.237 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/0/1 (not half duplex), with eduPaSS_wap.eduPaSS_wap FastEthernet0 (half duplex).

* sourcetype=syslog   Options|  
* source=/var/log/splunk/logsw1/syslog/syslog.log.20100725-0005.gz   Options

# 9 24/07/2010 00:50:05.000

Jul 24 00:15:32 host3.com.au 4484: Jul 24 00:15:30.952 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet1 (not full duplex), with BSLMELSWFO01.cscnms.bsl.net GigabitEthernet0/1 (full duplex).

* sourcetype=syslog   Options|  
* source=unknown   Options

# 10 24/07/2010 00:50:05.000

Jul 24 00:15:31 host4.com.au 504361: Jul 24 00:15:30.878 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet1 (not half duplex), with AHM-NSWHUR-R1 Ethernet0/0 (half duplex).

* sourcetype=syslog   Options|  
* source=unknown   Options

# 11 24/07/2010 00:50:05.000

Jul 24 00:15:31 host5.au 54387: *Jul 24 00:30:17.126 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/1 (not half duplex), with NTSW-LAKE1 FastEthernet0/1 (half duplex).

* sourcetype=syslog   Options|  
* source=unknown   Options

# 12 24/07/2010 00:50:05.000

Jul 24 00:15:31 host6.com.au 375229: 375237: Jul 24 00:15:29.993 UTC: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on GigabitEthernet0/0 (not half duplex), with host7.com.au GigabitEthernet0/1 (half duplex).

* sourcetype=syslog   Options|  
* source=unknown   Options
Tags (1)
0 Karma

surajana
Engager

Even i have same issue trying to resolve but didn't get any solution as of now.

0 Karma

wer1wer1
New Member

This is not a timestamp problem. It is a log rotation problem. Don't use /var/log/syslog* or /var/log/* and or rotate away from that directory. Or maybe use the TCP/UDP receiver that splunk has.

0 Karma

Lowell
Super Champion

So you have no idea where these events are coming from based on the event content?

Are the host/sourcetype of these events of any help and are the always the same?

Try a simple search like:

source=unknown | stats count by host, sourcetype, index | sort -count

And see if that gives you a better idea of what's going on.

Also make sure you don't have a "source=unknown" line in any of your inputs.conf files. That would certainly cause this.

0 Karma

stephanbuys
Path Finder

My recommendation would be to fix timestamp recognition on the originating data, there are lots of option, but the one that works best for me is explicitely setting TIME_FORMAT and TIME_PREFIX in props.conf.

Please see: http://www.splunk.com/base/Documentation/4.1.4/admin/Configuretimestamprecognition

If you can post a sanitized sample of the data we can make some further recommendations.

Lowell
Super Champion

Even if the timestamps are not setup properly that wouldn't change the source to "unknown".

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...