I am new to Splunk and I am sure my question is not new to the community. I have 220 Cisco endpoints reporting SYSLOG data to Splunk. All seems to be working well - except the logging of successful and failed login attempts. These messages are making it to Splunk but they are being identified as coming from host "pst" - not the host's IP address. Subsequently I cannot tell which message belongs to which host. I have about 100,000 of these failed login messages and I need to deal with them but I can't tell which Cisco devices are under attack.
That was my thought as well - yes we are in the PST time zone. What puzzles me is that the source type "syslog" does just fin figuring out all the other messages but it has a hard time with these.
Any data that is configured (by you) to have the sourcetype
syslog will pass through a transformation process, where the hostname is extracted from each event, on a per event basis. Typically this is something you would want to do, if you were collecting events from an existing syslog server. Instead of having the syslog server's hostname/ip-address as originator of these events, this process will extract and rewrite the hostname from each event. This is founded on the assumption that the hostname can be found in the expected place in the message.
Typically, syslog messages start with a timestamp, and that is followed by a hostname/ip-address. As you might expect, just by declaring a stream of events to be of type
syslog will not make this transformation work, unless the hostname information is actually there. Also, the sourcetype name
syslog has nothing to do with actual transport protocol.
It just means, "I want to call these events 'syslog', and for all messages of this type, please treat them as if they were standard syslog messages, and extract the hostname from where it should be."
You might want to read up on sourcetypes in general in the docs;
One solution to deal with your current problem is to make a search-time extraction of the relevant part of the events (i.e. the part that contains the actual hostname). This can be done inline in a search query with the
rex command, or through more permanent extractions in props.conf.
Please post a few sample events, and you'll probably get quick help with sorting out the regular expressions (if needed).
Hope this helps,
Here are four lines from a "Show logging" directly from my Cisco router -
Jan 29 16:01:56.330 pst: %SEC-6-IPACCESSLOGP: list RESTRICT-SSH-IN denied tcp 22.214.171.124(35608) -> 126.96.36.199(22), 1 packet
Jan 29 16:02:40.218 pst: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: fred] [Source: 10.129.7.192] [localport: 22] [Reason: Login Authentication Failed] at 16:02:40 pst Wed Jan 29 2014
The 1st message shows properly in Splunk. The 2nd shows a host of "pst". The source type is "SYSLOG".
@K, here is an example of the log format from a CISCO doc.
Sep 25 13:19:46.864 UTC: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: ] [Sourc
e: 172.26.158.234] [localport: 22] [Reason: Login Authentication Failed] at 13:1
9:46 UTC Thu Sep 25 2003
Thanks for your assistance. I modified the input.conf file in ...system\local. In it I added the following lines -
connection_host was set to dns for TCP. Almost all of my SYSLOG is coming in via VPN tunnels and TCP based SYSLOG works where UDP based SYSLOG does not.
I made the changes and restarted the Splunkd service (just in case) but but new SYSLOG messages of this type are still coming in with a host name of "pst". All other SYSLOG messages seem to have the correct host ip address.
If your Splunk indexer is actually listening for syslog traffic itself (not really recommended) you might be in luck. From the docs for inputs.conf for UDP/TCP inputs;
connection_host = [ip|dns|none]
* "ip" sets the host to the IP address of the system sending the data.
* "dns" sets the host to the reverse DNS entry for IP address of the system sending the data.
* "none" leaves the host as specified in inputs.conf, typically the splunk system hostname.
* Defaults to "ip".
Thus your inputs.conf could be written like;
sourcetype = not_syslog
A quick search host=pst | table _raw reveals that the source IP address isn't in the data stream. Reading the doc above points to the need to assign the source type on a per-event rather than a per-source basis. This was simple SYSLOG data coming in via UDP and TCP. THis is new territory for me - surely someone has run into this before and has already forged this trail. Is there some setting on the Switch that I need to manipulate or is this entirely done in Splunk?