Wrong host identifier in SYSLOG messages

[avamereit]

I am new to Splunk and I am sure my question is not new to the community. I have 220 Cisco endpoints reporting SYSLOG data to Splunk. All seems to be working well - except the logging of successful and failed login attempts. These messages are making it to Splunk but they are being identified as coming from host "pst" - not the host's IP address. Subsequently I cannot tell which message belongs to which host. I have about 100,000 of these failed login messages and I need to deal with them but I can't tell which Cisco devices are under attack.

[avamereit]

That was my thought as well - yes we are in the PST time zone. What puzzles me is that the source type "syslog" does just fin figuring out all the other messages but it has a hard time with these.

[lukejadamec]

I'm guessing that for some reason Splunk is thinking the timezone from the message is the host name, and I'm guessing that the switch is set to the PST timezone.

[avamereit]

SEC_LOGIN-4-LOGIN_FAILED

[lukejadamec]

What is the message ID? Cisco has a message ID for every type of log entry.

[avamereit]

I submitted a "host=pst | table _raw" and looked at the output. It seems the host info is not there. Not sure where this leaves me.

[kristian_kolb]

Any data that is configured (by you) to have the sourcetype syslog will pass through a transformation process, where the hostname is extracted from each event, on a per event basis. Typically this is something you would want to do, if you were collecting events from an existing syslog server. Instead of having the syslog server's hostname/ip-address as originator of these events, this process will extract and rewrite the hostname from each event. This is founded on the assumption that the hostname can be found in the expected place in the message.

Typically, syslog messages start with a timestamp, and that is followed by a hostname/ip-address. As you might expect, just by declaring a stream of events to be of type syslog will not make this transformation work, unless the hostname information is actually there. Also, the sourcetype name syslog has nothing to do with actual transport protocol.

It just means, "I want to call these events 'syslog', and for all messages of this type, please treat them as if they were standard syslog messages, and extract the hostname from where it should be."

You might want to read up on sourcetypes in general in the docs;

http://docs.splunk.com/Documentation/Splunk/latest/Data/Whysourcetypesmatter

One solution to deal with your current problem is to make a search-time extraction of the relevant part of the events (i.e. the part that contains the actual hostname). This can be done inline in a search query with the rex command, or through more permanent extractions in props.conf.

Please post a few sample events, and you'll probably get quick help with sorting out the regular expressions (if needed).

Hope this helps,

/K

[lukejadamec]

Did you try adding the sourcetype = not_syslog in the inputs.conf?

[avamereit]

Here are four lines from a "Show logging" directly from my Cisco router -

Jan 29 16:01:56.330 pst: %SEC-6-IPACCESSLOGP: list RESTRICT-SSH-IN denied tcp 82.221.102.177(35608) -> 173.160.208.161(22), 1 packet

Jan 29 16:02:40.218 pst: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: fred] [Source: 10.129.7.192] [localport: 22] [Reason: Login Authentication Failed] at 16:02:40 pst Wed Jan 29 2014

The 1st message shows properly in Splunk. The 2nd shows a host of "pst". The source type is "SYSLOG".

[lukejadamec]

@K, here is an example of the log format from a CISCO doc.
Sep 25 13:19:46.864 UTC: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: ] [Sourc
e: 172.26.158.234] [localport: 22] [Reason: Login Authentication Failed] at 13:1
9:46 UTC Thu Sep 25 2003

[kristian_kolb]

Not sure I've understood completely - but what sourcetype does your events have?

Also, can you post a few of them?

[avamereit]

Thanks for your assistance. I modified the input.conf file in ...system\local. In it I added the following lines -
[tcp]
acceptFrom=*
connection_host=ip

connection_host was set to dns for TCP. Almost all of my SYSLOG is coming in via VPN tunnels and TCP based SYSLOG works where UDP based SYSLOG does not.
I made the changes and restarted the Splunkd service (just in case) but but new SYSLOG messages of this type are still coming in with a host name of "pst". All other SYSLOG messages seem to have the correct host ip address.

[kristian_kolb]

If your Splunk indexer is actually listening for syslog traffic itself (not really recommended) you might be in luck. From the docs for inputs.conf for UDP/TCP inputs;

connection_host = [ip|dns|none]
* "ip" sets the host to the IP address of the system sending the data.
* "dns" sets the host to the reverse DNS entry for IP address of the system sending the data.
* "none" leaves the host as specified in inputs.conf, typically the splunk system hostname.
* Defaults to "ip".

Thus your inputs.conf could be written like;

[udp://514]
connection_host=ip
sourcetype = not_syslog

/K

[avamereit]

A quick search host=pst | table _raw reveals that the source IP address isn't in the data stream. Reading the doc above points to the need to assign the source type on a per-event rather than a per-source basis. This was simple SYSLOG data coming in via UDP and TCP. THis is new territory for me - surely someone has run into this before and has already forged this trail. Is there some setting on the Switch that I need to manipulate or is this entirely done in Splunk?

[lukejadamec]

Are all messages hostless?
Does the _raw data contain the IP of the correct host?
search host=pst | table _raw