Reporting

Wrong host identifier in SYSLOG messages

avamereit
New Member

I am new to Splunk and I am sure my question is not new to the community. I have 220 Cisco endpoints reporting SYSLOG data to Splunk. All seems to be working well - except the logging of successful and failed login attempts. These messages are making it to Splunk but they are being identified as coming from host "pst" - not the host's IP address. Subsequently I cannot tell which message belongs to which host. I have about 100,000 of these failed login messages and I need to deal with them but I can't tell which Cisco devices are under attack.

0 Karma

avamereit
New Member

That was my thought as well - yes we are in the PST time zone. What puzzles me is that the source type "syslog" does just fin figuring out all the other messages but it has a hard time with these.

0 Karma

lukejadamec
Super Champion

I'm guessing that for some reason Splunk is thinking the timezone from the message is the host name, and I'm guessing that the switch is set to the PST timezone.

0 Karma

avamereit
New Member

SEC_LOGIN-4-LOGIN_FAILED

0 Karma

lukejadamec
Super Champion

What is the message ID? Cisco has a message ID for every type of log entry.

0 Karma

avamereit
New Member

I submitted a "host=pst | table _raw" and looked at the output. It seems the host info is not there. Not sure where this leaves me.

0 Karma

kristian_kolb
Ultra Champion

Any data that is configured (by you) to have the sourcetype syslog will pass through a transformation process, where the hostname is extracted from each event, on a per event basis. Typically this is something you would want to do, if you were collecting events from an existing syslog server. Instead of having the syslog server's hostname/ip-address as originator of these events, this process will extract and rewrite the hostname from each event. This is founded on the assumption that the hostname can be found in the expected place in the message.

Typically, syslog messages start with a timestamp, and that is followed by a hostname/ip-address. As you might expect, just by declaring a stream of events to be of type syslog will not make this transformation work, unless the hostname information is actually there. Also, the sourcetype name syslog has nothing to do with actual transport protocol.

It just means, "I want to call these events 'syslog', and for all messages of this type, please treat them as if they were standard syslog messages, and extract the hostname from where it should be."

You might want to read up on sourcetypes in general in the docs;

http://docs.splunk.com/Documentation/Splunk/latest/Data/Whysourcetypesmatter

One solution to deal with your current problem is to make a search-time extraction of the relevant part of the events (i.e. the part that contains the actual hostname). This can be done inline in a search query with the rex command, or through more permanent extractions in props.conf.

Please post a few sample events, and you'll probably get quick help with sorting out the regular expressions (if needed).

Hope this helps,

/K

lukejadamec
Super Champion

Did you try adding the sourcetype = not_syslog in the inputs.conf?

0 Karma

avamereit
New Member

Here are four lines from a "Show logging" directly from my Cisco router -

Jan 29 16:01:56.330 pst: %SEC-6-IPACCESSLOGP: list RESTRICT-SSH-IN denied tcp 82.221.102.177(35608) -> 173.160.208.161(22), 1 packet

Jan 29 16:02:40.218 pst: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: fred] [Source: 10.129.7.192] [localport: 22] [Reason: Login Authentication Failed] at 16:02:40 pst Wed Jan 29 2014

The 1st message shows properly in Splunk. The 2nd shows a host of "pst". The source type is "SYSLOG".

0 Karma

lukejadamec
Super Champion

@K, here is an example of the log format from a CISCO doc.
Sep 25 13:19:46.864 UTC: %SEC_LOGIN-4-LOGIN_FAILED: Login failed [user: ] [Sourc
e: 172.26.158.234] [localport: 22] [Reason: Login Authentication Failed] at 13:1
9:46 UTC Thu Sep 25 2003

0 Karma

kristian_kolb
Ultra Champion

Not sure I've understood completely - but what sourcetype does your events have?

Also, can you post a few of them?

0 Karma

avamereit
New Member

Thanks for your assistance. I modified the input.conf file in ...system\local. In it I added the following lines -
[tcp]
acceptFrom=*
connection_host=ip

connection_host was set to dns for TCP. Almost all of my SYSLOG is coming in via VPN tunnels and TCP based SYSLOG works where UDP based SYSLOG does not.
I made the changes and restarted the Splunkd service (just in case) but but new SYSLOG messages of this type are still coming in with a host name of "pst". All other SYSLOG messages seem to have the correct host ip address.

0 Karma

kristian_kolb
Ultra Champion

If your Splunk indexer is actually listening for syslog traffic itself (not really recommended) you might be in luck. From the docs for inputs.conf for UDP/TCP inputs;

connection_host = [ip|dns|none]
* "ip" sets the host to the IP address of the system sending the data.
* "dns" sets the host to the reverse DNS entry for IP address of the system sending the data.
* "none" leaves the host as specified in inputs.conf, typically the splunk system hostname.
* Defaults to "ip".

Thus your inputs.conf could be written like;

[udp://514]
connection_host=ip
sourcetype = not_syslog

/K

0 Karma

avamereit
New Member

A quick search host=pst | table _raw reveals that the source IP address isn't in the data stream. Reading the doc above points to the need to assign the source type on a per-event rather than a per-source basis. This was simple SYSLOG data coming in via UDP and TCP. THis is new territory for me - surely someone has run into this before and has already forged this trail. Is there some setting on the Switch that I need to manipulate or is this entirely done in Splunk?

0 Karma

lukejadamec
Super Champion

Are all messages hostless?
Does the _raw data contain the IP of the correct host?
search host=pst | table _raw

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.