topic Re: Source Host instead of Relay Host in Splunk Search

Source Host instead of Relay Host

jyanga — Thu, 14 Jun 2012 20:36:53 GMT

Due to network restrictions, I needed to use a server as a relay. This relay server in turn forwards the logs to my Splunk server. The problem is that the Splunk server indexes relay server as host. Hence, even if there were actually a few servers that is sending its logs to the relay server, the spunk server shows only the relay server in the Hosts box in the Splunk Summary. Is there a way for the Splunk server to index the source hosts instead of the relay server?

Re: Source Host instead of Relay Host

Ayn — Thu, 14 Jun 2012 20:43:45 GMT

How is the relaying done, via Splunk forwarding, syslog, ...?

Re: Source Host instead of Relay Host

jyanga — Thu, 14 Jun 2012 21:03:06 GMT

The relaying is done via Syslog-ng. The logs are preserved when Syslog-ng receives the relay and then have Splunk reads these logs. However, I am attempting to save disk space by not having Syslog-ng in the middle and letting Splunk receive the logs directly from the relay server.

Re: Source Host instead of Relay Host

kristian_kolb — Thu, 14 Jun 2012 21:57:11 GMT

There are a couple of things that can be done in this situation. Depending on how your relay server is setup, some of these instructions may not apply.

On the Splunk indexer, set the sourcetype to 'syslog' for the incoming data. Splunk should then try to extract the host information on a per-event basis, and index that.
Install a heavy splunk forwarder on the relay host, along with the syslog-ng. Have syslog-ng write the incoming data to files on disk, where the forwarder can pick them up. In this case the forwarder will do the parsing, and extract the host info.
Same as for 2 above, but have the syslog daemon write separate files for each host (like /var/log/blaha/<hostname>/blaha.log . Then have the forwarder (can be a Universal Forwarder) monitor the /var/log/blaha/ directory structure, and extract the hostname through the host_segment=4 parameter for input stanzas - see the Admin manual for inputs.conf syntax.

Hope this helps,

Kristian

Re: Source Host instead of Relay Host

jyanga — Fri, 15 Jun 2012 14:36:39 GMT

Among the recommendations, I opted to try #1. Here are the steps I have performed.

01) Deleted previously created Data inputs
02) Deleted previously generated indexed logs
03) Added a new Data input with the follow config:
UDP port: 514
Set sourcetype: From list
Select source type from list: syslog
04) generated logs

Hosts box in the Summary page still only has the Syslog-ng relay server IP address.

Did I miss a configuration?

Re: Source Host instead of Relay Host

jyanga — Fri, 15 Jun 2012 14:52:50 GMT

FYI, I just ran a tcpdump on the Splunk server while generating the logs from the source. The logs it receives were pristine. Hence, the Splunk server is the one that is adding the relay server in the host field.

Re: Source Host instead of Relay Host

kristian_kolb — Sun, 17 Jun 2012 20:00:36 GMT

do your events look like:

timestamp relay-host timestamp original-host message
?

In that case, I believe you'll have to configure your syslog server to NOT write its own timestamps/hostname before relaying. Or you can have a look at this wiki post for stripping them at the indexer:

http://wiki.splunk.com/Community:StripSyslog

Re: Source Host instead of Relay Host

jyanga — Tue, 19 Jun 2012 15:55:28 GMT

Thank you for the wiki URL. It indeed deleted the time stamp and host IP of the relay server from the logs. However, when I go to the Summary page, the Hosts box still shows the relay server IP address instead of the source server.

FYI, per my last log, tcpdump shows that the relay server is not adding time stamp and itself in the log.

Re: Source Host instead of Relay Host

scarface_01 — Thu, 13 Sep 2012 13:22:51 GMT

You can try to change the "keep_hostname" option in global section of the relay syslog-ng.conf. If you are using multiple relays chain, it must be configured on all relay, if not it keeps the last relay IP address in the log server.

Pascal