Source Host instead of Relay Host

[jyanga]

Due to network restrictions, I needed to use a server as a relay. This relay server in turn forwards the logs to my Splunk server. The problem is that the Splunk server indexes relay server as host. Hence, even if there were actually a few servers that is sending its logs to the relay server, the spunk server shows only the relay server in the Hosts box in the Splunk Summary. Is there a way for the Splunk server to index the source hosts instead of the relay server?

[scarface_01]

You can try to change the "keep_hostname" option in global section of the relay syslog-ng.conf. If you are using multiple relays chain, it must be configured on all relay, if not it keeps the last relay IP address in the log server.

Pascal

[kristian_kolb]

There are a couple of things that can be done in this situation. Depending on how your relay server is setup, some of these instructions may not apply.

1. On the Splunk indexer, set the sourcetype to 'syslog' for the incoming data. Splunk should then try to extract the host information on a per-event basis, and index that.
2. Install a heavy splunk forwarder on the relay host, along with the syslog-ng. Have syslog-ng write the incoming data to files on disk, where the forwarder can pick them up. In this case the forwarder will do the parsing, and extract the host info.
3. Same as for 2 above, but have the syslog daemon write separate files for each host (like /var/log/blaha/<hostname>/blaha.log . Then have the forwarder (can be a Universal Forwarder) monitor the /var/log/blaha/ directory structure, and extract the hostname through the host_segment=4 parameter for input stanzas - see the Admin manual for inputs.conf syntax.

Hope this helps,

Kristian

[jyanga]

Thank you for the wiki URL. It indeed deleted the time stamp and host IP of the relay server from the logs. However, when I go to the Summary page, the Hosts box still shows the relay server IP address instead of the source server.

FYI, per my last log, tcpdump shows that the relay server is not adding time stamp and itself in the log.

[kristian_kolb]

do your events look like:

timestamp relay-host timestamp original-host message
?

In that case, I believe you'll have to configure your syslog server to NOT write its own timestamps/hostname before relaying. Or you can have a look at this wiki post for stripping them at the indexer:

http://wiki.splunk.com/Community:StripSyslog

/K

[jyanga]

FYI, I just ran a tcpdump on the Splunk server while generating the logs from the source. The logs it receives were pristine. Hence, the Splunk server is the one that is adding the relay server in the host field.

[jyanga]

Among the recommendations, I opted to try #1. Here are the steps I have performed.

01) Deleted previously created Data inputs
02) Deleted previously generated indexed logs
03) Added a new Data input with the follow config:
UDP port: 514
Set sourcetype: From list
Select source type from list: syslog
04) generated logs

Hosts box in the Summary page still only has the Syslog-ng relay server IP address.

Did I miss a configuration?

[jyanga]

The relaying is done via Syslog-ng. The logs are preserved when Syslog-ng receives the relay and then have Splunk reads these logs. However, I am attempting to save disk space by not having Syslog-ng in the middle and letting Splunk receive the logs directly from the relay server.

[Ayn]

How is the relaying done, via Splunk forwarding, syslog, ...?