Due to network restrictions, I needed to use a server as a relay. This relay server in turn forwards the logs to my Splunk server. The problem is that the Splunk server indexes relay server as host. Hence, even if there were actually a few servers that is sending its logs to the relay server, the spunk server shows only the relay server in the Hosts box in the Splunk Summary. Is there a way for the Splunk server to index the source hosts instead of the relay server?
You can try to change the "keep_hostname" option in global section of the relay syslog-ng.conf. If you are using multiple relays chain, it must be configured on all relay, if not it keeps the last relay IP address in the log server.
There are a couple of things that can be done in this situation. Depending on how your relay server is setup, some of these instructions may not apply.
/var/log/blaha/<hostname>/blaha.log. Then have the forwarder (can be a Universal Forwarder) monitor the
/var/log/blaha/directory structure, and extract the hostname through the
host_segment=4parameter for input stanzas - see the Admin manual for inputs.conf syntax.
Hope this helps,
Thank you for the wiki URL. It indeed deleted the time stamp and host IP of the relay server from the logs. However, when I go to the Summary page, the Hosts box still shows the relay server IP address instead of the source server.
FYI, per my last log, tcpdump shows that the relay server is not adding time stamp and itself in the log.
do your events look like:
timestamp relay-host timestamp original-host message
In that case, I believe you'll have to configure your syslog server to NOT write its own timestamps/hostname before relaying. Or you can have a look at this wiki post for stripping them at the indexer:
FYI, I just ran a tcpdump on the Splunk server while generating the logs from the source. The logs it receives were pristine. Hence, the Splunk server is the one that is adding the relay server in the host field.
Among the recommendations, I opted to try #1. Here are the steps I have performed.
01) Deleted previously created Data inputs
02) Deleted previously generated indexed logs
03) Added a new Data input with the follow config:
UDP port: 514
Set sourcetype: From list
Select source type from list: syslog
04) generated logs
Hosts box in the Summary page still only has the Syslog-ng relay server IP address.
Did I miss a configuration?
The relaying is done via Syslog-ng. The logs are preserved when Syslog-ng receives the relay and then have Splunk reads these logs. However, I am attempting to save disk space by not having Syslog-ng in the middle and letting Splunk receive the logs directly from the relay server.