Splunk Search

Source Host instead of Relay Host

jyanga
New Member

Due to network restrictions, I needed to use a server as a relay. This relay server in turn forwards the logs to my Splunk server. The problem is that the Splunk server indexes relay server as host. Hence, even if there were actually a few servers that is sending its logs to the relay server, the spunk server shows only the relay server in the Hosts box in the Splunk Summary. Is there a way for the Splunk server to index the source hosts instead of the relay server?

Tags (1)
0 Karma

scarface_01
New Member

You can try to change the "keep_hostname" option in global section of the relay syslog-ng.conf. If you are using multiple relays chain, it must be configured on all relay, if not it keeps the last relay IP address in the log server.

Pascal

0 Karma

kristian_kolb
Ultra Champion

There are a couple of things that can be done in this situation. Depending on how your relay server is setup, some of these instructions may not apply.

  1. On the Splunk indexer, set the sourcetype to 'syslog' for the incoming data. Splunk should then try to extract the host information on a per-event basis, and index that.
  2. Install a heavy splunk forwarder on the relay host, along with the syslog-ng. Have syslog-ng write the incoming data to files on disk, where the forwarder can pick them up. In this case the forwarder will do the parsing, and extract the host info.
  3. Same as for 2 above, but have the syslog daemon write separate files for each host (like /var/log/blaha/<hostname>/blaha.log . Then have the forwarder (can be a Universal Forwarder) monitor the /var/log/blaha/ directory structure, and extract the hostname through the host_segment=4 parameter for input stanzas - see the Admin manual for inputs.conf syntax.

Hope this helps,

Kristian

0 Karma

jyanga
New Member

Thank you for the wiki URL. It indeed deleted the time stamp and host IP of the relay server from the logs. However, when I go to the Summary page, the Hosts box still shows the relay server IP address instead of the source server.

FYI, per my last log, tcpdump shows that the relay server is not adding time stamp and itself in the log.

0 Karma

kristian_kolb
Ultra Champion

do your events look like:

timestamp relay-host timestamp original-host message
?

In that case, I believe you'll have to configure your syslog server to NOT write its own timestamps/hostname before relaying. Or you can have a look at this wiki post for stripping them at the indexer:

http://wiki.splunk.com/Community:StripSyslog

/K

0 Karma

jyanga
New Member

FYI, I just ran a tcpdump on the Splunk server while generating the logs from the source. The logs it receives were pristine. Hence, the Splunk server is the one that is adding the relay server in the host field.

0 Karma

jyanga
New Member

Among the recommendations, I opted to try #1. Here are the steps I have performed.

01) Deleted previously created Data inputs
02) Deleted previously generated indexed logs
03) Added a new Data input with the follow config:
UDP port: 514
Set sourcetype: From list
Select source type from list: syslog
04) generated logs

Hosts box in the Summary page still only has the Syslog-ng relay server IP address.

Did I miss a configuration?

0 Karma

jyanga
New Member

The relaying is done via Syslog-ng. The logs are preserved when Syslog-ng receives the relay and then have Splunk reads these logs. However, I am attempting to save disk space by not having Syslog-ng in the middle and letting Splunk receive the logs directly from the relay server.

0 Karma

Ayn
Legend

How is the relaying done, via Splunk forwarding, syslog, ...?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...