Hi,
i have a weird problem with forwarding logs from my apache servers to both spunk and a 3rd party syslog server. As soon as i involve the Splunk Forwarder installed on the apache host i loose the original hostname of my apache server when i forward the messages to a central standard syslog-ng server, this syslog server has a custom input written that listens on port 5514 that will use the originating host from the syslog header so i can index the log data to the right host in the db I'm storing it in. But if i skip the splunk forwarder and use standard syslog from apache to the indexer the forwarding works as expected and the origination hostname is retained. I have tried multiple configuration options on the indexer, they are detailed further down. In this environment it is not possible to send the logs straight from the splunkForwarder to the syslog server for other reasons.
Any idea what to do to solve this? This should be standard stuff imho so i think i must have missed something basic.
apache - 192.168.7.166 # has default log options apart from hostname first added (%V)
SplunkIndexer - 192.168.7.157
SyslogServer - 192.168.7.15
Client - 192.168.7.1
The original log on file is:
notroot@apache:~$ tail -f /var/log/apache2/access.log
apache 192.168.7.166 192.168.7.1 - - [25/Mar/2014:18:59:31 +0100] "GET / HTTP/1.1" 304 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36"
NON WORKING SETUP:
apache--SplunkUnivForwarder-->SplunkIndexerport9997--forwardToSyslogTCPoutput-->SyslogServerport5514
No matter what i do the log always come in with a source of 192.168.7.157, which should be 192.168.7.166 is the source of the log is my apache server.
Log arrives at central syslog server:
2014-03-24T21:01:45+01:00 192.168.7.157 apache 192.168.7.166 192.168.7.1 - - [24/Mar/2014:19:52:48 +0000] "GET / HTTP/1.1" 304 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36"
No matter what config options for forwarding i try on the indexer it always comes in with .157, see below for confg files.
WORKING SETUP:
apache--syslog (syslog-ng)-->SplunkIndexerPort514---forwardToUnomalyasTCPoutput-->UnomalyPort5514.
In Splunk it looks like this, the timestamp and originating host (apache) correctly seen:
apache apache 192.168.7.166 192.168.7.1 - - [25/Mar/2014:18:59:31 +0100] "GET / HTTP/1.1" 304 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36"
After forwarding to the Syslog server the originating host is intact (apache):
unomaly@unomaly:~$ tail -f /var/log/unomaly.in | grep apache
2014-03-25T18:59:32+01:00 apache apache 192.168.7.166 192.168.7.1 - - [25/Mar/2014:18:59:31 +0100] "GET / HTTP/1.1" 304 209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36"
I suspect this would also work if the Universal Forwarder could do standard syslog to the indexer as the syslog-ng is doing in the working example…
I have tried with both syslog and tcpout options (some now commented out below) in the outputs.conf as well as syslog_routing with props and transform files on the indexer:
outputs.conf
#[syslog]
#defaultGroup=allElseGroup
#[syslog:syslogGroup]
#server = 192.168.7.111:5514
#type = tcp
#[syslog:allElseGroup]
#server = 192.168.7.111:5514
#type=tcp
[tcpout:unomaly]
server = 192.168.7.111:5514
sendCookedData = false
====
transform.conf
REGEX=.
DEST_KEY=_SYSLOG_ROUTING
FORMAT=syslogGroup
===
props.conf
[syslog]
TRANSFORMS-routing=syslogRouting
... View more