Splunk indexer is not sending to syslog - please help

[awalesa]

Hi,

I've tried everything. I have read all the answers and docs. A cannot force splunk indexer to forward all events to syslog server. I even tried to look at tcpdump output and here is no trace of communication on desired port.

Config s simple - UniversalForwarder (Windows Events) -> Splunk Indexer (Linux) -> Syslog (Linux). UF to SI works, SI to Syslog not. tecpdump on SI is not showing any outbound communication to syslog.

outputs.conf from SI - 192.168.9.22 is IP of Syslog

[syslog]

defaultGroup = mysyslog

[syslog:mysyslog]

server = 192.168.9.22:514

type = udp

I don't want fo filter anything so I'm not using props.conf and transforms.conf - but with them the situation is the same - no communication between SI and Syslog.

Maybe I have some component disabled or something? I have tried this config on both linux and Windows (all version 4.3) and no luck.

Anyone has working config files to share? Any ideas?

Thanks,
Alex

[awalesa]

Hi, I like the format of windows events send directly form UF to indexer and the UF cannot send to syslog 😞 I tried to use snare as syslog client to syslog ant then to splunk but the output at the end is messed (internatiolan letters, one single line etc)

Alex

[awalesa]

Sorry 🙂

I want to try to use OSSIM at the end to warn on anomalies

Alex

[Drainy]

But why do you need the syslog functionality at the end? is that just where you have all your other logs centralised? Also you can just click on comment below this instead of doing another answer 🙂

[awalesa]

In the free license the syslog forward feature is disabled - that's why I cannot force splunk to talk to syslog 😞

Alex

[Drainy]

Hah! You learn something new every day. Ok, as an aside why did you want to forward onto syslog in the first place?

[awalesa]

Is there any debug log or debug switch to see how the routing goes? I just cannot understand why splunk is not forwarding events to syslog...

ALex

[awalesa]

Hi, nothing like that - both servers are on LAN. I can succesfully forward events with syslog-ng between both (disabling splunk for that time). When I try to do the same with splunk nothing happens. When I run tcpdump on indexer (which should send events to syslog) also nothing - even one packet 😞

Thanks,
Alex

[wcolgate_splunk]

You mention "no communication". Do you have a firewall, or similar software that is blocking/dropping network packets?

[awalesa]

Hi,

I have changed props as you suggested - sill no communication - it's look like splunk doesn't even try to send anything to syslog. Just to be sure I have tried to forward events with syslog-ng - with success. For sure there is something wrong with my splunk...

Alex

[awalesa]

Hi,

all files edited in SPLUNK_HOME/etc/system/local/

outputs.conf

[syslog]

defaultGroup = mysyslog

[syslog:mysyslog]

server = 192.168.9.151:514

type = udp

props.conf

[host::*]

TRANSFORMS-routing = send_to_syslog

transforms.conf

[send_to_syslog]

REGEX = .

DEST_KEY = _SYSLOG_ROUTING

FORMAT = mysyslog

output of

/opt/splunk/bin/splunk cmd btool outputs list --debug
system [syslog]
system defaultGroup = mysyslog
system [syslog:mysyslog]
system server = 192.168.9.151:514
system type = udp
system [tcpout]
system autoLB = true
system autoLBFrequency = 30
system blockOnCloning = true
system compressed = false
system connectionTimeout = 20
system disabled = false
system dropClonedEventsOnQueueFull = 5
system dropEventsOnQueueFull = -1
system forwardedindex.0.whitelist = .*
system forwardedindex.1.blacklist = _.*
system forwardedindex.2.whitelist = _audit
system forwardedindex.filter.disable = false
system heartbeatFrequency = 30
system indexAndForward = false
system maxConnectionsPerIndexer = 2
system maxFailuresPerInterval = 2
system maxQueueSize = 500KB
system readTimeout = 300
system secsInFailureInterval = 1
system sendCookedData = true
system useACK = false
system writeTimeout = 300

output of

/opt/splunk/bin/splunk cmd btool props list --debug|grep syslog
system [anaconda_syslog]
system REPORT-syslog = syslog-extractions
system [cisco_syslog]
system REPORT-syslog = syslog-extractions
system TRANSFORMS = syslog-host
system [delayedrule::syslog]
system sourcetype = syslog
system TRANSFORMS-routing = send_to_syslog
system [linux_messages_syslog]
system REPORT-syslog = syslog-extractions
system TRANSFORMS = syslog-host
system REPORT-syslog = syslog-extractions
system [postfix_syslog]
system REPORT-syslog = syslog-extractions
system TRANSFORMS-host = syslog-host
system [rule::postfix_syslog]
system sourcetype = postfix_syslog
system [rule::sendmail_syslog]
system sourcetype = sendmail_syslog
system [sendmail_syslog]
system REPORT-syslog = sendmail-extractions
system TRANSFORMS = syslog-host
system sourcetype = syslog
system sourcetype = syslog
system sourcetype = syslog
system [source::.../syslog(.\d+)?]
system sourcetype = syslog
system [source::.../var/log/anaconda.syslog(.\d+)?]
system sourcetype = anaconda_syslog
system [syslog]
system REPORT-syslog = syslog-extractions
system TRANSFORMS = syslog-host
system [windows_snare_syslog]
system REPORT-syslog = syslog-extractions
system TRANSFORMS = syslog-host

output of
/opt/splunk/bin/splunk cmd btool transforms list --debug
system [send_to_syslog]
system CAN_OPTIMIZE = True
system CLEAN_KEYS = True
system DEFAULT_VALUE =
system DEST_KEY = _SYSLOG_ROUTING
system FORMAT = mysyslog
system KEEP_EMPTY_VALS = False
system LOOKAHEAD = 4096
system MV_ADD = False
system REGEX = .
system SOURCE_KEY = _raw
system WRITE_META = False

I can see that in props there is delayedrule and not host::* - what does it mean?
Splunk of course restarted. I'm feeding indexer with UF (windows) and syslog generator (UDP:514 - source syslog) - sill no effects 😞

Thanks
Alex

[Drainy]

I'll update my answer above, have a looksie

[Drainy]

A few things to check.
Have you restarted the Splunk indexer? this is required for it to read in the updated config.

Secondly, have you defined what traffic you want to send? you will need to specify the data even if its everything, e.g. define it for the host field of the UF. Or use wildcards with regex to specify all.
http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf

The need to define the traffic is explained here; (I have pasted a snippet below)
http://docs.splunk.com/Documentation/Splunk/latest/admin/Outputsconf

#---- Routing Data to Syslog Server -----
# To route data to syslog server:
# 1) Decide which events to route to which servers.
# 2) Edit the props.conf, transforms.conf, and outputs.conf files on the forwarders.

# Edit $SPLUNK_HOME/etc/system/local/props.conf and set a TRANSFORMS-routing attribute as shown here:

 [<spec>]
 TRANSFORMS-routing=<unique_stanza_name>

* <spec> can be: 
  * <sourcetype>, the source type of an event 
  * host::<host>, where <host> is the host for an event 
  * source::<source>, where <source> is the source for an event 

* Use the <unique_stanza_name> when creating your entry in transforms.conf.

# Edit $SPLUNK_HOME/etc/system/local/transforms.conf and set rules to match your props.conf stanza: 

  [<unique_stanza_name>]
  REGEX=<your_regex>
  DEST_KEY=_SYSLOG_ROUTING
  FORMAT=<unique_group_name>

* <unique_stanza_name> must match the name you created in props.conf. 
* Enter the regex rules in <your_regex> to determine which events get conditionally routed. 
* DEST_KEY should be set to _SYSLOG_ROUTING to send events via SYSLOG.
* Set FORMAT to <unique_group_name>. This should match the syslog group name you create in outputs.conf.

Finally. If you have defined traffic then run this command and what is the output;

./splunk cmd btool outputs list --debug

This will list all the outputs.conf detail it has read in, debug forces it to pre-pend each line with the App name that the config has taken effect from.
Presumably you are editing outputs.conf in SPLUNK_HOME/etc/system/local/?

EDIT:
Ok, so what source does your incoming syslog data (incoming to the indexer) have? E.g. on my system its just syslog-data.
In which case, why don't you try applying the props to source::syslog-data and see how that performs? Having a search around I have seen a few others had issue using host.