Getting Data In

Windows: Sending Splunk logs to third party server

spectrum2035
Explorer

I need to send Windows Event logs to the third party syslog solutions. Logs from Windows Universal Forwarder is sent to HFWD and from there it is routed both Splunk IDX and Syslog Aggregator. For some reasons its not hitting the syslog server. I have checked btool for input, output, props and transforms and couldn't find anything there.

Config on the HFWD to accept logs from the Windows server and to send it to syslog

=========================================================================

props.conf

[host::10.20.10.10]
TRANSFORMS-routing_syslog = fwd_data_to_syslog

transforms.conf

[fwd_data_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = to_syslog

outputs.conf
#Sent to Indexer

[syslog:to_syslog]
server = 10.172.148.186:1514
#type = udp

inputs.conf
[splunktcp://10.20.10.10:9997]
#machine not part of the domain so need to use the IP address
#_SYSLOG_ROUTING = to_syslog

=========================================================================
Config on Windows UF

[tcpout]
defaultGroup = send_to_syslog
maxQueueSize = 7MB
autoLBFrequency=15


[tcpout:send_to_syslog]
server = 10.175.108.40:9997
#sendCookedData = false
=========================================================================

One of the base app to send logs from Heavy FWD to INDX

[tcpout]
default_group = indexer_fwd
axQueueSize = 7MB
autoLBFrequency=15

[tcpout:indexer_fwd]
server = IDX1.abcd.com:9997, IDX2.abcd.com:9997, IDX3.abcd.com:9997
0 Karma
1 Solution

koshyk
Super Champion

Please try following

ALL below settings have to be done on the Heavy Forwarder

In props.conf

#props.conf
[host::10.20.10.10]
TRANSFORMS-routing_syslog = fwd_data_to_syslog

in transforms.conf

#transforms.conf
[fwd_data_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = to_syslog

In outputs.conf (alongside your indexer outputs, you need to add syslog stanza separately)

# outputs.conf
[syslog:to_syslog]
server = 10.172.148.186:1514
#type = udp

If this is not working,
1. you need to get tcpdump of HeavyForwarder to see if there is some network connection issue. You need to see destination traffic going to 10.172.148.186 on port 1514
2. Check for firewall issues
3. Check tcpdump at the destination server level to ensure the message is captured at wire
4. Try setting up this connection to another server which you own with no firewall etc.
5. Try removing the indexer setting just to see if there is any conflict of stanza. You can double check using btool

View solution in original post

koshyk
Super Champion

Please try following

ALL below settings have to be done on the Heavy Forwarder

In props.conf

#props.conf
[host::10.20.10.10]
TRANSFORMS-routing_syslog = fwd_data_to_syslog

in transforms.conf

#transforms.conf
[fwd_data_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = to_syslog

In outputs.conf (alongside your indexer outputs, you need to add syslog stanza separately)

# outputs.conf
[syslog:to_syslog]
server = 10.172.148.186:1514
#type = udp

If this is not working,
1. you need to get tcpdump of HeavyForwarder to see if there is some network connection issue. You need to see destination traffic going to 10.172.148.186 on port 1514
2. Check for firewall issues
3. Check tcpdump at the destination server level to ensure the message is captured at wire
4. Try setting up this connection to another server which you own with no firewall etc.
5. Try removing the indexer setting just to see if there is any conflict of stanza. You can double check using btool

View solution in original post

spectrum2035
Explorer

The issue was with the firewall, Network team didnt open the firewall rule (though they said its been done).

0 Karma

codebuilder
SplunkTrust
SplunkTrust

You have a typo in your tcpout stanza in your app (based on the values you provided):

 [tcpout]
 default_group = indexer_fwd
 axQueueSize = 7MB
 autoLBFrequency=15

Should be:

 [tcpout]
 default_group = indexer_fwd
 maxQueueSize = 7MB
 autoLBFrequency=15

That is likely causing errors/issues. Also, make sure that you have an index created to receive your data. This should match your DEST_KEY.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

evania
Community Manager
Community Manager

Hi @codebuilder ,

Did you have a chance to check out answers? If it worked, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help you.

Thanks for posting!

0 Karma

spectrum2035
Explorer

It didnt work.. as commented by koshyk we are still waiting for tcpdump data from the linux team.

0 Karma

spectrum2035
Explorer

Thanks codebuilder for your response. I did check "maxQueueSize" settings, it was the typo while i put here. Logs are receiving at the indexer without any issues.

We have issues only at the syslog aggregator (10.172.148.186 at port 1514)

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Do you have the pass4SymmKey set correctly on your HF?

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

spectrum2035
Explorer

To add, we are not using SSL as we are currently testing phase. HF are working for other data sources and is able to send data to the Indexers. So pass4SymmKey wont be an issue here.
The issue is with the HF to syslog.

0 Karma

spectrum2035
Explorer

Just to give one more update: When i have checked the internal index with port 1514 i am getting following output..

06-06-2019 16:46:36.584 +0100 INFO Metrics - group=syslog_connections, ingest_pipe=1, to_syslog:10.172.148.186:1514:10.172.148.186:1514, sourcePort=8089, destIp=10.172.148.186, destPort=1514, _tcp_Bps=2744.97, _tcp_KBps=2.68, _tcp_avg_thruput=2.74, _tcp_Kprocessed=46355, _tcp_eps=12.68

0 Karma

spectrum2035
Explorer

Forgot to add, i have also added to_syslog to the tcpout default_group

[tcpout]
default_group = indexer_fwd, to_syslog as some else mentioned that it has resolved their issue. But for me No Luck 😞

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!