How can I configure Heavy Forwarder to sent to two different Splunk instances?

[jwalzerpitt]

I have the following Splunk architecture

Server A has Splunk installed. It also has Sysmon installed, which I am forwarding to our on-prem Splunk instance. I'd like to configure Server A to function as a heavy forwarder so I can forward Windows Security Event logs from a Domain controller and then route them (not index on Server A) to our Splunk instance on Azure.

I've been reviewing docs and other questions/answers to try and configure my inputs and outputs.conf file correctly and here is what I have:

outputs.conf

[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout:default-autolb-group]
server = splunk-onprem:9997

[tcpout-server://splunk-onprem:9997]

[tcpout:azure]
server = x.x.x.x:9997
indexAndForward = 0
clientCert = opt/splunk/etc/auth/server.pem.
sslPassword = xxxxxxxxxx
sslVerifyServerCert = false

inputs.conf

[default]
host = servername

[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = false
renderXml = 1
index=sysmon

[WinEventLog://Security] 
 _TCP_ROUTING = azure

I believe the outputs.conf file is good, but not sure about the inputs.conf file.

Any help would be greatly appreciated
Thx

[somesoni2]

See this

http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Filter_and_route_ev...

[jwalzerpitt]

Thx for the link.

I created a props.conf file as follows:
[azure]
TRANSFORMS-routing=azure

I created a transforms.conf file as follows:
[azure]
REGEX=WinEventLog:Security
DEST_KEY=_TCP_ROUTING
FORMAT=azure

Restarted the splunkd service, and checked the splunkd.log and see the same messages:

02-16-2018 09:01:17.183 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:65235. Read Timeout Timed out after 600 seconds.
02-16-2018 09:01:17.183 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:56929. Read Timeout Timed out after 600 seconds.
02-16-2018 09:01:17.183 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:59042. Read Timeout Timed out after 600 seconds.
02-16-2018 09:11:18.196 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:49407. Read Timeout Timed out after 600 seconds.
02-16-2018 09:11:18.196 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:57472. Read Timeout Timed out after 600 seconds.
02-16-2018 09:11:18.196 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:59586. Read Timeout Timed out after 600 seconds.
02-16-2018 09:21:19.214 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:49969. Read Timeout Timed out after 600 seconds.
02-16-2018 09:21:19.214 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:58009. Read Timeout Timed out after 600 seconds.
02-16-2018 09:21:19.214 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:60157. Read Timeout Timed out after 600 seconds.

[somesoni2]

What about the outputs.conf? Do you've a tcpout group defined for azure with correct host :port information?

[jwalzerpitt]

outputs.conf defined as follows:

[tcpout:azure]
server = x.x.x.x:9997
indexAndForward = 0
clientCert = opt/splunk/etc/auth/server.pem.
sslPassword = xxxxx
sslVerifyServerCert = false

[iandrews_splunk]

_TCP_ROUTING should be all you need. have you tried it out, yet?

[jwalzerpitt]

Modified the domain controller to send Windows security event logs to server A (HF). In Wireshark I see traffic coming into port 9997 from the domain controller and on the Azure Splunk server I see traffic from server A via tcpdump on port 9997, but the winevt index is not being populated. The inputs.conf file on Azure Splunk is as follows:

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:(?!\s*groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:(?!\s*groupPolicyContainer)"
index = wineventlog
renderXml=false

With that, where can I troubleshoot to see if the logs are being sent on by server A, or if there is an error/failure on server A?

Thx

[iandrews_splunk]

Does the new indexer destination have the same index defined in indexes.conf? Also, are there any errors in splunkd.log?

[jwalzerpitt]

I am seeing the following errors in the splunkd.log on server A:

02-15-2018 15:20:50.704 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:58553. Read Timeout Timed out after 600 seconds.
02-15-2018 15:20:50.704 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:63109. Read Timeout Timed out after 600 seconds.
02-15-2018 15:20:51.972 -0500 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:61772. Read Timeout Timed out after 600 seconds.

I have the Microsoft Add-on installed on the Azure Splunk instance and the indexes.conf reads:

[windows]
homePath = $SPLUNK_DB/windows/db
coldPath = $SPLUNK_DB/windows/colddb
thawedPath = $SPLUNK_DB/windows/thaweddb

[wineventlog]
homePath = $SPLUNK_DB/wineventlog/db
coldPath = $SPLUNK_DB/wineventlog/colddb
thawedPath = $SPLUNK_DB/wineventlog/thaweddb

[perfmon]
homePath = $SPLUNK_DB/perfmon/db
coldPath = $SPLUNK_DB/perfmon/colddb
thawedPath = $SPLUNK_DB/perfmon/thaweddb

Thx

[jwalzerpitt]

I am going to test later today - I will update this thread after I test

Thx