All Apps and Add-ons

Technology Add-on for Windows: Heavy Forwarder to Clustered Indexers

willadams
Contributor

I believe I have managed to get myself confused and would like to request assistance in resolving a conundrum I have with my SPLUNK infrastructure.

I have a new SPLUNK Clustered Environment with 1 Search Head, 1 Deployment / License server, 1 Cluster manager and 2 Indexers. I also have a corporate network and my DMZ network. I am using the SPLUNK TA for Windows as well as the SPLUNK TA for SYSMON.

On my SPLUNK infrastructure, I have the SPLUNK TA for Windows and SPLUNK TA for SYSMON installed on the Cluster Manager, The Deployment Server, and the Search Head. The cluster manager deploys a "master app" to each of the Indexers negating the need for me to manually do it. I have created a new application called "mydomain_windows_events". I have created 4 folders in this application (default, bin, local and meta). I have created inputs.conf file cloned off the SPLUNK TA for Windows. I have modified the inputs.conf file to capture events I would like.

For my corporate connected machine, it has a Universal Forwarder installed to it that sends directly to the CM and the data transmits successfully and all my data is indexed properly and extracts the fields as required.

I also have a SPLUNK Heavy Forwarder in our DMZ network that is also a Deployment server (similar application configured as non-DMZ above). All the DMZ Windows devices will send their logs to the Heavy Forwarder to then be forwarded to a test standalone SPLUNK Enterprise instance. I confirmed that my standalone Enterprise instance receives the logs fine and all of the fields are extracted properly. However, I then configured my SPLUNK Heavy Forwarder via "Forwarding and Receiving" to send to my clustered indexers (I added indexer1:9997, indexer2:9997) and I am no longer getting the extract fields as before.

What have I done incorrectly? I think this may be my understanding as to whether I should configure my Heavy Forwarder to send to the CM via port 8089 as per my non-DMZ configuration? Or whether I should be sending the Heavy Forwarder directly to the indexers. I suspect this is where the problem is. As I said the data gets there but I am not getting the extraction and just a block of logs.

The sourcetype on both the extracted and unextracted show the same. For example, when viewing when extraction has worked, the sourcetype reads "WinEventLog". When viewing when the extraction has NOT worked the sourcetype reads "WinEventLog". So it appears the same. Is this a problem with cooked data (i.e. not raw)? I read the SPLUNK article about Heavy Forwarders and it states that "By default, forwarders send cooked data (universal forwarders send unparsed data and heavy forwarders send parsed data.)". So what do I do with my heavy forwarder so that the data being received by my clustered environment shows correctly?

0 Karma
1 Solution

willadams
Contributor

I think I have solved the issue. The problem was to do with my deployment app on the DMZ Heavy Forwarder including the "sourcetype" line. Once I removed this line and re-pointed my DMZ Heavy Forwarder to the new clustered indexers I found that now all of my data is being extracted as it should be.

View solution in original post

willadams
Contributor

I think I have solved the issue. The problem was to do with my deployment app on the DMZ Heavy Forwarder including the "sourcetype" line. Once I removed this line and re-pointed my DMZ Heavy Forwarder to the new clustered indexers I found that now all of my data is being extracted as it should be.

willadams
Contributor

I thought I would experiment and change the SPLUNK HF to send to the CM via 8089 and see what result I got. This resulted in the TCP output processor pausing the data flow and no more data being sent. I did notice a field called "ta_windows_action" that is showing failure. I changed back to the indexer cluster IPs and data started flowing again

** Incidentally my inputs.conf" file for my custom app in the DMZ is configured as:

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
whitelist1 = 4624 [and a bunch of other event ids]
index=windows
sourcetype=WinEventLog:Security
renderXml=true

[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
index=windows
sourcetype=XMLWinEventLog:Microsoft-Windows-Sysmon/Operational
renderXml=true

I reviewed my configuration for my non-DMZ and the configuration for the DMZ application and tried also changing the DMZ application to remove the "sourcetype" line so that it reads as follows:

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
whitelist1 = 4624 [and a bunch of other event ids]
index=windows
renderXml=false

[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
index=windows
renderXml=false

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...