Universal Forwarder Installation

[knewter]

Quick question, I'm still getting my feet wet with Splunk but I was wondering how long does it typically take to receive data after installing a universal forwarder? Does it depend on the how much or type of data?

I'm trying to setup a test environment for testing the Splunk for AD app. I have a Win 2008 R2 Box setup as a domain controller forwarding to a Splunk 5.0 instance on linux. I'm testing what happens when I modify the inputs.conf file. From what I'm seeing after I modify the inputs.conf file and restart the service it can take hours before any data is received. Is that normal? I've tried adjusting the input.conf files so I only get the new data but nothing seems to happen.

I have done a default install of the Splunk for AD app in my Windows system. Installed the universal forwarder without selecting any inputs. After it was installed I stopped the service and copied the Windows, DNS and Domain Controller TA to the apps folder. Below is a sample output from the btool of some of the inputs that were enabled. The documentation stated if you didn't need to update the indexes then the default would work. That's what should be displayed below.

Sample inputs.conf:

C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DomainController-NT6\default\inputs.conf [WinEventLog:Key Management Service]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf _rcvbuf = 1572864
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf counters = *
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DomainController-NT6\default\inputs.conf disabled = 0
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dc_name =
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dns_name =
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_resolve_ad_obj = 0
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf host = ********
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DomainController-NT6\default\inputs.conf index = winevents
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf instances = *
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf interval = 10
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf object = Processor
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DomainController-NT6\default\inputs.conf queue = parsingQueue
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DomainController-NT6\default\inputs.conf sourcetype = "WinEventLog:Key Management Service"
C:\Program Files\SplunkUniversalForwarder\etc\apps\MSICreated\local\inputs.conf [WinEventLog:Security]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf _rcvbuf = 1572864
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf checkpointInterval = 5
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf counters = *
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf current_only = 0
C:\Program Files\SplunkUniversalForwarder\etc\apps\MSICreated\local\inputs.conf disabled = 0
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dc_name =
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dns_name =
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_resolve_ad_obj = 1
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf host = ********
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf index = perfmon
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf instances = *
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf interval = 10
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf object = Processor
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf start_from = oldest
C:\Program Files\SplunkUniversalForwarder\etc\apps\MSICreated\local\inputs.conf [WinEventLog:Setup]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf _rcvbuf = 1572864
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf checkpointInterval = 5
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf counters = *
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf current_only = 0
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf disabled = 1
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dc_name =
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dns_name =
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_resolve_ad_obj = 0
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf host = ********
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf index = perfmon
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf instances = *
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf interval = 10
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf object = Processor
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf start_from = oldest
C:\Program Files\SplunkUniversalForwarder\etc\apps\MSICreated\local\inputs.conf [WinEventLog:System]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf _rcvbuf = 1572864
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf checkpointInterval = 5
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf counters = *
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf current_only = 0
C:\Program Files\SplunkUniversalForwarder\etc\apps\MSICreated\local\inputs.conf disabled = 0
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dc_name =
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf evt_dns_name =
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_resolve_ad_obj = 0
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf host = ********
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf index = perfmon
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf instances = *
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf interval = 10
C:\Program Files\SplunkUniversalForwarder\etc\apps\TA-DNSServer-NT6\default\inputs.conf object = Processor
C:\Program Files\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf start_from = oldest

[kristian_kolb]

Assuming that data is actually being generated where the forwarder is installed, it should in general be coming fairly quickly. If nothing has happened within a minute or so, it's time to start troubleshooting.

First, Second and Third:

How do you KNOW you are NOT getting data? Might seem silly, but many times data is actually coming in, though perhaps not where you'd want it to. So,
- Sniff the network
- Ensure firewalls are open (local too)
- Correct ports on both ends
- Does the data end up in an index for which you do not have permissions, or one that is not searched by default?
- Are timestamps parsed correctly. If not, you'll maybe not see them unless you search over 'All time'
- Have you made changes that require a restart? Are the cables connected?

Fourth:

Take a look at what the forwarder is doing.
- check the splunkd.log on the forwarder
- you can also connect to the splunkd-port to see what the TailingProcessor (reads files) is up to: https://your_forwarder:8089/services/admin/inputstatus/TailingProcessor:FileStatus

Also, please edit your question to also include your inputs.conf (forwarder and indexer) and outputs.conf.

Hope this helps,

Kristian

[kristian_kolb]

Sorry, your edited post makes me realize that i need to look further into the AD-app.

However, if this is the first time you installed this on an AD server, it could take some time before all the historical data is in. Does the deployment monitor show that data is coming in? In that case, just wait a little.

Check for error messages in the splunkd.log on the indexer.

/K

[knewter]

Thanks for your quick response.

I've updated my original question with a sample of my inputs.conf from btool. In looking at several of the stanza's the sourcetype doesn't appear to be set. Is that normal?

I also looked at my Splunk Deployment app and it does see the forwarder and it says that it's active so it must be communicating something. I'm wondering if there is an error in the sourcetype?