Why is Windows TA not working with Deployment App?

[ChristianF]

In my previous post I was advised to deploy Windows TA via Deployment Server which I did and the app is installed on the servers I want. However the issue is I deployed the app and there is no information being forwarded to the server with Windows events.

Both client and server are able to communicate with one another and the default port for Splunk is open on 9997/tcp. I have edited the inputs.conf file in both the app and the actual SplunkForwarder local folder. I have set the various logs I want in the inputs file to disabled = 0 and still no data comes through to my indexes.

[gcusello]

Hi @ChristianF,

the best debugging approach is the one described by @PickleRick .

In addition I hint to check if you're receiving the Splunk internal logs from that server.

If not, check the outputs.conf, if yes, you can perform the following checks:

• check the hostname of your server is correct: in $SPLUNK_HOME\etc\sytstem\local\server.conf
• check if you're receiving logs in a different index.
• enlarge the time search to understand if you're receiving logs with a wrong timestamp.

The usual issue is the first.

Ciao.

Giuseppe

[ChristianF]

Hey Gisueppe, I have my UF pointed to 10.1.70.24:9997 which is the correct IP for the first of my indexers in that cluster. I checked for the past week and checked my data summary in the Search App. I have the logs pointed to the main index for now. But no data in any of those three locations.

[gcusello]

Hi @ChristianF,

at first, if you have a cluster, it's better to point to all the Indexers in autoloadbalancing or (better) to use Indexer Discovery (https://docs.splunk.com/Documentation/Splunk/9.1.0/Indexer/indexerdiscovery) instead to point to uno Indexer.

Then, it isn't a best practice to use the main index, but it's better to crete your own index for your data, without exceeding in the number of Indexes.

Have you Splunk internal logs from that server?

if not, there's a connection issue, if yes, we can concentrate debug on the Windows Add-On.

Another very stupid question: did you restarted the Forwarder after conf files update?

In other words, did you configured (in the ServerClass) the restart after update option for your apps?

Ciao.

Giuseppe

[ChristianF]

Hey Giuseppe, thank you for the documentation on the load balancers, i do have my forwarders set to my manager node which is 10.1.70.27:9997 and from there I believe looking at the documentation will load balance to the other two indexers I have.

I do have other indexes created but I'm primarily concerned with just getting the data in the first place and then separate from there seems like the best idea currently to me. Haha there are no stupid questions where I'm concerned, I'm convinced I'm missing something simple.

But to answer your question, yes I do have the deployment apps set to restart the forwarder after install. I am receiving internal logs from my forwarders. 

[PickleRick]

OK. You can verify if your UF is properly pushing events to the whole cluster by doing

index=_internal host=your_UF_hostname | stats count by splunk_server

for the last 24h or so.

Also, if you're receiving the UF's internal events, you can do

index=_internal host=your_UF_name source=*metrics.log group IN (per_source_thruput, per_sourcetype_thruput) series=wineventlog*

And see if your metrics show any logs ingested or if they are at constant zero.

You can also search the metrics.log for per_index_thruput and see if there are events pushed into your destination index (main in your case).

If they are, and your source is generating the events in a more or less constant stream, you can run a real-time(All time) search (that's the only practical use case I have for the real-time searches; don't use them otherwise :-)) for

index=main host=your_UF_hostname

You should see your events as they come. Pay attention especially to the timestamps (typical problem - misconfgured timezone on the forwarder).

[gcusello]

Hi @PickleRick,

if you're pointing to the Master Node, you enabled Indexers Discovery, as correct.

If you're receiving Splunk Internal Logs and you enabled the restart option in the ServerClass, the issue is in the TA_Windows.

What user are you using to run Splunk? does it have the grants to access wineventlog?

Ciao.

Giuseppe

[PickleRick]

1. Check if you're getting the logs from the UF itself (check your _internal index)

2. Check your effective inputs configuration on the UF

splunk btool inputs list --debug

You described it a bit vaguely so we have no idea what and where exactly did you enable and if those settings were actually deployed or not.

[ChristianF]

Hey PickleRick, my UF is sending _internal logs to my indexers and my effective inputs are all the ones I defined in my inputs.conf file that I deployed with the app.

C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf disabled = 0
host = $decideOnStartup
index = default
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf interval = 60
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf sourcetype = MSAD:NT6:DN
S
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf [SSL]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf _rcvbuf = 1572864
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf allowSslRenegotiation =
true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf certLogMaxCacheEntries =
10000
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf certLogRepeatFrequency =
1d
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf cipherSuite = ECDHE-ECDS
A-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AE
S256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf ecdhCurves = prime256v1,
secp384r1, secp521r1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_dc_name =
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_dns_name =
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_resolve_ad_obj = 0
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf host = $decideOnStartup
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf index = default
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf logCertificateData = tru
e
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf sslQuietShutdown = false

C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf sslVersions = tls1.2
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf [WinEventLog://Applicati
on]
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf checkpointInterval = 5
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf current_only = 0
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf disabled = 0
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_dc_name =
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_dns_name =
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf evt_resolve_ad_obj = 0
host = $decideOnStartup
index = default
C:\Program Files\SplunkUniversalForwarder\etc\system\default\inputs.conf interval = 60
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf renderXml = true
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf start_from = oldest
C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf [WinEventLog://DFS Repli
cation]

[Simple_Search]

You are expecting all of your data to be going to an index named default based on the inputs.conf on the local system, should this be like wineventlog or something else?