After upgrading our Windows Splunk forwarder from Splunk 6.1.2 to 6.3.1, why are application and system logs not being sent?

[jhingley]

Hello

I upgraded to a 6.3.1 Splunk forwarder on a Windows 2012 server. Connectivity is fine and Security logs are coming through, but I can't see Application or System logs (I ensured all three boxes had been checked during the installation process) - I checked 'system>local>inputs.conf' and added the stanzas detailed on this site [WinEventLog://Application], etc, but no joy .

The previous version was 6.1.2 and all logs were coming through. I uninstalled the new version and put this back on - all logs were seen (I didn't need to change the inputs.conf file either) .

Have checked splunkd.log after restarting the service, but I can't see a message that details what I am doing wrong - all help appreciated!

btw, the Splunk Indexer and the web server are running OEL6, if this has any bearing. They are working correctly .

[jagould]

OK, I'm running Splunk Light on windows, using the Universal Forwarders, NOT using Deployment server setting, but the Index Forwarder setting.
So my problem was the fact that the Splunk Light be Default, only searches the Main and OS Indexes. Me being New to Splunk, this was very frustrating as i didn't know that.
Once you enable the Windows Addon (which you need) it creates a wineventlog index. All of the data from windows goes into those indexes (Application, System, Security only).
The only way to get this data to show, would be to search using "index=* host=bla" to see the data i was expecting to see.

I created a "authorize.conf" under Splunk/etc/system/local/ and added this to the file:

[role_admin]
srchIndexesDefault = *

restarted Splunk Light.

Now everything shows up by default, and all my hosts show up with the correct information.

Not sure if this is your issue, but i wanted to share mine since they were "similar"

[jagould]

So i have come to realise that splunk is horrible software.
The only way i can get it to work, is if i setup the Forwarders to be Deployment Clients, and then "Add data" from a Forwarder and select the client that's a deployment client.
Problem is, Once i a create a server class, the only way i can add to it is using the deployment*.conf on the server, which defeats the entire purpose of splunk IMO.
I want everything to be fully automated (i've got scripted install of the Forwarder via WSUS using LocalUpdatePublisher) and when installed i just want the data to be send to the Indexer.

[jhingley]

I have rolled back to 6.1.8 . Its working correctly with all three logs coming through . Trouble is we need the 'fixed' feature of data integrity (hashing of logs) for compliance that is in 6.3 .

[jagould]

I know what you mean.
I tried 6.2.7 and it doesn't work there either for me.

[jhingley]

Linux works seamlessly - have 6.3 running on all OEL boxes no problem . Out of interest are you running the Indexer on Windows or Linux ?

[jagould]

its running on windows.
I think i just realized my issue, and i'm slightly embarrassed because it seems to simple.
I'm running Splunk Light for my indexer on windows, when you enable the addon for Windows (which you need to index windows logs) it creates a Separate wineventlog index. Apparently the ONLY way to get the Search to show this index is to manually specify it in the search "index=* host=xx" and then i can see all my source types.

It's probably been working the entire time. Now i need to find out how to include that index by default.

[jhingley]

I tried 'index=_internal host=xxxxxx' but can only see 'metrics.log' and 'splunkd.log' in the source .

[jagould]

So i have come to realise that splunk is horrible software.
The only way i can get it to work, is if i setup the Forwarders to be Deployment Clients, and then "Add data" from a Forwarder and select the client that's a deployment client.
Problem is, Once i a create a server class, the only way i can add to it is using the deployment*.conf on the server, which defeats the entire purpose of splunk IMO.
I want everything to be fully automated (i've got scripted install of the Forwarder via WSUS using LocalUpdatePublisher) and when installed i just want the data to be send to the Indexer.

[jagould]

OK, I'm running Splunk Light on windows, using the Universal Forwarders, NOT using Deployment server setting, but the Index Forwarder setting.
So my problem was the fact that the Splunk Light be Default, only searches the Main and OS Indexes. Me being New to Splunk, this was very frustrating as i didn't know that.
Once you enable the Windows Addon (which you need) it creates a wineventlog index. All of the data from windows goes into those indexes (Application, System, Security only).
The only way to get this data to show, would be to search using "index=* host=bla" to see the data i was expecting to see.

I created a "authorize.conf" under Splunk/etc/system/local/ and added this to the file:

[role_admin]
srchIndexesDefault = *

restarted Splunk Light.

Now everything shows up by default, and all my hosts show up with the correct information.

Not sure if this is your issue, but i wanted to share mine since they were "similar"

[jhingley]

Many thanks for that - will give it a go and report back .

[jagould]

I'm having the same exact problem, except worse.
This is a brand new install and only the Windows Setup event logs are being forwarded.

[jhingley]

Glad its not just me! I am running W2K12 R1 - do you have the same setup ?

[jagould]

i'm running 2k8R2

[jhingley]

Ok - I have just installed the forwarder on our Domain Controller (W2K12 r1) , it sends 'system' and 'application' but not 'security ' - even though I selected the same settings as the previous server . There is plenty of space on the indexer , and again I cant find a specific error message within 'splunkd.log' to pinpoint why certain event logs are not being sent .