Getting Data In

Why are Windows Event Logs not forwarded after installing a new Windows server?

krusty
Contributor

Hi there,

I have the following issue detected in our environment and I'm not sure where the problem comes from.
We have several Windows Server monitored with a heavy forwarder. The Event logs are grabbed remotely by WMI.
So far everything works as expected.

Now we have done a new installation of one Windows Server.
The Server has the same name and IP address. Only the OS has changed from Windows Server 2008 R2 to Windows Server 2012 R2.
If I do a wbemtest with the user on the Splunk heavy forwarder, the Splunk service is running, and I can see the events from the fresh installed server. So there are no permission or firewall issues between the forwarder and the Windows Server.
But I can't see any events from this server on the indexer.

Does someone has an idea what is going wrong or how I can figure out the problem?

For your information. I removed the configuration of the Windows Server on the forwarder, restarted the forwarder and add the Windows Server again and restarted the forwarder. Nothing happens.
I removed the index of the Windows Server from the indexer, restart the indexer and added the index again. Nothing happens.

Could it be possible that the Splunk forwarder stores Information of grabbed events in another file?

For any ideas I'll be very thankful.

[edit: it's a heavy forwarder not a universal one]

0 Karma
1 Solution

dvwijk
Explorer

Hi. Splunk keeps track of which eventid it has indexed for each server. You should search for fish buckets en resetting these.

So it will start working when it reaches the number it last indexed on the old server 🙂

View solution in original post

0 Karma

dvwijk
Explorer

Hi. Splunk keeps track of which eventid it has indexed for each server. You should search for fish buckets en resetting these.

So it will start working when it reaches the number it last indexed on the old server 🙂

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi krusty,
to better understand you situation, you should make some test, to understand if your Indexers receive logs from your new Forwarder or not:

On Indexers:
- run the following search index=_internal host=your_hostname and index=_internal host=your_hostname*

On forwarder:
- verify indexers name (or IP address) in $SPLUNK_HOME\etc\system\local\outputs.conf
- verify hostname in $SPLUNK_HOME\etc\system\local\server.conf
- verify hostname in $SPLUNK_HOME\etc\system\local\inputs.conf
- by BLI telnet indexers_IP 9997
If eventually for Indexers you use username, try with IP address.

If all the checks are OK run on forwarder (from CLI) $SPLUNK_HOME\bin\splunk cmd btool props list --debug > ppp.txt and verify in this file the correct addressing to Indexers.

If you don't receive logs in _internal there is a problem in communication between Forwarder and Indexers.
If you have logs in _internal but not in wineventlog (or a different index you use), modify on forwarder inputs.conf in $SPLUNK_HOME\etc\apps\TA_Windows inserting in one stanza crcSalt = <SOUCE> (literally, including the angle brackets).
In this way you reload all events (see https://docs.splunk.com/Documentation/Splunk/6.5.2/Admin/Inputsconf).

Let me know.

Bye.
Giuseppe

0 Karma

krusty
Contributor

Hi Giuseppe,

thanks for your answer.
I have to clarify something first. I monitor the remote eventlog with wmi, so I have on the forwarder all Windows Servers configured in the wmi.conf file.
Example:
[WMI:win2k12r2-vm]
disabled = 0
event_log_file = Application, Security, System, HardwareEvents, Internet Explorer, Key Management Service, Windows PowerShell
index = win2k12r2-vm
interval = 5
server = win2k12r2-vm

You assume that the forwarder is the new Installation, but this isn't true. The forwarder runs over more than 2 years without any problem. It is only the monitored Windows Server which was new installed and now I cannot get any events into the indexer.

For me it is importand to find out if any events will be send to the indexer or if the forwarder has the issue with receiving the data via wmi from the new server.

Kind regards,
Thomas

0 Karma

woodcock
Esteemed Legend

Do you have a really, REALLY good reason for going with WMI? There is a reason that Splunk created the WinEventLog facility (actually probably hundreds). I would NEVER use WMI direct to get data into Splunk.

0 Karma

krusty
Contributor

Hmm, a really good reason to do so? I think not really.
We start with our splunk environment many years ago. If I'm right at the beginning there were no other choice as monitor the Windows events by using WMI or by installing a light forwarder on each server.
We didn't like to install another agent on the server so we decided to go with a heavy forwarder and the WMI (remote event log grabbing) solution.

I do my very best to change the situation now, because I read a lot in the forum about the disadvantage of using WMI and the advantage of using UF instead.

Do you have on each Windows server a UF installed where you want to grab the Event logs?

0 Karma

gcusello
SplunkTrust
SplunkTrust

OK it's really different!
I never used WMI for remote monitoring, always Forwarders.
Sorry I cannot help you!
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...