Forwarding windows event log data across domains with free license

[asmercer2004]

My company purchased an enterprise license and we got it working on one domain. We want to consolidate logs from another domain onto a machine in the first domain. I tried to pull the logs directly but kept getting WMI errors, so I thought I'd try forwarding the event logs. I put a free license on one machine in the 2nd domain as a test. I set up the data input and set it to forward to port 60000 (this is what the receiver was set to). I never get any data on the receiver. I get a message on the forwarder that says "Search scheduler is disabled in free version." What can I do in this situation. Am I able to duplicate the enterprise license that we purchased on the other machines? Would I only need 1 forwarder or a forwarder on each machine in the 2nd domain?

[ftk]

Lowell is correct about using the forwarder license for the forwarders. It is essentially a enterprise license that allows only a minuscule amount of indexing -- but it does allow you to forward as much data to another splunk instance as you like.

I recommend installing a forwarder, preferably a light-weight forwarder, on each of your 2nd domain servers and have them forward to your enterprise licensed indexer in the 1st domain.

Alternatively you could install just a single forwarder in the 2nd domain, and have it aggregate all 2nd domain logs via WMI and then forwarded it to the 1st domain. However, as Lowell mentioned, WMI is generally not recommended as it is built more as a "best effort" service and is not as reliable as splunk forwarders.

Yet a third option would be to establish a trust relationship between your domains so you can pull logs from domain 2 from your indexer in domain 1 via WMI, but again, WMI is generally not recommended.

In regards to not receiving any data on the receiver, make sure that you open the appropriate ports in your firewalls (60000 in your case). Also keep in mind that you can enable SSL tunnels from the forwarders to the receivers, this can appease most firewall admins.

[ftk]

When searching on your main indexer, can you add | dedup at the end of your search and see if the number of results drops to the level you see on the forwarder? Also, when manually reviewing the results on your indexer, do you actually see duplicate events? Splunk should not be sending dupes unless you have duplicate monitors (for example a monitor and a WMI monitor for the same event log source).

[asmercer2004]

Additionally, I can see recent logs in the indexer (for example I can see where I restarted the splunk service after copying the license). I can't see those on the forwarder machine.

[asmercer2004]

yes I have 160000 events that claim to come from the host that has the forwarder installed on it. The indexer is monitoring other computers on its domain with WMI.

[ftk]

Do you mean that you have 160000 events that claim to be from that forwarder or 160000 total? Are you monitoring any other sources on this indexer?

[asmercer2004]

Ok after running the forwarder, I am noticing that on the box itself there are 24000 event log entries if I search. On the receiver I have over 160000. Could I may be forwarding multiple copies of the event logs?

[asmercer2004]

Thank you both for your responses. I switched over to the forwarding license and I also upgraded the enterprise licensed one to the most recent version. I also had to change the port, since admin policy blocks most application ports. It seems to be working now. I think that I will be installing the forwarders on the other machines on 2nd domain.

[Lowell]

No simply setup the forwarder license on your forwarding splunk instance(s). In the "etc" folder, simply replace the license file:

move splunk-forwarder.license splunk.license

Oh, and I recommend that you don't use the WMI-based event log collection if at all possible. The WinEventLog inputs using a splunk forwarder is much more reliable.