Hi all, I have this situation:
I have installed splunk universal forwarder to forward the logs of Windows seven, to my splunk in ubuntu.
The universal forwarder works good, and use the port 8089, its config file input and outputs (set in c:\ ..system/local..) are these:
Input:
[default]
host = FROSSI-LT
[WinEventLog://Security]
disabled = 0
[WinEventLog://Application]
disabled = 0
[WinEventLog://System]
disabled = 0
[perfmon://FreeDiskSpace]
disabled = 0
[perfmon://Memory]
disabled = 0
[perfmon://LocalNetwork]
disabled = 0
[perfmon://CPUTime]
disabled = 0
Output:
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = 10.28.4.154:9997
[tcpout-server://10.28.4.154:9997]
Where 10.28.4.154 is the IP of my ubuntu VM, and 9997 is open and listen. Enable from the splunk platform.
The same conf file in ubuntu VM are these:
input:
[default]
host = ubuntu
[tcp://:8089]
connection_host = 10.28.4.143
source = tcp:8089
Where 10.28.4.143 is the IP address of my Windows7
output:
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = 10.28.4.143:9997
[tcpout-server://10.28.4.143.1:9997]
And, yeah, windows doesn't send data, and this is the error message:
"forwarding to indexer group default-autolb-group blocked for N seconds."
And later so much time, the message became:
"skipped indexing of internal audit event will keep dropping events until indexer congestion is remedied. Check disk space and other issues that may cause indexer to block"
I have tried to follow all your step, but "no results found"
I have tried to reinstall splunk and follow all your step, but "No results found"
I just walked through the process and these are my notes from doing so.
This is from a fresh install of Ubuntu 14.04 and a Windows 2008 R2 VM.
Splunk 6.2.1 was downloaded in a DEB package and installed to the Ubuntu VM with the command
sudo dpkg -i splunk-6.2.1-245427-linux-2.6-amd64.deb
Splunk was then started for the first time with
sudo /opt/splunk/bin/splunk start
Read though the license agreement and enter "y" when prompted.
Start up your web browser and connect to your Splunk instance at http://ip.address:8000
Enter "admin" and "changeme" for the username and password combination. Change it to something you know when prompted.
You should now be at the Splunk home screen.
Click on the Settings menu and then click "Forwarding and Receiving" from the list of options.
On the Forwarding and Receiving page, click "Configure Receiving". This will show the current receiver settings, if any, and allow you to create a new one.
Click the "New" button and enter "9997" to the "Listen on this port" box and then Save.
Splunk is now listening for forwarders on port 9997.
Nothing else needs to be done on the indexer/search head.
Validate that there are no events in Splunk by searching for “*” over all time.
Turn your attention to your forwarder. I am assuming the Spunk Universal Forwarder is already installed.
Create or Edit the inputs.conf file in $SPLUNK_HOME\etc\system\local. It should look like this:
[default]
host = windows_hostname
[WinEventLog::Application]
checkpointInterval = 5
current_only = 0
disabled = 0
evt_resolve_ad_obj = 1
start_from = oldest
[WinEventLog::Security]
checkpointInterval = 5
current_only = 0
disabled = 0
evt_resolve_ad_obj = 1
start_from = oldest
[WinEventLog::System]
checkpointInterval = 5
current_only = 0
disabled = 0
evt_resolve_ad_obj = 1
start_from = oldest
Create or Edit the outputs.conf file in $SPLUNK_HOME\etc\system\local. It should look like this:
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = 192.168.84.145:9997
[tcpout-server://192.168.84.145:9997]
Restart the Splunk Universal Forwarder.
Go back to the web interface on the Splunk indexer/search head and re-run your search. You should now see events from Windows.
Yeah I have done it!
But doesn't work in any case. For now I have not touch the config file in splunk, only the files in splunk forwarders
have you tried to verify connectivity from the windows VM to the ubuntu vm?
ping ubuntu_ip_address
if that works, then try connecting to port 9997 on the ubuntu vm with:
telnet ubuntu_ip_address 9997
Work : ( the port and the IP are ok
One thing I forgot to add.... the IP address in outputs.conf should be updated to the IP address of your Splunk indexer/search head.
I have no idea If this could help anyone, but I have solve simple enabling my websplunk service, but the forwarder have problem yet.
When you reinstalled splunk, did you remove the /opt/splunk directory first, or just re-install splunk on top of what was there?
if you re-installed on top of what was there, Splunk would have kept all the modifications you made to config files in the "local" directories.
Ok, I have tried, I have delete the outputs.conf and edit the inputs.conf, only in my ubuntu VM, but now, when I try to open SPLUNK, it doesn't start, the message on terminal is the same of the splunk forwarder in windows:
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
.
Stopping splunk helpers...
Done.
Splunk> The IT Search Engine.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _thefishbucket msad perfmon sos sos_summary_daily windows wineventlog winevents
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
And is splunk is not available at ubuntu:8000, What can I do?
First thing I see is that you should not need an outputs.conf on the indexer. The outputs.conf tells Splunk where to send it's logs. In this case you're telling your indexer to send the logs back to the windows 7 computer. So remove the outputs.conf on the Ubuntu VM.
Second, your inputs.conf on the Ubuntu should identify the protocol and port the indexer is listening on. for the case you've described, this should be sufficient for your inputs.conf.
[splunktcp:9997]
Make those changes on the Ubuntu server and restart Splunk on it.
Ok, I have tried, I have delete the outputs.conf and edit the inputs.conf, only in my ubuntu VM, but now, when I try to open SPLUNK, it doesn't start, the message on terminal is the same of the splunk forwarder in windows:
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
.
Stopping splunk helpers...
Done.
Splunk> The IT Search Engine.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _thefishbucket msad perfmon sos sos_summary_daily windows wineventlog winevents
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
And is splunk is not available at ubuntu:8000, What can I do?