I installed the universal forwarder 4.2.5 on my remote Linux machine and set it to monitor my squid access logs.
After installing it, I run the following commands to have the data sent to my Splunk server:
/opt/splunkforwarder/bin/splunk add forward-server 192.168.2.2:9997
/opt/splunkforwarder/bin/splunk monitor /var/log/squid/access.log -sourcetype squid
Immediately after issuing the last command which monitors the access.log file, the events start flowing into the server and I can view them on the Squid App and Search app. Everything works fine.
The problem starts when I restart Splunk. After issuing the ./splunk restart command, Splunk starts up okay, but the logs are no longer forwarded to the server. I have not seen any errors. I am not sure why it stops sending logs to the server after restarting it.
After issuing "splunk monitor /var/log/squid/access.log -sourcetype squid" command, the following is appended to the inputs.conf file in the "/opt/splunkforwarder/etc/apps/search/local" directory:
disabled = false
sourcetype = squid
And after the "splunk add forward-server 192.168.2.2:9997" command, the following gets appended to the outputs.conf file in the "/opt/splunkforwarder/etc/system/local" directory:
defaultGroup = 192.168.2.2_9997
disabled = false
server = 192.168.2.2:9997
So I believe that what you are talking about is being done when I give splunk the commands mentioned above. I am not sure what other inputs.conf or outputs.conf are there for me to put the info in.
Even after a restart, if I issue the add monitor command again, it won't let me because it says the file is already being monitored. So there must be a file somewhere that knows I already issued the add monitor command. Even after I delete the info in the inputs.conf file and reissue the add monitor command, it still won't let me because it thinks I am already monitoring it.
OK, yes I see that those commands do add the correct stanza to the conf files. Run a splunk list forward-server after you restart splunk on the universal forwarder. And let us know the output.