Hello!
It's my first time writing here so forgive me if my question may lack information.
What I want to do:
I want to execute a batch file via scripted input and write the output of this script into a specific log. then I want to send this log to be indexed in another server. all of this should later on be deployed within an app to a universal forwarder which executes the script, writes the log and sends it to a specific server into a specific index.
What I've done so far:
I've created an app which as a script in /bin that is basically changing the passwd of the universal forwarder and creating a log in which it echos certain statements. the script itself looks like this:
#!bin/sh
FILE=/opt/splunkforwarder/etc/passwd
if test -f "$FILE"; then
echo $(date) " $FILE existiert." >> /opt/splunkforwarder/etc/apps/myapp/logging/changepw.log
#mv /opt/splunkforwarder/etc/apps/myapp/local/inputs.conf /opt/splunkforwarder/etc/apps/myapp/local/inputs.conf.bak
mv /opt/splunkforwarder/etc/passwd /opt/splunkforwarder/etc/passwd.bak
echo $(date) " $FILE wurde umbenannt und wird neu erstellt.Inputs.conf wurde deaktiviert" >> /opt/splunkforwarder/etc/apps/myapp/logging/changepw.log
/opt/splunkforwarder/bin/splunk restart
else
echo $(date) " $FILE existiert nicht." >> /opt/splunkforwarder/etc/apps/myapp/logging/changepw.log
fi
so as of now it should do the following:
- check if there is a passwd
- if yes, rename it to passwd.bak , renaming my inputs.conf to inputs.conf.bak (so it uses the inputs.conf in default which has a deactivated scripted input) and then restart splunk. after each previous step it writes a message into changepw.log
the inputs.conf looks like this:
[script://./bin/change.sh]
disabled = 0
interval= -1
[monitor:///opt/splunkforwarder/etc/apps/myapp/logging/*]
disabled = 0
index = main
my outputs.conf looks like this:
[tcpout]
defaultGroup = splunk_indexer
[tcpout-server://<ip>]
[tcpout:splunk_indexer]
disabled = false
server = <ip>:9997
what the problem is:
when I start the script it does at it was told. changing the passwd and renaming it to passwd.bak and writing all echos into a changepw.log then restarting splunk. for whatever reason it doesn't seem to send anything to my server.
did I leave somthing out? I feel like I'm standing right in front of a wall. hope someone can help!
edit: I've noticed that, when I deactivate the script in my inputs.conf and comment out the mv inputs.conf inputs.conf.bak part and start the change.sh, then it works just fine and my server shows the log. why can that be? I assume that, when I mv the inputs.conf the script ends even tho it already started. can that be? if so, the final question would be how does the script need to look like in order to do the following:
- check if there is a passwd, if so change it to passwd.bak , write everything in a log and restart splunk. after restarting splunk should not start the script again.
Ok, I solved the Problem.
In order to restart Splunk via scripted Input I need to use this in my script:
/opt/splunkforwarder/bin/splunk restart > /dev/null 2>&1
without /dev/null 2>&1 it results in stopping splunk but not starting again. Therefore not forwarding my logs
It's possibly due to the renaming of inputs.conf. Once you rename the inputs.conf and restart splunk , splunk does not know about monitor configurations and it just ignores the files.
For your use case, instead of removing the inputs.conf which has monitor stanza :
Either you can split the inputs conf and keep the monitor stanza in a permanent conf in separate app context or local inputs
Or
Use/create a separate dummy "lock" file as soon as you perform the necessary actions. Then include a condition in your script to check for the presence of this file before performing the action, ie. no action if the file is present.
It all depends on how often you need to perform the passwd file change.
Now I've drilled down the problem:
- in my scripted input, the bash script has to , after all changes, restart splunk
- the restart doesn't work because it also kills the process the script needs to function. so it stops splunk, therefore kills the processes and that's it
Question: how can I restart splunk after my script did all the changes?
Ok, I solved the Problem.
In order to restart Splunk via scripted Input I need to use this in my script:
/opt/splunkforwarder/bin/splunk restart > /dev/null 2>&1
without /dev/null 2>&1 it results in stopping splunk but not starting again. Therefore not forwarding my logs
okay I've split the inputs.conf. one with the monitor stanza in default and one with the scripted input in local.
the script does mv the inputs.conf in local into backupinput after all other loops are good.
the thing I've noticed is:
my script runs fine and changes the passwd just as I want and writes the echo in my log file. still, my splunk doesn't get any new input. BUT when I manually vim into the log file and write down "test test" for instance - immediately it appears in my splunk.