topic Re: Universal Forwarder doesn't forward specific log in Getting Data In

Universal Forwarder doesn't forward specific log

misterduke — Tue, 07 Jul 2020 10:08:30 GMT

Hello!

It's my first time writing here so forgive me if my question may lack information.

What I want to do:

I want to execute a batch file via scripted input and write the output of this script into a specific log. then I want to send this log to be indexed in another server. all of this should later on be deployed within an app to a universal forwarder which executes the script, writes the log and sends it to a specific server into a specific index.

What I've done so far:

I've created an app which as a script in /bin that is basically changing the passwd of the universal forwarder and creating a log in which it echos certain statements. the script itself looks like this:

#!bin/sh FILE=/opt/splunkforwarder/etc/passwd if test -f "$FILE"; then echo $(date) " $FILE existiert." >> /opt/splunkforwarder/etc/apps/myapp/logging/changepw.log #mv /opt/splunkforwarder/etc/apps/myapp/local/inputs.conf /opt/splunkforwarder/etc/apps/myapp/local/inputs.conf.bak mv /opt/splunkforwarder/etc/passwd /opt/splunkforwarder/etc/passwd.bak echo $(date) " $FILE wurde umbenannt und wird neu erstellt.Inputs.conf wurde deaktiviert" >> /opt/splunkforwarder/etc/apps/myapp/logging/changepw.log /opt/splunkforwarder/bin/splunk restart else echo $(date) " $FILE existiert nicht." >> /opt/splunkforwarder/etc/apps/myapp/logging/changepw.log fi

so as of now it should do the following:
- check if there is a passwd

- if yes, rename it to passwd.bak , renaming my inputs.conf to inputs.conf.bak (so it uses the inputs.conf in default which has a deactivated scripted input) and then restart splunk. after each previous step it writes a message into changepw.log

the inputs.conf looks like this:

[script://./bin/change.sh] disabled = 0 interval= -1 [monitor:///opt/splunkforwarder/etc/apps/myapp/logging/*] disabled = 0 index = main

my outputs.conf looks like this:

[tcpout] defaultGroup = splunk_indexer [tcpout-server://<ip>] [tcpout:splunk_indexer] disabled = false server = <ip>:9997

what the problem is:

when I start the script it does at it was told. changing the passwd and renaming it to passwd.bak and writing all echos into a changepw.log then restarting splunk. for whatever reason it doesn't seem to send anything to my server.

I've already checked whether my forwarder is active. it is
I can ping the server from the UF
I've created a test.log in the same folder in which my changepw.log resides and filled it with some text. after a few moments it appeared on my server, indexed.
splunk is starting with user splunk and has all the necessary rights to execute, read and write anything within /splunkforwarder

did I leave somthing out? I feel like I'm standing right in front of a wall. hope someone can help!

edit: I've noticed that, when I deactivate the script in my inputs.conf and comment out the mv inputs.conf inputs.conf.bak part and start the change.sh, then it works just fine and my server shows the log. why can that be? I assume that, when I mv the inputs.conf the script ends even tho it already started. can that be? if so, the final question would be how does the script need to look like in order to do the following:

- check if there is a passwd, if so change it to passwd.bak , write everything in a log and restart splunk. after restarting splunk should not start the script again.

Re: Universal Forwarder doesn't forward specific log

renjith_nair — Tue, 07 Jul 2020 11:12:36 GMT

@misterduke,

It's possibly due to the renaming of inputs.conf. Once you rename the inputs.conf and restart splunk , splunk does not know about monitor configurations and it just ignores the files.

For your use case, instead of removing the inputs.conf which has monitor stanza :

Either you can split the inputs conf and keep the monitor stanza in a permanent conf in separate app context or local inputs

Use/create a separate dummy "lock" file as soon as you perform the necessary actions. Then include a condition in your script to check for the presence of this file before performing the action, ie. no action if the file is present.

It all depends on how often you need to perform the passwd file change.

Re: Universal Forwarder doesn't forward specific log

misterduke — Tue, 07 Jul 2020 15:22:42 GMT

okay I've split the inputs.conf. one with the monitor stanza in default and one with the scripted input in local.

the script does mv the inputs.conf in local into backupinput after all other loops are good.

the thing I've noticed is:

my script runs fine and changes the passwd just as I want and writes the echo in my log file. still, my splunk doesn't get any new input. BUT when I manually vim into the log file and write down "test test" for instance - immediately it appears in my splunk.

Re: Universal Forwarder doesn't forward specific log

misterduke — Wed, 08 Jul 2020 10:44:15 GMT

Now I've drilled down the problem:

- in my scripted input, the bash script has to , after all changes, restart splunk

- the restart doesn't work because it also kills the process the script needs to function. so it stops splunk, therefore kills the processes and that's it

Question: how can I restart splunk after my script did all the changes?

Re: Universal Forwarder doesn't forward specific log

misterduke — Wed, 08 Jul 2020 14:44:03 GMT

Ok, I solved the Problem.

In order to restart Splunk via scripted Input I need to use this in my script:

/opt/splunkforwarder/bin/splunk restart > /dev/null 2>&1

without /dev/null 2>&1 it results in stopping splunk but not starting again. Therefore not forwarding my logs