Getting Data In

Universal Forwarder doesn't forward specific log

misterduke
Explorer

Hello!

It's my first time writing here so forgive me if my question may lack information.

What I want to do: 

I want to execute a batch file via scripted input and write the output of this script into a specific log. then I want to send this log to be indexed in another server. all of this should later on be deployed within an app to a universal forwarder which executes the script, writes the log and sends it to a specific server into a specific index. 

 

What I've done so far:

I've created an app which as a script in /bin that is basically changing the passwd of the universal forwarder and creating a log in which it echos certain statements. the script itself looks like this: 

 

 

#!bin/sh

FILE=/opt/splunkforwarder/etc/passwd


if test -f "$FILE"; then
     echo $(date) " $FILE existiert." >> /opt/splunkforwarder/etc/apps/myapp/logging/changepw.log

    #mv /opt/splunkforwarder/etc/apps/myapp/local/inputs.conf /opt/splunkforwarder/etc/apps/myapp/local/inputs.conf.bak
    mv /opt/splunkforwarder/etc/passwd /opt/splunkforwarder/etc/passwd.bak

    echo $(date) " $FILE wurde umbenannt und wird neu erstellt.Inputs.conf wurde deaktiviert" >> /opt/splunkforwarder/etc/apps/myapp/logging/changepw.log


    /opt/splunkforwarder/bin/splunk restart
else
    echo $(date) " $FILE existiert nicht." >> /opt/splunkforwarder/etc/apps/myapp/logging/changepw.log
fi

 

 

 

so as of now it should do the following:
- check if there is a passwd

- if yes, rename it to passwd.bak , renaming my inputs.conf to inputs.conf.bak (so it uses the inputs.conf in default which has a deactivated scripted input) and then restart splunk. after each previous step it writes a message into changepw.log 

 

the inputs.conf looks like this: 

 

 

[script://./bin/change.sh]
disabled = 0
interval= -1

[monitor:///opt/splunkforwarder/etc/apps/myapp/logging/*]
disabled = 0
index = main

 

 

 

 

my outputs.conf looks like this:

 

 

[tcpout]
defaultGroup = splunk_indexer

[tcpout-server://<ip>]

[tcpout:splunk_indexer]
disabled = false
server = <ip>:9997

 

 

 

 

what the problem is:

when I start the script it does at it was told. changing the passwd and renaming it to passwd.bak and writing all echos into a changepw.log then restarting splunk. for whatever reason it doesn't seem to send anything to my server.

  • I've already checked whether my forwarder is active. it is 
  • I can ping the server from the UF
  • I've created a test.log in the same folder in which my changepw.log resides and filled it with some text. after a few moments it appeared on my server, indexed. 
  • splunk is starting with user splunk and has all the necessary rights to execute, read and write anything within /splunkforwarder

 

did I leave somthing out? I feel like I'm standing right in front of a wall. hope someone can help!

edit: I've noticed that, when I deactivate the script in my inputs.conf and comment out the mv inputs.conf inputs.conf.bak part and start the change.sh, then it works just fine and my server shows the log. why can that be? I assume that, when I mv the inputs.conf the script ends even tho it already started. can that be? if so, the final question would be how does the script need to look like in order to do the following:

- check if there is a passwd, if so change it to passwd.bak , write everything in a log and restart splunk. after restarting splunk should not start the script again.

 

Labels (2)
0 Karma
1 Solution

misterduke
Explorer

Ok, I solved the Problem. 

 

In order to restart Splunk via scripted Input I need to use this in my script: 

 

 

/opt/splunkforwarder/bin/splunk restart > /dev/null 2>&1

 

 

without /dev/null 2>&1 it results in stopping splunk but not starting again.  Therefore not forwarding my logs

View solution in original post

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

@misterduke,

It's possibly due to the renaming of inputs.conf. Once you rename the inputs.conf and restart splunk , splunk does not know about monitor configurations and it just ignores the files.

For your use case, instead of removing the inputs.conf which has monitor stanza : 

Either you can split the inputs conf and keep the monitor stanza in a permanent conf in separate app context or local inputs

Or

Use/create a separate dummy "lock" file as soon as you perform the necessary actions. Then include a condition in your script to check for the presence of this file before performing the action, ie. no action if the file is present.

It all depends on how often you need to perform the passwd file change.

Happy Splunking!
0 Karma

misterduke
Explorer

Now I've drilled down the problem:

- in my scripted input, the bash script has to , after all changes, restart splunk

- the restart doesn't work because it also kills the process the script needs to function. so it stops splunk, therefore kills the processes and that's it

 

Question: how can I restart splunk after my script did all the changes?

0 Karma

misterduke
Explorer

Ok, I solved the Problem. 

 

In order to restart Splunk via scripted Input I need to use this in my script: 

 

 

/opt/splunkforwarder/bin/splunk restart > /dev/null 2>&1

 

 

without /dev/null 2>&1 it results in stopping splunk but not starting again.  Therefore not forwarding my logs

0 Karma

misterduke
Explorer

okay I've split the inputs.conf. one with the monitor stanza in default and one with the scripted input in local. 

 

the script does mv the inputs.conf in local into backupinput after all other loops are good. 

the thing I've noticed is:

my script runs fine and changes the passwd just as I want and writes the echo in my log file. still, my splunk doesn't get any new input. BUT when I manually vim into the log file and write down "test test" for instance - immediately it appears in my splunk. 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...