Scripted input not indexing fully

[siva_cg]

Hi Team,

I have deployed a scripted input on the Cluster Master which will gives us the last accessed, modified time and user details of the configuration files like indexes.conf, server.conf and props.conf using stat Linux command.

I am able to see the logs in Splunk GUI. But the issue here, I am able to see the details of the only one file which means only first line of the output. When I run it manually it is running fine. Could you please help me in rectifying it?

Sample Output:
/opt/splunk/etc/system/local/server.conf 1762 splunk 2017-08-11 19:12:46 2017-05-16 19:10:09 2017-05-16 19:10:09
/opt/splunk/etc/master-apps/_cluster/local/indexes.conf 1762 splunk 2017-08-11 19:12:46 2017-05-16 19:10:09 2017-05-16 19:10:09

Logs in Splunk GUI:
/opt/splunk/etc/system/local/server.conf 1762 splunk 2017-08-11 19:12:46 2017-05-16 19:10:09 2017-05-16 19:10:09

Thanks in advance.

[mattymo · Accepted solution]

I took your script and altered it slightly for my environment (just changed the args):

[splunker@n00bserver bin]$ cat whereIsMyData.sh 
#!/bin/sh

arr=(/home/splunker/splunk/etc/apps/n00blab/local/indexes.conf /home/splunker/splunk/etc/system/local/server.conf)
n=0
while (( n < ${#arr[*]} )); do
stat -c '%n %s %U %.19x %.19y %.19z' "${arr[n]}"
((n++))
done
[splunker@n00bserver bin]$

Then i took the resulting output and i saved it to a file and ran it through the add data wizard:

/home/splunker/splunk/etc/apps/n00blab/local/indexes.conf 205 splunker 2017-08-13 00:00:02 2017-05-07 01:10:21 2017-05-30 23:18:51
/home/splunker/splunk/etc/system/local/server.conf 725 splunker 2017-08-12 23:34:00 2017-03-22 13:04:11 2017-05-30 23:18:51

and created a sourcetype called stat, which set the timestamp to current

[stat]
DATETIME_CONFIG = CURRENT
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Custom
pulldown_type = true

working like a charm for me:

This could actually a pretty cool data source. I have never used the stat command before.

Anyways I think your root cause is the lack of props. I suggest ALWAYS taking your output and running it through the add data wiz to ensure you are in control of the particulars. In this case I have no idea what sourcetype is hitting when you set up your input...but it could be incorrectly identifying the timestamp, and if the last mod time on the file is really old, could be rolling right into frozen.

Try configuring the props and reloading and see if you are good to go!

[siva_cg]

Hi @mmodestino,

Thank you very much. It worked as expected after re-deploying it with the configurations given by you. As you said, it might be the issue with the lack of props.conf. It was accessed very long back.

Thank you once again.

[mattymo]

Did you search alltime? Could be a timestamping issue. Can you share your props.conf, or even better a link to the script so I could run it in my lab?

[siva_cg]

Hi @mmodestino,

Yes, I have searched for All Time too. Please find the script I am using below:

!/bin/sh

arr=(/opt/splunk/etc/master-apps/_cluster/local/indexes.conf /opt/splunk/etc/system/local/server.conf)
n=0
while (( n < ${#arr[*]} )); do
stat -c '%n %s %U %.19x %.19y %.19z' "${arr[n]}"
((n++))
done

Till now, we didn't deploy props.conf. Thank you.

[cpetterborg]

Have you looked closely at the permissions on the .conf files your script is using, and run the script with the same user that your splunkd runs as? What I mean to say is that if your splunkd runs as a different user than the one that you are using when you run it on the command line manually, then you need to try to use the same user account that the splunkd process is using to run the script. I know this might seem obvious, but sometimes it is the simple things that can trip us up. I know that I've looked around the permissions problems until I get desperate.

[siva_cg]

Hi @cpetterborg,
Yes, I have run manually as the same user as the .conf files are. It gave me results as expected.

[mattymo]

so what sourcetype is being set for this?? not setting a props leaves you open to all sorts of foolery. going to try this in my lab now...