How to monitor a file on a remote unix machine via inputs.conf ?

[Hemnaath]

Hi Team,

Currently we need to add additional monitoring stanza to monitor the audit files that are in the remote UNIX nodes. These are the files "wtmpx utmpx, wtmp and btmp" that needs to be included in the monitoring stanza along with the below inputs.conf stanza.

wtmpx and utmpx - These two files present under /var/adm/ directory.
wtmp and btmp - These two files present under /var/log/ directory.

inputs stanza details:

[monitor:///var/adm]
whitelist=(\.log|log$|wtmpx|utmpx|message)
index=nix
disable=0

inputs stanza details:

 [monitor:///var/log]
whitelist=(\.log|log$|secure|message|auth|wtmp|btmp|cron$|\.out)
blacklist=(lastlog)
index=nix
disable=0

Kindly let me know whether above input stanza is correct to fetch the newly added files from the remote machine and also it will be really helpful if you guide me what is the purpose of using $ and .out in the inputs.conf stanza.

thanks in advance

[nickhills · Accepted solution]

Do you see any errors - I suspect if you look in your logs you may see warnings about these files being binary .
normally one uses a tool to read the contents of these files - so my initial guess is that could be your issue.

Edit: updated answer to include my comment
Take a look at this post: https://answers.splunk.com/answers/5844/can-i-splunk-my-wtmp-files.html

The data in wtmp (etc) is used by the who and last tools, and is not directly readable (as they are binary)
There are two good solutions in that post which explain how you can monitor the relevant data

[Hemnaath]

Hi Nickhill, I am unable to see Error message but could see the Warning message in splunk when I search index="_internal" host=test01 log_level=WARN

01-16-2018 08:06:47.560 -0500 WARN FileClassifierManager - The file '/var/adm/utmpx' is invalid. Reason: binary

01-16-2018 08:06:46.560 -0500 WARN FileClassifierManager - The file '/var/adm/wtmpx' is invalid. Reason: binary

Kindly let me know how to monitor these files in splunk.

[nickhills]

Sure - thats the message i was expecting.

Take a look at this post: https://answers.splunk.com/answers/5844/can-i-splunk-my-wtmp-files.html

The data in wtmp (etc) is used by the who and last tools, and is not directly readable (as they are binary)

There are two good solutions in that post which explain how you can monitor the relevent data

[Hemnaath]

thanks Nickhill for the update, but now I am not sure how to write a scripted input to monitor this file and index into splunk by following the below approach.

1) Set up a scripted input calling a shell script that executes "who" or "last" with the options you need and that will index the generated output. This is the simplest approach.

So can you please guide me how to create an scripted inputs for my requirement please.

[Hemnaath]

Hi Nickhill, can you guide me on how to create an scripted inputs for getting the data file into splunk.

thanks in advance.

[nickhills]

I would open another ticket for the question "how do I collect the results of who/last on linux"
As its a more of a useful heading for this issue.
Hopefully I have answered your question on what $ means, and "why" your collection is not working

[Anam]

@hemnaath

Was @nickhillscpl able to answer your question "on what $ means, and "why" your collection is not working"? If he was, please click accept on his answer to resolve the post. If not please provide more information that people can use to help troubleshoot further.

Thanks

[Hemnaath]

Hi asiddique, Nickhillscpl was able to clarify the doubts both of my questions. So I am accepting this answer.

But I am not sure how to get this is done using the link shared by nickhillscpl. If you need me to open a new question for the same, I can do that.

https://answers.splunk.com/answers/5844/can-i-splunk-my-wtmp-files.html

thanks in advance.

[493669]

Hi @Hemnaath
in whitelist you need to provide regex to match filename which you want to monitor

The "$" anchors the regular expression to the end of the line.For ex. |log$ will check if your filename ends with log
.out specifies if filename extension is 'out' like filename.out
refer this document https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Whitelistorblacklistspecificincomingdat...

as per splunk docs,
you need to write disabled instead of disable

disabled = [0|1]
* Whether or not the input is enabled.
* Defaults to 0 (enabled).

[Hemnaath]

Hi, thanks for you support on this but I am unable to monitor the "wtmpx|utmpx" file from the path /var/adm/ on the remote host. But we could monitor other files ".log|log$|message" are being ingested into splunk .

Similarly for the second stanza also we are unable to monitor the "wtmp|btmp" files from the path /var/log/ on the remote host. But we could monitor other files ".log|log$|secure|message|auth|cron$|.out" are being ingested into splunk .

Kindly guide me how to fix this issue, we need to pull the wtmpx, utmpx, wtmp,btmp in splunk.

[493669]

what is the extension of these wtmpx, utmpx, wtmp,btmp files?

[Hemnaath]

Hi wtmpx,utmpx are the data files which does not have any extension and I could see some warning message in the splunkd.log .

01-16-2018 08:06:47.560 -0500 WARN FileClassifierManager - The file '/var/adm/utmpx' is invalid. Reason: binary

01-16-2018 08:06:46.560 -0500 WARN FileClassifierManager - The file '/var/adm/wtmpx' is invalid. Reason: binary

[493669]

The NO_BINARY_CHECK is a props.conf configuration, and so you will want to create a stanza in props like:

 [<sourcetype_name>]
 NO_BINARY_CHECK = true

[mayurr98]

hey I think your first regex seems okay but the second is not. do you want to monitor cron.log and cron.out as well? for the second one .Also there are syntax problems. Refer the one I have given.

[monitor:///var/adm]
whitelist =   wtmpx\.log$|utmpx\.log$|message\.log$
index = nix
disabled =0

[monitor:///var/log]
whitelist = (secure|message|auth|wtmp|btmp|cron)(\.log$|\.out$)
index = nix
disabled = 0

Also $ means end of string
If you want to learn more about regex use this link https://regex101.com/

let me know if it helps!

[Hemnaath]

Hi Mayurrr98, thanks for you support on this but I am unable to monitor the "wtmpx|utmpx" file from the path /var/adm/ on the remote host. But we could monitor other files ".log|log$|message" are being ingested into splunk . Similarly for the second stanza also we are unable to monitor the "wtmp|btmp" files from the path /var/log/ on the remote host. But we could monitor other files ".log|log$|secure|message|auth|cron$|.out" are being ingested into splunk .

Since already we are able to get the other files "message|auth|cron" etc info in splunk, I did not change the regex, just added the new file name in the stanza along with other file name.

Kindly guide me how to fix this issue, we need to pull the wtmpx, utmpx, wtmp,btmp in splunk.

[mayurr98]

did you use wtmpx\.log$|utmpx\.log$|message\.log$ to monitor your first stanza ?

[Hemnaath]

Hi wtmpx,utmpx are the data files which does not have any extension and I could see some warning message in the splunkd.log .

01-16-2018 08:06:47.560 -0500 WARN FileClassifierManager - The file '/var/adm/utmpx' is invalid. Reason: binary

01-16-2018 08:06:46.560 -0500 WARN FileClassifierManager - The file '/var/adm/wtmpx' is invalid. Reason: binary.

So let me know whether, I can use the new regex which you had mentioned in the comment.
thanks in advance.

[mayurr98]

yes try that and for binary
check this link and my accepted solution on this
https://answers.splunk.com/answers/610499/why-uf-think-my-file-is-binary.html#comment-611591

[Hemnaath]

Hi Mayurr, I tried by creating a test app and pushed the below configuration for the test machine and it was fetching the some data but the data was not in a readable format.

Inputs.conf
[monitor:///var/adm/wtmpx]
index = unix
sourcetype = unix:host:wtmpx

Props.conf

[unix:host:wtmpx]
CHARSET = AUTO
NO_BINARY_CHECK = true

And the output in splunk console :

\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00ts/1pts/1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00No\x00\x00\x00\x00\x00\x00\x00Z^>Q\x00P

Kindly let me know how to correct this.

[mayurr98]

I guess the data is binary you need to put appropriate CHARSET