whitelisting data from a unix log

[nickhaj]

HI, I'm relatively new to Splunk and need a bit of guidance around whitelisting specific data via a Unix inputs.conf. The log being monitored contains some non-standard entries to confirm a NFS f/system is not 'stale' - these events are getting taken across to Splunk but NOT in a timely fashion as the sourcetype isnt recognising them, so they are not being sent until any standard input is written to the log/ingested - this can result in delays in getting these events into Splunk and consequent time mismatches. These events only occur on one server hence thinking it is much better to control the input via that box rather than changing the sourcetype.

So, inputs.conf currently looks like this;
[monitor:///path-to-log]
disabled = 0
index = index
sourcetype = sourcetype
blacklist = list of logs to ignore

The non-standard data looks like this :

(and a line of stars)

PULSE CHECK - Fri Dec 6 07:30:00 GMT 2019

(and a line of stars)

Can someone advise the appropriate whitelist entry and where to place it in inputs.conf please ?

[gcusello · Accepted solution]

Hi @nickhaj,
if reading the standard logs you have to use a well defined time format (e.g. dd/mm/yyy HH:MM:SS) or other characteristics and the non standard logs are really different from the first (with special regard to timestamp format), the only way is to use a different sourcetype to assign in a different stanza blacklisting the files of the other kind.

If instead you don't need to define timestamp format (e.g. it's yyyy-mm-dd HH:MM:SS) in both your kind of logs, you can use the same sourcetype in one stanza.

But probably to use one sourcetype for two really different kind of logs isn't the best solution.

Ciao.
Giuseppe

[nickhaj]

Hi Giuseppe - many thanks for the prompt response! All the events are in one (type of) log which I am ingesting (both the OK events and the 'non-standard' ones). I can probably get the non-standard log output brought into line(ish) with the standard, was just wondering if there was a way round it via splunk config, but reading your response it looks like I ideally need to add the timestamp....if thats my correct interpretation please confirm and I will accept your response as the answer.

Many Thanks again!
Nick

[gcusello]

Hi @nickhaj,
timestamp is a data that you must have in all events, infact when yu haven't it in an event, at index time, Splunk gives to an event the timestamp of the previous or the current time because you must have it!
As I said, choose the way to ingest you logs between the two choices I hinted.

Ciao.
Giuseppe

[nickhaj]

Yep, thats exactly whats occurring Guiseppe ! We will get the correct Timestamp format added to the non-standard data.

Many Thanks!
Nick