Getting Data In

whitelisting data from a unix log

nickhaj
New Member

HI, I'm relatively new to Splunk and need a bit of guidance around whitelisting specific data via a Unix inputs.conf. The log being monitored contains some non-standard entries to confirm a NFS f/system is not 'stale' - these events are getting taken across to Splunk but NOT in a timely fashion as the sourcetype isnt recognising them, so they are not being sent until any standard input is written to the log/ingested - this can result in delays in getting these events into Splunk and consequent time mismatches. These events only occur on one server hence thinking it is much better to control the input via that box rather than changing the sourcetype.

So, inputs.conf currently looks like this;
[monitor:///path-to-log]
disabled = 0
index = index
sourcetype = sourcetype
blacklist = list of logs to ignore

The non-standard data looks like this :

(and a line of stars)

PULSE CHECK - Fri Dec 6 07:30:00 GMT 2019

(and a line of stars)

Can someone advise the appropriate whitelist entry and where to place it in inputs.conf please ?

Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @nickhaj,
if reading the standard logs you have to use a well defined time format (e.g. dd/mm/yyy HH:MM:SS) or other characteristics and the non standard logs are really different from the first (with special regard to timestamp format), the only way is to use a different sourcetype to assign in a different stanza blacklisting the files of the other kind.

If instead you don't need to define timestamp format (e.g. it's yyyy-mm-dd HH:MM:SS) in both your kind of logs, you can use the same sourcetype in one stanza.

But probably to use one sourcetype for two really different kind of logs isn't the best solution.

Ciao.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @nickhaj,
if reading the standard logs you have to use a well defined time format (e.g. dd/mm/yyy HH:MM:SS) or other characteristics and the non standard logs are really different from the first (with special regard to timestamp format), the only way is to use a different sourcetype to assign in a different stanza blacklisting the files of the other kind.

If instead you don't need to define timestamp format (e.g. it's yyyy-mm-dd HH:MM:SS) in both your kind of logs, you can use the same sourcetype in one stanza.

But probably to use one sourcetype for two really different kind of logs isn't the best solution.

Ciao.
Giuseppe

0 Karma

nickhaj
New Member

Hi Giuseppe - many thanks for the prompt response! All the events are in one (type of) log which I am ingesting (both the OK events and the 'non-standard' ones). I can probably get the non-standard log output brought into line(ish) with the standard, was just wondering if there was a way round it via splunk config, but reading your response it looks like I ideally need to add the timestamp....if thats my correct interpretation please confirm and I will accept your response as the answer.

Many Thanks again!
Nick

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @nickhaj,
timestamp is a data that you must have in all events, infact when yu haven't it in an event, at index time, Splunk gives to an event the timestamp of the previous or the current time because you must have it!
As I said, choose the way to ingest you logs between the two choices I hinted.

Ciao.
Giuseppe

0 Karma

nickhaj
New Member

Yep, thats exactly whats occurring Guiseppe ! We will get the correct Timestamp format added to the non-standard data.

Many Thanks!
Nick

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...