topic Re: Help with wildcard inputs issues in Getting Data In

Help with wildcard inputs issues

a212830 — Mon, 28 Sep 2020 18:20:24 GMT

Hi,

I have some new inputs configured with wildcards and whitelists, but they aren't pulling in the data.

The path to the files is: /xt112qdal0/log/app/XTRAC/6.9.1.1/XTRAC6.9.1.1_ClusterMbr1/xtrac.error.log

(Note that this paticular file can exist in many different sub-directories off the ..log parent, hence the wildcard)

I've confirmed that the filed is readable. I can see the connection being made to the indexer, so, I'm assuming that it's my inputs:

[monitor:///xt*qa*/log/]
recursive = Yes
index = Xtrac_ceops_qa_pm_logs
sourcetype = xtrac_error
followTail = 0
disabled = 0
whitelist = xtrac.error.log
crcSalt =

Is something wrong with this setup?

Re: Help with wildcard inputs issues

lguinn2 — Mon, 08 Dec 2014 00:54:19 GMT

Try this

[monitor:///xt*qa*/log/]
index = Xtrac_ceops_qa_pm_logs
sourcetype = xtrac_error
whitelist = xtrac.error.log

You don't need most of the settings, as you are using the defaults. Plus recursive = Yes is not a valid setting.

Finally, in your path example, the first directory name is tx112qdal0 and in the regular expression, you are missing the d between the q and the a. This may be the real problem...

Re: Help with wildcard inputs issues

a212830 — Mon, 28 Sep 2020 18:20:28 GMT

Thanks. Picked up the typo, and it found one of the files(trace), but not the others, even after making your recommended changes. The full inputs is the following:

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
sourcetype = xtrac_error
whitelist = xtrac.error.log

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
sourcetype = xtrac_perf
whitelist = xtrac.perf.log

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
sourcetype = xtrac_trace
whitelist = xtrac.trace.log

Re: Help with wildcard inputs issues

lguinn2 — Mon, 08 Dec 2014 07:45:05 GMT

Aha - this is the problem. You cannot have multiple stanzas with the same monitor. Do this instead:

inputs.conf

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = xtrac\.(error|trace|perf)\.log$

props.conf

[source::.../xtrac.error.log]
sourcetype = xtrac_error

[source::.../xtrac.perf.log]
sourcetype = xtrac_perf

[source::.../xtrac.trace.log]
sourcetype = xtrac_trace

Here is a reference in the Getting Data In manual for more information.

Re: Help with wildcard inputs issues

a212830 — Mon, 08 Dec 2014 12:22:09 GMT

Thank you, this is fantastic. I'm noticing that the splunkd is recommending that I set a crcsalt setting, because the error log is small. In this case, since they are all using the same inputs stanza, how would I do that?

Re: Help with wildcard inputs issues

lguinn2 — Mon, 08 Dec 2014 22:46:41 GMT

To do that, add the following line to your inputs.conf stanza:

crcSalt=<SOURCE>

This tells Splunk to consider the full path name of your file, as well as the contents of the file, to determine if a file is unique. (Splunk looks at the contents of each file to determine if it has already indexed the data.)

Re: Help with wildcard inputs issues

a212830 — Mon, 28 Sep 2020 18:20:55 GMT

Thanks. I was aware of crcSalt, but I'm never quite sure if it should be applied to all sources, or just certain ones.

That said, is this valid? I'm going to have a lot of files for this feed, so I'm concerned that the whitelist will reach a size limit.

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = xtrac.(error|trace|perf).log$

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = xwb.(error|trace|perf).log$

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = fuse.log$

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = TimerManager.log$

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = System(Err|Out).log$

Re: Help with wildcard inputs issues

lguinn2 — Wed, 10 Dec 2014 17:32:47 GMT

Wait!! You CANNOT have monitor stanzas that overlap. You have 5 identical monitor stanzas - this will not work!

Don't worry about the length of the whitelist...

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = (xtrac|xwb).(error|trace|perf).log$|fuse.log$|TimeManager.log$|System(Err|Out).log$

is not that long. But you could add more information into the monitor stanza as well:

[monitor:///xt*q*/log/.../fuse.log]

for example.

Re: Help with wildcard inputs issues

a212830 — Thu, 11 Dec 2014 14:58:55 GMT

Thanks - yes, I realized that wouldn't work.

The whitelist could get quite large - hence my concern. Would the example above work for a fuse.log that exists in any directory off ../log? The number and level of subdirectories varies - could be 2, could be 5... - want to make sure it would work.