Getting Data In
Highlighted

Help with wildcard inputs issues

Champion

Hi,

I have some new inputs configured with wildcards and whitelists, but they aren't pulling in the data.

The path to the files is: /xt112qdal0/log/app/XTRAC/6.9.1.1/XTRAC6.9.1.1_ClusterMbr1/xtrac.error.log

(Note that this paticular file can exist in many different sub-directories off the ..log parent, hence the wildcard)

I've confirmed that the filed is readable. I can see the connection being made to the indexer, so, I'm assuming that it's my inputs:

[monitor:///xtqa/log/]
recursive = Yes
index = Xtracceopsqapmlogs
sourcetype = xtrac_error
followTail = 0
disabled = 0
whitelist = xtrac.error.log
crcSalt =

Is something wrong with this setup?

Tags (2)
0 Karma
Highlighted

Re: Help with wildcard inputs issues

Legend

Try this

[monitor:///xt*qa*/log/]
index = Xtrac_ceops_qa_pm_logs
sourcetype = xtrac_error
whitelist = xtrac.error.log

You don't need most of the settings, as you are using the defaults. Plus recursive = Yes is not a valid setting.

Finally, in your path example, the first directory name is tx112qdal0 and in the regular expression, you are missing the d between the q and the a. This may be the real problem...

0 Karma
Highlighted

Re: Help with wildcard inputs issues

Champion

Thanks. Picked up the typo, and it found one of the files(trace), but not the others, even after making your recommended changes. The full inputs is the following:

[monitor:///xtq/log/]
index = Xtracceopsqawflogs
sourcetype = xtrac_error
whitelist = xtrac.error.log

[monitor:///xtq/log/]
index = Xtracceopsqawflogs
sourcetype = xtrac_perf
whitelist = xtrac.perf.log

[monitor:///xtq/log/]
index = Xtracceopsqawflogs
sourcetype = xtrac_trace
whitelist = xtrac.trace.log

0 Karma
Highlighted

Re: Help with wildcard inputs issues

Legend

Aha - this is the problem. You cannot have multiple stanzas with the same monitor. Do this instead:

inputs.conf

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = xtrac\.(error|trace|perf)\.log$

props.conf

[source::.../xtrac.error.log]
sourcetype = xtrac_error

[source::.../xtrac.perf.log]
sourcetype = xtrac_perf

[source::.../xtrac.trace.log]
sourcetype = xtrac_trace

Here is a reference in the Getting Data In manual for more information.

0 Karma
Highlighted

Re: Help with wildcard inputs issues

Champion

Thank you, this is fantastic. I'm noticing that the splunkd is recommending that I set a crcsalt setting, because the error log is small. In this case, since they are all using the same inputs stanza, how would I do that?

0 Karma
Highlighted

Re: Help with wildcard inputs issues

Legend

To do that, add the following line to your inputs.conf stanza:

crcSalt=<SOURCE>

This tells Splunk to consider the full path name of your file, as well as the contents of the file, to determine if a file is unique. (Splunk looks at the contents of each file to determine if it has already indexed the data.)

0 Karma
Highlighted

Re: Help with wildcard inputs issues

Champion

Thanks. I was aware of crcSalt, but I'm never quite sure if it should be applied to all sources, or just certain ones.

That said, is this valid? I'm going to have a lot of files for this feed, so I'm concerned that the whitelist will reach a size limit.

[monitor:///xtq/log/]
index = Xtracceopsqawflogs
whitelist = xtrac.(error|trace|perf).log$

[monitor:///xtq/log/]
index = Xtracceopsqawflogs
whitelist = xwb.(error|trace|perf).log$

[monitor:///xtq/log/]
index = Xtracceopsqawflogs
whitelist = fuse.log$

[monitor:///xtq/log/]
index = Xtracceopsqawflogs
whitelist = TimerManager.log$

[monitor:///xtq/log/]
index = Xtracceopsqawflogs
whitelist = System(Err|Out).log$

0 Karma
Highlighted

Re: Help with wildcard inputs issues

Legend

Wait!! You CANNOT have monitor stanzas that overlap. You have 5 identical monitor stanzas - this will not work!

Don't worry about the length of the whitelist...

[monitor:///xt*q*/log/]
index = Xtrac_ceops_qa_wf_logs
whitelist = (xtrac|xwb).(error|trace|perf).log$|fuse.log$|TimeManager.log$|System(Err|Out).log$

is not that long. But you could add more information into the monitor stanza as well:

[monitor:///xt*q*/log/.../fuse.log]

for example.

0 Karma
Highlighted

Re: Help with wildcard inputs issues

Champion

Thanks - yes, I realized that wouldn't work.

The whitelist could get quite large - hence my concern. Would the example above work for a fuse.log that exists in any directory off ../log? The number and level of subdirectories varies - could be 2, could be 5... - want to make sure it would work.

0 Karma