Does splunk have any limitation to index recursive directories on NFS from sub directories ?

[pimco_rgoyal]

Hi all, I was trying to configure a log pattern main.log from using recursive option. However splunk is failing to pick up any new files created and I dont see any errors as part of the _internal log as well. When I tried to enable the recursive option for it seemed to work fine. Is there any workaround or solution to fixing this ? From what I saw wildcard or regex also do not allow more than 2 directories.

index = app_elixir_rg
recursive = true
sourcetype = elixir
whitelist = main.log

[ddrillic]

Are there many files involved in this directory tree?

[pimco_rgoyal]

Yes there are large number of files as part of this directory.

[ddrillic]

Interesting. A fascinating "article" about the dangers of large number of files to monitor at is there a limit on the number of files splunk can monitor?

[jkat54]

Monitor too many with a wildcard approach and Splunk will tack the CPU at 100%. In this case he just wants main.log so it shouldn't be too bad unless there are many thousands+

[pimco_rgoyal]

No dont have count of main.log's counting to 1000's not does CPU take a spike on the node where the forwarder is installed.

[jkat54]

Try this instead:

[monitor:///appl/proddata/tmp/U/LOG_FILES/elixir_logs/.../main.log]
index = app_elixir_rg
sourcetype = elixir
crcSalt =

Recursive=true by default it's the ... that makes it look through sub directories.

[pimco_rgoyal]

No this failed to help work as well.

[jkat54]

Then you have a permissions issue or typo on your config.

Check the internal index for error messages:

index=_internal host=yourForwarder main.log

[pimco_rgoyal]

Dont see any errors reported that can help point out to the root cause for this.

[jkat54]

Do you see any messages with the search above? Can you paste them here?

[pimco_rgoyal]

My bad, I do see an error reported as below.

ERROR TailingProcessor - File will not be read, is too small to match seekptr checksum (file=/appl/proddata/tmp/U/LOG_FILES/elixir_logs/FXModel/20170804_143057_28122/TWDCNY/2017-07-26_1/main.log). Last time we saw this initcrc, filename was different. You may wish to use a CRC salt on this source. Consult the documentation or file a support case online at http://www.splunk.com/page/submit_issue for more info.

[pimco_rgoyal]

But just to add I did try adding CRC salt as well which did not seem to have fixed the issue.

[jkat54]

You added

crcSalt=<SOURCE>

In inputs.conf?

[pimco_rgoyal]

Yes correct