Splunk Forwarder doesn't monitor new files

[IlianYotov]

Hello,

I need some help. 

I have a folder and an app that writes logs in NDJSON format and creates a new log file every 15 minutes. 

The configuration that I use is this:

[monitor:///Users/yotov/app/.logs/.../*.log]
disabled = false
sourcetype = ndjson
crcSalt = <SOURCE>
alwaysOpenFile = 1

 The problem is that Splunk Forwarder doesn't detect newly added files. It reads only the files at the start, and detects newly added content in them, but when a new file is added they are ignored until restart of Splunk Forwarder.

I'm using the latest version of Splunk Forwarder and tried under Linux and MacOs

What am I missing?

[IlianYotov]

How Splunk detect a new file? Is it using polling or does it depend on Inotify in Linux for example?

[PickleRick]

splunk list monitor

and

splunk list inputstatus

are your friends here.

Also - crcSalt = <SOURCE> is a setting often used by newcomers to Splunk but in reality it's rarely needed (usually raising initCrcLength suffices).

alwaysOpenFile is most typically not needed. Leave it at default unless you're doing some weird stuff on Windows.

My suspicion would be that since you have many files (almost a hundred files for each day), you're running out of file descriptors.

[IlianYotov]

@PickleRick Yes rolling files every 15 minutes could produce hundreds of files, but my tests were executed with a very small number of files ( 10 - 20 ) and even with these files Splunk doesn't monitor the newly created. I will check the commands you wrote and hope to find what is the problem

[VatsalJagani]

@IlianYotov - Just to clarify the path you are trying to look at is

/Users/yotov/app/.logs/.../*.log

• Inside /Users/youtov/app
• There is a hidden folder named ".log"
• inside that, there are sub-folders
• inside which there are files with .log extention at the end.

Also, is there any specific reason for using alwaysOpenFile parameter?

* https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf 

[IlianYotov]

Yes, and here is an example:

/Users/yotov/app/.logs/
- 1/
   - 2024-05-14/
      - 10_00_00.log
      - 10_15_00.log
      ( every 15 minutes a new file is created )
      - 15_00_00.log
- 2/
   - 2024-05-14/
      - 10_00_00.log
      - 10_15_00.log
     ....

About alwaysOpenFile - no, I tried with and without it. but nothing happens

[gcusello]

Hi @IlianYotov,

do new files have the same name of the previous or a different one?

did you checked without the "crcSalt = <SOUCE>" option?

Is it possible that the new files have the same content of the previous ones?

Ciao.

Giuseppe

[IlianYotov]

Hi @gcusello 

No, the new file has a different name ( the name is the time when they are generated ). The content of the files is not the same because they contain. I tried different options of crcSalt but nothing happened.

I also checked logs in $SPLUNK_FORWARDER/var/log/splunk/metrics.log but there are no logs about new files