Is there a way for Splunk to read from one directory and capture everything being written to that directory?

[dreschke]

Hello Everyone,

The issue is that we are collecting CyberArk logs using the CyberArk add on 1.0.0. CyberArk is creating multiple sub directories on the monitored location and we are only capturing what is being written in the 2 vault directories. We feel that we are not capturing all of the CYberArk logs as they create a new directories for every log. The only constant is that all of sub directories are being created in the Directiory folder named xxx. Is there a away that Splunk can just read from the xxx folder and capture all of the logs that being written in the sub directories? We have open a ticket with CyberArk on the CyberArk creating the multiple sub directories but no work able solution has been provided. Please let me know if you need anymore information. Thank you for your help everyone.

[solarboyz1]

what is the format of the cyberArk files?

Splunk uses a method to track which files have already been indexed.
Splunk makes a hash of the first row of the file, to keep track of the files.

If the files all have a header row, and they are identical....the hash's will be the same and Splunk will not try to index the file.

If that's the case try adding the following to your the monitor stanza:

initCrcLength = 4096

[solarboyz1]

https://docs.splunk.com/Documentation/Splunk/7.1.0/Data/Monitorfilesanddirectorieswithinputs.conf

Look under the monitor files and directories section:
recursive = true|false

You just need to ensure the splunk process has access to the files/directories

[dreschke]

Thanks solarboyz1 for the help. I just have another follow up question for you.

If the monitored file/folder is directed to a specific file i.e c:/documents/testdata/test30/log123.log if by adding the recursive parameter to the input file and if i change the monitored file to just the parent folder/directory (c:/documents/testdata/) will it capture all of the file that are captured with in that directory?

[solarboyz1]

Correct, if you monitor c:/documents/testdata/ splunk should try to ingest any files dropped there that it can read.

You can also use wild card: c:/documents/testdata/*.log

To ensure you only pickup log files in that directory, if there is risk of other files you don't want read showing up in that file.

[dreschke]

Thanks again. Last question, Do i have to add the recursive parameter since the default setting is true? Also if i monitor this location c:/documents/testdata and there is a sub-directory create within a log file in that that folder will I still be able to read that log file?

[solarboyz1]

No, you don't need to specify it should pickup the default (as long as the default hasn't been changed).

Yes, It should recurse through any subdirectories it finds that it has access to.

[dreschke]

That's awesome, thank you for your help. I will let you know when it works. Thanks again for your help.

[dreschke]

solarboyz01,

I added the recursive stanza to my input config, and I changed the monitored file path to include the parent folder and not specific log file but I am still only seeing the one log file. I made the changes about 2 weeks ago. Could this because the input config file has a host stanza set to only read from the on host?