topic Re: How to monitor multiple unrelated directories in Getting Data In

How to monitor multiple unrelated directories

michaellightfoo — Mon, 03 Feb 2020 04:49:02 GMT

Using the universal forwarder I need to monitor multiple directories in separate parts of the filesystem.

Specifically (obfuscated so as not to identify our customer):
[monitor:///var/log]
[monitor:///home//logs]

It seems that multiple monitor stanzas are not working (at least our customer is reporting that the second monitor stanza is not forwarding any files to their Splunk instance.

Is there a workable solution?

Re: How to monitor multiple unrelated directories

gcusello — Mon, 03 Feb 2020 08:10:09 GMT

Hi michaellightfoot,
the best approach to monitor many different folders is to plan the ingestion before to start, in other words: use an Excel file to define monitoring perimeter: all the hosts to monitor and, for each server, the folders to ingest.
In this way you can create your own inputs.conf that permits to ingest all the logs you want.

To debug eventual not ingested logs you have to map the monitoring perimeter with the logs you're receiving so you can define which are the missing folders.
At this point you have to see one by one each folder to understand if there are logs to ingest or not.
In this way you can limit the folders to check.

At first check file permissions: maybe the user you're using to run Splunk Universal Forwarder hasn't read grants on that folder or files.
Then check the time format of your logs: maybe the logs have time format dd/mm/yyyy and by default Splunk uses the time format mm/dd/yyyy, so you ingested logs, but instead to have as timestamp 3rd of february they have 2nd of march.

Lety me know if in this way you solved.

Ciao.
Giuseppe

Re: How to monitor multiple unrelated directories

michaellightfoo — Wed, 05 Feb 2020 23:22:24 GMT

I suspect that the problem is at the customer splunk end as I have run a tcpdump and can see the data from both monitors being sent to their instance. Unfortunately I do not have access to that splunk instance so I cannot verify anything.

I will mention that they might need to check that they are ingesting the timestamps correctly.