Getting Data In
Highlighted

UF not sending logs from all folders monitored

Communicator

Hello Splunkers.

I have an issue that I've been dealing with for the past 2 days but no success in solving it.
I'm working on a Splunk cluster environment, 3 SH and 2 IDX.

I have an UF installed in a SunOS machine.
This UF monitors a file called runlog.098880020 (the number is actually just an ID, it doesn't really matters).
This log can be found at the path /export/tsi/tsi/tsiout.1509/runlog.098880020
The thing is: the application creates a new folder every month (tsiout.1505, tsiout.1506, tsiout.1507, tsiout.1508, tsiout.1509....)

this is how I've setted my inputs.conf:

[monitor:///export/home/tsi/tsi/.../runlog*]
index = tsi
sourcetype = tsi_logs

However when Splunk starts to indexing the files, it indexes only a few folders (e.g., tsiout.1406 and tsiout.1409).
If I set my inputs.conf as following, I can see the current log beeing indexed:

[monitor:///export/home/tsi/tsi/tsiout.1509/runlog*]
index = tsi
sourcetype = tsi_logs

Do you guys know why this is happening?
Shouldn't the ... tell Splunk to search in every folder for the runlog* file?

Thank you guys!
Regards!

0 Karma
Highlighted

Re: UF not sending logs from all folders monitored

Esteemed Legend

Have you ever used ignoreolderthan on these inputs?

0 Karma
Highlighted

Re: UF not sending logs from all folders monitored

Communicator

I've never used ignoreolderthan

0 Karma
Highlighted

Re: UF not sending logs from all folders monitored

SplunkTrust
SplunkTrust

For your purposes, [monitor:///export/home/tsi/tsi/*/runlog*] should work.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: UF not sending logs from all folders monitored

Communicator

No luck in that either..

The files indexed where:
/export/home/tsi/tsi/tsiout.1409/runlog.0901000145
/export/home/tsi/tsi/tsiout.1209/runlog.0905001532
/export/home/tsi/tsi/tsiout.1209/runlog.0901001129
/export/home/tsi/tsi/tsiout.1209/runlog.0904213433

Maybe is there somthing to do with the month..
Only the folders tsiout.1209 and tsiout.1409 were indexed...

0 Karma
Highlighted

Re: UF not sending logs from all folders monitored

SplunkTrust
SplunkTrust

Have you checked the permissions on the folders that are not indexed?

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: UF not sending logs from all folders monitored

Esteemed Legend

What makes you think that it is not indexing? Tell the WHOLE story (did it ever get indexed before and now you are trying to reindex it)?

0 Karma
Highlighted

Re: UF not sending logs from all folders monitored

Communicator

OK..
I don't need old logs..
I need to index this month logs and the ones that are coming.
I can see at the server that runlog.0901000208 exists under the folder tsiout.1509, but Splunk is not indexing it (never indexed).

0 Karma
Highlighted

Re: UF not sending logs from all folders monitored

Contributor

Let's start with some basics:

Do all the directories and files underneath /export/home/tsi/tsi have the same permissions? Does the user the UF is running as have appropriate permissions for those files and directories? Directories will need at least x to traverse them; files will need both r and x.

You may also need to look at the privileges assigned using usermod on Solaris if the UF is not running as root (it should not be, that is a security risk). Look at /etc/user_attrand review the bottom portion of this docs page to see if permissions match:
Run Splunk as Non-Root User

Are any errors being generated on the UF? You can search index = _internal for the UF's hostname or IP; if nothing is showing up there, check on the UF itself. Logs will be in /opt/splunkforwarder/var/log/splunk/

View solution in original post