topic Re: After installing the File/Directory Information Input add-on, why are no logs being indexed? in All Apps and Add-ons

After installing the File/Directory Information Input add-on, why are no logs being indexed?

smudge797 — Tue, 24 Jan 2017 17:03:22 GMT

After installing File/Directory Information Input add-on on Splunk 6.5.1 for Windows and configuring to UNC path, I see the following:

INFO Time is later than filter, st_mtime=1459251861.8578942, must_be_later_than=None

Re: After installing the File/Directory Information Input add-on, why are no logs being indexed?

aaraneta_splunk — Tue, 24 Jan 2017 17:17:44 GMT

@smudge797 - Please provide more information and context as to what you need help with as it is not clear. Generally, the more information you provide, the better chance of being answered by experts in the Answers community. Thank you.

Re: After installing the File/Directory Information Input add-on, why are no logs being indexed?

woodcock — Tue, 24 Jan 2017 19:06:26 GMT

Make sure that NTP is set on both your forwarder (the Windows machine) and the Indexers. Things cannot happen in the future.

Re: After installing the File/Directory Information Input add-on, why are no logs being indexed?

smudge797 — Tue, 29 Sep 2020 12:32:35 GMT

This is the app: https://splunkbase.splunk.com/app/2776/

This is the configured input:

[file_meta_data://ACETestFolder]
file_hash_limit = 500MB
file_path = \\acetest
include_file_hash = 0
interval = 15m
only_if_changed = 1
recurse = 1

There is an initial pull of events seen in the main index, some 3k events
similar to this:
time="Tue Jan 24 16:37:39 2017" is_directory=1 file_count=0 directory_count=16 path=\\acetest atime="Thu Dec 22 12:52:26 2016" atime_epoch=1482429146.82 ctime="Thu Oct 15 16:15:49 2015" ctime_epoch=1444940149.33 dev=0 gid=0 ino=0 mode=16895 mtime="Thu Dec 22 12:52:26 2016" mtime_epoch=1482429146.82 nlink=0 size=4096 uid=0 owner=Administrators\BUILTIN owner...(lots more fields)

All events have same timestamp time="Tue Jan 24 16:37:39 2017" which is correctly indexed in main.

In the index= _internal source="C:\Program Files\Splunk\var\log\splunk\file_meta_data_modular_input.log"
Events like:
2017-01-24 16:37:34,867 INFO Time is later than filter, st_mtime=1482429146.8163483, must_be_later_than=0, path='\\\acetest'

Have second input:

[file_meta_data://fileTest]
file_hash_limit = 500MB
file_path = \\someshare_archive06$
include_file_hash = 0
interval = 15m
only_if_changed = 1
recurse = 1
disabled = 0

No events in main.

index=_internal source="C:\Program Files\Splunk\var\log\splunk\file_meta_data_modular_input.log"

2017-01-24 17:35:24,240 INFO Time is later than filter, st_mtime=1333460403.592, must_be_later_than=0, path="\\\someshare_archive06$\~filedetails.xlsm3.xlsm"

Also lost of these:

2017-01-24 17:29:38,009 ERROR Error when processing path="blah", reason="(1332, 'LookupAccountSid', 'No mapping between account names and security IDs was done.')" Traceback (most recent call last): File "C:\Program Files\Splunk\etc\apps\file_meta_data\bin\file_meta_data.py", line 381, in get_file_data windows_acl_info = cls.get_windows_acl_data(file_path, logger) File "C:\Program Files\Splunk\etc\apps\file_meta_data\bin\file_meta_data.py", line 254, in get_windows_acl_data sid_resolved = win32security.LookupAccountSid(None, sid) error: (1332, 'LookupAccountSid', 'No mapping between account names and security IDs was done.')

Re: After installing the File/Directory Information Input add-on, why are no logs being indexed?

smudge797 — Thu, 26 Jan 2017 15:34:26 GMT

NTP is working.

Re: After installing the File/Directory Information Input add-on, why are no logs being indexed?

LukeMurphey — Fri, 17 Mar 2017 04:54:23 GMT

The only thing I can think of is that something in your environment isn't allowing the account SID to be looked up.

I'm going to make input succeed even if the SID lookup fails.

Re: After installing the File/Directory Information Input add-on, why are no logs being indexed?

LukeMurphey — Fri, 17 Mar 2017 05:05:07 GMT

This happens when there is no mapping between the account names and the SIDs available on the host. See http://www.rebeladmin.com/2016/01/how-to-fix-error-no-mapping-between-account-names-and-security-ids-in-active-directory/ for information.

I am going to modify the input so that it proceeds even if SID lookup fails. This will done under this ticket and will be released in version 1.1.1.

Update

I just released version 1.1.1 which should allow the input to work even if the SID and account name mapping doesn't exist on the host. You won't get the Windows ACL data if this condition exists, but the input will still run.

Please let me know if that fixes the problem (or just accept this answer).

Re: After installing the File/Directory Information Input add-on, why are no logs being indexed?

smudge797 — Tue, 29 Sep 2020 14:13:49 GMT

Using latest version I'm still having issues. Only seeing occasional logging to the correctly configured index on restarting the service. The interval is being ignored here.

Looking at internal logs:
Index=_internal source="C:\Program Files\Splunk\var\log\splunk\file_meta_data_modular_input.log"

Seeing events like:

INFO Completed retrieval of file data....
WARNING Unable to get the ACL data, reason=(5, 'GetFileSecurity', 'Access is denied.')...
INFO Time is later than filter, st_mtime=1322859631.165719, must_be_later_than=None, path='...
INFO Time is later than filter, st_ctime=1330974351.030764, must_be_later_than=None, path=...

Re: After installing the File/Directory Information Input add-on, why are no logs being indexed?

LukeMurphey — Tue, 23 May 2017 17:33:26 GMT

A few things to look into:

Do the "Time is later than filter" logs include the files that you want logs for? This would indicate that the input is skipping the files because it doesn't detect that they have changed.
Are you sure that Splunk has access to the files you want it to monitor? The permission errors seen previously might be an indicator that Splunk doesn't have access to the files.
You might try disabling the option to only include new results to see if you get the results you want.