topic Re: File Metadata only indexing on Splunk Startup in All Apps and Add-ons

File Metadata only indexing on Splunk Startup

jhankee — Tue, 29 Sep 2020 13:23:00 GMT

So I've deployed the File Meta Data app to a heavy forwarder running on Linux, If I restart splunk the directory information is sent to the indexer at that time and never again. I created a 2nd app and deployed that to a Windows server with a Heavy Forwarder and that will index the contents of the directory on the scheduled interval.

Any suggestions on how to get the Linux data indexed would be great.

[file_meta_data://host_scripts]
file_hash_limit = 500MB
file_path = /path/scripts
host = host
include_file_hash = 0
index = filemon
interval = 10m
only_if_changed = 0
recurse = 1

Re: File Metadata only indexing on Splunk Startup

LukeMurphey — Thu, 23 Mar 2017 17:34:48 GMT

Could you let me know if you see anything worth noting when you run the following search?

index=_internal (sourcetype=file_meta_data_modular_input OR (ExecProcessor "file_meta_data"  sourcetype=splunkd))

Tracebacks would be of particular interest.

Re: File Metadata only indexing on Splunk Startup

jhankee — Tue, 29 Sep 2020 13:23:22 GMT

thanks for getting back to me. so nothing in splunkd after the startup relating to "file_meta_data", I found this in the "splunk/var/log/splunk/file_meta_data_modular_input.log", these are the last lines:

2017-03-22 18:19:02,856 INFO Time is later than filter, st_mtime=1490213086.0038838, must_be_later_than=None, path='/lawprd/scripts'
2017-03-22 18:19:02,856 INFO Completed retrieval of file data, count=202, path=/lawprd/scripts
2017-03-22 18:19:02,861 ERROR Execution failed
Traceback (most recent call last):
File "/lawprd/splunk/etc/apps/finapps_lawson_filemon/bin/file_info_app/modular_input.py", line 1320, in execute
self.do_run(in_stream, log_exception_and_continue=True)
File "/lawprd/splunk/etc/apps/finapps_lawson_filemon/bin/file_info_app/modular_input.py", line 1220, in do_run
input_config)
File "/lawprd/splunk/etc/apps/finapps_lawson_filemon/bin/file_meta_data.py", line 508, in run
result['time'] = time.strftime("%a %b %d %H:%M:%S %Y")
TypeError: list indices must be integers, not str

Re: File Metadata only indexing on Splunk Startup

LukeMurphey — Thu, 23 Mar 2017 18:36:55 GMT

This looks like a bug. I'm fixing it under this ticket: https://lukemurphey.net/issues/1797

I'm planning on releasing it in a maintenance release (1.1.2).

Update

This is now fixed in version 1.1.2 (or later). Download it here: https://splunkbase.splunk.com/app/2776/

Re: File Metadata only indexing on Splunk Startup

LukeMurphey — Fri, 24 Mar 2017 05:34:54 GMT

I'm struggling to figure this out.

Could you let me know a few things?

What version of the app are you using?
Does the input seem to fail on the same file? You might be able to tell by seeing if the logs report the same file before the input fails.

Re: File Metadata only indexing on Splunk Startup

jhankee — Fri, 24 Mar 2017 17:59:17 GMT

For the record - I'm running 1.1.1. Upon closer examination there is one directory that I didn't notice that splunk doesn't have access to, I selected a different directory and get the expected results. So.....my next question is, any recommendations on how to blacklist a directory? In my case, the directory contains files and several subdirectories, I want to index everything except the the directory called 'admin'

thanks!

Re: File Metadata only indexing on Splunk Startup

LukeMurphey — Sun, 09 Apr 2017 05:20:24 GMT

I have a fix for this; just running the changes through the final tests.