All Apps and Add-ons

File Metadata only indexing on Splunk Startup

jhankee
New Member

So I've deployed the File Meta Data app to a heavy forwarder running on Linux, If I restart splunk the directory information is sent to the indexer at that time and never again. I created a 2nd app and deployed that to a Windows server with a Heavy Forwarder and that will index the contents of the directory on the scheduled interval.

Any suggestions on how to get the Linux data indexed would be great.

[file_meta_data://host_scripts]
file_hash_limit = 500MB
file_path = /path/scripts
host = host
include_file_hash = 0
index = filemon
interval = 10m
only_if_changed = 0
recurse = 1

0 Karma
1 Solution

LukeMurphey
Champion

This looks like a bug. I'm fixing it under this ticket: https://lukemurphey.net/issues/1797

I'm planning on releasing it in a maintenance release (1.1.2).

Update

This is now fixed in version 1.1.2 (or later). Download it here: https://splunkbase.splunk.com/app/2776/

View solution in original post

LukeMurphey
Champion

This looks like a bug. I'm fixing it under this ticket: https://lukemurphey.net/issues/1797

I'm planning on releasing it in a maintenance release (1.1.2).

Update

This is now fixed in version 1.1.2 (or later). Download it here: https://splunkbase.splunk.com/app/2776/

LukeMurphey
Champion

I have a fix for this; just running the changes through the final tests.

0 Karma

LukeMurphey
Champion

Could you let me know if you see anything worth noting when you run the following search?

index=_internal (sourcetype=file_meta_data_modular_input OR (ExecProcessor "file_meta_data"  sourcetype=splunkd))

Tracebacks would be of particular interest.

0 Karma

jhankee
New Member

For the record - I'm running 1.1.1. Upon closer examination there is one directory that I didn't notice that splunk doesn't have access to, I selected a different directory and get the expected results. So.....my next question is, any recommendations on how to blacklist a directory? In my case, the directory contains files and several subdirectories, I want to index everything except the the directory called 'admin'

thanks!

0 Karma

jhankee
New Member

thanks for getting back to me. so nothing in splunkd after the startup relating to "file_meta_data", I found this in the "splunk/var/log/splunk/file_meta_data_modular_input.log", these are the last lines:

2017-03-22 18:19:02,856 INFO Time is later than filter, st_mtime=1490213086.0038838, must_be_later_than=None, path='/lawprd/scripts'
2017-03-22 18:19:02,856 INFO Completed retrieval of file data, count=202, path=/lawprd/scripts
2017-03-22 18:19:02,861 ERROR Execution failed
Traceback (most recent call last):
File "/lawprd/splunk/etc/apps/finapps_lawson_filemon/bin/file_info_app/modular_input.py", line 1320, in execute
self.do_run(in_stream, log_exception_and_continue=True)
File "/lawprd/splunk/etc/apps/finapps_lawson_filemon/bin/file_info_app/modular_input.py", line 1220, in do_run
input_config)
File "/lawprd/splunk/etc/apps/finapps_lawson_filemon/bin/file_meta_data.py", line 508, in run
result['time'] = time.strftime("%a %b %d %H:%M:%S %Y")
TypeError: list indices must be integers, not str

0 Karma

LukeMurphey
Champion

I'm struggling to figure this out.

Could you let me know a few things?

  1. What version of the app are you using?
  2. Does the input seem to fail on the same file? You might be able to tell by seeing if the logs report the same file before the input fails.
0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...