All Apps and Add-ons

Can you use regex in file_path

nhicks
Explorer

Is it possible to use regex in the file_path setting for the File/Directory Information Input app.

Here is what I am trying to get to

  • E:\Folder\Folder2\20160808\InvalidFile\*.cdi_Error1
  • E:\Folder\Folder2\20160809\InvalidFile\*.cdi_Error1
  • E:\Folder\Folder2\20160810\InvalidFile\*.cdi_Error1 etc.

I have tried

  • file_path = E:\Folder\Folder2\*\InvalidFiles\*.cdi_Error1
  • file_path = E:\Folder\Folder2\...\InvalidFiles\*.cdi_Error1

I have also tried several different regex options for *.cdi_Error1. To many to list.

When I try the above options I am receiving this message in the file_meta_data_modular_input.log

  • 2016-08-26 10:34:45,864 WARNING Unable to access path="E:\Folder\Folder2\*\InvalidFiles\*.cdi_Error1", reason="[Error 123] The filename, directory name, or volume label syntax is incorrect: 'E:\Folder\Folder2\*\InvalidFiles\*.cdi_Error1'"
  • 2016-08-26 10:34:45,864 INFO Completed retrieval of file data, count=0, path=E:\Folder\Folder2\*\InvalidFiles\*.cdi_Error1

Not sure why the 2nd message shows it was complete but it definitely did not pull in the information.

I also tried using whitelist

  • file_path = E:\Folder\Folder2
  • recurse = 1
  • whitelist = *.cdi_Error1

But then I get this message

  • 2016-08-26 12:54:28,592 ERROR The input stanza 'file_meta_data://APPNAME' is invalid: The parameter 'whitelist' is not a valid argument

I know that I can set the file_path setting to E:\Folder\Folder2 and set recurse = 1 but this then pulls in some 50000 files and I only need the .cdi_Error1 files.

I also know that if I pull in the 50000 files I can just use logic in the search parameters to filter out only the .cdi_Error1 files but this server is already heavily used and I do not want to put more stress on it by grabbing metadata for 50000 files. Plus its just a lot of data that I do not need to index.

I did try restarting splunk on both the indexer, search head and forwarder many times but it did not help.

Any help is appreciated. Thank you

0 Karma
1 Solution

LukeMurphey
Champion

Regular expressions and wild-cards are not currently supported. That is a good idea though. I created an enhancement request: http://lukemurphey.net/issues/1453 for it.

View solution in original post

0 Karma

LukeMurphey
Champion

Regular expressions and wild-cards are not currently supported. That is a good idea though. I created an enhancement request: http://lukemurphey.net/issues/1453 for it.

0 Karma

nhicks
Explorer

Thank you Luke for the reply and the enhancement request. This will be very helpful for us if it is implemented.

0 Karma

nhicks
Explorer

It was a typo. Sorry, I was trying to make sure I typed it correctly and missed that. The file name and the error in the log file are the exact same.

0 Karma

inventsekar
SplunkTrust
SplunkTrust

ohk.. that File/Directory Information Input was built by Luke Murphey.
https://splunkbase.splunk.com/app/2776/

as per the above reply from Luke Murphey, Regular expressions and wild-cards are not currently supported.

0 Karma

inventsekar
SplunkTrust
SplunkTrust

The actual filename says ".....\InvalidFile\" and the error msg says ..\InvalidFiles\
was it a typo?!?!

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...