All Apps and Add-ons

Can you use regex in file_path

Explorer

Is it possible to use regex in the file_path setting for the File/Directory Information Input app.

Here is what I am trying to get to

  • E:\Folder\Folder2\20160808\InvalidFile\*.cdi_Error1
  • E:\Folder\Folder2\20160809\InvalidFile\*.cdi_Error1
  • E:\Folder\Folder2\20160810\InvalidFile\*.cdi_Error1 etc.

I have tried

  • filepath = E:\Folder\Folder2\*\InvalidFiles\*.cdiError1
  • filepath = E:\Folder\Folder2\...\InvalidFiles\*.cdiError1

I have also tried several different regex options for *.cdi_Error1. To many to list.

When I try the above options I am receiving this message in the filemetadatamodularinput.log

  • 2016-08-26 10:34:45,864 WARNING Unable to access path="E:\Folder\Folder2\*\InvalidFiles\*.cdiError1", reason="[Error 123] The filename, directory name, or volume label syntax is incorrect: 'E:\Folder\Folder2\*\InvalidFiles\*.cdiError1'"
  • 2016-08-26 10:34:45,864 INFO Completed retrieval of file data, count=0, path=E:\Folder\Folder2\*\InvalidFiles\*.cdi_Error1

Not sure why the 2nd message shows it was complete but it definitely did not pull in the information.

I also tried using whitelist

  • file_path = E:\Folder\Folder2
  • recurse = 1
  • whitelist = *.cdi_Error1

But then I get this message

  • 2016-08-26 12:54:28,592 ERROR The input stanza 'filemetadata://APPNAME' is invalid: The parameter 'whitelist' is not a valid argument

I know that I can set the filepath setting to E:\Folder\Folder2 and set recurse = 1 but this then pulls in some 50000 files and I only need the .cdiError1 files.

I also know that if I pull in the 50000 files I can just use logic in the search parameters to filter out only the .cdi_Error1 files but this server is already heavily used and I do not want to put more stress on it by grabbing metadata for 50000 files. Plus its just a lot of data that I do not need to index.

I did try restarting splunk on both the indexer, search head and forwarder many times but it did not help.

Any help is appreciated. Thank you

0 Karma
1 Solution

Champion

Regular expressions and wild-cards are not currently supported. That is a good idea though. I created an enhancement request: http://lukemurphey.net/issues/1453 for it.

View solution in original post

0 Karma

Champion

Regular expressions and wild-cards are not currently supported. That is a good idea though. I created an enhancement request: http://lukemurphey.net/issues/1453 for it.

View solution in original post

0 Karma

Explorer

Thank you Luke for the reply and the enhancement request. This will be very helpful for us if it is implemented.

0 Karma

Explorer

It was a typo. Sorry, I was trying to make sure I typed it correctly and missed that. The file name and the error in the log file are the exact same.

0 Karma

Champion

ohk.. that File/Directory Information Input was built by Luke Murphey.
https://splunkbase.splunk.com/app/2776/

as per the above reply from Luke Murphey, Regular expressions and wild-cards are not currently supported.

0 Karma

Champion

The actual filename says ".....\InvalidFile\" and the error msg says ..\InvalidFiles\
was it a typo?!?!

0 Karma