topic Hunk and virtual indexes in All Apps and Add-ons

Hunk and virtual indexes

jwalzerpitt — Fri, 06 Nov 2015 20:38:59 GMT

We are using Hunk in a POC and the way our HDFS file structure is set up is we have a folder for every date, so for example our firewall logs are set up like:

/logs/fwsm (parent dir)
--/2015-11-06
--/2015-11-05
--/2015-11-04
…
--/2015-10-31

We set up a main virtual index at the parent so we’re searching all logs under /logs/fwsm. An issue we’re running into is there is a need to search per day so I find myself creating a virtual index for every date, and with that I had two questions

• Is there any other way to search by date using the virtual indexes?
• Is there any limit to the amount of virtual indexes that can be created (as one can imagine, this will get real ugly when we start creating virtual indexes by date for multiple sourcetypes)?

Thx

Re: Hunk and virtual indexes

rdagan_splunk — Mon, 09 Nov 2015 18:30:04 GMT

Have you tried to use the Time Capturing Regex as shown in this document?
http://docs.splunk.com/Documentation/Hunk/latest/Hunk/Addavirtualindex

Re: Hunk and virtual indexes

jwalzerpitt — Mon, 09 Nov 2015 18:53:20 GMT

I did not see that option/document - I assume the time capturing regex means I'd be able to search by date/time within the main virtual index? Am I basing the regex on the file structure, or the log's date/time format?

Thx

Re: Hunk and virtual indexes

kschon_splunk — Mon, 09 Nov 2015 19:43:38 GMT

Yes, this will allow you to efficiently search by time within a single virtual index. The capturing regex will allow Hunk to choose which files to search based on the directories they are in, so it should match that, not the log structure.

Re: Hunk and virtual indexes

rdagan_splunk — Mon, 09 Nov 2015 19:47:38 GMT

The option to capture the Regex is part of the Virtual Index UI, Select the Customize Timestamp Format button.
Your assumption is correct, once you set it up you can use the search and the search time picker to select a specific day within the HDFS data.
Here is an example:
path = /logs/fwsm/...
accept =
regex = .?/fwsm/(\d+)-(\d+)-(\d+)/.
format = yyyyMMdd

Re: Hunk and virtual indexes

jwalzerpitt — Mon, 09 Nov 2015 20:06:34 GMT

Thx for the reply and info

Re: Hunk and virtual indexes

jwalzerpitt — Mon, 09 Nov 2015 20:12:55 GMT

Thx

Apologies as the actual dir structure is /LogCentral/Firewall, so I set my 'Time capturing regex' as follows - ?/Firewall/(d+)-(d+)-(d+)/. (leaving Time Format, Time Adjustment, and Time Zone untouched), but when I run a query - index=fwsm - using the Date/Time picker (I'm selecting Date Range|Before 11/3/2015), I'm getting 'No results found'

Thx

Re: Hunk and virtual indexes

kschon_splunk — Tue, 10 Nov 2015 19:22:58 GMT

First, make certain there is a '.' char in front of your leading '?' char. (I realize that may just be a typo.)

Also, try setting the format to yyyyMMdd.

Re: Hunk and virtual indexes

jwalzerpitt — Wed, 11 Nov 2015 18:22:08 GMT

I added the '.' in front of the time leading '?', and added yyyyMMdd to the time format and it worked! I can't thank you enough!!

Greatly appreciated!

Re: Hunk and virtual indexes

kschon_splunk — Wed, 11 Nov 2015 18:40:15 GMT

Happy it worked!

Re: Hunk and virtual indexes

jwalzerpitt — Wed, 02 Dec 2015 21:08:57 GMT

Was hoping to revisit this issue if possible as I'm seeing some weirdness with the time regex.

We have three directories on HDFS:

• /LogCentral/Firewall
• /LogCentral/ISE
• /LogCentral/ WindowsEvent

I have the following regex applied to our Firewall virtual index and I can use the time picker no problem (slightly modified from the original recommendation):

.?/Firewall/(d+)-(d+)-(d+)/.?)

However, applying the same format to the other two logs (below) I get no events at all no matter what dates I select in the time picker, yet I'm using the same format.

.?/ISE/(d+)-(d+)-(d+)/.?)
.?/WindowsEvent/(d+)-(d+)-(d+)/.?)

Tried the following regex and got a match on regex101.com:

.+ISE/(d+)-(d+)-(d+)

Yet when I enter that and try and run a search, it errors out:

[cdhprovider] Error while running external process, return_code=255. See search.log for more info
[cdhprovider] IOException - No input paths specified in job.

Thx

Re: Hunk and virtual indexes

jwalzerpitt — Wed, 09 Dec 2015 14:29:24 GMT

Working with Splunk Support, the solution was to change the 'Time Range' setting under the Time section to 1 day. Once this change was applied, the date/time picker worked.

Thx for everyone's feedback and help