What can I put in a vix.input.1.et.format so that hunk searches only look at a limited number of files

[alexmc]

I fear this is a question where I must have missed the answer in the manual

We have a directory tree of data in HDFS looking like

/data/systemname/tablename/2015/02/27/05

so I want to create a virtual index such that Hunk knows that certain directories are associated with certain dates only. (Partition pruning I think)

My understanding is that I need to define my virtual index something like this... Is this right?

vix.input.1.et.regex = /(\d+)/(\d+)/(\d+)/(\d+)
vix.input.1.et.format = YYYYmmddHH
vix.input.1.et.offset = 0
vix.input.1.lt.regex = /(\d+)/(\d+)/(\d+)/(\d+)
vix.input.1.lt.format = YYYYmmddHH
vix.input.1.lt.offset = 3600

(Am I right in thinking that 3600 is the number of seconds in my bucket - one hour)

I really need some examples of using "earliest", "latest" as well as "earliest_time" and "latest_time" in hunk.

Can I do this AND set _time by looking at a UTC field within my data?

Thanks!

[Ledion_Bitincka]

(Am I right in thinking that 3600 is the number of seconds in my bucket - one hour)
Yes

I really need some examples of using "earliest", "latest" as well as "earliest_time" and "latest_time" in hunk.

Are you asking about how to specify earliest/latest in searches? If so you could simply use the time range picker which is the recommended way, or you can use search time modifiers

[hyan_splunk · Accepted solution]

The "Customize timestamp format" checkbox on virtual index page is for configuring timestamp and partition pruning by time.

There are examples under each input box in that "Time" section. Here is the setting for your case:

vix.input.1.et.regex=/data/systemname/tablename/(\d+)/(\d+)/(\d+)/(\d+)
vix.input.1.et.format=yyyyMMddHH
vix.input.1.et.offset=0
vix.input.1.lt.regex=/data/systemname/tablename/(\d+)/(\d+)/(\d+)/(\d+)
vix.input.1.lt.format=yyyyMMddHH
vix.input.1.lt.offset=3600