Dashboards & Visualizations

Hive partitions and timepicker

pierre_corbel
Engager

Hello,

I got a partitionned Hive table by field dt (in the YYYYMMDD format)

Example :

/mywarehouse/my.db/foo/dt=20160207/part-m-00000

I got a Hunk Index on top of that :

[foo]
vix.provider = my_hive_provider
vix.input.1.path = /mywarehouse/my.db/foo/...
vix.input.1.splitter.hive.dbname = my
vix.input.1.splitter.hive.tablename = foo
vix.input.1.splitter.hive.fileformat = orc

The problem is, when I select a date from the timepicker, I would like Hunk to go directly to the dt partition (because now it makes a full scan of the DB)

I try to modify the following:

vix.input.1.path = /mywarehouse/my.db/foo/${dt}/...

and to add in props.conf :

[foo]
TIME_PREFIX="dt":
TIME_FORMAT = %Y%m%d

[source::.../mywarehouse/my.db/foo/*/*]
sourcetype = foo

But none of it matter...

Could someone lend me a hand on that?

Thanks

0 Karma
1 Solution

pierre_corbel
Engager

I finally did like with HDFS Indexes, i.e. add in indexes.conf

vix.input.1.et.format = yyyyMMdd
vix.input.1.et.regex = /mywarehouse/my.db/foo/dt=(\d+)
vix.input.1.lt.format = yyyyMMdd
vix.input.1.lt.offset = 86400
vix.input.1.lt.regex = /mywarehouse/my.db/foo/dt=(\d+)

View solution in original post

0 Karma

pierre_corbel
Engager

I finally did like with HDFS Indexes, i.e. add in indexes.conf

vix.input.1.et.format = yyyyMMdd
vix.input.1.et.regex = /mywarehouse/my.db/foo/dt=(\d+)
vix.input.1.lt.format = yyyyMMdd
vix.input.1.lt.offset = 86400
vix.input.1.lt.regex = /mywarehouse/my.db/foo/dt=(\d+)

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Instead of vix.input.1.path = /mywarehouse/my.db/foo/${dt}/...
try
In the VIX UI, select the option to customize timestamp format
See this document: http://docs.splunk.com/Documentation/Hunk/latest/Hunk/Addavirtualindex

0 Karma
Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

  Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

Splunk Decoded: Business Transactions vs Business IQ

It’s the morning of Black Friday, and your e-commerce site is handling 10x normal traffic. Orders are flowing, ...

Fastest way to demo Observability

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting ...