Dashboards & Visualizations

Hive partitions and timepicker

pierre_corbel
Engager

Hello,

I got a partitionned Hive table by field dt (in the YYYYMMDD format)

Example :

/mywarehouse/my.db/foo/dt=20160207/part-m-00000

I got a Hunk Index on top of that :

[foo]
vix.provider = my_hive_provider
vix.input.1.path = /mywarehouse/my.db/foo/...
vix.input.1.splitter.hive.dbname = my
vix.input.1.splitter.hive.tablename = foo
vix.input.1.splitter.hive.fileformat = orc

The problem is, when I select a date from the timepicker, I would like Hunk to go directly to the dt partition (because now it makes a full scan of the DB)

I try to modify the following:

vix.input.1.path = /mywarehouse/my.db/foo/${dt}/...

and to add in props.conf :

[foo]
TIME_PREFIX="dt":
TIME_FORMAT = %Y%m%d

[source::.../mywarehouse/my.db/foo/*/*]
sourcetype = foo

But none of it matter...

Could someone lend me a hand on that?

Thanks

0 Karma
1 Solution

pierre_corbel
Engager

I finally did like with HDFS Indexes, i.e. add in indexes.conf

vix.input.1.et.format = yyyyMMdd
vix.input.1.et.regex = /mywarehouse/my.db/foo/dt=(\d+)
vix.input.1.lt.format = yyyyMMdd
vix.input.1.lt.offset = 86400
vix.input.1.lt.regex = /mywarehouse/my.db/foo/dt=(\d+)

View solution in original post

0 Karma

pierre_corbel
Engager

I finally did like with HDFS Indexes, i.e. add in indexes.conf

vix.input.1.et.format = yyyyMMdd
vix.input.1.et.regex = /mywarehouse/my.db/foo/dt=(\d+)
vix.input.1.lt.format = yyyyMMdd
vix.input.1.lt.offset = 86400
vix.input.1.lt.regex = /mywarehouse/my.db/foo/dt=(\d+)

0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Instead of vix.input.1.path = /mywarehouse/my.db/foo/${dt}/...
try
In the VIX UI, select the option to customize timestamp format
See this document: http://docs.splunk.com/Documentation/Hunk/latest/Hunk/Addavirtualindex

0 Karma
Get Updates on the Splunk Community!

Splunk + ThousandEyes: Correlate frontend, app, and network data to troubleshoot ...

 Are you tired of troubleshooting delays caused by siloed frontend, application, and network data? We've got a ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

🔐 Trust at Every Hop: How mTLS in Splunk Enterprise 10.0 Makes Security Simpler

From Idea to Implementation: Why Splunk Built mTLS into Splunk Enterprise 10.0  mTLS wasn’t just a checkbox ...