All Apps and Add-ons

Why do I get different performance in Hunk based on how I specify time?

EricLloyd79
Builder

We have a setup with MapR File System and Splunk Hadoop for Analytics (HUNK) ...MapRFS is using an NFS mount to have all logs centralized.

We have our virtual index set up with this file directory format:
maprdata/sourcetype/year/month/day/hour/foo.log

I have included the indexes. conf file at the bottom without the provider.

When I run a basic index search such as index=mapr1 | stats count, for the last 3 hours using Splunk Web, it runs optimally. It finds the relevant events quickly and then finished. This is ideal.

When I run the same search for a specific hour in the past (say 3 hours ago) either using the Splunk Web or "earliest=-3h@h latest=-2h@h", it will search through a very large number of events and then find events finally (and its not the correct number of events even).

[mapr1]
vix.input.1.accept =
vix.input.1.et.format = yyyyMMddHH
vix.input.1.et.regex = /user/mapr/maprdata/.?/(\d+)/(\d+)/(\d+)/(\d+)/.
vix.input.1.lt.format = yyyyMMddHH
vix.input.1.lt.offset = 3600
vix.input.1.lt.regex = /user/mapr/maprdata/.?/(\d+)/(\d+)/(\d+)/(\d+)/.
vix.input.1.path = /user/mapr/maprdata/${sourcetype}/...
vix.provider = maproly

0 Karma
1 Solution

EricLloyd79
Builder
0 Karma

EricLloyd79
Builder

This was resolved in another question that I asked which can be found here:
https://answers.splunk.com/answers/669336/need-help-optimizing-search-in-hunk.html?childToView=67861...

0 Karma

soumyasaha25
Contributor

are you receiving the data in a json , generated by some application, in that case can you change the timestamp field at the json schema from string to long.
probably your issue is like below:
your Timestamp:
timestamp: "276257257257265"
Splunk expects:
timestamp: 276257257257265

Alternatively, you can handle this via a calculated field. For example, you could add this to props.conf:
EVAL-_time = strptime(timestamp, "%s")
However "%s" expects a 10-digit epoch time string, so you would probably need to use substr/trim too. Hence in this case if possible, it is best to get the timestamp type changed before it reaches splunk.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...