That is definitely a inefficiency of the app that primarily comes from the need to keep the app setup simple - otherwise the app would need to ask the admin/installer to specify the index. I will bring this up with these app owners. ... View more

I'd recommend you simply set vix.splunk.home.datanode to a directory that hunk would have access to on the Hadoop nodes. ... View more

Can you elaborate on what you mean by "My Hunk queries work fine on the virtual index when I run them in search "? Is timestamp extracted correctly when you run a simple search like "index=vix | head 10", but not when using timechart? Also, what data format are you working with? ... View more

That error message usually means that the mapper task was not able to run the search process (or the process crashed) on the Hadoop nodes. Can you check on the Hadoop nodes if a splunk package is properly installed in " /tmp/splunk/splunk-search1/"? (ie does /tmp/splunk/splunk-search1/splunk/bin/splunkd exitst?) Another possible reason for this would be that /tmp is sometimes mounted without execute permissions, in which case you should change the dir in the provider by setting vix.splunk.home.datanode to a dir with execute permissions and a few GB of space. ... View more

we're already tracking a similar enhancement request internally, for your reference SPL-94381 ... View more

Hmm, that's strange, does any of the following configs actually get the charset set right? Basically trying to see if there's either another source:: stanza that matches and sets the CHARSET (due to wildcarding) or an automatic sourcetype that overrides your setting [source::/data/com_data_log*/...] TIME_PREFIX="created": SHOULD_LINEMERGE=false NO_BINARY_CHECK=true CHARSET = Latin-1 # setting priority should ensure that the settings of this stanza are layered on *top* of other matches priority = 100 How about the following? [source::/data/com_data_log*/...] TIME_PREFIX="created": SHOULD_LINEMERGE=false NO_BINARY_CHECK=true sourcetype = my-sourcetype priority = 100 [my-sourcetype] CHARSET=Latin-1 ... View more

Two more things: a) make sure to add header.time in the required fields for the vix b) you'd need to fix TIME_FORMAT, probably need "%s%3N" (or maybe that's what you have and it doesn't render right here) ... View more

I'd also recommend revising the time extraction rule based on this answer - eval based timestamp extraction causes time based partition pruning to be disabled. ... View more

Please do ask a separate question here, as what you describe is supposed to work no matter what the search mode is - ie we should always extract the required (referenced in the search) fields. Please remember to include details about how the field extraction is defined (props/transforms.conf) ... View more

Yes it is. Are you seeing something different or experiencing any issues? ... View more

Two questions first though: is the data in Hive partitioned by time in HDFS? If so, what does the partition scheme look like, an example would be great what do the events returned by Hunk look like? Again, an anonymized example should be sufficient ... View more

@wingmanwingman which version of Hunk are you referring to? We don't publicise future release dates due to various reasons, but if your concern is about Hunk supporting HDP 2.2 - we do try to keep up with the cadence of the Hadoop vendors to add their newly released versions to the list of supported versions - in the vast majority of cases it's a matter of us running the full test suite on the latest public Hunk version against new Hadoop versions. ... View more

It should work for Hunk too ... View more

Given that | is not part of commonly used CSV formats out there you'd have to use Hunk's delimiter based KV extraction by using props/transforms.conf $SPLUNK_HOME/etc/apps/search/local/props.conf [my-sourcetype] REPORT-delim = pipe-extractor $SPLUNK_HOME/etc/apps/search/local/transforms.conf [pipe-extractor] FIELDS = field1,field2, .... DELIMS = | You can read a blog post about the technique here ... View more

Hunk is licensed based on the number of compute nodes (in Hadoop 1.x lingo TaskTracker nodes) ... View more

When using AWS our recommendation is to use the OEM/hourly integration which takes care of a lot of the underlying auth issues - more info here and here Regarding the error message you're seeing it's basically telling you that we cannot connect to S3 to fetch the data. To solve that you need to do one of the following: a. provide credentials in the vix path (notice using s3n instead of s3) input path: s3n://[awsAccessKeyId]:[awsSecretAccessKey]@bucket_name/path/to/data b. provide the credentials in the provider via "Additional settings": vix.fs.s3n.awsAccessKeyId = vix.fs.s3n.awsSecretAccessKey = then in your vix input path use s3n://bucket_name/path/to/data ... View more

How do I get the skipped reason? (I know events must be 100k per bucket to qualify, but how do I know if this was the reason for skipping?) Given that verification amounts to computing the summary again and comparing with what's already present (ie it's pretty expensive) we randomly pick buckets to verify, otherwise if we picked all we'd mind as well rebuild the summary. Regarding the suspended status - we suspend report acceleration if the size of a summary is above a certain threshold (summary size / bucket size > threshold) look at the following entries in savedsearches.conf.spec auto_summarize.max_summary_size = <unsigned int> - The minimum summary size when to start testing it's helpfulness - Defaults to 52428800 (5MB) auto_summarize.max_summary_ratio = <positive float> - The maximum ratio of summary_size/bucket_size when to stop summarization and deem it unhelpful for a bucket - NOTE: the test is only performed if the summary size is larger than auto_summarize.max_summary_size - Defaults to: 0.1 auto_summarize.max_disabled_buckets = <unsigned int> - The maximum number of buckets with the suspended summarization before the summarization search is completely - stopped and the summarization of the search is suspended for auto_summarize.suspend_period - Defaults to: 2 ... View more

I've tried both - there's gotta be some errors in search.log. First though, can you make sure that there are no network connectivity issues? e.g. telnet [mongo-db-host] [port] ... View more

While lookups help in forward translating the values, when we perform reverse lookups the search gets translated into (environmentid=123 OR environmentname=Test) , which unfortunately means that the search based partition pruning cannot help. We'll take that in as an enhancement request and do some research on how we can solve this problem. In the mean time one workaround that I think of would be using form searches to aid users in picking up an environment (show a user friendly string, but use the id to populate the search) ... View more

(Am I right in thinking that 3600 is the number of seconds in my bucket - one hour) Yes I really need some examples of using "earliest", "latest" as well as "earliest_time" and "latest_time" in hunk. Are you asking about how to specify earliest/latest in searches? If so you could simply use the time range picker which is the recommended way, or you can use search time modifiers ... View more