By "forwarding to Hunk" I assume you mean placing the data on HDFS (or another Hadoop-compatible file system) to be searched via a virtual index? The most common workflow is to add the log files to a regular Splunk index (using any input method, such as a Splunk forwarder), and set up that index to archive to HDFS. This means you can use a Splunk-managed index to get fast performance on the most recent data, and you can use Hadoop (via Hunk) to search the much larger pool of older data. You can find more information about getting data into Splunk indexes here: http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Getstartedwithgettingdatain You can find documentation about archiving here: https://docs.splunk.com/Documentation/Hunk/6.4.3/Hunk/ArchivingSplunkindexes If you want to copy the data directly to HDFS without first adding it to a regular Splunk index, you cannot currently do this via a Splunk forwarder. There are a number of third party tools that can be used to do this, e.g. Apache Flume (https://flume.apache.org/). ... View more

In your limits.conf file, try setting maintenance_period to something larger than the default, which is 1800 (i.e. 30 min). That default seems like it would explain the 10-20 minute lifespan you're seeing. If you change it to, say, 5400 (i.e. 90 min) do the DM files last longer? This won't fix your problem, but will help narrow down what is happening. ... View more

I believe you can handle this case via a calculated field. For example, you could add this to props.conf: [] EVAL-_time = strptime(timestamp, "%s") However "%s" expects a 10-digit epoch time string, and it looks like you have more precision than that, so you would probably need to use substr too. Since you can change the generating schema, that's probably a lot simpler. ... View more

I'm glad that it's working! For my own information, how did you change the field type? Is it a change in the sourcetype? ... View more

Last mod time usually is not used for anything. I was just making an inference about the events in those files. For each file in a virtual index, the "earliest time" and "latest time" should correspond to the timestamps of the first and last events in the file, respectively. They are used to determined whether Hunk needs to read that file. If the file presumably has no events in the time range of the query, then we might as well skip it. But the actual timestamp of each event in that file is determined according to the event's contents and the event's sourcetype. (The file's last modification time might be used if no other information is available.) An event whose timestamp is outside the time-range of the search will still be rejected, no matter what "et" and "lt" its file has. So if "et" and "lt" are not configured correctly, then Hunk will incorrectly reject the files containing events it wants, and correctly reject events from the files it reads. An offset determines the difference between a time read from a file's path, and the correct value. In your example stanza at the top, you use the same value for "vix.input.1.et.regex" as "vix.input.1.lt.regex", and the same value for "vix.input.1.et.format" as "vix.input.1.lt.format". Without an offset, et would be the same as lt for each file. Adding "vix.input.1.lt.offset = 3600" means that the presumed latest time for the file will be one hour after the earliest time. In my example, I just added 3 hours to both. ... View more

I notice that your directory name indicates that it contains files for hour=15, which I believe means 3pm-4pm, and your files have a last-modified date in the range 6pm-7pm, which would be hour=18. This might be because the process creating the files is using a different timezone than the process which moves the files to HDFS. Do the events in those files also have timestamps with hour=15 or hour=18? If the latter, then you need to either change the timezone, as Ledion suggested, or else use an earliest time offset of 10800 (i.e. 3 hours) and a latest time offset of 14400 (i.e. 4 hours). ... View more

I agree, statistics and brain health are at odds. Hopefully this will help. You can, of course, use the simpler solution and add a "head 100" command. It just won't spread the events around as evenly. ... View more

Great! If there are no other issues, can you "accept" the answer? ... View more

Hi Becky. That's correct. The default value still assumes you are using a Hadoop 1.x distribution, most of which came with Hive 0.x. If you create a new provider and change the drop-down to Yarn, it will change the value of that property to this: vix.env.HUNK_THIRDPARTY_JARS = $SPLUNK_HOME/bin/jars/thirdparty/common/avro-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/avro-mapred-1.7.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-compress-1.10.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/commons-io-2.4.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/libfb303-0.9.2.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/parquet-hive-bundle-1.5.0.jar,$SPLUNK_HOME/bin/jars/thirdparty/common/snappy-java-1.1.1.7.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive_1_2/hive-exec-1.2.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive_1_2/hive-metastore-1.2.1.jar,$SPLUNK_HOME/bin/jars/thirdparty/hive_1_2/hive-serde-1.2.1.jar This value refers to Hive 1.2.1. The reason I asked about the Hadoop version is that, unfortunately, Hive 1.x changed some method signatures, so the Splunk jar you use must be compiled against both the Hadoop version (1.x or 2.x) and the Hive version (0.x or 1.x). Assuming that you are using Hadoop 2.x, you should be able to just add the above line to you provider stanza, and hopefully that will fix your problem. Or you could change it to point to your own Hive jars (it looks like you are using 1.2.0, and Hunk 6.4 ships with 1.2.1), but I don't think it will be necessary. ... View more

It looks like your Hive version is 1.2.0. If vix.env.HUNK_THIRDPARTY_JARS is not specified in your provider, then you are using the default value, which probably assumes Hive version 0.12. Which Hadoop distribution are you using, and which version? And which version of Hunk are you using? ... View more

The reason I didn't suggest something like that before is that random() picks a number uniformly distributed from 0 to 2,147,483,647. Numbers ending in 00 to 47 show up one more time in that range than the numbers 48 to 99. In this case, there are 21,474,836 full sets of numbers and one incomplete set, so it will be very hard to detect the difference. But if you did: ...| eval rand=random() % 2000000000 where rand < 100 Then everything from 0 to 147,483,647 will show up twice as often as everything from 147,483,648 to 2,147,483,647, and 00 - 99 will be significantly over-represented, and you will get too many events. ... View more

In that case (i.e. speed is not important, you just need to make sure the query does not fail), then there are indeed some ways to do this. If I wanted a random sample of 1/1000th of my data-set, I could try this: index=foo | eval synthId=random()/2147483647 | table _time synthId | search synthId < 0.001 | table _raw —The middle command creates a new synthetic ID field called synthId. The description for the random() function (http://docs.splunk.com/Documentation/Splunk/6.4.0/SearchReference/CommonEvalFunctions) states that it creates a psuedo-random number between 0 and 2^31-1 = 2147483647, so random()/2147483647 creates a pseudo-random decimal number between 0.0 and 1.0. I want 1/1000th, so I take events with a value less than 1/1000. —The final table guarantees that filtering will happen on the task nodes, instead of bringing all events to the search head. Any “aggregating” command will do. In your case, we could try “search synthId < 0.00000005” (100 / 2 billion = 5*10-7). But it’s now reasonable to start worrying about round-off error, so it’s probably better if we calculate for ourselves that (2^31-1) * 100 / 2 billion = 107.4 , so we get: index=foo | eval synthId=random() | table _time synthId | search synthId <= 107 | …. This will get you a sample that is approximately the right size, is different every time, and is statically correct. If you want a sample of exactly the right size, you could get too many items and take the first 100, e.g.: index=foo | eval synthId=random() | table _time synthId | search synthId <= 150 | head 100 | …. This will slightly bias the sample based on iteration order. On the other hand, if you want a repeatable sample, you can use a hash instead of a random number. For example, you could use synthId=tonumber(substr(md5(_raw), -8), 16) to get a pseudo-random number between 0 and 4294967296 that will be the same for a given event, every time you calculate it. Then you can use all the tricks above. Hopefully this helps. ... View more

On this one, I believe the documentation is correct as it stands. We want to change the behavior of the original index, so that it does not delete its buckets when they roll to frozen. We don't need to change the behavior of the archiving index--it already knows to look for renamed buckets if they are there. The coldToFrozenScript property is actually a generalized mechanism that you can use without archiving. For instance, if you want to write a script that encrypts old buckets and transfers them via scp to another system, you trigger that script with this property. There is more information about it in the spec for the indexes.conf file: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Indexesconf ... View more

Thanks Becky! I believe you are correct on both counts. I'm bringing this to the attention of our Documentation team. ... View more

Do you mean 100 events fairly sampled from a virtual index? If so, the problem is that, in order to do it efficiently, you would need an external index of some sort. Since Hunk does not directly manage your data, it does not maintain an index. Hunk cannot know how many events are in each file in your virtual index without reading them, and it does not know the starting offsets of the events within a file, so it cannot do fair sampling without reading all the files that your search would hit. If you are willing to sample inefficiently, you could have Hunk read all the events and pipe them to a custom command that would do the sampling by randomly deciding whether to keep each event. I believe such a command is included in the Machine Learning Toolkit on Splunkbase. But if the whole reason you are sampling is to speed up the search, then this may not be what you want. ... View more

There is more information about this feature here: http://docs.splunk.com/Documentation/Hunk/latest/Hunk/Setanarchivescript Ordinarily, buckets will be archived long before they roll to frozen, so the system will delete frozen buckets normally. However, if you set vix.output.buckets.older.than to very large number, or if an indexer has a very heavy load, it is possible for a bucket to be deleted before it is archived. As a back-up feature, we provide a coldToFrozen script you can use which instead of deleting the bucket, just renames it. The bucket won't be searchable, but archiving will still find it, and will delete it once it has been successfully archived. Assuming that you are using this feature, these panels will show you how many frozen-and-renamed buckets have been found, and how many have been deleted, by the archiving process. ... View more

Thanks Becky. I've passed along a request to my Documentation group. ... View more

Sorry for losing track of this thread. I did not meant to leave your question unanswered so long. You can tell how a search was done via either the job inspector or the search log. To see the first, after running your search, on the results page go to Job -> Inspect Job. From there, you can also click on the link "search.log". If you run a search with a reporting command, like "index=my_vix | stats count", you will normally get a "mixed mode" search. This means Hunk will launch a Map Reduce job, and stream data while waiting for the job to complete. In the job inspector, you should see entries for:  erp..MR erp..stream.bytes which give you information about the Map Reduce and streaming parts of the search, respectively. If you look in search.log, you will find a lines that look like this (note the word "mixed"): 03-14-2016 11:25:27.508 INFO ExternalResultProvider - provider=, mode.config=report, mode.search=mixed ... 03-14-2016 11:25:29.225 INFO ERP. - SplunkMR$SearchHandler - Search mode: mixed Now if you run a search like "index=my_vix", it should run as a pure streaming search. You should see that the job inspector has a line for stream.bytes, but not for MR. In the search.log, you should find something like: 03-14-2016 11:25:17.403 INFO ExternalResultProvider - provider=, mode.config=report, mode.search=stream .... 03-14-2016 11:25:18.272 INFO ERP. - SplunkMR$SearchHandler - Search mode: stream Finally, we can force the original search ("index=my_vix | stats count") to be a pure reporting search (i.e. MR job but no streaming of data), by adding this line to the provider stanza: vix.splunk.search.mixedmode = 0 Now the job inspector will have an entry for MR, but not for stream.bytes, and the search.log should have these lines: 03-14-2016 11:32:20.462 INFO ExternalResultProvider - provider=, mode.config=report, mode.search=report 03-14-2016 11:32:21.599 INFO ERP. - SplunkMR$SearchHandler - Search mode: report Hope that helps. ... View more

Yes, Fast mode does work with Hunk. Fast mode disables field-discovery, so that Hunk will only know about index-time fields and required fields configured in the provider or index. This may speed up your searches, depending on the search itself, how the data is stored, the size of the data to searched, etc. ... View more

I'm not certain why you get this issue in some cases and not others, but it does sound like like your JVM is configured to disallow the cipher Hunk is using for SSL connections. If you prefer using OpenJDK, try editing this file: /lib/security/java.security And comment out the line that starts with: jdk.tls.disabledAlgorithms=.... And see whether that gets rid of the error. I think the issue with not getting any events back from your ORC/snappy files is likely a separate issue. ... View more

Based your error message, I'm guessing that this issue is not related to the data you are searching. Can you try searching some of the data that worked for you before, from the same Hunk instance? It sounds like the issue is a mismatch between the security features Hunk is trying to use, and the security features allowed by your JVM. Which version of Hunk are you using? And which Java version? ... View more

A few points: --I just realized we're ignoring the obvious way to handle this: change the vix.env.HADOOP_CLIENT_OPTS setting in the provider. That will overwrite any value inherited from the user environment. --To test what value is being used, you can edit your hadoop script. This will usually be in /bin/hadoop. You can insert a line that looks like: echo $HADOOP_CLIENT_OPTS > ~/my_hadoop_client_opts.txt Then run a search in Hunk, and see what that file contains. --Depending on your Hadoop distribution, you may need to set $HADOOP_OPTS instead. Hope that helps. ... View more