topic Re: Hunk timestamp problem in All Apps and Add-ons

Hunk timestamp problem

felipetavares — Wed, 04 May 2016 20:30:22 GMT

Hello there guys!

We have here a Yarn Hadoop cluster running fine and also we have Hunk 6.4.0 running, which connects to our haddop cluster and can make queries on it.
If I do a query like this: index=fs_vindex | dedup computerName | table computerName on All Time, I get a list with all servers that are sending logs to the location specified for this vindex.
But, if I change the time picker to a specified amount of time (today or whatever), the query won't work and, worse, I don't see any errors at the logs.

I've searched a lot, but I still can't find a solution, hope anyone can help me.
Here comes the configuration files:

indexes.conf

[fs_vindex]
vix.description = Index virtual do FS
vix.input.1.et.format = yyyyMMddHH
vix.input.1.et.regex = /storage/data/BR_SIEM_success.EventLogFS/hourly/(\d{4})/(\d{2})/(\d{2})/(\d{2})/.*
vix.input.1.lt.format = yyyyMMddHH
vix.input.1.lt.regex = /storage/data/BR_SIEM_success.EventLogFS/hourly/(\d{4})/(\d{2})/(\d{2})/(\d{2})/.*
vix.input.1.path = /storage/data/BR_SIEM_success.EventLogFS/hourly/...
vix.provider = hadoop_producao
vix.provider.description = Ambiente
vix.input.1.lt.offset = 3600

[provider:hadoop_producao]
vix.command.arg.3 = $SPLUNK_HOME/bin/jars/SplunkMR-hy2.jar
vix.description = Ambiente
vix.env.HADOOP_HOME = /usr/lib/hadoop
vix.env.JAVA_HOME = /opt/java
vix.family = hadoop
vix.fs.default.name = hdfs://srvmasterofpuppets:8020
vix.mapreduce.framework.name = yarn
vix.output.buckets.max.network.bandwidth = 0
vix.splunk.home.hdfs = /user/hunk
vix.hadoop.security.authorization = 0
vix.splunk.impersonation = 0

Thank you in advance!

Re: Hunk timestamp problem

Ledion_Bitincka — Wed, 04 May 2016 20:37:54 GMT

Are the events time stamped correctly? ie does the event timestamp correspond to the path of the file where the event is found?

Unrelated, I'd recommend using the following more efficient search to get a table of computerName (and a count of events)

index=fs_vindex | stats count BY computerName

Re: Hunk timestamp problem

felipetavares — Wed, 04 May 2016 20:50:11 GMT

Hello there @Ledion! Yes, there are events and the timestamp is correct, I can see them when I run the All Time query and when I navigate with the "Explore Data".
I didn't use the stats, because I don't need the count of event, I only need the name of each server, is it really faster? Is it because it's only one operation and the way I'm doing I'm using two?

Thanks

Re: Hunk timestamp problem

Ledion_Bitincka — Wed, 04 May 2016 22:00:04 GMT

I assume your don't see any events if you run a search over a given time period (hour, day, week etc) - correct? If so this is generally an indication that the time extracted from the events is different from the one extracted from the path. Is there a timezone difference between the path and Splunk's or your user's setting in Splunk?

I highly recommend that you do specify the timezone of the extracted time from the path

vix.input.[N].et/lt.timezone - the timezone in which to interpret the extracted time. E.g. "America/Los_Angeles" or "GMT-8:00",

see http://docs.oracle.com/javase/7/docs/api/java/util/TimeZone.html for more info on the supported strings

Re: Hunk timestamp problem

felipetavares — Thu, 05 May 2016 16:49:23 GMT

Great Ledion, now I got what you said, going to check that and report later!

Re: Hunk timestamp problem

felipetavares — Mon, 09 May 2016 18:27:01 GMT

Hello @Ledion.
I've played with the timestamp adjustment, but without success.
Our data is on BRT Timezone (GMT -3), but our HDFS Timezone is GMT, so I can see that our data has it's directory, on the same Timezone that the event's hour, but the files written on HDFS have +3 hours (GMT).
So, at the Virtual Index configuration, I've used the GMT, GMT +3, GMT -3 and some others, but none of them seemed to work for us.
Maybe I'm doing the time extraction on a wrong way?

vix.input.1.et.format = yyyyMMddHH
vix.input.1.et.regex = /storage/data/BR_SIEM_success.EventLogFS/hourly/(\d{4})/(\d{2})/(\d{2})/(\d{2})/.*
vix.input.1.lt.format = yyyyMMddHH
vix.input.1.lt.regex = /storage/data/BR_SIEM_success.EventLogFS/hourly/(\d{4})/(\d{2})/(\d{2})/(\d{2})/.*

I think that I'm missing something, any advices will help us a lot!
Ahh, this is what I get on "Explore Data":

storage/data/BR_SIEM_success.EventLogFS/hourly/2016/05/09/15/
Type    Name    Owner   Size    Permissions Last Modified Time 
BR_SIEM_success.0.0.66.73582308.1462816800000.avro  hdfs    56.32 KB    rw-r--r--   May 9, 2016 6:01:36 PM
BR_SIEM_success.10.10.21.76501680.1462816800000.avro    hdfs    19.82 KB    rw-r--r--   May 9, 2016 6:01:49 PM
BR_SIEM_success.11.11.42.75316419.1462816800000.avro    hdfs    37.62 KB    rw-r--r--   May 9, 2016 6:01:59 PM
BR_SIEM_success.2.2.23.80555652.1462816800000.avro  hdfs    20.89 KB    rw-r--r--   May 9, 2016 6:02:36 PM
BR_SIEM_success.3.3.92.80099611.1462816800000.avro  hdfs    77.57 KB    rw-r--r--   May 9, 2016 6:01:49 PM
BR_SIEM_success.5.5.23.78514890.1462816800000.avro  hdfs    20.66 KB    rw-r--r--   May 9, 2016 6:01:54 PM
BR_SIEM_success.7.7.21.78649173.1462816800000.avro  hdfs    19.22 KB    rw-r--r--   May 9, 2016 6:01:59 PM
BR_SIEM_success.8.8.44.74151067.1462816800000.avro  hdfs    38.13 KB    rw-r--r--   May 9, 2016 6:01:54 PM

Re: Hunk timestamp problem

kschon_splunk — Mon, 09 May 2016 22:56:22 GMT

I notice that your directory name indicates that it contains files for hour=15, which I believe means 3pm-4pm, and your files have a last-modified date in the range 6pm-7pm, which would be hour=18. This might be because the process creating the files is using a different timezone than the process which moves the files to HDFS. Do the events in those files also have timestamps with hour=15 or hour=18? If the latter, then you need to either change the timezone, as Ledion suggested, or else use an earliest time offset of 10800 (i.e. 3 hours) and a latest time offset of 14400 (i.e. 4 hours).

Re: Hunk timestamp problem

felipetavares — Tue, 10 May 2016 17:19:59 GMT

Hello there @kschon! The events have hour=15. As I said, I tried to use timezones GMT -3, GMT and even GMT +3, but it won't work. About the time offsets, it is used to consider the difference between modification time and event time?
Maybe you could help me?
I can't find more detailed information about this at the knowledge base.

Re: Hunk timestamp problem

kschon_splunk — Wed, 11 May 2016 01:51:32 GMT

Last mod time usually is not used for anything. I was just making an inference about the events in those files.

For each file in a virtual index, the "earliest time" and "latest time" should correspond to the timestamps of the first and last events in the file, respectively. They are used to determined whether Hunk needs to read that file. If the file presumably has no events in the time range of the query, then we might as well skip it. But the actual timestamp of each event in that file is determined according to the event's contents and the event's sourcetype. (The file's last modification time might be used if no other information is available.) An event whose timestamp is outside the time-range of the search will still be rejected, no matter what "et" and "lt" its file has. So if "et" and "lt" are not configured correctly, then Hunk will incorrectly reject the files containing events it wants, and correctly reject events from the files it reads.

An offset determines the difference between a time read from a file's path, and the correct value. In your example stanza at the top, you use the same value for "vix.input.1.et.regex" as "vix.input.1.lt.regex", and the same value for "vix.input.1.et.format" as "vix.input.1.lt.format". Without an offset, et would be the same as lt for each file. Adding "vix.input.1.lt.offset = 3600" means that the presumed latest time for the file will be one hour after the earliest time. In my example, I just added 3 hours to both.

Re: Hunk timestamp problem

felipetavares — Wed, 11 May 2016 17:53:13 GMT

Hello again @kschon!

Well, I've got what you mean about the offset, well, my files will hold only some minutes, as inside each dir we have 1 hour (as the configuration that follows below).
Even though, I've configured the et.offset to 3h and the the lt.offset to 4h, also used GMT and GMT-3 timezones, but it still doesn't work. So I played with time for the offset entrances, but still haven't got any improvement.

I also saw here something that I believe that was an error: I had 5~ virtual indexes, and all of them had the same "vix.xxx.1.xxx.xxx", so I thought that this should be causing the problems. I've excluded all the virtual indexes, except for below.

[wintel_fs_index]
vix.description = Index virtual do FS
vix.input.1.et.format = yyyyMMddHH
vix.input.1.et.regex = /storage/data/BR_SIEM_success.EventLogFS/hourly/(\d{4})/(\d{2})/(\d{2})/(\d{2})/.*
vix.input.1.et.offset = 3600
vix.input.1.et.timezone = America/Sao_Paulo
vix.input.1.lt.format = yyyyMMddHH
vix.input.1.lt.regex = /openbus/data/BR_SIEM_success.EventLogFS/hourly/(\d{4})/(\d{2})/(\d{2})/(\d{2})/.*
vix.input.1.lt.offset = 3600
vix.input.1.path = /openbus/data/BR_SIEM_success.EventLogFS/hourly/...
vix.input.1.lt.timezone = America/Sao_Paulo
vix.provider = hadoop_producao
vix.provider.description = Ambiente

But, I'm still not being able to generate outputs from time ranged queries,

Re: Hunk timestamp problem

felipetavares — Thu, 02 Jun 2016 19:03:16 GMT

Hello there @kschon and @Ledion (and for anyone else that should come to this problem)

I've finally solved the problem: the reason that Hunk wasn't able to parse the timestamp field is because it had the string type. I've changed it to long now and as a trick of magic, Hunk started to understand and parse it.
Hope I can help anyone else facing this problem.

Best Regards!

Re: Hunk timestamp problem

kschon_splunk — Mon, 06 Jun 2016 20:27:34 GMT

I'm glad that it's working! For my own information, how did you change the field type? Is it a change in the sourcetype?

Re: Hunk timestamp problem

burwell — Mon, 06 Jun 2016 20:42:23 GMT

I've been having timestamp problems too and see my timestamps are strings as well. Please do share how you worked around this issue exactly. Thanks!

Re: Hunk timestamp problem

felipetavares — Thu, 09 Jun 2016 22:22:24 GMT

Hello there @kschon!
No, I've changed it "before" hunk, on my data source, that is a json generated by our Big Data bus, so I only changed the field schema from string to long 🙂

Re: Hunk timestamp problem

felipetavares — Thu, 09 Jun 2016 22:25:50 GMT

Hello @burwell, sorry for the late answer, I was traveling and couldn't answer before.
Well, my sources here are some jsons generated by some other softwares that I control.
So, to fix this, we change the timestamp field at the json schema from string to long. For example:
We had:
timestamp: "276257257257265"
Now we have:
timestamp: 276257257257265
I don't really know if this kind of change can be done on Splunk before the timestamp recognition phase, maybe @kschon could tell us 😄

Re: Hunk timestamp problem

kschon_splunk — Fri, 10 Jun 2016 00:11:31 GMT

I believe you can handle this case via a calculated field. For example, you could add this to props.conf:

[]
EVAL-_time = strptime(timestamp, "%s")

However "%s" expects a 10-digit epoch time string, and it looks like you have more precision than that, so you would probably need to use substr too. Since you can change the generating schema, that's probably a lot simpler.