topic Time Conversion and indexing in Deployment Architecture

Time Conversion and indexing

RobertRi — Mon, 21 Jan 2013 14:44:11 GMT

We index logfiles which have date time strings on the beginning of each event in hex format.

50FD316C = Mon Jan 21 13:15:40 2013

Is it possible to change this hex datetimestamp to an readable timestamp with default splunk features?

Or is it possible to execute a script on each event and change this timestamps in the _raw field?

Thanks
Rob

Re: Time Conversion and indexing

jonuwz — Mon, 21 Jan 2013 21:29:45 GMT

Do you want to use the hex code as the timestamp for the event ?
Thats not possible without pre-processing the event beforehand ( before splunk sees it )

If all you want to do is convert the code to a readable time after the event has been indexed then assuming the field you want to convert is called 'hex_time'

... | eval my_time=tonumber(hex_field,16) | convert ctime(my_time) | table my_time

1st part converts the hex to epoch seconds

2nd part formats the time into a readable date format

If the 'hex_field' is not automatically extracted, this'll do it (based on the fact the hex is at the start of the line) :

... | rex ^(?<hex_field>\S+) | ...

Update

If you need to convert the hex to a timestamp beforehand, a (very) hacky way to achieve this would be to configure syslog-ng to follow the files, and run each line through a program.

i.e. (in syslog-ng.conf)

source ibm_file {
    file("/opt/IBM/log/kbb.log" follow_freq(1)
    flags(no-parse));
};

destination hex_converter { program("/home/user/hex_converter.pl"); };
log { source(ibm_file); destination(hex_converter); };

the 'hex_converter' program would read from STDIN, convert the hex, and output the converted line to a file that splunk would then index ( or write it out to syslog, or send it as a TCP stream to splunk )

IBM normally ship a binary called ras1log that'll you could use in the hex_converter before redirecting to another file.

Re: Time Conversion and indexing

RobertRi — Tue, 22 Jan 2013 06:00:10 GMT

Thanks jonuwz for your answer.
Yes I want to use this hex date for the actual event.
I'm also consider to make the time conversion on the forwarder side,
cause in case of debuging, there will be produced a lot of events on top.Do you see a possiblity to do so?Thanks
Robert

Re: Time Conversion and indexing

jonuwz — Tue, 22 Jan 2013 08:20:41 GMT

How do the logs get to the indexer ? syslog-ng ? universal forwarder ? Whats the application (looks IBM .. ) ?

Re: Time Conversion and indexing

RobertRi — Tue, 22 Jan 2013 14:37:51 GMT

Yes thats IBM logfiles, which came from a heavy forwarder.

Re: Time Conversion and indexing

RobertRi — Wed, 23 Jan 2013 06:50:52 GMT

Thank you jonuwz for your detailed explainaition.
I'm not firm with syslog but I will give it a try and check how this works.
I know the ras1log tool, this was my prefered tool to troubleshoot.