Archive

Time Conversion and indexing

Communicator

Hi

We index logfiles which have date time strings on the beginning of each event in hex format.

50FD316C = Mon Jan 21 13:15:40 2013

Is it possible to change this hex datetimestamp to an readable timestamp with default splunk features?

Or is it possible to execute a script on each event and change this timestamps in the _raw field?

Thanks
Rob

Tags (1)
0 Karma
1 Solution

Influencer

Do you want to use the hex code as the timestamp for the event ?
Thats not possible without pre-processing the event beforehand ( before splunk sees it )

If all you want to do is convert the code to a readable time after the event has been indexed then assuming the field you want to convert is called 'hex_time'

... | eval my_time=tonumber(hex_field,16) | convert ctime(my_time) | table my_time

1st part converts the hex to epoch seconds

2nd part formats the time into a readable date format

If the 'hex_field' is not automatically extracted, this'll do it (based on the fact the hex is at the start of the line) :

... | rex ^(?<hex_field>\S+) | ...

Update

If you need to convert the hex to a timestamp beforehand, a (very) hacky way to achieve this would be to configure syslog-ng to follow the files, and run each line through a program.

i.e. (in syslog-ng.conf)

source ibm_file {
    file("/opt/IBM/log/kbb.log" follow_freq(1)
    flags(no-parse));
};

destination hex_converter { program("/home/user/hex_converter.pl"); };
log { source(ibm_file); destination(hex_converter); };

the 'hex_converter' program would read from STDIN, convert the hex, and output the converted line to a file that splunk would then index ( or write it out to syslog, or send it as a TCP stream to splunk )

IBM normally ship a binary called ras1log that'll you could use in the hex_converter before redirecting to another file.

View solution in original post

0 Karma

Influencer

Do you want to use the hex code as the timestamp for the event ?
Thats not possible without pre-processing the event beforehand ( before splunk sees it )

If all you want to do is convert the code to a readable time after the event has been indexed then assuming the field you want to convert is called 'hex_time'

... | eval my_time=tonumber(hex_field,16) | convert ctime(my_time) | table my_time

1st part converts the hex to epoch seconds

2nd part formats the time into a readable date format

If the 'hex_field' is not automatically extracted, this'll do it (based on the fact the hex is at the start of the line) :

... | rex ^(?<hex_field>\S+) | ...

Update

If you need to convert the hex to a timestamp beforehand, a (very) hacky way to achieve this would be to configure syslog-ng to follow the files, and run each line through a program.

i.e. (in syslog-ng.conf)

source ibm_file {
    file("/opt/IBM/log/kbb.log" follow_freq(1)
    flags(no-parse));
};

destination hex_converter { program("/home/user/hex_converter.pl"); };
log { source(ibm_file); destination(hex_converter); };

the 'hex_converter' program would read from STDIN, convert the hex, and output the converted line to a file that splunk would then index ( or write it out to syslog, or send it as a TCP stream to splunk )

IBM normally ship a binary called ras1log that'll you could use in the hex_converter before redirecting to another file.

View solution in original post

0 Karma

Communicator

Thank you jonuwz for your detailed explainaition.
I'm not firm with syslog but I will give it a try and check how this works.
I know the ras1log tool, this was my prefered tool to troubleshoot.

0 Karma

Communicator

Yes thats IBM logfiles, which came from a heavy forwarder.

0 Karma

Influencer

How do the logs get to the indexer ? syslog-ng ? universal forwarder ? Whats the application (looks IBM .. ) ?

0 Karma

Communicator

Thanks jonuwz for your answer.
Yes I want to use this hex date for the actual event.
I'm also consider to make the time conversion on the forwarder side,
cause in case of debuging, there will be produced a lot of events on top.Do you see a possiblity to do so?Thanks
Robert

0 Karma