Deployment Architecture

Time Conversion and indexing

RobertRi
Communicator

Hi

We index logfiles which have date time strings on the beginning of each event in hex format.

50FD316C = Mon Jan 21 13:15:40 2013

Is it possible to change this hex datetimestamp to an readable timestamp with default splunk features?

Or is it possible to execute a script on each event and change this timestamps in the _raw field?

Thanks
Rob

Tags (1)
0 Karma
1 Solution

jonuwz
Influencer

Do you want to use the hex code as the timestamp for the event ?
Thats not possible without pre-processing the event beforehand ( before splunk sees it )

If all you want to do is convert the code to a readable time after the event has been indexed then assuming the field you want to convert is called 'hex_time'

... | eval my_time=tonumber(hex_field,16) | convert ctime(my_time) | table my_time

1st part converts the hex to epoch seconds

2nd part formats the time into a readable date format

If the 'hex_field' is not automatically extracted, this'll do it (based on the fact the hex is at the start of the line) :

... | rex ^(?<hex_field>\S+) | ...

Update

If you need to convert the hex to a timestamp beforehand, a (very) hacky way to achieve this would be to configure syslog-ng to follow the files, and run each line through a program.

i.e. (in syslog-ng.conf)

source ibm_file {
    file("/opt/IBM/log/kbb.log" follow_freq(1)
    flags(no-parse));
};

destination hex_converter { program("/home/user/hex_converter.pl"); };
log { source(ibm_file); destination(hex_converter); };

the 'hex_converter' program would read from STDIN, convert the hex, and output the converted line to a file that splunk would then index ( or write it out to syslog, or send it as a TCP stream to splunk )

IBM normally ship a binary called ras1log that'll you could use in the hex_converter before redirecting to another file.

View solution in original post

0 Karma

jonuwz
Influencer

Do you want to use the hex code as the timestamp for the event ?
Thats not possible without pre-processing the event beforehand ( before splunk sees it )

If all you want to do is convert the code to a readable time after the event has been indexed then assuming the field you want to convert is called 'hex_time'

... | eval my_time=tonumber(hex_field,16) | convert ctime(my_time) | table my_time

1st part converts the hex to epoch seconds

2nd part formats the time into a readable date format

If the 'hex_field' is not automatically extracted, this'll do it (based on the fact the hex is at the start of the line) :

... | rex ^(?<hex_field>\S+) | ...

Update

If you need to convert the hex to a timestamp beforehand, a (very) hacky way to achieve this would be to configure syslog-ng to follow the files, and run each line through a program.

i.e. (in syslog-ng.conf)

source ibm_file {
    file("/opt/IBM/log/kbb.log" follow_freq(1)
    flags(no-parse));
};

destination hex_converter { program("/home/user/hex_converter.pl"); };
log { source(ibm_file); destination(hex_converter); };

the 'hex_converter' program would read from STDIN, convert the hex, and output the converted line to a file that splunk would then index ( or write it out to syslog, or send it as a TCP stream to splunk )

IBM normally ship a binary called ras1log that'll you could use in the hex_converter before redirecting to another file.

0 Karma

RobertRi
Communicator

Thank you jonuwz for your detailed explainaition.
I'm not firm with syslog but I will give it a try and check how this works.
I know the ras1log tool, this was my prefered tool to troubleshoot.

0 Karma

RobertRi
Communicator

Yes thats IBM logfiles, which came from a heavy forwarder.

0 Karma

jonuwz
Influencer

How do the logs get to the indexer ? syslog-ng ? universal forwarder ? Whats the application (looks IBM .. ) ?

0 Karma

RobertRi
Communicator

Thanks jonuwz for your answer.
Yes I want to use this hex date for the actual event.
I'm also consider to make the time conversion on the forwarder side,
cause in case of debuging, there will be produced a lot of events on top.Do you see a possiblity to do so?Thanks
Robert

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...