Getting Data In

How to transform a numeric field at index-time (from microseconds to milliseconds)?

Path Finder

This is my situation:

I am currently using an older version of Apache which does not allow request times to be logged in milliseconds. I can't update the version of Apache until our next official release of our application that is in over a year. Eventually, the requests will be logged in milliseconds. In order to prevent conflicts at the point where the field becomes milliseconds, I want to transform the current value of the field (it's in microseconds) to milliseconds at index-time. This would mean that once the new version of Apache logs the requests in milliseconds, it will not affect the older data that isn't in the same format.

Is there a way to implement a temporary transformation of the field that gets indexed that could be removed once the modification has taken place?

For context: I am using a single indexer with multiple forwarders that send the logs to be indexed.

Any help would be greatly appreciated.


0 Karma

Esteemed Legend

The answer from @diogofgm creates a new field but this solution updates the _raw event by putting this in props.conf:

SEDCMD-1digitTo2 =s/ \(.\)$/ 0\1/
SEDCMD-2digitsTo3 = s/ \(..\)$/ 0\1/
SEDCMD-micro2milli = s/ \(.*\)\(...\)$/ \1\.\2/
0 Karma


In the props.conf in that sourcetype stanza you can do:

EVAL-request_seconds = request_time /1000000

Just replace the request_time with the field you already have being extracted for that number

Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

Revered Legend

What is the format of logs you have? Can you post some sample events?

0 Karma

Path Finder

They are standard Apache logs: - - [26/Aug/2015:14:26:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 9565 - - [26/Aug/2015:14:31:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 22270 - - [26/Aug/2015:14:36:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 17775 - - [26/Aug/2015:14:41:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 19384 - - [26/Aug/2015:14:46:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 15199 - - [26/Aug/2015:14:51:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 13081 - - [26/Aug/2015:14:56:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 14866 - - [26/Aug/2015:15:01:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 14962 - - [26/Aug/2015:15:06:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 14313 - - [26/Aug/2015:15:11:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 18321 - - [26/Aug/2015:15:16:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 18693 - - [26/Aug/2015:15:21:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 15301 - - [26/Aug/2015:15:26:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 15142 - - [26/Aug/2015:15:31:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 18524

The last field being the request time in microseconds.

0 Karma
Get Updates on the Splunk Community!

Data Preparation Made Easy: SPL2 for Edge Processor

By now, you may have heard the exciting news that Edge Processor, the easy-to-use Splunk data preparation tool ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Tips & Tricks When Using Ingest Actions

Tune in to learn about:Large scale architecture when using Ingest ActionsRegEx performance considerations ...