Getting Data In

How to transform a numeric field at index-time (from microseconds to milliseconds)?

Path Finder

This is my situation:

I am currently using an older version of Apache which does not allow request times to be logged in milliseconds. I can't update the version of Apache until our next official release of our application that is in over a year. Eventually, the requests will be logged in milliseconds. In order to prevent conflicts at the point where the field becomes milliseconds, I want to transform the current value of the field (it's in microseconds) to milliseconds at index-time. This would mean that once the new version of Apache logs the requests in milliseconds, it will not affect the older data that isn't in the same format.

Is there a way to implement a temporary transformation of the field that gets indexed that could be removed once the modification has taken place?

For context: I am using a single indexer with multiple forwarders that send the logs to be indexed.

Any help would be greatly appreciated.


0 Karma

Esteemed Legend

The answer from @diogofgm creates a new field but this solution updates the _raw event by putting this in props.conf:

SEDCMD-1digitTo2 =s/ \(.\)$/ 0\1/
SEDCMD-2digitsTo3 = s/ \(..\)$/ 0\1/
SEDCMD-micro2milli = s/ \(.*\)\(...\)$/ \1\.\2/
0 Karma


In the props.conf in that sourcetype stanza you can do:

EVAL-request_seconds = request_time /1000000

Just replace the request_time with the field you already have being extracted for that number

Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

Revered Legend

What is the format of logs you have? Can you post some sample events?

0 Karma

Path Finder

They are standard Apache logs: - - [26/Aug/2015:14:26:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 9565 - - [26/Aug/2015:14:31:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 22270 - - [26/Aug/2015:14:36:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 17775 - - [26/Aug/2015:14:41:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 19384 - - [26/Aug/2015:14:46:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 15199 - - [26/Aug/2015:14:51:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 13081 - - [26/Aug/2015:14:56:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 14866 - - [26/Aug/2015:15:01:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 14962 - - [26/Aug/2015:15:06:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 14313 - - [26/Aug/2015:15:11:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 18321 - - [26/Aug/2015:15:16:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 18693 - - [26/Aug/2015:15:21:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 15301 - - [26/Aug/2015:15:26:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 15142 - - [26/Aug/2015:15:31:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 18524

The last field being the request time in microseconds.

0 Karma
Get Updates on the Splunk Community!

Tips & Tricks When Using Ingest Actions

Tune in to learn about:Large scale architecture when using Ingest ActionsRegEx performance considerations ...

Announcing Our Splunk MVPs

We are excited to announce the first cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...