Getting Data In

How to transform a numeric field at index-time (from microseconds to milliseconds)?

splunkmasterfle
Path Finder

This is my situation:

I am currently using an older version of Apache which does not allow request times to be logged in milliseconds. I can't update the version of Apache until our next official release of our application that is in over a year. Eventually, the requests will be logged in milliseconds. In order to prevent conflicts at the point where the field becomes milliseconds, I want to transform the current value of the field (it's in microseconds) to milliseconds at index-time. This would mean that once the new version of Apache logs the requests in milliseconds, it will not affect the older data that isn't in the same format.

Is there a way to implement a temporary transformation of the field that gets indexed that could be removed once the modification has taken place?

For context: I am using a single indexer with multiple forwarders that send the logs to be indexed.

Any help would be greatly appreciated.

Thanks

0 Karma

woodcock
Esteemed Legend

The answer from @diogofgm creates a new field but this solution updates the _raw event by putting this in props.conf:

[YourSourceTypeHere]
SEDCMD-1digitTo2 =s/ \(.\)$/ 0\1/
SEDCMD-2digitsTo3 = s/ \(..\)$/ 0\1/
SEDCMD-micro2milli = s/ \(.*\)\(...\)$/ \1\.\2/
0 Karma

diogofgm
SplunkTrust
SplunkTrust

In the props.conf in that sourcetype stanza you can do:

EVAL-request_seconds = request_time /1000000

Just replace the request_time with the field you already have being extracted for that number

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

somesoni2
Revered Legend

What is the format of logs you have? Can you post some sample events?

0 Karma

splunkmasterfle
Path Finder

They are standard Apache logs:

179.31.12.34 - - [26/Aug/2015:14:26:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 9565
179.31.12.34 - - [26/Aug/2015:14:31:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 22270
179.31.12.34 - - [26/Aug/2015:14:36:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 17775
179.31.12.34 - - [26/Aug/2015:14:41:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 19384
179.31.12.34 - - [26/Aug/2015:14:46:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 15199
179.31.12.34 - - [26/Aug/2015:14:51:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 13081
179.31.12.34 - - [26/Aug/2015:14:56:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 14866
179.31.12.34 - - [26/Aug/2015:15:01:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 14962
179.31.12.34 - - [26/Aug/2015:15:06:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 14313
179.31.12.34 - - [26/Aug/2015:15:11:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 18321
179.31.12.34 - - [26/Aug/2015:15:16:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 18693
179.31.12.34 - - [26/Aug/2015:15:21:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 15301
179.31.12.34 - - [26/Aug/2015:15:26:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 15142
179.31.12.34 - - [26/Aug/2015:15:31:04 -0400] "GET /AnApplication/ HTTP/1.1" 200 84 18524

The last field being the request time in microseconds.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...