topic Re: Add and remove text from a field at extraction time in Splunk Search

Add and remove text from a field at extraction time

splunkmasterfle — Fri, 12 Sep 2014 17:57:47 GMT

Hi,

Is there a way to add text to a field that matches a specific pattern?

Example:

log:

2014-09-12 13:40:12,359 "GET /app/rest/1.0/payableinvoicelink/100-129880-2fc616a5-33d5-4567-9abf-71b1234a3d36 HTTP/1.1"

What I would like to have:

requestType: "GET /app/rest/1.0/payableinvoicelink/getById"

Another example:

log:
2014-09-12 13:40:12,359 "GET /app/rest/1.0/application/674536 HTTP/1.1"

What I would like to have:

requestType: "GET /app/rest/1.0/application/getById"

Is there an elegant way of achieving this?

Thanks!

Re: Add and remove text from a field at extraction time

lguinn2 — Fri, 12 Sep 2014 18:09:01 GMT

An INDEX time solution - which means that all of your data would need to be reindexed. Also, the original value would be gone...

In the Splunk manuals, there is a section called Anonymize data, which shows how to replace certain patterns in your input with "XXXX" or "####". Follow those instructions, but use "getById" (or whatever you want) instead of "XXX".

Example 1 (using transforms.conf)

[fix-application]
REGEX = (.*?)GET /app/rest/1.0/application/674536 HTTP/1.1(.*)
FORMAT = $1GET /app/rest/1.0/application/getById$2
DEST_KEY = _raw

Example 2 (using only props.conf)

SEDCMD-fix2 = s/(GET \/app\/rest\/1.0\/payableinvoicelink\/)100-129880-2fc616a5-33d5-4567-9abf-71b1234a3d36 HTTP\/1.1\/\1 getById/

Note that I had to escape the / in the second example, which makes it a bit harder to read

Re: Add and remove text from a field at extraction time

splunkmasterfle — Fri, 12 Sep 2014 18:55:02 GMT

Does this work at index time? Meaning I have to re-import all of my data ?

Re: Add and remove text from a field at extraction time

lguinn2 — Fri, 12 Sep 2014 21:12:55 GMT

Ah, good point, all of this is at index time. Another answer follows for how to do it at search time.

Re: Add and remove text from a field at extraction time

lguinn2 — Fri, 12 Sep 2014 21:21:21 GMT

A SEARCH time solution (or two)

Option 1: Use the replace command (documentation here.) This works great if you only have a few changes to make, and if you only want to do it for a single report.

Option 2: Use an automatic lookup. This allows you to create an entirely new field. You can have both the new field and the original field. A lookup table can contain thousands of values and is easier than a series of replacements. If you set up the lookup as automatic, the new field will always be present.

Use field lookups is a tutorial that uses the Splunk UI. I recommend it.