Getting Data In

How to replace field during parsing?

bosseres
Contributor

Hello everyone, 

I have logs like 

 

 

2022-11-23 12:47:42.000 id="123" event="some text text2 text3   text4"

 

 

I want to trim everything that goes after three consecutive spaces, so I want to get raw logs

 

 

2022-11-23 12:47:42.000 id="123" event="some text text2 text3"

 

 

 

I did such props.conf

[my_sourcetype]
...
EXTRACT-event = event="(?<event>.+?)\s{3,}.*"
...

 

It's working fine, I get event field what I want, but I still get old logs with 3+ spaces.

What should I add to props conf to get correct logs?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Splunk can work on events in two separate points.

One thing is the so-called index time processing. Which means that an event is being ingested from the source, some transforms are being applied, possibly rewriting some metadata (a relatively common use - to cast event to another sourcetype, or filter some events out completely by redirecting them to nullqueue), extracting indexed fields or trimming/rewriting raw data. Then event is written do the index and its _raw value is immutable from now on - you can't change it anymore.

With search-time operations you can extract some parts of the raw message as fields, calculate other fields based on those values and/or indexed fields and so on.

So if you define a search-time extraction (that's what the EXTRACT setting does), you can get your field value populated but the original _raw event still contains the event as it was when it was ingested and written to the index. If you want to trim the raw event value before writing it to the index, you have to use TRANSFORMS or SEDCMD.

gcusello
SplunkTrust
SplunkTrust

Hi @bosseres,

if you are speaking at search time, the same rules can be applied also to old events.

If you're speaking at index time, you can only apply rules to new indexed events not to already indexed events.

If you want to modify already indexed events, it isn't possible in Splunk.

As a workaround you could extract and reindex them, but it's a very hard work.

Ciao.

Giuseppe

bosseres
Contributor

I want to replace/trim only new logs, that not indexed

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @bosseres,

if you're sure that in each event you have only one group of three spaces, you could try this command to insert in props.conf:

SEDCMD-reduce_event = s/^.*\s\s\s.*/^.*\s\s\s/g

Ciao.

Giuseppe

antob
Engager

What if I want to replace all the values in a specific field (number 3) during ingestion with a fixed value for all rows?

0 Karma
Get Updates on the Splunk Community!

Credit Card Data Protection & PCI Compliance with Splunk Edge Processor

Organizations handling credit card transactions know that PCI DSS compliance is both critical and complex. The ...

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...