All Apps and Add-ons

How do I make a part of a field lowercase at index-time?

sistemistiposta
Path Finder

Hello Splunk users,

I run Splunk for Postfix, and I have a props.conf containing

EXTRACT-message_id = message-id\=\<(?<message_id>[^\>]*)\>
EXTRACT-Message_Id = Message-ID\:\s*\<(?<message_id>[^\>]*)\>

I index log lines like these:

2015-11-18T15:20:46.250700+01:00 host postfix/smtpd[10067]: 3p15sB1mdLzFpW3: client=my.example.org[10.10.10.168]
2015-11-18T15:21:14.704192+01:00 host postfix/cleanup[10000]: 3p15sB1mdLzFpW3: message-id=<dskjfsdlkfjslkDFDFSF@gMail.coM>
2015-11-18T15:21:15.073639+01:00 host postfix/qmgr[11106]: 3p15sB1mdLzFpW3: from=<postmaster@example.org>, size=313, nrcpt=1 (queue active)
2015-11-18T15:21:16.032231+01:00 host postfix/lmtp[10177]: 3p15sB1mdLzFpW3: to=<me@example.org>, relay=127.0.0.1[127.0.0.1]:20024, delay=43, delays=42/0/0.04/0.91, dsn=2.6.0, status=sent (250 2.6.0 <me@example.org> Message accepted for delivery)
2015-11-18T15:21:16.032538+01:00 host postfix/qmgr[11106]: 3p15sB1mdLzFpW3: removed
2015-11-18T15:21:16.020006+01:00 host amavis[10103]: (10103-07) Passed SPAMMY {RelayedTaggedInbound}, [127.0.0.1] [10.10.10.168] <postmaster@example.org> -> <me@example.org>, Message-ID: <dskjfsdlkfjslkDFDFSF@gmail.com>, mail_id: wFMEnpSkkuTV, Hits: 6.078, size: 1391, queued_as: 3p15sl74zczFpW4, 426 ms

As you can see, the process Amavis lowercases the domain part of Message-Id ('dskjfsdlkfjslkDFDFSF@gMail.coM' became 'dskjfsdlkfjslkDFDFSF@gmail.com'). This is a problem when I run transaction on message_id.

I don't care about case on domain part of Message-Id, so I would like to lowercase the domain part of message_id field at index-time. I have modified my props.conf as follow (like already suggested in similar answers):

EXTRACT-message_id = message-id\=\<(?<message_id>[^\>]*)\>
EXTRACT-Message_Id = Message-ID\:\s*\<(?<message_id>[^\>]*)\>
SEDCMD-message_id = s/\(@[A-Za-z0-9\.]*\)/\L\1/g
SEDCMD-Message_Id = s/\(@[A-Za-z0-9\.]*\)/\L\1/g

and I have restarted Splunk.
Unfortunately, this doesn't work. Domain part of Message-Id is not lowercased, but sed linux command line works:

# echo 'dskjfsdlkfjslkDFDFSF@gMail.coM' | sed 's/\(@[A-Za-z0-9\.]*\)/\L\1/g'
dskjfsdlkfjslkDFDFSF@gmail.com

Could you tell me how to do this with Splunk? I'm sure I have misunderstood something.
Thank you very much

Best Regards
Marco

0 Karma
1 Solution

jluo_splunk
Splunk Employee
Splunk Employee

Hi sistemistiposta,

I'm unsure if it's possible to force a lower-case standard during index-time, however you can use "eval lower()" to force your field values into lower case before running a transaction. For example..

... | eval message_id=lower(message_id)

View solution in original post

0 Karma

rnandasana
Splunk Employee
Splunk Employee

We can make field value in lowercase at index-time using the below config in transforms.conf 

Here is the syntax that you can use it.

--> For Example: If we want to make source field to lower case then use below syntex
INGEST_EVAL = source=lower(source)

 

Fro more info refer -https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Data/IngestEval

Tags (1)
0 Karma

jluo_splunk
Splunk Employee
Splunk Employee

Hi sistemistiposta,

I'm unsure if it's possible to force a lower-case standard during index-time, however you can use "eval lower()" to force your field values into lower case before running a transaction. For example..

... | eval message_id=lower(message_id)

0 Karma
Get Updates on the Splunk Community!

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...

Splunk With AppDynamics - Meet the New IT (And Engineering) Couple

Wednesday, November 20, 2024  |  10AM PT / 1PM ET Register Now Join us in this session to learn all about ...