How do I make a part of a field lowercase at index-time?

[sistemistiposta]

Hello Splunk users,

I run Splunk for Postfix, and I have a props.conf containing

EXTRACT-message_id = message-id\=\<(?<message_id>[^\>]*)\>
EXTRACT-Message_Id = Message-ID\:\s*\<(?<message_id>[^\>]*)\>

I index log lines like these:

2015-11-18T15:20:46.250700+01:00 host postfix/smtpd[10067]: 3p15sB1mdLzFpW3: client=my.example.org[10.10.10.168]
2015-11-18T15:21:14.704192+01:00 host postfix/cleanup[10000]: 3p15sB1mdLzFpW3: message-id=<dskjfsdlkfjslkDFDFSF@gMail.coM>
2015-11-18T15:21:15.073639+01:00 host postfix/qmgr[11106]: 3p15sB1mdLzFpW3: from=<postmaster@example.org>, size=313, nrcpt=1 (queue active)
2015-11-18T15:21:16.032231+01:00 host postfix/lmtp[10177]: 3p15sB1mdLzFpW3: to=<me@example.org>, relay=127.0.0.1[127.0.0.1]:20024, delay=43, delays=42/0/0.04/0.91, dsn=2.6.0, status=sent (250 2.6.0 <me@example.org> Message accepted for delivery)
2015-11-18T15:21:16.032538+01:00 host postfix/qmgr[11106]: 3p15sB1mdLzFpW3: removed
2015-11-18T15:21:16.020006+01:00 host amavis[10103]: (10103-07) Passed SPAMMY {RelayedTaggedInbound}, [127.0.0.1] [10.10.10.168] <postmaster@example.org> -> <me@example.org>, Message-ID: <dskjfsdlkfjslkDFDFSF@gmail.com>, mail_id: wFMEnpSkkuTV, Hits: 6.078, size: 1391, queued_as: 3p15sl74zczFpW4, 426 ms

As you can see, the process Amavis lowercases the domain part of Message-Id ('dskjfsdlkfjslkDFDFSF@gMail.coM' became 'dskjfsdlkfjslkDFDFSF@gmail.com'). This is a problem when I run transaction on message_id.

I don't care about case on domain part of Message-Id, so I would like to lowercase the domain part of message_id field at index-time. I have modified my props.conf as follow (like already suggested in similar answers):

EXTRACT-message_id = message-id\=\<(?<message_id>[^\>]*)\>
EXTRACT-Message_Id = Message-ID\:\s*\<(?<message_id>[^\>]*)\>
SEDCMD-message_id = s/\(@[A-Za-z0-9\.]*\)/\L\1/g
SEDCMD-Message_Id = s/\(@[A-Za-z0-9\.]*\)/\L\1/g

and I have restarted Splunk.
Unfortunately, this doesn't work. Domain part of Message-Id is not lowercased, but sed linux command line works:

# echo 'dskjfsdlkfjslkDFDFSF@gMail.coM' | sed 's/\(@[A-Za-z0-9\.]*\)/\L\1/g'
dskjfsdlkfjslkDFDFSF@gmail.com

Could you tell me how to do this with Splunk? I'm sure I have misunderstood something.
Thank you very much

Best Regards
Marco

[rnandasana]

We can make field value in lowercase at index-time using the below config in transforms.conf 

Here is the syntax that you can use it.

--> For Example: If we want to make source field to lower case then use below syntex
INGEST_EVAL = source=lower(source)

Fro more info refer -https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Data/IngestEval

[jluo_splunk · Accepted solution]

Hi sistemistiposta,

I'm unsure if it's possible to force a lower-case standard during index-time, however you can use "eval lower()" to force your field values into lower case before running a transaction. For example..

... | eval message_id=lower(message_id)