topic Re: Regex to Normalize data in Getting Data In

Help with configuration: How to use Regex to Normalize data?

ramprakash — Wed, 11 May 2022 22:49:52 GMT

Dear All,

I have a requirement to parse the data correctly. I am getting merged events and wants separate events for the below events. Could someone help me what configuration needs to be changed and how can i learn regex.

I need events to break from [22/05/11@08:13:58.246+0200] P-20316642 T-000001....Timeframe, P and T values can be different. Appreciate your help

[22/05/11@08:14:25.252+0200] P-37945744 T-000001 1 AS -- (Procedure: 'olb-stp-monitoring.r' Line:273) DML TRACE ERROR : use of refreshUsrRig , decomissioning ongoing
[22/05/11@08:14:03.266+0200] P-29491506 T-000001 1 AS -- (Procedure: 'olb-stp-monitoring.r' Line:273) DML TRACE ERROR : use of refreshUsrRig , decomissioning ongoing
[22/05/11@08:13:58.246+0200] P-20316642 T-000001 1 AS -- (Procedure: 'olb-stp-monitoring.r' Line:273) DML TRACE ERROR : use of refreshUsrRig , decomissioning ongoing

Re: Regex to Normalize data

PickleRick — Wed, 11 May 2022 07:19:37 GMT

First question is - where are you getting those events from. If it's - for example - file source, it's about setting proper line breakers in props.conf. If they are coming in from HEC and they are pushed this way... Well, you can't do much about it - you should check the source.

Re: Regex to Normalize data

ITWhisperer — Wed, 11 May 2022 07:34:43 GMT

For the regex side of things, regex101.com is a good place to practice regex - however, it is worth noting that the way Splunk implements regex, particularly in searches, often requires additional escaping beyond what regex101 deems as a correct regex expression particularly for backslashes.

Re: Regex to Normalize data

ramprakash — Wed, 11 May 2022 07:38:21 GMT

It is coming fro the source file. Do you happen to know what props.conf i can use here.

Re: Regex to Normalize data

ramprakash — Wed, 11 May 2022 07:39:40 GMT

Thanks for the suggestion. I will definitely have a look at this. Meanwhile can you share me what config change i can make on the indexer.

Re: Regex to Normalize data

PickleRick — Wed, 11 May 2022 07:56:41 GMT

The default line breaker should break the contents into multiple events so the question is what are your current settings - do you have something set explicitly for this sourcetype?

Re: Regex to Normalize data

ramprakash — Wed, 11 May 2022 08:00:53 GMT

No. I have not setup anything yet. The weird thing is some events are parsing correctly and some are bring merged

Re: Regex to Normalize data

PickleRick — Wed, 11 May 2022 08:20:34 GMT

Check your source file. The default line breaker is ([\r\n]+) so it should match any end-of-line character sequence.

Use btool to verify your config (both on UF - see if it sets proper sourcetype, and on indexer(s) - verify the settings for that sourcetype/source/host).

Re: Regex to Normalize data

ramprakash — Wed, 11 May 2022 08:50:07 GMT

As of now now I don't find the default line breaker is ([\r\n]+) in any sourcetype.

Can I put it under sourcetype of both UF and Indexer and try again ?

Re: Regex to Normalize data

PickleRick — Wed, 11 May 2022 08:58:55 GMT

Interesting. I always thought it's explicitly specified in the default props.conf but I see (on my installation) that it isn't.

Try adding it on the indexer.

Re: Regex to Normalize data

venky1544 — Wed, 11 May 2022 10:03:13 GMT

Hi @ramprakash

if you want a line or event break before the date [22/05/11@08:13:58.246+0200]

you could use either of the below setting in props.conf

BREAK_ONLY_BEFORE_DATE = true

BREAK_ONLY_BEFORE=<regex>

inyour case should be something like this

BREAK_ONLY_BEFORE=\[[\d+\/]+@[\d+:]+.\d+\+\d+\]

Re: Regex to Normalize data

ramprakash — Wed, 11 May 2022 10:30:28 GMT

Thanks you so much and I assume I need to make the changes at UF level only or at the Indexer(In that case I will contact Splunk support]

Re: Regex to Normalize data

PickleRick — Wed, 11 May 2022 10:41:01 GMT

Line breaking is done on the indexer/heavy-forwarder level (depending on whether you use HF's or not in your path of events). So you need to push your settings there. In case of the Splunk Cloud, where you don't have direct access to the indexers, you need to prepare and deploy an app containing the settings.

Re: Regex to Normalize data

venky1544 — Wed, 11 May 2022 11:22:42 GMT

Hi @ramprakash

is your instance on splunk cloud if yes and you do have to contact the splunk support and if you have splunk enterprise on premise or splunk enterpise on cloud then you need to do it on the indexer not on forwarder and in HF(if you have this component and files are processing through HF)

Thanks

Venky

P.S if it helps karms points are appreciated/if it resolves solution acceptance is appreciated

Re: Regex to Normalize data

ramprakash — Wed, 11 May 2022 13:35:05 GMT

Thanks for the solution. Let me check with the Support team as the instance is on Splunk cloud.

So I just need to provide them the sourcetype and the config that's all and they will make the changes on our behalf ?

We have UF Intermediate forwarder in between, I don't think so that's gonna be any issue.

Re: Regex to Normalize data

venky1544 — Wed, 11 May 2022 13:39:59 GMT

Hi @ramprakash

Yes it should be pretty straight forward you provide them the config details and they should be able to do it

Thanks

venky