Help with configuration: How to use Regex to Normalize data?

[ramprakash]

Dear All,

I have a requirement to parse the data correctly. I am getting merged events and wants separate events for the below events. Could someone help me what configuration needs to be changed and how can i learn regex.

I need events to break from [22/05/11@08:13:58.246+0200] P-20316642 T-000001....Timeframe, P and T values can be different. Appreciate your help

[22/05/11@08:14:25.252+0200] P-37945744 T-000001 1 AS -- (Procedure: 'olb-stp-monitoring.r' Line:273) DML TRACE ERROR : use of refreshUsrRig , decomissioning ongoing
[22/05/11@08:14:03.266+0200] P-29491506 T-000001 1 AS -- (Procedure: 'olb-stp-monitoring.r' Line:273) DML TRACE ERROR : use of refreshUsrRig , decomissioning ongoing
[22/05/11@08:13:58.246+0200] P-20316642 T-000001 1 AS -- (Procedure: 'olb-stp-monitoring.r' Line:273) DML TRACE ERROR : use of refreshUsrRig , decomissioning ongoing

[venky1544]

Hi @ramprakash 

if you want a line or event break before the date  [22/05/11@08:13:58.246+0200] 

you could use either of the below setting  in props.conf 

BREAK_ONLY_BEFORE_DATE = true

OR 

BREAK_ONLY_BEFORE=<regex>

inyour case should be something like this 

BREAK_ONLY_BEFORE=\[[\d+\/]+@[\d+:]+.\d+\+\d+\]

[ramprakash]

Thanks you so much and I assume I need to make the changes at UF level only or at the Indexer(In that case I will contact Splunk support]

[venky1544 · Accepted solution]

Hi @ramprakash 

is your instance on splunk cloud  if yes and you do have to contact the splunk support and if you have splunk enterprise on premise or splunk enterpise on cloud then you need to do it on the indexer not on forwarder and in HF(if you have this component and files are processing through HF)

Thanks

Venky

P.S if it helps karms points are appreciated/if it resolves solution acceptance is appreciated 

[ramprakash]

Thanks for the solution. Let me check with the Support team as the instance is on Splunk cloud.

So I just need to provide them the sourcetype and the config that's all and they will make the changes on our behalf ?

We have UF Intermediate forwarder in between, I don't think so that's gonna be any issue.

[venky1544]

Hi @ramprakash 

Yes it should be pretty straight forward you provide them the config details and they should be able to do it

Thanks

venky

[PickleRick]

Line breaking is done on the indexer/heavy-forwarder level (depending on whether you use HF's or not in your path of events). So you need to push your settings there. In case of the Splunk Cloud, where you don't have direct access to the indexers, you need to prepare and deploy an app containing the settings.

[ITWhisperer]

For the regex side of things, regex101.com is a good place to practice regex - however, it is worth noting that the way Splunk implements regex, particularly in searches, often requires additional escaping beyond what regex101 deems as a correct regex expression particularly for backslashes.

[ramprakash]

Thanks for the suggestion. I will definitely have a look at this. Meanwhile can you share me what config change i can make on the indexer.

[PickleRick]

First question is - where are you getting those events from. If it's - for example - file source, it's about setting proper line breakers in props.conf. If they are coming in from HEC and they are pushed this way... Well, you can't do much about it - you should check the source.

[ramprakash]

It is coming fro the source file. Do you happen to know what props.conf i can use here.

[PickleRick]

The default line breaker should break the contents into multiple events so the question is what are your current settings - do you have something set explicitly for this sourcetype?

[ramprakash]

No. I have not setup anything yet. The weird thing is some events are parsing correctly and some are bring merged 

[PickleRick]

Check your source file. The default line breaker is ([\r\n]+) so it should match any end-of-line character sequence.

Use btool to verify your config (both on UF - see if it sets proper sourcetype, and on indexer(s) - verify the settings for that sourcetype/source/host).

[ramprakash]

As of now now I don't find the default line breaker is ([\r\n]+) in any sourcetype.

Can I put it under sourcetype of both UF and Indexer and try again ?

[PickleRick]

Interesting. I always thought it's explicitly specified in the default props.conf but I see (on my installation) that it isn't.

Try adding it on the indexer.