Getting Data In

Help with configuration: How to use Regex to Normalize data?

ramprakash
Explorer

Dear All,

I have a requirement to parse the data correctly. I am getting merged events and wants separate events for the below events. Could someone help me what configuration needs to be changed and how can i learn regex.

I need events to break from [22/05/11@08:13:58.246+0200] P-20316642 T-000001....Timeframe, P and T values can be different. Appreciate your help

 

[22/05/11@08:14:25.252+0200] P-37945744 T-000001 1 AS -- (Procedure: 'olb-stp-monitoring.r' Line:273) DML TRACE ERROR : use of refreshUsrRig , decomissioning ongoing
[22/05/11@08:14:03.266+0200] P-29491506 T-000001 1 AS -- (Procedure: 'olb-stp-monitoring.r' Line:273) DML TRACE ERROR : use of refreshUsrRig , decomissioning ongoing
[22/05/11@08:13:58.246+0200] P-20316642 T-000001 1 AS -- (Procedure: 'olb-stp-monitoring.r' Line:273) DML TRACE ERROR : use of refreshUsrRig , decomissioning ongoing

Labels (1)
Tags (3)
0 Karma
1 Solution

venky1544
Contributor

Hi @ramprakash 

is your instance on splunk cloud  if yes and you do have to contact the splunk support and if you have splunk enterprise on premise or splunk enterpise on cloud then you need to do it on the indexer not on forwarder and in HF(if you have this component and files are processing through HF)

Thanks

Venky

P.S if it helps karms points are appreciated/if it resolves solution acceptance is appreciated 

 

View solution in original post

0 Karma

venky1544
Contributor

Hi @ramprakash 

if you want a line or event break before the date  [22/05/11@08:13:58.246+0200] 

you could use either of the below setting  in props.conf 

BREAK_ONLY_BEFORE_DATE = true

OR 

BREAK_ONLY_BEFORE=<regex>

inyour case should be something like this 

BREAK_ONLY_BEFORE=\[[\d+\/]+@[\d+:]+.\d+\+\d+\]

0 Karma

ramprakash
Explorer

Thanks you so much and I assume I need to make the changes at UF level only or at the Indexer(In that case I will contact Splunk support]

0 Karma

venky1544
Contributor

Hi @ramprakash 

is your instance on splunk cloud  if yes and you do have to contact the splunk support and if you have splunk enterprise on premise or splunk enterpise on cloud then you need to do it on the indexer not on forwarder and in HF(if you have this component and files are processing through HF)

Thanks

Venky

P.S if it helps karms points are appreciated/if it resolves solution acceptance is appreciated 

 

0 Karma

ramprakash
Explorer

Thanks for the solution. Let me check with the Support team as the instance is on Splunk cloud.

So I just need to provide them the sourcetype and the config that's all and they will make the changes on our behalf ?

We have UF Intermediate forwarder in between, I don't think so that's gonna be any issue.

0 Karma

venky1544
Contributor

Hi @ramprakash 

Yes it should be pretty straight forward you provide them the config details and they should be able to do it

Thanks

venky

0 Karma

PickleRick
Ultra Champion

Line breaking is done on the indexer/heavy-forwarder level (depending on whether you use HF's or not in your path of events). So you need to push your settings there. In case of the Splunk Cloud, where you don't have direct access to the indexers, you need to prepare and deploy an app containing the settings.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

For the regex side of things, regex101.com is a good place to practice regex - however, it is worth noting that the way Splunk implements regex, particularly in searches, often requires additional escaping beyond what regex101 deems as a correct regex expression particularly for backslashes.

0 Karma

ramprakash
Explorer

Thanks for the suggestion. I will definitely have a look at this. Meanwhile can you share me what config change i can make on the indexer.

0 Karma

PickleRick
Ultra Champion

First question is - where are you getting those events from. If it's - for example - file source, it's about setting proper line breakers in props.conf. If they are coming in from HEC and they are pushed this way... Well, you can't do much about it - you should check the source.

0 Karma

ramprakash
Explorer

It is coming fro the source file. Do you happen to know what props.conf i can use here.

0 Karma

PickleRick
Ultra Champion

The default line breaker should break the contents into multiple events so the question is what are your current settings - do you have something set explicitly for this sourcetype?

0 Karma

ramprakash
Explorer

No. I have not setup anything yet. The weird thing is some events are parsing correctly and some are bring merged 

0 Karma

PickleRick
Ultra Champion

Check your source file. The default line breaker is ([\r\n]+) so it should match any end-of-line character sequence.

Use btool to verify your config (both on UF - see if it sets proper sourcetype, and on indexer(s) - verify the settings for that sourcetype/source/host).

0 Karma

ramprakash
Explorer

As of now now I don't find the default line breaker is ([\r\n]+) in any sourcetype.

Can I put it under sourcetype of both UF and Indexer and try again ?

0 Karma

PickleRick
Ultra Champion

Interesting. I always thought it's explicitly specified in the default props.conf but I see (on my installation) that it isn't.

Try adding it on the indexer.

Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...