topic Re: Applying line breakers to data from universal forwarder in Getting Data In

How to apply line breakers to data from universal forwarder?

Poojitha — Thu, 26 May 2022 15:12:16 GMT

Hi All,

I have setup a universal forwarder in windows machine to monitor static file which is in json format.

The logs are being forwarded but the point is it is forwarded as single event like below :

{"Env": "someenv12”, "Name": "test12”, "feature": "TestFeature12”, "logLevel": "info", "Id": "1234", "date": 1652187242.57, "productName": “testproduct”, "process_name": “test process, "pid": 695, "process_status": "sleeping", "process_cpu_usage": 0.0, "process_ram_usage": 0.0, "metric_type": "system_process"} {"Env": "someenv1”3, "Name": "test13”, "feature": "TestFeature12”, "logLevel": “error”, "Id": "234", "date": 1652187342.57, "productName": “testproduct12”, "process_name": “test process, "pid": 685, "process_status": "sleeping", "process_cpu_usage": 0.0, "process_ram_usage": 0.0, "metric_type": “application_process} {"Env": "someenv14”, "Name": "test14”, "feature": "TestFeature13”, “info”: “error”, "Id": "2344", "date": 1672187342.57, "productName": “testproduct13”, "process_name": “test process, "pid": 695, "process_status": "sleeping", "process_cpu_usage": 0.0, "process_ram_usage": 0.0, "metric_type": “security”}

This entire thing is coming as one event.

I have applied line breakers in props.conf file :

[test_sourcetype] SHOULD_LINEMERGE =false NO_BINARY_CHECK=true BREAK_ONLY_BEFORE={"Env" MUST_BREAK_AFTER=\"\} TIME_PREFIX=date TIMEFORMAT=%s%4N MAX_TIMESTAMP_LOOKAHEAD = 14

I have added it under /SplunkUniversalForwarder/etc/apps/splunk_TA_windows app/local/props.

None of my line breaking is getting applied , please help me on this.

Should I add props.conf under default folder ?

Regards,
NVP

Re: Applying line breakers to data from universal forwarder

richgalloway — Thu, 26 May 2022 12:31:57 GMT

The universal forwarder cannot perform line breaking. That is done by the indexer or HF.

Try using these settings in props.conf:

[test_sourcetype] SHOULD_LINEMERGE =false NO_BINARY_CHECK=true LINE_BREAKER=()\{"Env" TIME_PREFIX=date TIMEFORMAT=%s%4N MAX_TIMESTAMP_LOOKAHEAD = 14

Re: Applying line breakers to data from universal forwarder

Poojitha — Thu, 26 May 2022 13:18:46 GMT

@richgalloway - Thanks for your response .

So, this props.conf I will be applying in the path I have mentioned right ?

I mean inside the TA - /SplunkUniversalForwarder/etc/apps/splunk_TA_windows app/local/props.

Re: Applying line breakers to data from universal forwarder

richgalloway — Thu, 26 May 2022 14:09:49 GMT

As I mentioned, the props.conf settings must be on an indexer or Heavy Forwarder so the path as given is incorrect. It should be $SPLUNK_HOME/etc/apps/splunk_TA_windows app/local/props.conf, where $SPLUNK_HOME usually is /opt/splunk, but can change depending on where you installed Splunk.

Re: Applying line breakers to data from universal forwarder

Poojitha — Thu, 26 May 2022 14:33:07 GMT

Apologies for my repeated question.

I understand this folder path $SPLUNK_HOME usually is /opt/splunk, so this will be our linux environment file path.

I also understand the parsing/line_breaking cannot be done at universal forwarder.

In my case, I have installed the Universal forwarder on windows machine , I dont have opt/path there, so it means I have to add it either on heavy forwarder or indexer ?

Re: Applying line breakers to data from universal forwarder

richgalloway — Thu, 26 May 2022 14:46:00 GMT

These props settings have no effect on a UF so we can ignore it in this case.

The settings go on the indexers, which is on a Linux server in your environment so the /opt/splunk path applies. If your Splunk is installed in a different directory (/Splunk is not uncommon) then use that instead.

Re: Applying line breakers to data from universal forwarder

Poojitha — Fri, 27 May 2022 09:42:22 GMT

Thanks for clarifying 🙂