Getting Data In

How to apply line breakers to data from universal forwarder?

Poojitha
Path Finder

Hi All,

I have setup a universal forwarder in windows machine to monitor static file which is in json format.

The logs are being forwarded but the point is it is forwarded as single event like below :

 

 

{"Env": "someenv12”, "Name": "test12”, "feature": "TestFeature12”, "logLevel": "info", "Id": "1234", "date": 1652187242.57, "productName": “testproduct”,  "process_name": “test process, "pid": 695, "process_status": "sleeping", "process_cpu_usage": 0.0, "process_ram_usage": 0.0, "metric_type": "system_process"}
{"Env": "someenv1”3, "Name": "test13”, "feature": "TestFeature12”, "logLevel": “error”, "Id": "234", "date": 1652187342.57, "productName": “testproduct12”,  "process_name": “test process, "pid": 685, "process_status": "sleeping", "process_cpu_usage": 0.0, "process_ram_usage": 0.0, "metric_type": “application_process}
{"Env": "someenv14”, "Name": "test14”, "feature": "TestFeature13”, “info”: “error”, "Id": "2344", "date": 1672187342.57, "productName": “testproduct13”,  "process_name": “test process, "pid": 695, "process_status": "sleeping", "process_cpu_usage": 0.0, "process_ram_usage": 0.0, "metric_type": “security”}

 

 

This entire thing is coming as one event.

I have applied line breakers in props.conf file :

 

 

[test_sourcetype]
SHOULD_LINEMERGE =false
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE={"Env"
MUST_BREAK_AFTER=\"\}
TIME_PREFIX=date
TIMEFORMAT=%s%4N
MAX_TIMESTAMP_LOOKAHEAD = 14

 

 


I have added it under /SplunkUniversalForwarder/etc/apps/splunk_TA_windows app/local/props.

None of my line breaking is getting applied , please help me on this.

Should I add props.conf under default folder ?

Regards,
NVP

Labels (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The universal forwarder cannot perform line breaking.  That is done by the indexer or HF.

Try using these settings in props.conf:

[test_sourcetype]
SHOULD_LINEMERGE =false
NO_BINARY_CHECK=true
LINE_BREAKER=()\{"Env"
TIME_PREFIX=date
TIMEFORMAT=%s%4N
MAX_TIMESTAMP_LOOKAHEAD = 14
---
If this reply helps you, Karma would be appreciated.
0 Karma

Poojitha
Path Finder


@richgalloway  - Thanks for your response .

So, this props.conf I will be applying in the path I have mentioned right ?

I mean inside the TA - /SplunkUniversalForwarder/etc/apps/splunk_TA_windows app/local/props.

 

Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

As I mentioned, the props.conf settings must be on an indexer or Heavy Forwarder so the path as given is incorrect.  It should be $SPLUNK_HOME/etc/apps/splunk_TA_windows app/local/props.conf, where $SPLUNK_HOME usually is /opt/splunk, but can change depending on where you installed Splunk.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Poojitha
Path Finder

Apologies for my repeated question.

I understand this folder path $SPLUNK_HOME usually is /opt/splunk, so this will be our linux environment file path. 

I also understand the parsing/line_breaking cannot be done at universal forwarder.

In my case, I have installed the Universal forwarder on windows machine , I  dont have opt/path there, so it means I have to add it either on heavy forwarder or indexer ?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

These props settings have no effect on a UF so we can ignore it in this case.

The settings go on the indexers, which is on a Linux server in your environment so the /opt/splunk path applies.  If your Splunk is installed in a different directory (/Splunk is not uncommon) then use that instead.

---
If this reply helps you, Karma would be appreciated.

Poojitha
Path Finder

Thanks for clarifying 🙂 

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...