How to apply line breakers to data from universal forwarder?

[Poojitha]

Hi All,

I have setup a universal forwarder in windows machine to monitor static file which is in json format.

The logs are being forwarded but the point is it is forwarded as single event like below :

{"Env": "someenv12”, "Name": "test12”, "feature": "TestFeature12”, "logLevel": "info", "Id": "1234", "date": 1652187242.57, "productName": “testproduct”,  "process_name": “test process, "pid": 695, "process_status": "sleeping", "process_cpu_usage": 0.0, "process_ram_usage": 0.0, "metric_type": "system_process"}
{"Env": "someenv1”3, "Name": "test13”, "feature": "TestFeature12”, "logLevel": “error”, "Id": "234", "date": 1652187342.57, "productName": “testproduct12”,  "process_name": “test process, "pid": 685, "process_status": "sleeping", "process_cpu_usage": 0.0, "process_ram_usage": 0.0, "metric_type": “application_process}
{"Env": "someenv14”, "Name": "test14”, "feature": "TestFeature13”, “info”: “error”, "Id": "2344", "date": 1672187342.57, "productName": “testproduct13”,  "process_name": “test process, "pid": 695, "process_status": "sleeping", "process_cpu_usage": 0.0, "process_ram_usage": 0.0, "metric_type": “security”}

This entire thing is coming as one event.

I have applied line breakers in props.conf file :

[test_sourcetype]
SHOULD_LINEMERGE =false
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE={"Env"
MUST_BREAK_AFTER=\"\}
TIME_PREFIX=date
TIMEFORMAT=%s%4N
MAX_TIMESTAMP_LOOKAHEAD = 14

I have added it under /SplunkUniversalForwarder/etc/apps/splunk_TA_windows app/local/props.

None of my line breaking is getting applied , please help me on this.

Should I add props.conf under default folder ?

Regards,
NVP

[richgalloway]

The universal forwarder cannot perform line breaking.  That is done by the indexer or HF.

Try using these settings in props.conf:

[test_sourcetype]
SHOULD_LINEMERGE =false
NO_BINARY_CHECK=true
LINE_BREAKER=()\{"Env"
TIME_PREFIX=date
TIMEFORMAT=%s%4N
MAX_TIMESTAMP_LOOKAHEAD = 14

[Poojitha]

@richgalloway  - Thanks for your response .

So, this props.conf I will be applying in the path I have mentioned right ?

I mean inside the TA - /SplunkUniversalForwarder/etc/apps/splunk_TA_windows app/local/props.

[richgalloway]

As I mentioned, the props.conf settings must be on an indexer or Heavy Forwarder so the path as given is incorrect.  It should be $SPLUNK_HOME/etc/apps/splunk_TA_windows app/local/props.conf, where $SPLUNK_HOME usually is /opt/splunk, but can change depending on where you installed Splunk.

[Poojitha]

Apologies for my repeated question.

I understand this folder path $SPLUNK_HOME usually is /opt/splunk, so this will be our linux environment file path. 

I also understand the parsing/line_breaking cannot be done at universal forwarder.

In my case, I have installed the Universal forwarder on windows machine , I  dont have opt/path there, so it means I have to add it either on heavy forwarder or indexer ?

[richgalloway]

These props settings have no effect on a UF so we can ignore it in this case.

The settings go on the indexers, which is on a Linux server in your environment so the /opt/splunk path applies.  If your Splunk is installed in a different directory (/Splunk is not uncommon) then use that instead.

[Poojitha]

Thanks for clarifying 🙂