Getting Data In

Where to edit props.config for breaking log into multiple events?

itdeptPFS
New Member

I am using universal forwarders to move log data from remote servers to a centralized Splunk Light server. Where do I edit the props.config? On the remote server or on the centralized Splunk Light server? When I search for props.config, I am returned server files, which one should I choose? I am a surprised that Splunk doesn't automatically break these into events because my log file has a blank space between each event.

MessageType: INFO | ApplicationName: MicroSaleDataTransporter | ApplicationVersion: 1.3.0.0 | Message: Hourly...

MessageType: INFO | ApplicationName: MicroSaleDataTransporter | ApplicationVersion: 1.3.0.0 | Message: Successfully....

MessageType: INFO | ApplicationName: MicroSaleDataTransporter | ApplicationVersion: 1.3.0.0 | Message: File created...

MessageType: INFO | ApplicationName: MicroSaleDataTransporter | ApplicationVersion: 1.3.0.0 | Message: People Counter...

MessageType: INFO | ApplicationName: MicroSaleDataTransporter | ApplicationVersion: 1.3.0.0 | Message: Successfully...

Thanks,
Chris

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Edit the props.conf file wherever event parsing is done. This is usually your indexer, but could also be a heavy forwarder.

You probably have several different props.conf files. The one to edit is the one in the app the corresponds to the data you are indexing. Be sure to edit local/props.conf (create it if you need to) rather than default/props.conf.

Splunk expects events to be separated by line-end characters (\n on Linux, \r\n on Windows) and to have a timestamp. It's best, however, not to allow Splunk to make guesses about the format of your events. Instead, use props.conf to describe your data. At the very least, include the TIME_PREFIX, TIME_FORMAT, LINE_BREAKER, SHOULD_LINEMERGE, TRUNCATE, and MAX_TIMESTAMP_LOOKAHEAD attributes.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

pruthvikrishnap
Contributor

Hi itdept,

Description by richgalloway is very details and will work, it should be something like this basing on your data.
[sourcetype]
SHOULD_LINEMERGE=true
NO_BINARY_CHECK=true
BREAK_ONLY_BEFORE=Regex
TIME_FORMAT=%H:%M:%S.%3N
TIME_PREFIX=^
MAX_TIMESTAMP_LOOKAHEAD=12

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Edit the props.conf file wherever event parsing is done. This is usually your indexer, but could also be a heavy forwarder.

You probably have several different props.conf files. The one to edit is the one in the app the corresponds to the data you are indexing. Be sure to edit local/props.conf (create it if you need to) rather than default/props.conf.

Splunk expects events to be separated by line-end characters (\n on Linux, \r\n on Windows) and to have a timestamp. It's best, however, not to allow Splunk to make guesses about the format of your events. Instead, use props.conf to describe your data. At the very least, include the TIME_PREFIX, TIME_FORMAT, LINE_BREAKER, SHOULD_LINEMERGE, TRUNCATE, and MAX_TIMESTAMP_LOOKAHEAD attributes.

---
If this reply helps you, Karma would be appreciated.
0 Karma

itdeptPFS
New Member

Thank you! I created the props.conf file on the indexer and it is working. I am still not exactly sure how this fowarder creates a SourceType name but I am finally getting my logs broken correctly.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The sourcetype is specified in the forwarder's inputs.conf file.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Want a chance to win $500 to the Splunk shop? Take our IT Incident Management Survey!

  Top Trends & Best Practices in Incident ManagementSplunk is partnering up with Constellation Research to ...