should_linemerge for json data using universal forwarder

[vinceskahan]

I have an application writing out JSON formatted logfile entries that we're using the universal forwarder to get over to the indexer system. The log entries (client-side) could be several lines per second. I didn't define a sourcetype when I added the monitor on the forwarder system (yes, I know 'now'). So the result is that the indexer is 'helping' too much and it sometimes puts multiple entries into one event as seen on the splunk console.

Questions:

• would specifying a sourcetype on the 'forwarder' system be the right thing to do here ?
• which sourcetype do I pick ? It's a internally written app that writes one-line records to a logfile we monitor. Those lines just 'happen' to be valid JSON

or

• do I need to create a custom sourcetype to specify, and set SHOULD_LINEMERGE=false in a props.conf file ?
• and do I do the props.conf file on the server side, or the client side ? We deploy the forwarders using puppetlabs-splunk and really do not want to have to touch the splunk server at all when adding a forwarder system feeding it data

I poked around a little with "splunk btool props list <sourcetype_here>" and can see which types do or don't linemerge, but there are a 'lot' of known sourcetypes. Any suggestions on which one to pick if we can (hopefully) not need to create our own ?

[woodcock]

Generally what you do, if you are not using somebody else's configuration files (e.g. from an app on apps.splunk.com), is you create your own app directory like $SPLUNK_HOME/etc/apps/MyApp/default (yes, since you are the developer of this app, you use default, not local) and you create your files there. Inside this directory, you should put your inputs.conf file and inside this file you should have something like this:

[monitor:///path/to/my/file.log]
sourcetype=MyApp

You might also add index=MyIndex if you would like to get your events out of index=main.

In the same directory structure, you should put your props.conf file and inside this file you should have something like this:

[MyApp]
INDEXED_EXTRACTIONS = json

This set of files needs to be put on your Forwarders and the Splunk instances there all restarted.

That is mostly it but you will probably like to do some other things, too. For example, there's a TIMESTAMP_FIELDS setting that exploits the JSON structure rather than specifying TIME_FORMAT or TIME_PREFIX expressions to manually walk through the structure; see more here:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
http://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileheadersatindextime

[somesoni2]

If your'e using a universal forwarder, the sourcetype definition should be on Indexers (server side). If your data is in just single line, SHOULD_LINEMERGE should be false.

[vinceskahan]

Makes a devops deployment kind of hard to do. No way to control should_linemerge from the client side using the universal forwarder ? Does the forwarder support props/transforms ? Setting a known sourcetype that has should_linemerge=false on the server side already ?