Line breaking source types

[mstodola]

I am trying to fix the issue of my zeek logs not being broken into separate events. These logs are in json format and start with '{"ts":' and end with '}' (excluding single quotes). Given they are on separate lines, I would expect the code below to work.

# In /opt/splunk/etc/system/local/props.conf
# which I copied from the ../default/props.conf
[default]
...
LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = True
...

 At this point I think my issue may be not knowing which stanza to place that in. I do have the SPLUNK_TA_ZEEK add-on, but that is in a specific app (not S&R). Looking under sourcetypes in the Web UI, there are zeek, zeek:conn, bro, bro_conn, etc sourcetypes, but my sourcetypes in my events are zeek_conn, etc. I went ahead and applied the above code to zeek, zeek:conn, bro, and bro_conn.

TLDR:

1. What stanza do I edit?
2. Is the code snippet the correct settings?
3. Do I need to restart the cluster to apply these changes?

[PickleRick]

OK. There are several issues we need to touch here.

1. As a rule of thumb, you shouldn't fiddle with system/local directory. Organize your settings into apps and place your settings there. See https://docs.splunk.com/Documentation/Splunk/latest/Admin/Wheretofindtheconfigurationfiles for details

2. In index-time, application scope doesn't have as much meaning as during search-time (except for precedence of settings between apps - see the above linked article again).

3. Your stanza should be named either based on sourcetype, source or host. See https://docs.splunk.com/Documentation/Splunk/latest/Admin/Attributeprecedencewithinafile

4. You should never use SHOULD_LINEMERGE=True. Honestly. There might be some border cases where line-merging makes sense but I haven't seen any so far during my Splunk experience.

5. This line breaker will break your data stream at each line end. I don't think this is what you want.

6. How are you getting your data? (reading it from files? receiving over syslog? Is UF involved or HF?)

[mstodola]

For 1-5, my conclusion is that I should add the following stanza in /opt/splunk/etc/apps/Splunk_TA_zeek/local/props.conf:

[source::/zeek/logs/current/conn.log]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)

and repeat for any other sources having the issue.

In regards to 6, I have a universal forwarder monitoring the ./zeek/logs/current/ directory and forwarding directly to all the indexers.

[isoutamo]

One comment about apps. When you are creating and using your own app then conf files should be in app/default directory when you are installing it. That situation should be stored also in git or other VCS. Then when you modify those using GUI those changes will be stored into app/local folder.
BUT when you are taking any apps e.g. from splunkbase then you should put your local configuration into app/local or even create an additional app for your local change. In that way you could easily update into a newer version.

[PickleRick]

That might indeed be the way to go 🙂 Just adjust your line breaker. I don't remember how zeek logs to files but if your logs are multilined as you wrote in the original post, you might want to do something like

LINE_BREAKER = ([\r\n]+)\{"ts"

But if they are single line, indeed the newline line breaker should be OK.