Getting Data In

Sourcetype not being applied

gabrsar
Loves-to-Learn Lots

Hi everyone. I'm using Splunk Enterprise (Trial) to understand how things works.

I'm trying to configure some sourcetype for my python/flask application, logs where getting merged incorrectly, with two or more line logs being joined inside a single event and sourcetype is not being applied.

For example, this is a single event in splunk:

[2021-06-23 12:05:09,807] {/program.py:make_request:452} DEBUG - https://localhost:443 "POST /create/user HTTP/1.1" 201 None
[2021-06-23 12:05:09,810] {/program.py:make_request:493} INFO - user created with success id=1234

I also have some logs with this format:

[2021-06-24 17:48:37,490] {/program/main.py:authorize:69} INFO - Host: localhost:5000
  User-Agent: curl/7.64.0
  Accept: */*

I tried creating a new sourcetype on Settings->Data->Source Types. But I noticed two weird things.

1 - If I go on Advanced and configure as I want, It don't save my new regex for LINE_BREAKER. I need it to be: ([\n\r]+)[ but every time I press save, and open again, its the default one ([\n\r]+). If I go on "Events Break" instead and just type my regex it saves. What I'm doing wrong?

2 - It doesn't apply my new sourcetype to my logs. I check on Search->Event lists and my logs are being sourcetyped as output-too_small, now I changed something and it is output-2

Then, I started googling around, and reading some docs, they tell to edit some files on splunk server then I did:

3 - Also tried creating a new sourcetype on $SPUNK_HOME/etc/system/local/props.conf as follow:

[python_flask]
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\n\r]+)\[
NO_BINARY_CHECK = true
disabled = false

4 - Also changed my $SPUNK_HOME/etc/system/local/inputs.conf, and added:

[monitor:///var/log/program/output.log]
sourcetype = python_flask

I restarted with splunk restart both server and universal forwarder, and the only thing that changed is that it started to put sourcetype=output-2 on my events .

I'm quite noob in splunk management, so sorry if any question is dumb, I have already checked the docs, google and so on. Thank you so much in advance.

Labels (2)
0 Karma

codebuilder
Influencer

Check the permissions on the props.conf that you created. I'm going to guess it's owned by root (or another user) and Splunk can't access it. If you are running Splunk as the splunk user you'll need to chmod splunk:splunk $SPUNK_HOME/etc/system/local/props.conf (e.g.) and then cycle Splunk.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Persistent Queue at TcpOut — One of Splunk's Most Practical Features

Splunk introduced persistent queueing at the tcpout layer as one of the most practical resilience features in ...

Skip the Awkward Silence: Have a .conf-ersation at .conf26

Picture this. You arrive at .conf26 already having your socializing and networking plans mapped out. No ...

Rethinking Zero Trust: From Product Purchases to Logical Control Evidence

Note: This post outlines a proposed architecture and serves as an interest check. If we secure commitments ...