Getting Data In

Why is our props.conf configuration for our universal forwarder and clustered indexers not breaking events properly?

Communicator

We have a few access log files from our SecureMedia application that we are attempting to ingest. I've been able to get most of them to ingest properly, but one particular sort of access log keeps giving me problems. The entries in the log look something like this:

Jun23-18:27:52.213 ESAM
<rpksmsresp><rc>0</rc><msg>OK</msg><type>Live</type></rpksmsresp>
Jun23-18:27:52.214 ESAM: sn=xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx add: --oatvra del: wait: 27 sec
Jun23-18:27:52.258 ESAM: v=3&sname=AUTOKEY&sn=xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx&huid=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx&maxkeylen=2048&...

There is a Splunk Universal Fowarder installed on the system sending to our clustered indexers, all running 6.2.3.

When ingested into Splunk, all of these are combined into a single event instead of broken out into separate events. We have the following props.conf in place on both the forwarder and on the indexer cluster:

[sm-esam-access]
TIME_FORMAT=%b%d-%H:%M:%S.%N
TIME_PREFIX=^
TZ=UTC
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
disabled = false

I've played around with various applications of the props.conf stanza for this sourcetype - including and not including statements like BREAK_ONLY_BEFORE=^\w+\d\d-\d\d:\d\d:\d\d with no success. I've even pulled the log data up into a standalone instance of 6.2.3 and played with the "Add Data" section to try to get it to break properly. Problem is, when I do it this way it appears to break properly but the props.conf from that app doesn't work when put into production.

Any thoughts on what might be going on here or how to fix it?

0 Karma

SplunkTrust
SplunkTrust

Hi burras,

these are only hints to give: You're doing linebreaking which is a parsing operation and always happens on the indexer or on a heavy weight forwarder (see the wiki on this topic http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F )
To troubleshoot this you can check at first if the stanza name in the props.conf exactly matches; is your sourcetype really called sm-esam-access?
Next check if any other props.conf is taking precedences over yours or if your is applied at all with btool:

$SPLUNK_HOME/bin/cmd btool props list --debug

or

$SPLUNK_HOME/bin/cmd btool props list sm-esam-access --debug

Next would be to check if any defined regex is matching or not and last but not least remember this props setting is only valid for any new incoming events.

Hope that helps ...

cheers, MuS

Communicator

Thanks MuS. I verified that the stanza name matches up directly with the sourcetype. And yes, it is actually called sm-esam-access 🙂

Thanks for the link to the Wiki - I checked that out previously but hadn't had any luck getting the breaks to work correctly so that's how I ended up with props.conf in different locations.

I tried to run the btool commands you list to check props.conf precedences but it kept coming back with the following error: /opt/splunk/bin/splunkd: error while loading shared libraries: libjemalloc.so.1: cannot open shared object file: No such file or directory

0 Karma

SplunkTrust
SplunkTrust

run it as the user splunk or run this first source /opt/splunk/bin/setSplunkEnv to set all needing environment settings.

0 Karma

Communicator

Okay, that fixed the run problem. The only props.conf I show on the indexers that contains the sourcetypes that I'm working on are the correct one:

[root@resvasplindex06 ~]# btool props list --debug |grep -v default | grep esam |more
/opt/splunk/etc/slave-apps/cluster/local/props.conf [sm-esam-access]
/opt/splunk/etc/slave-apps/
cluster/local/props.conf [sm-esam-audit]
/opt/splunk/etc/slave-apps/_cluster/local/props.conf [sm-esam-error]

From what I can tell the one specifically for sm-esam-access looks okay as well (everything is either from cluster props.conf or from the default):

[root@resvasplindex06 ~]# btool props list sm-esam-access --debug
/opt/splunk/etc/slave-apps/cluster/local/props.conf [sm-esam-access]
/opt/splunk/etc/system/default/props.conf ANNOTATE
PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTOKVJSON = true
/opt/splunk/etc/slave-apps/cluster/local/props.conf BREAKONLYBEFORE = ^\w+\d\d-\d\d:\d\d:\d\d
/opt/splunk/etc/system/default/props.conf BREAK
ONLYBEFOREDATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/system/default/props.conf DATETIMECONFIG = /etc/datetime.xml
/opt/splunk/etc/system/default/props.conf HEADER
MODE =
/opt/splunk/etc/system/default/props.conf LEARNSOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE
BREAKERLOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MAX
DAYSAGO = 2000
/opt/splunk/etc/system/default/props.conf MAX
DAYSHENCE = 2
/opt/splunk/etc/system/default/props.conf MAX
DIFFSECSAGO = 3600
/opt/splunk/etc/system/default/props.conf MAXDIFFSECSHENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX
EVENTS = 256
/opt/splunk/etc/system/default/props.conf MAXTIMESTAMPLOOKAHEAD = 128
/opt/splunk/etc/system/default/props.conf MUSTBREAKAFTER =
/opt/splunk/etc/system/default/props.conf MUSTNOTBREAKAFTER =
/opt/splunk/etc/system/default/props.conf MUST
NOTBREAKBEFORE =
/opt/splunk/etc/slave-apps/cluster/local/props.conf NOBINARYCHECK = true
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/slave-apps/
cluster/local/props.conf SHOULDLINEMERGE = false
/opt/splunk/etc/slave-apps/
cluster/local/props.conf TIMEFORMAT = %b%d-%H:%M:%S.%N
/opt/splunk/etc/slave-apps/
cluster/local/props.conf TIMEPREFIX = ^
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/system/default/props.conf TRUNCATE = 10000
/opt/splunk/etc/slave-apps/
cluster/local/props.conf TZ = UTC
/opt/splunk/etc/system/default/props.conf detecttrailingnulls = false
/opt/splunk/etc/slave-apps/_cluster/local/props.conf disabled = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/system/default/props.conf sourcetype =

I also reverified that the stanzas in props.conf match up to the stanzas in inputs.conf:
[root@revaapp01 local]# cat inputs.conf |grep esam
[monitor:///opt/securemedia/var/app-8082/logs/esam-access.log]
sourcetype = sm-esam-access
[monitor:///opt/securemedia/var/app-8082/logs/esam-audit.log]
sourcetype = sm-esam-audit
[monitor:///opt/securemedia/var/app-8082/logs/esam-error.log]
sourcetype = sm-esam-error

0 Karma

SplunkTrust
SplunkTrust

props.conf and transforms.conf troubleshooting is hard I know and mostly like the hardest thing is that you must figure out what's wrong, because I can only provide some basic hints like these:

http://answers.splunk.com/answers/4075/whats-the-best-way-to-track-down-props-conf-problems.html
http://docs.splunk.com/Documentation/Splunk/6.2.3/Troubleshooting/Enabledebuglogging

0 Karma

Communicator

Thanks MuS - I'll continue to dig into it and let everyone know what I find...

0 Karma