All Apps and Add-ons

Splunk ISE TA fails when distributed via Cluster Master

bjoernhansen
Path Finder

Hey,

I use this TA on an Indexer cluster, and it fails to actually process any event. After some troubleshooting, the core of the problem lies here:

05-05-2017 16:49:01.514 +0200 ERROR AggregatorMiningProcessor - Uncaught exception in Aggregator, skipping an event: Can't open DateParser XML configuration file "/opt/splunk/etc/apps/Splunk_TA_cisco-ise/default/datetime_udp.xml": No such file or directory - data_source="/opt/logfiles/10.1.2.3/2017-05-05.log", data_host="10.1.2.3", data_sourcetype="cisco:ise:syslog"

When the app is distributed via the cluster master, it is stored in /etc/slave-apps on the indexers, and the reference in the default/props.conf does not work:

[cisco:ise]
DATETIME_CONFIG = /etc/apps/Splunk_TA_cisco-ise/default/datetime_udp.xml

[cisco:ise:syslog]
DATETIME_CONFIG = /etc/apps/Splunk_TA_cisco-ise/default/datetime_udp.xml

I fixed this by creating a local/props.conf like this:

[cisco:ise]
DATETIME_CONFIG = /etc/slave-apps/Splunk_TA_cisco-ise/default/datetime_udp.xml

[cisco:ise:syslog]
DATETIME_CONFIG = /etc/slave-apps/Splunk_TA_cisco-ise/default/datetime_udp.xml

This obviously only fixes it for cluster environments.

From my understanding, DATETIME_CONFIG allows for no variable that includes the app path, so I see no obvious solution to fix this.
If it can't be fixed, at least an eye-catching hint about this in the install guide could be helpful.

ChrisBell04
Communicator

I too have encountered this with the CM. Finally decided to remove it entirely.

custom prefix stuff for our syslogs, modify to meet your env.

[cisco:ise:syslog]
DATETIME_CONFIG = 
MAX_TIMESTAMP_LOOKAHEAD = 40
NO_BINARY_CHECK = false
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N %:z
TIME_PREFIX = \S+\s\S+\s\S+\s\d+\s\d+\s\d+\s
0 Karma

highsplunker
Contributor

OMG. Seems i have similar problem. But i'm dealing with heavy forwarder problem now. Nevertheless the error is the same. Not fixed it yet.

0 Karma

jcspigler2010
Path Finder

Just dealt with this today at a customer site. Luckily I tailed the splunkd.logs on one of the indexers while their syslog w/ UF was firing data at this particular indexer. Saw the error and spiraled down to your same conclusion. I submitted feed back on this TA to have the doc site updated with instructions to modify the DATETIME_CONFIG variable in clustered index environments...

Get Updates on the Splunk Community!

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...