Cisco eStreamer eNcore 4.0.9 Add-on for Splunk 8.1.0.1- Why am I not receiving any results?

[alcman]

I'm running splunk 8.1.0.1 and Cisco eStreamer eNcore 4.0.9 and configured cisco FMC for estream integration but it doent show any logs. I have some Errors in splunkd.log and estreamer.log.

I dont  receive any result when I search for

sourcetype="cisco:estreamer:data"

splunkd.log:

12-01-2020 10:55:45.104 +0330 INFO DatabaseDirectoryManager - Finished writing bucket manifest in hotWarmPath=/opt/splunk/var/lib/splunk/_telemetry/db duration=0.000
12-01-2020 10:56:16.088 +0330 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
12-01-2020 10:56:35.888 +0330 WARN CalcFieldProcessor - Invalid eval expression for 'EVAL-first_pkt_sec' in stanza [cisco:estreamer:data]: The expression is malformed. Expected AND.
12-01-2020 10:56:43.574 +0330 WARN CalcFieldProcessor - Invalid eval expression for 'EVAL-first_pkt_sec' in stanza [cisco:estreamer:data]: The expression is malformed. Expected AND.
12-01-2020 11:00:00.002 +0330 INFO ExecProcessor - setting reschedule_ms=3599998, for command=/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
12-01-2020 11:00:45.541 +0330 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/TA-eStreamer/bin/splencore.sh clean" find: ‘../../data’: No such file or directory
12-01-2020 11:04:45.710 +0330 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
12-01-2020 11:09:16.851 +0330 WARN CalcFieldProcessor - Invalid eval expression for 'EVAL-first_pkt_sec' in stanza [cisco:estreamer:data]: The expression is malformed. Expected AND.
12-01-2020 11:09:47.042 +0330 WARN CalcFieldProcessor - Invalid eval expression for 'EVAL-first_pkt_sec' in stanza [cisco:estreamer:data]: The expression is malformed. Expected AND.

estreamer.log

2020-12-01 10:57:47,097 Service ERROR [no message or attrs]: PID file already exists
2020-12-01 10:58:58,905 Monitor INFO Running. 3465700 handled; average rate 1604.32 ev/sec;
2020-12-01 10:59:47,105 Service ERROR [no message or attrs]: PID file already exists
2020-12-01 11:00:58,856 Monitor INFO Running. 3642600 handled; average rate 1597.5 ev/sec;
2020-12-01 11:01:47,003 Service ERROR [no message or attrs]: PID file already exists
2020-12-01 11:02:59,543 Monitor INFO Running. 3729700 handled; average rate 1553.92 ev/sec;
2020-12-01 11:03:46,998 Service ERROR [no message or attrs]: PID file already exists
2020-12-01 11:04:59,259 Monitor INFO Running. 3744100 handled; average rate 1485.59 ev/sec;
2020-12-01 11:05:47,086 Service ERROR [no message or attrs]: PID file already exists
2020-12-01 11:06:59,648 Monitor INFO Running. 3759600 handled; average rate 1423.95 ev/sec;
2020-12-01 11:07:47,049 Service ERROR [no message or attrs]: PID file already exists
2020-12-01 11:08:59,299 Monitor INFO Running. 3773900 handled; average rate 1367.29 ev/sec;
2020-12-01 11:09:47,126 Service ERROR [no message or attrs]: PID file already exists
2020-12-01 11:10:59,220 Monitor INFO Running. 3788200 handled; average rate 1315.21 ev/sec;

[gurlest]

Me again... While deploying this we noted that the TA-eStreamer/bin/encore/data/splunk directory being the data directory causes more problems than not.

The newest problem being that the /bin directory is replicated, so if the heavy-forwarder has any searchpeers, it will cause bundle replication issues because Splunk will be attempting to replicate 200gb+ data directory all over the place.

We have opted to move the data directory back to the old location of TA-eStreamer/data in the TA--eStreamer/bin/encore/estreamer.conf file.  This addresses the issues with:

• /bin/splencore.sh clean() location being incorrect
• /default/inputs.conf monitor location for data files being incorrect
• /bin/encore/data/splunk being 200gb+ causes replication issues if the hfw has any searchpeers

[mattcosa]

Hi all, have been working with 4.2.2 and now 4.6.0 release and there are numerous other discrepancies in the wrapper's that are shipped with the add-on.

In splencore.sh, missing cases that are still present in the default inputs.conf . I have restored two additional cases to support the clean and status functions on line 197 (in 4.6.0 of splencore.sh);

clean)
clean
;;

status)
status
;;

You can optionally add additional text to describe the available functions in the echo block at the bottom.

Both these should output to sourcetype IN (cisco:estreamer:status cisco:estreamer:clean)

The clean() function has no output. Appending a -print will log what you're actually deleting under the sourcetype cisco:estreamer:clean

find /var/log/my/custom/path -type f -mmin +720 -delete -print

The 4.6.0 stop() function does not work properly. Line 141 appears to have been added in 4.6.0 but is commented out. Allegedly this was fixed in the 4.6.0 release. 4.2.2 has a much more robust code block. I have replaced the 4.6.0 code block with the 4.2.2 code, which works assuming the estreamer service is actually responsive to kill -s INT.

Using /local/inputs.conf to specify a custom log path and custom index, we have the following full configuration which enables all the scripts, and monitors the correct custom path for log data (more a Splunk conf than Cisco).

[script://./bin/splencore.sh clean]
disabled = 0
index = myindex

[script://./bin/splencore.sh status]
disabled = 0
index = myindex

[script://./bin/splencore.sh start]
disabled = 0
index = myindex

[monitor://$SPLUNK_HOME/etc/apps/TA-eStreamer/bin/encore/data]
disabled = 0
index = myindex

[monitor:///var/log/my/custom/path/encore.log*]
disabled = 0
index = myindex
source = encore
sourcetype = cisco:estreamer:data
crcSalt = <SOURCE>

Lastly, on deployment when configuring the pkcs12 keys, we required an additional export to get the rather than the documented 2. We were receiving the error:

WARNING: can't open config file: /opt/splunk-home/openssl/openssl.conf

This path obviously doesnt exist (specifically /opt/splunk-home), and we were unable to determine where it was set.

Assuming a default install, the following fixed the issue:

export SPLUNK_HOME=/opt/splunk
export LD_LIBRARY_PATH=$SPLUNK_HOME/lib
export OPENSSL_CONF=/opt/splunk/openssl/openssl.cnf

IMvHO, this TA out of the box is not production ready without these changes, and the code over the last 3 months has degraded to the point of being non-functional.

To this effect, I encourage you to reach out to the Cisco Splunk development team. I have had positive response from: fp-4-splunk@cisco.com and encore-community@cisco.com.

Thank you to @gurlest  for all the commentary thus far.

[_joe]

I am on 6.4.2 and I just had to edit the clean statement to find the events (and I didn't customize the location). I've been scratching my head thinking I must have missed a variable somewhere but then I came across this post... what is going on?...