Here is another technique to do fuzzy matching on multi-row outputs but without using a temp file, to avoid same-file-use for instances where many searches may call the code at once. We are using versions of this for some RBA enrichment macros. Thanks @japger_splunk for a tip on multireport and @woodcock for the map example! | makeresults | eval src="1.1.1.1;2.2.2.2;3.3.3.3;4.4.4.4;5.5.5.5" | makemv src delim=";" | mvexpand src `comment(" BEGIN ENRICHMENT BLOCK")` `comment("REMEMBER ROW ORDER AND MARK AS ORIGINAL DATA")` | eval original_row=1 | streamstats count AS marker `comment("FORK THE SEARCH: FIRST TO PRESERVE RESULTS, SECOND TO COMPARE EACH ROW AGAINST LOOKUP")` | multireport [ ] `comment("FOR EACH ROW, RUN FUZZY MATCH AGAINST LOOKUP AND SUMMARIZE THE RESULTS")` [| map maxsearches=99999 search=" | inputlookup notable_cache | eval marker=$marker$, src=$src$ | eval match=if(like(raw,\"%\".src.\"%\"), 1, 0) | where match==1 | eval age_days = (now()-info_search_time)/86400 | eval in_notable_7d=if(age_days<=7,1,0), in_notable_30d=if(age_days<=30,1,0) | stats values(marker) AS marker, sum(in_notable_7d) AS in_notable_7d_count, sum(in_notable_30d) AS in_notable_30d_count BY src "] `comment("INTERLEAVE THE ORIGINAL RESULTS WITH THE LOOKUP MATCH RESULTS")` | sort 0 marker, in_notable_30d_count, in_notable_7d_count `comment("TRANSPOSE DATA FROM ABOVE")` | streamstats current=f window=1 last(in_notable_30d_count) AS prev_in_notable_30d_count, last(in_notable_7d_count) AS prev_in_notable_7d_count `comment("GET RID OF THE LOOKUP RESULTS")` | where original_row==1 `comment("CLEAN UP THE DATA")` | rename prev_in_notable_30d_count AS in_notable_30d_count, prev_in_notable_7d_count AS in_notable_7d_count | fillnull value=0 in_notable_30d_count, in_notable_7d_count | fields - original_row, marker ... View more

My two cents on this. I tried using setfacl in our environment, but auditd kept changing the group permissions on audit.log when it rolls the log, which in turn jacks up the setfacl permissions. I tried running incrond as a watchdog to automatically change the permissions back if auditd messed them up, but found that incrond would crash after a while. Here is the solution I settled on. This is in the context of RHEL7/CentOS7, running Splunk as non-root user "splunk". From some of my old notes: #the following steps make the audit.log readable by splunk #add adm group if dne, add splunk user to adm group, set log_group to adm in auditd.conf, fix dir perms, restart auditd id -g adm &>/dev/null || groupadd adm usermod -a -G adm splunk sed -i.bak 's/log_group = .*/log_group = adm/g' /etc/audit/auditd.conf chgrp -R adm /var/log/audit/ chmod 0750 /var/log/audit/ chmod 0640 /var/log/audit/* service auditd restart ... View more

Are you talking about this one? https://splunkbase.splunk.com/app/2937/ If so, I'm curious if it works with Server 2008 flavors. Says it's built for 2012r2 and up. ... View more

Darn - I spoke too soon. Looks the searches behind the panels in the ES dashboard want more fields. Going to see how much work will be involved in normalizing this data source further. Best option might just be to let the Splunk devs do the work and wait for an official update to the TA. ... View more

Mikealbje, I just implemented this workaround in our Splunk instance. Thank you! This just saved me hours of work! BTW, I had to fix a typo - the stanza in transforms.conf: [msad_dns_vendor_query_type_lookup] should be: [dns_vendor_query_type_lookup] Another modification I made to your method was copying the default DNSServer-NT6 TA to a custom TA "DNSServer-NT6_cimfix", implementing your fixes, and deleting the default/inputs.conf. I have the original vanilla DNSServer-NT6 TA pushing down to the AD boxes for log collection, and the DNSServer-NT6_cimfix band-aid version of the TA pushing to the search head and indexer for index-time and search-time field extraction. The extractions look like they're working. Just waiting on Network Sessions (DNS) data model acceleration to rebuild and the DNS ES dashboard should be open for business. ... View more

Per Kyle from support's recommendation, I modified the init script to use /bin/su like in the older 6.0.x scripts and it looks like it's fixed the issue as a stop-gap solution. I am running Splunk Enterprise 6.1.3. #!/bin/sh # # /etc/init.d/splunk # init script for Splunk. # generated by 'splunk enable boot-start'. # # chkconfig: 2345 90 60 # description: Splunk indexer service # RETVAL=0 . /etc/init.d/functions splunk_start() { echo Starting Splunk... /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" start --no-prompt --answer-yes" RETVAL=$? [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk } splunk_stop() { echo Stopping Splunk... /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" stop " RETVAL=$? [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/splunk } splunk_restart() { echo Restarting Splunk... /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" restart " RETVAL=$? [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk } splunk_status() { echo Splunk status: /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" status " RETVAL=$? } case "$1" in start) splunk_start ;; stop) splunk_stop ;; restart) splunk_restart ;; status) splunk_status ;; *) echo "Usage: $0 {start|stop|restart|status}" exit 1 ;; esac exit $RETVAL ... View more

Weird. The last part of that got screwed up. Should read: Regarding the ifconfig error you were having, try this on the host with the forwarder installed. On some distros (debian), non-root accounts don't know the path to ifconfig. Switch to splunk user: sudo su - splunk which ifconfig Might return nothing. ifconfig Might return: -bash: ifconfig: command not found It's probably in sbin. /sbin/ifconfig If that is the case, maybe you could add something in the pushed nix app that lets it know about that path, or calls ifconfig directly. ... View more

I'm necromancing this question. You said you ran chmod +x *.sh on the app's bin dir, but not sure if you ran it on the app's bin dir on the host with the forwarder installed, or the app's bin dir in your deployment-apps on the deployment server, so I'll post this in case someone finds it useful. I would imagine running chmod on the scripts in the app directory on the host with the forwarder installed would allow the files to get overwritten when a change is detected and the deployment server re-pushes the old files that don't have the execute bit set. Anyway, I had the same issue over here. My solution was the same as yours I think. On your forwarder, check out the scripts directory that has been pushed down from the deployment server: cd /opt/splunkforwarder/etc/apps/Splunk_TA_nix/bin ls -lh Will probably look something like this: -rw------- 1 splunk splunk 3.6K Jun 26 23:39 common.sh -rw------- 1 splunk splunk 3.5K Jun 26 23:39 cpu.sh -rw------- 1 splunk splunk 4.9K Jun 26 23:39 df.sh ... The files do not have execute permission by the splunk user. On your deployment server, you need to give the scripts execute permission, then reload deploy-server: cd /opt/splunk/etc/deployment-apps/Splunk_TA_nix/bin ls -lh Same deal - will probably look something like this: -rw------- 1 splunk splunk 3.6K Jun 26 23:39 common.sh -rw------- 1 splunk splunk 3.5K Jun 26 23:39 cpu.sh -rw------- 1 splunk splunk 4.9K Jun 26 23:39 df.sh ... Fix the permissions: chmod u+x *.sh Reload the deployment server: /opt/splunk/bin/splunk reload deploy-server Wait for the files to push down. You could run this on the forwarder and watch the date on the Splunk_TA_nix dir: watch -n 1 'ls -lh /opt/splunkforwarder/etc/apps' It should push down eventually. Now, on your search head, run this over the last 60 minutes or so to look for results from one of the check scripts: index=main host="YOUR-HOST" eventtype=cpu If you don't see anything, wait a few minutes and run the search again. You could also run this splunk search to see if you are still having the errors: index=_internal host="YOUR-HOST" "ERROR ExecProcessor" Regarding the ifconfig error you were having, try this on the host with the forwarder installed: On some distros (debian), non-root accounts don't know the path to ifconfig. Switch to splunk user: sudo su - splunk which ifconfig (might return nothing) ifconfig Might return: -bash: ifconfig: command not found It's probably in sbin /sbin/ifconfig If that is the case, maybe you could add something in the pushed nix app that lets it know about that path, or calls ifconfig directly. ... View more