All Apps and Add-ons

Weird behavior of addon

a_naoum
Path Finder

Hi, I'm seeing a very strange behavior in the search after the installation of the addon.

Without auditd addon or secure-addon

I'm doing a simple search in a strict period ex. 12.7.2019 09:00 to 12.7.2019 10:00
index=linux_sec host=blablabla1
I'm getting from this result 102 events from both /var/log/secure and /var/log/audit/audit.log which is ok. Sourcetypes are linux:audit and linux_secure. Data is get from the official Splunk addon (with sourcetypes there as linux_audit and linux_secure)

After I install the addon, for exact same period instead of getting 102 events I'm getting much less (around 32) and the linux_secure sourcetype is missed. Tabling the raw data I'm getting 102.
If I remove the addon everything is getting back to normal. The behavior happen with any kind of server in any kind of period.

Any ideas? Do I miss something?

Thanks

0 Karma

woodcock
Esteemed Legend

I am sure that @doksu can help.

0 Karma

doksu
Contributor

Indeed, thanks @woodcock. @a_naoum that's a curious situation. Could you please let me know the version of Splunk and the apps you're using with any customisations? Also, if you're using ES in that search environment and its version.

a_naoum
Path Finder

Splunk is 7.0.3 and ES is 5.0.1.

After couple of hours of troubleshooting (usually there is an enlightenment after I create a question here) I found the issue.
This eval : EVAL-process_name = urldecode(replace(proctitle,"([0-9A-F]{2})","%\1")) because there is the case of null characters (%00). If founds one or more times this one it is just.... don't want to work and doing the above behavior. Not sure if it is issue with our environment or OS (I found it for now only in RHEL7).
I managed to "fix" it by doing an extra replace and remove the characters: EVAL-process_name = urldecode(replace(replace(proctitle,"([0-9A-F]{2})","%\1"),"%00",""))
but I'm not sure about the "correct" way.

btw: the rest urldecode evals doesn't have any issue most probably because the data doesn't contain null characters.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...

Design, Compete, Win: Submit Your Best Splunk Dashboards for a .conf26 Pass

Hello Splunkers,  We’re excited to kick off a Splunk Dashboard contest! We know that dashboards are a primary ...

May 2026 Splunk Expert Sessions: Security & Observability

Level Up Your Operations: May 2026 Splunk Expert Sessions Whether you are refining your security posture or ...