Splunk is 7.0.3 and ES is 5.0.1.
After couple of hours of troubleshooting (usually there is an enlightenment after I create a question here) I found the issue.
This eval : EVAL-process_name = urldecode(replace(proctitle,"([0-9A-F]{2})","%\1")) because there is the case of null characters (%00). If founds one or more times this one it is just.... don't want to work and doing the above behavior. Not sure if it is issue with our environment or OS (I found it for now only in RHEL7).
I managed to "fix" it by doing an extra replace and remove the characters: EVAL-process_name = urldecode(replace(replace(proctitle,"([0-9A-F]{2})","%\1"),"%00",""))
but I'm not sure about the "correct" way.
btw: the rest urldecode evals doesn't have any issue most probably because the data doesn't contain null characters.
... View more