So the eval command's if works just like the IF macro in Excel. The idea here the search is trying to "zero-fill" a field if the contents of the field is the empty string. This is a fairly common idiom, and can sometimes be replaced with the coalesce function like so: | eval verifiedButBounced = coalesce('detail.verifiedButBounced',0) The trickery here is the use of both " and ' . In eval , a . (dot) character can be used to mean string concatenation. It can also be used in a field name, like if a field comes from JSON data. So if you have the string detail.verifiedButBounced do you mean a field named detail.verifiedButBounced or do you mean detail concatenated with verifiedButBounced ? It is ambiguous! The eval command uses a single-quote around a field name to "escape" otherwise special characters within that field name. Double quotes are used to represent static strings. So your original eval could be expressed as: If the field named detail.verifiedButBounced is not equal to the string "" then set the new field verifiedButBounced equal to the value of detail.verifiedButBounced . Otherwise, set the value of the new field verifiedButBounced equal to 0. ... View more

So generally the answer here is a resounding "no you cannot." Remember that on almost all modern operating systems, the kernel enforces a process boundary wall. It is not trivial for process 1 to reach over, under, or around that wall to read the memory of your Java process. Yes, there is the concept of SYSV shared memory or memory mapped files - but these generally require lots of coordination and agreement between processes so that locks are taken when the shared memory is being updated and so that the layout of the shared memory is understood and makes sense to both processes. The layout of the Java heap and in-memory objects is not entirely clear to non-Java processes and different JVMs format both heaps and objects differently! It is just too much to ask Splunk to be able to yoink log events out of the address space of another process. Write them to a file, or a socket, or use the HTTP event collector and a log4j output .. just not memory, please. (It is also entirely possible I have misunderstood your question) ... View more

(Warning: Medium length troubleshooting section follows, enjoy or skip to the Teal Deer at the end) Normally introspection data is collected by a helper process that starts as a child when splunkd does. This process is a copy of splunkd with the argument "instrument-resource-usage". On some hosts we see this process: $ ps -ef | grep splunk | grep -v search splunk 18731 1 36 Feb26 ? 6-03:18:49 splunkd -p 8089 restart splunk 18732 18731 0 Feb26 ? 01:28:01 [splunkd pid=18731] splunkd -p 8089 restart [process-runner] splunk 18957 18732 0 Feb26 ? 00:39:53 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore On some we don't: $ ps -ef | grep splunkd | grep -v "search" splunk 30198 1 39 02:44 ? 03:07:52 splunkd -p 8089 restart splunk 30199 30198 0 02:44 ? 00:02:00 [splunkd pid=30198] splunkd -p 8089 restart [process-runner] And this missing process correlates with the hosts where we are not getting introspection data. Normally this process should get started every time at Splunk start. We can see it get started in splunkd.log during our last restart: $ egrep instrument-resource-usage $SPLUNK_HOME/var/log/splunk/splunkd.log 03-14-2016 02:44:10.822 -0400 INFO ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage But it's not running long after the restart, as it's gone when we look only seconds later. We can try to run it ourselves and see if we get output: $ /opt/splunk/bin/splunkd instrument-resource-usage INFO RU_main - I-data gathering (Resource Usage) starting; period=10s Cond 'swapSize_free <= ru._swap.get()' false; line 443, collect_hostwide__Linux() Oof. An assertion it looks like, related to swap? Strange. Let's see what our swap looks like: $ free -m total used free shared buffers cached Mem: 12921 12629 291 0 237 10158 -/+ buffers/cache: 2234 10686 Swap: 14855 1759218580 256081 Okay. Weird. We have LOTS more swap free than we do used. How does free come up with those numbers? Time to dig out the source. (tick tock tick tock tick tock) Turns out free gets those numbers from /proc/meminfo - and that is messed up too. $ grep Swap /proc/meminfo SwapCached: 19784 kB SwapTotal: 15212520 kB SwapFree: 262221008 kB (And here's the TL;DR) Now we're getting somewhere. If /proc/meminfo is wrong, then it's most likely a kernel bug (whee!). A little googling finds https://access.redhat.com/solutions/1571043. If you don't have access behind Red Hat's paywall - the gist is that there was a bug in kernel-2.6.32-573.1.1.el6.x86_64 that causes SwapFree to get counted wrong. You can reboot to fix it temporarily but it's only temporary. Fixing it permanently requires updating to a newer kernel. This problem likely also occurs in CentOS at the same kernel level. ... View more

Martin is right. There are two (maybe three) fundamental facts here: [1] The process of authorizing a search head to search-peer with an indexer requires a user name and password on the remote indexer that is ONLY used to authenticate the linking process. Once the peering has been done, the account can be deleted entirely as the indexer will have a cryptographic key that can be used to authenticate the remote search head itself. [2] From a user authentication and roles perspective, the search head has full and total autonomy. Once it is authorized to peer with an indexer, users on that search head can do ANYTHING they want to the indexers from a search perspective without the indexer's consent. By agreeing to let a search head peer with your indexer, you are confirming your total and absolute unwavering trust in the admins of that search head. There is simply no existing way for you to allow a search head to peer with your indexer and restrict the activity of its users from the indexer. [3] When you have a cluster, pointing a search head at the individual cluster peers is not a good idea. Always configure the search head as a mode=searchhead member of the cluster and point it at the cluster master. ... View more

I would suggest a support case. SIGFPE is a floating point exception, and you can see in your logs there a "trap divide error". It sounds like this new input may have issues. Disable it, see what difference that makes on your crashing. But, regardless, submit a support case with a diag. Also you could try upgrading to 6.3.1 - I don't see this as a known issue, but it never hurts to show support you've done all you can do before contacting them. ... View more

So, the name of the game here is "atomicity". What I mean is that if Splunk can, for a moment, see the file "in between states" during your overlay, then it can basically make a determination of .. "So, I know this file by CRC, but the last time I saw it, it was bigger than it is now. It must have been rotated and this is a new 'instance' of that file that I must read from the beginning." On most operating systems, a file COPY is no where near atomic. If you were fast enough to see it happen you would see the destination file growing a little bit every fraction of a second. But a file move (at least on the same drive / filesystem) is usually atomic (at least on Unix based OSs). What I would suggest is copying the files to a temporary directory in the same drive - and then move them into your monitored directory. This almost always works on Unixes - but I'm not sure how well it does on Windows TBH. ... View more