Old thread but no final solution explanation given so I wanted to share a solution as we were facing a similar issue: We experienced the behaviour when using iplocation in conjunction with eventstats. When iplocation was used before eventstats, the location was correct, when used after eventstats, it was wrong. Cause: Eventstats (as well as stats) is a data processing / non-streaming command (-> it runs on the search head). Iplocation is a distributable streaming command (-> it can run on the indexer). So using IP location BEFORE eventstats (and right after the base search) makes it run on the indexers. When using iplocation AFTER eventstats it runs on the search head. The reason for yielding different location results was due to different iplocation database versions (GeoLite2..) on search head and indexers. So make sure, your geo location db is up-to-date and identical on all your Splunk components then iplocation yields the same results regardless where it is used in the search. Where you place it depends on your search. Usually it is advisable to enrich AFTER transforming/aggregations commands like stats or eventstats. But as iplocation is a distributable streaming command it might perform better when it can run on the indexers instead of the search head. ... View more

We were facing the same issue and I stumbled over this thread in search for possible causes. As I did not find one but eventually found the cause for the different locations of IPs depending on where iplocation is used in the search, I wanted to share it. The given and accepted answer above is misleading / incomplete (it does not explain the different results depending on where in the search iplocation is used) We experienced the behaviour when using iplocation in conjunction with eventstats. When iplocation was used before eventstats, the location was correct, when used after eventstats, it was wrong. Cause: Eventstats (as well as stats) is a data processing / non-streaming command (-> it runs on the search head). Iplocation is a distributable streaming command (-> it can run on the indexer). So using IP location BEFORE eventstats (and right after the base search) makes it run on the indexers. When using iplocation AFTER eventstats it runs on the search head. The reason for yielding different location results was due to different iplocation database versions (GeoLite2..) on search head and indexers. So make sure, your geo location db is up-to-date and identical on all your Splunk components then iplocation yields the same results regardless where it is used in the search. Where you place it depends on your search. Usually it is advisable to enrich AFTER transforming/aggregations commands like stats or eventstats. But as iplocation is a distributable streaming command it might perform better when it can run on the indexers instead of the search head. ... View more

mike.randal, are you sure you are getting dublicate events (dublicate JSON events) or are you just seeing dublicate entries in the fields if you output events with | table ... In the second case, this is likely due to the Add-On doing index-time (via indexed_extractions = JSON) and search-time field extractions (via KV_MODE = JSON) resulting in dublicate field entries, if you have the addon installed on Heavy Forwarder/Indexer and Search Head. See: https://answers.splunk.com/answers/223095/why-is-my-sourcetype-configuration-for-json-events.html You could circumvent this by using spath on the fields you want to display and output to new fields. Or you adopt the props.conf settings of the Add-On (which might have other implications). ... View more

Hi ankithreddy777, not sure if the question is still relevant, but as I had a similar issue ( I think) I will share my solution. If I interpret your question correctly, you have something like: search ... | processing ... | outputlookup append=true file_1.csv | append [ inputlookup file_1.csv | lookup ... | processing and merging... ] | final processing or outputlookup If this is your szenario, then your first outputlookup will have the results of your initial search appended correctly, BUT the inputlookup file_1.csv in the append section is subsearch and will get parsed and dispatched before your outer search, so it will take the version of file_1.csv before the results from your search have been appended. The append command appends the results of a subsearch to the current results. To solve this, you can just replace append by appendpipe. This appends the result of the subpipeline to the search results. Unlike a subsearch, the subpipeline is not run first. The subpipeline is run when the search reaches the appendpipe command. ... View more

Hi, as there is no answer yet and I had the same question: Yes, proxy settings in /etc/splunk-launch.conf are passed to Splunk apps. Just adding http_proxy=<proxyname>:<port> or for https: https_proxy=<proxyname>:<port> to splunk-launch.conf worked for me and the Hipchat app happily uses the proxy for notification forwarding. ... View more

Hi MuS, sorry for the long delay in accepting the answer. It was the right approach with the following search I got it working the way I wanted: search ... | ldapfilter domain=mydomain search="(sAMAccountName=$user$)" attrs="DisplayName,givenName,mail" | eval firstName=givenName | eval mail=mail | eval fullName=DisplayName | eventstats values(mail) AS mail_adresses| eval recipients=mvjoin(mail_adresses, ",") | table eventTime, field1, firstName, fullName, mail, recipients |sendemail to="mailbox@example.net" bcc=$result.recipients$ from="sender@example.net" footer="My Mail Footer" server=localhost subject="Some Subject" message="Hallo $result.firstName$, some text" the evals to explicitly (re)assign the queried ldap attributes to field names are necessary as for some reason those fields are "special" in some way. They show up in a table but when sendmail is used in the search then the fields are empty. ... View more

I also have the same issue. Would be nice to know if this is a bug or works as designed? Maybe it would interfere with the scheduled search/alert mailing functionality? Functionality is almost identical, however, in the scheduled search mail alerting, one cannot set the mail sender (from). This is always the globaly definied mail sender. So a working sendemail command in scheduled searches would be helpful. ... View more

ok, thanks for the clarification. The traps I posted above is a tcpdump output. So likely the values are encoded and tcpdump just decodes them. Unfortunately, fixing the encoding on the source will not be that simple. The trap Sender is a BMC of a Quanta Server. That means for us updating the BMC firmware on hundreds of Servers (given an update is available) ... Would implementing the Workaround hack described in http://pysnmp.sourceforge.net/faq.html into snmp.py be a workaround Option or would that introduce other issues? ... View more

Hi Damien, thanks a lot for the quick answer. but I don't quite get it. Where is Encoding used here on the vendor side? The system.sysUpTime.0 field in the trap supplies the System uptime in miliseconds. system.sysUpTime.0=3084126100 (356 days) triggers the error system.sysUpTime.0=1113987000 (240 days) works ok. What exactly does snmp.py with those values? Looking at the error: "ValueRangeConstraint(0, 4294967295) failed at: "-1210841196"" at TimeTicks it seems that there is a calculation of sysUptime - upperBoundValue (3084126100 - 4294967295 -1) =-1210841196, which Triggers the error. But why is this done? And why does the other other trap gets processe ok as it would also result in a negative value if the above calculation would be done. ... View more

Yes, directory /opt/splunk on DCN was completely deleted. UF is installed in /opt/splunkforwarder. And yes, SH was restarted after DCN removal. ... View more

Yes it is still on and has to be. The VMWare App is still in use and there are two other DCNs active. It's just that apparently the old DCN ist still included in the scheduling and I can't figure out where or why. ... View more

Hi, Thanks for the answers so far but did not bring me further. I tried btool before without results. However, I could pinpoint it down to be related to the VMWare app. The Network Connections from the indexers are initiated by a python process running the ta_vmware_collection_sheduler: splunk 11012 10886 15 Mar11 ? 02:36:00 python /opt/splunk/etc/apps/Splunk_TA_vmware/bin/ta_vmware_collection_scheduler.py So I continued on the Search Head where the VMWare App is installed as this is where the scheduling happens. I searched all the VMware related apps and TAs for IP or hostname of the old DCN but did not find a config file where it still was present. In the relevant DCN config file (/opt/splunk/etc/apps/Splunk_TA_vmware/local/hydra_node.conf) only the correct Hosts are still present. In the logs of the seach head I could trace the deletion of the DCN: web_access.log:127.0.0.1 - xxxxxxx [09/Mar/2015:13:24:20.918 +0100] "DELETE /en-US/custom/splunk_for_vmware/splunk_for_vmware_setup/splunk_for_vmware/delete_collection_node/https:%2F%2Fdcn-hostname:8089 HTTP/1.1" 200 - "https://searchhead:8000/en-US/custom/splunk_for_vmware/splunk_for_vmware_setup/splunk_for_vmware/show_collection_setup" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36" - 54fd90f4eb7fbd79676150 887ms But apparently, it still is used by the Scheduler worker processes as acollection node. So maybe someone with expert Knowledge of the VMWare app can give a further hint on where the Problem could be. UPDATE (2015-03-16): As SA_Hydra is responsible for the data collection sheduling, we disabled it (-> move to disabled apps) on the indexers. That stopped the connections from the indexers to the old CDN. However, it had side effects. Now the indexers are throwing errors: Search peer index1.example.net has the following message: Unable to initialize modular input "ta_vmware_collection_worker" defined inside the app "Splunk_TA_vmware": Introspecting scheme=ta_vmware_collection_worker: script running failed (exited with code 1). After inspecting /opt/splunk/etc/apps/Splunk_TA_vmware/local I noticed config files which I then would also not expect on a indexer: -rw-------. 1 splunk splunk 979 Mar 5 16:33 app.conf -rw-------. 1 splunk splunk 357 Mar 6 09:07 hydra_node.conf -rw-rw-r--. 1 splunk splunk 238 Mar 6 11:03 indexes.conf -rw-------. 1 splunk splunk 150 Mar 12 15:20 inputs.conf -rw-------. 1 splunk splunk 737 Mar 6 09:07 ta_vmware_collection.conf -rw-------. 1 splunk splunk 177 Feb 17 09:20 ta_vmware_syslog_forwarder.conf -rw-------. 1 splunk splunk 333 Mar 6 09:07 vcenter_forwarder.conf I find this config identical on my Search Head, where the VMWare Collection Config is done via Splunk Web and which then controls the CDNs. So no idea how it came to the indexers... As far as distribution of VMWare app components to Splunk servers goes, we applied the matrix here: http://docs.splunk.com/Documentation/VMW/3.1.4/Configuration/Componentreference According to this, SA_Hydra is necessary on the indexers. †† Install SA-Utils and SA-Hydra on the Splunk indexers to stop modular input introspection from failing. This is a workaround to a known issue. These components do not affect the operation of your indexers. But apparently, it does somehow affect the operations of the indexers. So the next step would be to disable the hydra/collection config in Splunk_TA_vmware/local/ on the indexers. But I still wonder, how the config got there in the first place or if, why and which VMWare app components are really, REALLY necessary on the indexers. Any ideas or definite answers there? ... View more