About flle

flle · ‎03-31-2022

Hi, we have a severe performance issue with dbxlookup in DB Connect App 3.8 for a MySQL DB. dbxlookups in a Splunk Query take several minutes to return results. The strange thing is, that it only happens when using dbxlookup. Using dbxquery is blazing fast (1.4 seconds to return 100K results) and also when configuring the lookup in the DB Connect app (where you run the actual SQL query that is to be used for the lookup) it is extremely fast. Example of using dbxlookup in a search: | makeresults | eval page_id=510376245 | dbxlookup connection="myDB" query="SELECT C.CONTENTID as content_id,C.TITLE as page_title, S.SPACEKEY as space_key,S.SPACENAME as space_name FROM CONTENT AS C LEFT JOIN SPACES AS S ON (C.SPACEID=S.SPACEID) ORDER BY C.CONTENTID DESC" "content_id" AS "page_id" The search job inspector shows the long time the dbxlookup took. The search log does not yield any helpful information: 03-31-2022 16:27:11.988 INFO SearchParser [110743 StatusEnforcerThread] - PARSING: table _time, page_id 03-31-2022 16:27:11.988 INFO SearchParser [110743 StatusEnforcerThread] - PARSING: head 20 03-31-2022 16:27:11.988 INFO SearchParser [110743 StatusEnforcerThread] - PARSING: dbxlookup lookup="scwiki_db" 03-31-2022 16:27:11.988 INFO ChunkedExternProcessor [110743 StatusEnforcerThread] - Running process: /opt/splunk/jdk1.8.0_111/bin/java -Dlogback.configurationFile\=../config/command_logback.xml -DDBX_COMMAND_LOG_LEVEL\=DEBUG -cp ../jars/dbxquery.jar com.splunk.dbx.command.DbxLookupCommand 03-31-2022 16:27:12.585 INFO DispatchExecutor [110743 StatusEnforcerThread] - BEGIN OPEN: Processor=dbxlookup 03-31-2022 16:27:12.605 INFO DispatchExecutor [110743 StatusEnforcerThread] - END OPEN: Processor=dbxlookup 03-31-2022 16:27:12.605 INFO ChunkedExternProcessor [110743 StatusEnforcerThread] - Skipping custom search command since we are in preview mode: dbxlookup 03-31-2022 16:27:12.620 INFO PreviewExecutor [110743 StatusEnforcerThread] - Finished preview generation in 0.663433841 seconds. 03-31-2022 16:27:13.143 INFO DispatchExecutor [110818 phase_1] - END OPEN: Processor=table 03-31-2022 16:27:13.143 INFO DispatchExecutor [110818 phase_1] - BEGIN OPEN: Processor=dbxlookup 03-31-2022 16:27:13.364 INFO DispatchExecutor [110818 phase_1] - END OPEN: Processor=dbxlookup 03-31-2022 16:27:14.627 INFO ReducePhaseExecutor [110743 StatusEnforcerThread] - ReducePhaseExecutor=1 action=PREVIEW -> here the delay happens 03-31-2022 16:29:44.577 INFO PreviewExecutor [110743 StatusEnforcerThread] - Stopping preview triggers since search almost finished 03-31-2022 16:29:44.580 INFO DownloadRemoteDataTransaction [110818 phase_1] - Downloading logs from all remote event providers 03-31-2022 16:29:44.849 INFO ReducePhaseExecutor [110818 phase_1] - Downloading all remote search.log files took 0.270 seconds 03-31-2022 16:29:44.850 INFO DownloadRemoteDataTransaction [110818 phase_1] - Downloading logs from all remote event providers So it must have something to do with the way dbxlookup works. Could be an java issue or an mysql driver issue, a combination of both or something completely different :-). We are using the latest DB Connect MySQL Add-On. I am grateful for any hints or tipps on how to troubleshoot this furter. Or an actual solution :-).

flle · ‎01-27-2021

Hi, looked through documentation and Splunk answers but did not find reason/root cause for the following obervation: We have an index with 2 sourcetypes. one is JSON, the other plain text. Event examples: Sourcetype 1, JSON notation: { [-] context: xyz criteria: { [+] } device: desktop results: [ [-] 50832171 ] searchType: QuickSearch } Raw Text notation: {"context":"xyz","device":"desktop","searchType":"QuickSearch","results":["50832171"],"criteria":{"Item name":"Example"}} Sourcetype 2, None-JSON event: 2021-01-27 10:27:39.000, timestamp="2021-01-27 10:27:39.0", context="abc", searchType="Advanced Search", device="Mobile", criteria="Item Name", results="93751371" I want to do a lookup of the values in the results field. Problem: in the JSON event, this is an array, in the None-JSON a string. So I tried to use spath to extract results{} into field results and then do a lookup with that common field name for both sourcetypes: <base search> | spath output=results path=results{} | lookup myLookup id as results The problem is, when I do this, the results field in the None-JSON event disappears... So without the spath I have an auto-extracted results{} field in the JSON event and a results field in the None-JSON. Adding the spath removed the results field from the None-JSON event. --> Why? I have found a way to work around this, but I would like to understand the technical reasone behind the behaviour. My workaround is: | spath output=result path=results{} | eval results=coalesce(results, result) | lookup myLookup id as results

flle · ‎12-21-2020

Old thread but no final solution explanation given so I wanted to share a solution as we were facing a similar issue: We experienced the behaviour when using iplocation in conjunction with eventstats. When iplocation was used before eventstats, the location was correct, when used after eventstats, it was wrong. Cause: Eventstats (as well as stats) is a data processing / non-streaming command (-> it runs on the search head). Iplocation is a distributable streaming command (-> it can run on the indexer). So using IP location BEFORE eventstats (and right after the base search) makes it run on the indexers. When using iplocation AFTER eventstats it runs on the search head. The reason for yielding different location results was due to different iplocation database versions (GeoLite2..) on search head and indexers. So make sure, your geo location db is up-to-date and identical on all your Splunk components then iplocation yields the same results regardless where it is used in the search. Where you place it depends on your search. Usually it is advisable to enrich AFTER transforming/aggregations commands like stats or eventstats. But as iplocation is a distributable streaming command it might perform better when it can run on the indexers instead of the search head.

flle · ‎12-21-2020

We were facing the same issue and I stumbled over this thread in search for possible causes. As I did not find one but eventually found the cause for the different locations of IPs depending on where iplocation is used in the search, I wanted to share it. The given and accepted answer above is misleading / incomplete (it does not explain the different results depending on where in the search iplocation is used) We experienced the behaviour when using iplocation in conjunction with eventstats. When iplocation was used before eventstats, the location was correct, when used after eventstats, it was wrong. Cause: Eventstats (as well as stats) is a data processing / non-streaming command (-> it runs on the search head). Iplocation is a distributable streaming command (-> it can run on the indexer). So using IP location BEFORE eventstats (and right after the base search) makes it run on the indexers. When using iplocation AFTER eventstats it runs on the search head. The reason for yielding different location results was due to different iplocation database versions (GeoLite2..) on search head and indexers. So make sure, your geo location db is up-to-date and identical on all your Splunk components then iplocation yields the same results regardless where it is used in the search. Where you place it depends on your search. Usually it is advisable to enrich AFTER transforming/aggregations commands like stats or eventstats. But as iplocation is a distributable streaming command it might perform better when it can run on the indexers instead of the search head.

flle · ‎10-18-2019

mike.randal, are you sure you are getting dublicate events (dublicate JSON events) or are you just seeing dublicate entries in the fields if you output events with | table ... In the second case, this is likely due to the Add-On doing index-time (via indexed_extractions = JSON) and search-time field extractions (via KV_MODE = JSON) resulting in dublicate field entries, if you have the addon installed on Heavy Forwarder/Indexer and Search Head. See: https://answers.splunk.com/answers/223095/why-is-my-sourcetype-configuration-for-json-events.html You could circumvent this by using spath on the fields you want to display and output to new fields. Or you adopt the props.conf settings of the Add-On (which might have other implications).

flle · ‎12-22-2017

Hi ankithreddy777, not sure if the question is still relevant, but as I had a similar issue ( I think) I will share my solution. If I interpret your question correctly, you have something like: search ... | processing ... | outputlookup append=true file_1.csv | append [ inputlookup file_1.csv | lookup ... | processing and merging... ] | final processing or outputlookup If this is your szenario, then your first outputlookup will have the results of your initial search appended correctly, BUT the inputlookup file_1.csv in the append section is subsearch and will get parsed and dispatched before your outer search, so it will take the version of file_1.csv before the results from your search have been appended. The append command appends the results of a subsearch to the current results. To solve this, you can just replace append by appendpipe. This appends the result of the subpipeline to the search results. Unlike a subsearch, the subpipeline is not run first. The subpipeline is run when the search reaches the appendpipe command.

flle · ‎07-12-2017

Hi, to support several external lookups to internet services with APIs I added a proxy configuration to splunk-launch.conf: HTTP_PROXY=proxy.example.net:8080 HTTPS_PROXY=proxy.example.net:8080 Now I have an external lookup to an API on my internal network which should connect directly but not via the proxy. Forwarding the connection from the proxy to the internal network is not possible, so I need a solution on the Splunk Search Head. Question: Is it somehow possible to configure exceptions which should not go via the proxy configured in splunk-launch.conf? Thanks & regards

flle · ‎06-08-2016

Hi, When configuring the Data Collection Nodes (DCN) in the collection configuration for the Splunk App for VMware, a "Splunk Forwarder Username" has to be provided. This user is used by the Scheduler to push the collection jobs to the DCNs. By default, the admin user is used. Does anyone know precisely what capabilities that forwarder user needs to perform the scheduling tasks? I want to delegate the collection configuration setup to the VMWare Admins, but do not want to give them the admin password. So, I want to set up a dedicated user and role for the DCN scheduler with only the capabilities needed. Thanks & regards flle

flle · ‎05-24-2016

Hi, as there is no answer yet and I had the same question: Yes, proxy settings in /etc/splunk-launch.conf are passed to Splunk apps. Just adding http_proxy=<proxyname>:<port> or for https: https_proxy=<proxyname>:<port> to splunk-launch.conf worked for me and the Hipchat app happily uses the proxy for notification forwarding.

flle · ‎12-29-2015

Hi MuS, sorry for the long delay in accepting the answer. It was the right approach with the following search I got it working the way I wanted: search ... | ldapfilter domain=mydomain search="(sAMAccountName=$user$)" attrs="DisplayName,givenName,mail" | eval firstName=givenName | eval mail=mail | eval fullName=DisplayName | eventstats values(mail) AS mail_adresses| eval recipients=mvjoin(mail_adresses, ",") | table eventTime, field1, firstName, fullName, mail, recipients |sendemail to="mailbox@example.net" bcc=$result.recipients$ from="sender@example.net" footer="My Mail Footer" server=localhost subject="Some Subject" message="Hallo $result.firstName$, some text" the evals to explicitly (re)assign the queried ldap attributes to field names are necessary as for some reason those fields are "special" in some way. They show up in a table but when sendmail is used in the search then the fields are empty.

flle · ‎11-02-2015

woodcock, thanks for the update. I did figure out the issue in the meantime and there was more to it, hence I add an answer myself 🙂 If you foward structured data from a forwarder to an indexer, the indexer does NOT parse those events again (parsing, aggregation and typing queues are skipped). See the "Caveats" section here: http://docs.splunk.com/Documentation/Splunk/6.2.6/Data/Extractfieldsfromfileheadersatindextime In my case, INDEXED_EXTRACTIONS on the Universal Forwarder transform data to structured date, so the indexer does ignore any props or transforms on the indexers for this data. For the timestamp issue I could actually get around that with adding TIMESTAMP_FIELDS on the forwarder, but only if Splunk can auto-identify the time format. As the parsing capabilities of a universal forwarder are limited to the parsing functions in the INPUT phase (see http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F), and data is not being parsed again on the indexer when using indexed extractions on the UF this puts some constraints on my overall parsing capabilities. I basically loose all capabilities of the PARSING Phase. Conclusion: When using INDEXEC_EXTRACTIONS on a UF, be sure that you can achieve all desired parsing with the capabilities of the INPUT phase. Otherwise you have to use a Heavy Forwarder and do all the parsing there. Or with a UF forward unparsed data from the UF to the indexer and to all the parsing on the indexer.

flle · ‎10-21-2015

I also have the same issue. Would be nice to know if this is a bug or works as designed? Maybe it would interfere with the scheduled search/alert mailing functionality? Functionality is almost identical, however, in the scheduled search mail alerting, one cannot set the mail sender (from). This is always the globaly definied mail sender. So a working sendemail command in scheduled searches would be helpful.

flle · ‎08-05-2015

I stumbled across an interesting issue and need some advice / hints here. I have two sourcetypes where I need some time_format and time_prefix mangling to correctly parse the time stamps. When setting up the new data input (batch input for files) I did the configs in props.conf on the universal forwarder for the first input, and after some regex tuning for time_prefix, it worked fine. For the second input however, I could not get it to work. The difference is, that I have indexed_extractions for the first input. I then remembered that time_format & time_prefix are done in the parsing phase and thus can only be done on the indexers or heavy forwarders (see also: http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings). So now I am confused on whether time_format & time_prefix also work on a universal forwarder and I am just having an error in my second props.conf I am not seeing or that Splunk miraculously fixed the time_stamp extraction on its own regardless of my props.conf changes :-). Or time_* works in conjunction with indexed_extractions?? Log sample input 1: (desired timestamp is bold) "hostname_10.1.2.3_**2015-07-22T15_01_43Z**","HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run","2014-05-28T10:28:33Z","expand_sz","SysTrayApp","C:\Program Files\IDT\WDM\sttray64.exe","TRUE","FALSE","2013-11-06T22:07:24Z","2014-05-28T09:03:46Z","2014-05-28T09:03:46Z","C:\Program Files\IDT\WDM\sttray64.exe","1703424","IDT, Inc.","IDT PC Audio","1.0.6496.0","IDT PCA","Copyright © 2004 - 2009 IDT, Inc.","sttray64.exe","IDT PC Audio","1.0.6496.0","FALSE","FALSE","TRUST_E_NOSIGNATURE","The file is not signed","","","1f918ddae59e246b8f48ce5aa400b3aa","8896809e855ae08b43e41b25a6bdca8ed1905bbfc59e7b779070eaa0bbc1b319" Log sample input 2: (desired timestamp is bold) "05.08.2015 10:22:36";"3";"Network connection detected:;SequenceNumber: 161522;UtcTime: **05.08.2015 08:17:49.149 AM**;ProcessGuid: {6B887E38-96AB-55AC-0000-0010EB030000};ProcessId: 4;Image: System;User: NT-AUTORIT\xC4T\SYSTEM;Protocol: udp;Initiated: false;SourceIsIpv6: false;SourceIp: 10.1.2.3;SourceHostname: ;SourcePort: 137;SourcePortName: netbios-ns;DestinationIsIpv6: false;DestinationIp: 10.1.2.3;DestinationHostname: myhostnamet;DestinationPort: 137;DestinationPortName: netbios-ns" **Inputs.conf** [batch://d:\Splunk\sysmon\] disabled = 0 sourcetype = sysmon move_policy = sinkhole index=testing [batch://d:\Splunk\regdump\] disabled = 0 sourcetype = regdump move_policy = sinkhole crcSalt = <SOURCE> index=testing props.conf [regdump] INDEXED_EXTRACTIONS = CSV TIME_FORMAT = %Y-%m-%dT%H_%M_%S%Z TIME_PREFIX = ^([^_]*_){2} [sysmon] TIME_FORMAT = %d.%m.%Y %I:%M:%S.%3N %p TIME_PREFIX = ^([^;]*;){4}UtcTime:\s+ [source::...sysmon*csv] TZ = UTC The regdump sourcetyp was the first I integrated and Splunk, by default, extracted the timestamp further down in the event (2013-11-06T22:07:24Z). After I configured a matching time_prefix regex and time_format, the desired timestamp is now extracted. For the sysmon sourcetype however, it does not work. So what is the deal here? What am I missing? Thanks for any hints.

flle · ‎06-11-2015

ok, thanks for the clarification. The traps I posted above is a tcpdump output. So likely the values are encoded and tcpdump just decodes them. Unfortunately, fixing the encoding on the source will not be that simple. The trap Sender is a BMC of a Quanta Server. That means for us updating the BMC firmware on hundreds of Servers (given an update is available) ... Would implementing the Workaround hack described in http://pysnmp.sourceforge.net/faq.html into snmp.py be a workaround Option or would that introduce other issues?

flle · ‎06-11-2015

Hi Damien, thanks a lot for the quick answer. but I don't quite get it. Where is Encoding used here on the vendor side? The system.sysUpTime.0 field in the trap supplies the System uptime in miliseconds. system.sysUpTime.0=3084126100 (356 days) triggers the error system.sysUpTime.0=1113987000 (240 days) works ok. What exactly does snmp.py with those values? Looking at the error: "ValueRangeConstraint(0, 4294967295) failed at: "-1210841196"" at TimeTicks it seems that there is a calculation of sysUptime - upperBoundValue (3084126100 - 4294967295 -1) =-1210841196, which Triggers the error. But why is this done? And why does the other other trap gets processe ok as it would also result in a negative value if the above calculation would be done.

flle · ‎06-11-2015

Hi, I am experiencing an issue/bug with the SNMP trap receiver of the SNMP modular input. Working Trap: 09:22:56.553974 IP 172.17.xxx.xxx.43321 > myhost.snmptrap: V2Trap(87) system.sysUpTime.0=1113987000 S:1.1.4.1.0=E:7244.1.2.1.1651 E:7244.1.2.1.3.3="Test Trap" Non-working Trap: 09:24:14.598038 IP 172.17.xxx.yyy.58003 > myhost.snmptrap: V2Trap(87) system.sysUpTime.0=3084126100 S:1.1.4.1.0=E:7244.1.2.1.1651 E:7244.1.2.1.3.3="Test Trap" Error: 06-11-2015 09:24:14.600 +0200 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/snmp_ta/bin/snmp.py" Exception receiving trap ConstraintsIntersection(ConstraintsIntersection(), ValueRangeConstraint(0, 4294967295)) failed at: "ValueRangeConstraint(0, 4294967295) failed at: "-1210841196"" at TimeTicks snmp_stanza:snmp://snmptrap Apparently, its related to an out of bounds value while handling the system uptime. Th shown "failed at" value -1210841196 would be the result-1 when deducting the upper ValueRage limit (4294967295) from the sysUpTime (3084126100 ). I am not sure, why this is done or why it is not failing for the first trap but should be fixed in the app. Will this question be read by one of the authors (By Damien Dallimore and Scott Spencer) or can anybody with personal contacts point them to this? Regards Florian

flle · ‎03-12-2015

Yes, directory /opt/splunk on DCN was completely deleted. UF is installed in /opt/splunkforwarder. And yes, SH was restarted after DCN removal.

flle · ‎03-12-2015

Yes it is still on and has to be. The VMWare App is still in use and there are two other DCNs active. It's just that apparently the old DCN ist still included in the scheduling and I can't figure out where or why.

flle · ‎03-12-2015

Hi, Thanks for the answers so far but did not bring me further. I tried btool before without results. However, I could pinpoint it down to be related to the VMWare app. The Network Connections from the indexers are initiated by a python process running the ta_vmware_collection_sheduler: splunk 11012 10886 15 Mar11 ? 02:36:00 python /opt/splunk/etc/apps/Splunk_TA_vmware/bin/ta_vmware_collection_scheduler.py So I continued on the Search Head where the VMWare App is installed as this is where the scheduling happens. I searched all the VMware related apps and TAs for IP or hostname of the old DCN but did not find a config file where it still was present. In the relevant DCN config file (/opt/splunk/etc/apps/Splunk_TA_vmware/local/hydra_node.conf) only the correct Hosts are still present. In the logs of the seach head I could trace the deletion of the DCN: web_access.log:127.0.0.1 - xxxxxxx [09/Mar/2015:13:24:20.918 +0100] "DELETE /en-US/custom/splunk_for_vmware/splunk_for_vmware_setup/splunk_for_vmware/delete_collection_node/https:%2F%2Fdcn-hostname:8089 HTTP/1.1" 200 - "https://searchhead:8000/en-US/custom/splunk_for_vmware/splunk_for_vmware_setup/splunk_for_vmware/show_collection_setup" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36" - 54fd90f4eb7fbd79676150 887ms But apparently, it still is used by the Scheduler worker processes as acollection node. So maybe someone with expert Knowledge of the VMWare app can give a further hint on where the Problem could be. UPDATE (2015-03-16): As SA_Hydra is responsible for the data collection sheduling, we disabled it (-> move to disabled apps) on the indexers. That stopped the connections from the indexers to the old CDN. However, it had side effects. Now the indexers are throwing errors: Search peer index1.example.net has the following message: Unable to initialize modular input "ta_vmware_collection_worker" defined inside the app "Splunk_TA_vmware": Introspecting scheme=ta_vmware_collection_worker: script running failed (exited with code 1). After inspecting /opt/splunk/etc/apps/Splunk_TA_vmware/local I noticed config files which I then would also not expect on a indexer: -rw-------. 1 splunk splunk 979 Mar 5 16:33 app.conf -rw-------. 1 splunk splunk 357 Mar 6 09:07 hydra_node.conf -rw-rw-r--. 1 splunk splunk 238 Mar 6 11:03 indexes.conf -rw-------. 1 splunk splunk 150 Mar 12 15:20 inputs.conf -rw-------. 1 splunk splunk 737 Mar 6 09:07 ta_vmware_collection.conf -rw-------. 1 splunk splunk 177 Feb 17 09:20 ta_vmware_syslog_forwarder.conf -rw-------. 1 splunk splunk 333 Mar 6 09:07 vcenter_forwarder.conf I find this config identical on my Search Head, where the VMWare Collection Config is done via Splunk Web and which then controls the CDNs. So no idea how it came to the indexers... As far as distribution of VMWare app components to Splunk servers goes, we applied the matrix here: http://docs.splunk.com/Documentation/VMW/3.1.4/Configuration/Componentreference According to this, SA_Hydra is necessary on the indexers. †† Install SA-Utils and SA-Hydra on the Splunk indexers to stop modular input introspection from failing. This is a workaround to a known issue. These components do not affect the operation of your indexers. But apparently, it does somehow affect the operations of the indexers. So the next step would be to disable the hydra/collection config in Splunk_TA_vmware/local/ on the indexers. But I still wonder, how the config got there in the first place or if, why and which VMWare app components are really, REALLY necessary on the indexers. Any ideas or definite answers there?

flle · ‎03-11-2015

Hi, I am currently experiencing strange indexer to Universal Forwarder network connection requests on port 8089. I installed a Universal Forwarder on a server which was previously acting as Data Collection Node (DCN) for the Splunk VMWare App. The DCN was decommissioned and another server was brought online under the same IP, now with a UF installed. As soon as the UF went online I (continuously) noticed tons of error messages in splunkd.log: 03-11-2015 22:54:51.612 +0100 ERROR AuthenticationManagerSplunk - Login failed. Incorrect login for user: admin 03-11-2015 22:54:51.638 +0100 ERROR AuthenticationManagerSplunk - Login failed. Incorrect login for user: admin 03-11-2015 22:54:51.645 +0100 ERROR AuthenticationManagerSplunk - Login failed. Incorrect login for user: admin I first suspected a local issue but then saw corresponding errors in splunkd_access.log: X.X.0.39 - - [11/Mar/2015:22:54:51.630 +0100] "POST /services/auth/login HTTP/1.0" 401 129 - - - 8ms X.X.0.18 - - [11/Mar/2015:22:54:51.631 +0100] "POST /services/auth/login HTTP/1.0" 401 129 - - - 14ms X.X.0.18 - - [11/Mar/2015:22:54:51.666 +0100] "POST /services/auth/login HTTP/1.0" 401 129 - - - 11ms The Systems continuously trying to connect multiple times per second are my indexers. This goes on even when the UF is not running. I have not found a reason why the indexers are doing this as this does not happen with any other of the hundreds of active forwarders. I suspected something in conjunction with the servers DCN role before, but neither on the indexers nor on the Search Head (VMWare App config) did I find any reference to the decommissioned DCN. Ideas anyone? Thanks & best regards!

flle · ‎02-04-2015

Hi, Can the "sendemail" command be used to send multiple emails based on receiver information in the search result? So if I have a result with 10 events and each event contains an email address, I want to send 10 mails with specific information from each Event to 10 different receivers. When I try this, only one mail is sent based on the data of the first event in the search result set. Example: search <something>|ldapfilter domain=domainname search="(sAMAccountName=$user$)" attrs="DisplayName,title,givenName,sn,mail"|sendemail to=$result.mail$ server=localhost subject="Mail subject" message="Hallo $result.givenName$, Some mail text" With that search I get a result with usernames and lookup user attributes from AD and want to send one email per search result event based on the specific information of each event. Unfortunately, only one email is sent based on the data of the first event in the result. Is that a limitation of the sendemail command, a bug or am I missing something? Thanks for any hints!

flle · ‎11-11-2013

Hello gfuente, we did not solve the issue yet. We could reduce the impact in reducing the load (concurrent searches) but did not find the root cause. Did you solve it by now? Did the the web config parameters yield any new insight? Regards

flle · ‎10-10-2013

Updated post with additional info/answers to follow-up questions. @Lucas K: what do you mean with mounted or replicated bundles and what would be the impact (or benefit) of each?

flle · ‎10-05-2013

Hi, we are experiencing severe performance problems with a search head and could not really find a cause for this. So I hope to get a few more hints or ideas. The problem shows in the SH being extreeemly slow in responding or being unresponsive at all. It can take minutes for the search interface to fully load. If Splunk is restarted on the SH the problem is gone and performance is good again. But eventually the problem comes back, either within minutes but it can also take several hours. SH maschine: Virtual Linux Red Hat 5.5 on ESX Host, 8 vCPUs, 32 GB RAM, (shared) GBit network interface Splunk version 5.0.2. SH does distributed searches to 2 indexers. When the problem is present the SH displays the messages: Your network connection may have been lost or Splunk web may be down. Shortly after that the following message appears and then both of them stay or get refreshed. Your network connection was either restored or Splunk web was restarted. Its definitely not a network problem. Interface utitlization is always below 1MB/s The phyisical interface of the ESX host is also far from being utilized. From a machine resources point of view things don't look bad (stats taken when issue is present): top - 16:06:00 up 102 days, 4:33, 1 user, load average: 0.47, 1.09, 1.00 Tasks: 180 total, 1 running, 178 sleeping, 0 stopped, 1 zombie Cpu(s): 23.7%us, 1.9%sy, 0.0%ni, 74.1%id, 0.1%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 32959872k total, 16477580k used, 16482292k free, 7452048k buffers Swap: 7340000k total, 32968k used, 7307032k free, 6447520k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9815 splunk 19 0 210m 125m 9572 S 88.1 0.4 0:03.32 splunkd 28431 splunk 15 0 1358m 851m 15m S 23.9 2.6 345:40.15 splunkd 8714 tzhlefl1 15 0 10864 1092 760 R 0.7 0.0 0:00.24 top 2757 root 10 -5 0 0 0 S 0.3 0.0 212:19.67 kjournald 8566 splunk 20 0 90480 24m 9188 S 0.3 0.1 0:01.05 splunkd 32401 splunk 19 0 106m 27m 9204 S 0.3 0.1 0:42.16 splunkd ... $ vmstat 5 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 32968 16253304 7452748 6464240 0 0 0 210 0 0 16 1 82 0 0 2 0 32968 16127468 7452748 6464252 0 0 0 193 2091 1137 20 1 79 0 0 4 0 32968 16130308 7452748 6464864 0 0 0 759 1746 709 21 1 78 0 0 1 0 32968 16067828 7452764 6465200 0 0 0 487 1588 610 17 1 82 0 0 1 0 32968 16005332 7452764 6465188 0 0 0 118 1453 528 15 0 85 0 0 1 0 32968 16034304 7452768 6466060 0 0 0 198 1599 534 17 1 83 0 0 ... We have assigned dedicated CPU ressources and RAM to the VM to be sure it is not a ESX ressource allocation problem. Checks and info Splunk wise: The SH has quite a lot of scheduled searches configured, concurrent searches are averaging around 12, with peaks upt to 20 (taken from metrics.log). Additionally, there are 4 realtime searches running. In the logs I don't have any errors indicating performance issues but there are a lot of "Search not executed" errors from the Sheduled due to exeeding of concurrent searches or disk quota of users. /var/run/splunk/dispatch contains around 1000 directories during normal operations with the oldes entries dating back 3 days. I suspect this could cause trouble but it is still far from the default maximum. And running a "splunk cmd splunkd clean-dispatch ... -1hours" does not help during the issue Inspecting job run times show, that jobs are executed with ok performance but it takes lots of time to display the results. A search run from the GUI can have a Run Time of 1-2 seconds but takes a minute to show the first results. Running searches from the CLI returns results much faster. S.o.S. (Splunk on Splunk) only hint to a possible issue are the high numer of search artifacts in the dispatch directory. And there seems to be a memory leak in splunkd (memory usage slowly rises consistantly) but the problem occurence does not correlate with that. So I suspect some issue with result preparation and display but I don't have any more clues on how to speed it up or where to troubleshoot/tune to get a grip on the issue. As it goes away when restarting Splunk points towards an application issue but I have no idea what causes it. Any thoughts and hints are highly appreciated. Regards Flo Updated Post with answers to followup questions: CPU reservation is used ESX Server has 4 10Core CPUs with active hyperthreading. so: 40 physical cores / 80 logical processors There is only CPU overcommiting (NO memory overcommiting) ESX on which the SH runs has 27 VMs with alltogether 100 vCPUs. CPU ready percentage for SH VM is around 3% network metrics are ok. No errors, dropped packets, retransmits, overruns, etc. VMWare tools are installed and running. Within VM checked everything that vmware-toolbox offers. All ok from outside VM (ESX management) checked for ressource issues (network, CPU, RAM). Only thing was peaks in CPU ready states. Thats when we introduced CPU reservation. Did not help CPU ready is at around 3.5% for the SH VM SH is NOT acting as indexer (not even summaries). VMWare infrastructure is ok. No alerts/warnings on ESX side. Time is set via ntp not VMWare timekeeping (its disabled) There are some long runing searches but they are well distributed across the day (or rather night). As said, concurrent searche during the day is between 10 and 20 (taken from metrics log) I still suspect some wired splunkd/splunk-web behaviour when the issue kicks in but inspecting the different internal SH logs did not yield any helpful clues yet.

flle · ‎08-21-2013

Oracle. Added that info to question.

Posts	32
Solutions	1
Karma Given	5
Karma Received	12
Member Since	‎10-19-2011

Online Status	Offline
Date Last Visited	‎04-04-2022 04:57 PM

dbxlookup with MySQL (DB Connect app 3.8) huge del...

Extracting field with spath from JSON sourcetype o...

How to configure exceptions to proxy config in spl...

Splunk App for VMware: What capabilities are neede...

Why are timestamps parsed correctly for only one o...

Errors for SNMP Trap Receiver

Why am I seeing a high number of indexer connectio...

How to get Splunk sendemail command to send multip...

Search Head severe performance issue

DB Connect - OutOfMemoryError for db input

dbxlookup with MySQL (DB Connect app 3.8) huge del...

Extracting field with spath from JSON sourcetype o...

Re: iplocation query returning wrong location for ...

Re: iplocation command showing wrong location for ...

Re: Getting duplicate data

Re: How to add delay between two commands in searc...

How to configure exceptions to proxy config in spl...

Splunk App for VMware: What capabilities are neede...

Re: Is there a way for the HipChat Room Notificati...

Re: How to get Splunk sendemail command to send mu...

Re: Why are timestamps parsed correctly for only o...

Re: Sending Email using sendemail works, but not f...

Why are timestamps parsed correctly for only one o...

Re: Errors for SNMP Trap Receiver

Re: Errors for SNMP Trap Receiver

Errors for SNMP Trap Receiver

Re: Why am I seeing a high number of indexer conne...

Re: Why am I seeing a high number of indexer conne...

Re: Why am I seeing a high number of indexer conne...

Why am I seeing a high number of indexer connectio...

How to get Splunk sendemail command to send multip...

Re: Search Head severe performance issue

Re: Search Head severe performance issue

Search Head severe performance issue

Re: DB Connect - OutOfMemoryError for db input