All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

@marnall   after using your query I am  getting error message as " connection refused" in my search results.. Should I change my localhost to something else in the mail server hostname in email sett... See more...
@marnall   after using your query I am  getting error message as " connection refused" in my search results.. Should I change my localhost to something else in the mail server hostname in email settings in splunk UI ?Please do let me know
Hi @maede_yavari , the message means that you have to copy the app.conf from the default folder to the local one. Then, there's an error in outputs.conf: check it, if you want share it, eventually ... See more...
Hi @maede_yavari , the message means that you have to copy the app.conf from the default folder to the local one. Then, there's an error in outputs.conf: check it, if you want share it, eventually masking IP addresses. Ciao. Giuseppe
Hi @NoSpaces, You can reduce the log level of WorkloadsHandler in %SPLUNK_HOME%\etc\log-local.cfg. Create the file if it does not exist and add the following: [splunkd] category.WorkloadsHandler=FA... See more...
Hi @NoSpaces, You can reduce the log level of WorkloadsHandler in %SPLUNK_HOME%\etc\log-local.cfg. Create the file if it does not exist and add the following: [splunkd] category.WorkloadsHandler=FATAL Restart Splunk to allow the change to take effect. You can temporarily change the active level on a running instance from Settings > Server settings > Server logging > WorkloadsHandler or using the REST API: curl -k -u admin https://localhost:8089/services/server/logger/WorkloadsHandler -d level=FATAL curl is shipped with all modern releases of Windows, but you can use whichever HTTP client you prefer.
From your description, it seems like the upgrade of RH from 7 to 9 has disrupted Splunk's assumptions about the full DNS name.  You could try being more specific in the configuration files until Spl... See more...
From your description, it seems like the upgrade of RH from 7 to 9 has disrupted Splunk's assumptions about the full DNS name.  You could try being more specific in the configuration files until Splunk starts sending emails properly. E.g. you could set the from= field in alert_action.conf to be a full email address so that Splunk does not have to figure out the hostname to tack on with the ampersand. NOTE:  alert_actions.conf is plural (alert_actions, not alert_action)  #/opt/splunk/etc/system/local/alert_actions.conf [email] from = yoursplunkemail@yourdomain.com You could also try a direct sendemail command: | sendemail from="youremail@place.com" to="targetemail@place.com" subject="Test Email" If you have different emails per alert, you could edit the alert in the Advanced Settings to explicitly set the from= field to a full email address.
Hi gcusello,   I did this method but when I restart Splunk Universal Forwarder, the following warning is appeared:   No spec file for: /opt/splunkforwarder/etc/apps/outputs/local/app.conf Checki... See more...
Hi gcusello,   I did this method but when I restart Splunk Universal Forwarder, the following warning is appeared:   No spec file for: /opt/splunkforwarder/etc/apps/outputs/local/app.conf Checking: /opt/splunkforwarder/etc/apps/outputs/local/outputs.conf Invalid key in stanza [general] in /opt/splunkforwarder/etc/apps/outputs/local/outputs.conf, line 2: site (value: site2).   By the way, the mentioned  architecture is multi site cluster and we want all of the  Splunk Universal Forwarder send data to site 2.   Many Thanks.
The UT toolbox app relies on some .dat files in the $SPLUNKDIR$/etc/apps/utbox/bin/ directory which list the known TLD suffixes. Unfortunately, ".shop" is not listed in them. To add the ".shop" tld,... See more...
The UT toolbox app relies on some .dat files in the $SPLUNKDIR$/etc/apps/utbox/bin/ directory which list the known TLD suffixes. Unfortunately, ".shop" is not listed in them. To add the ".shop" tld, you can edit the suffix_list_custom.dat file at: $SPLUNKDIR$/etc/apps/utbox/bin/suffix_list_custom.dat and add a line containing "shop". A restart is not required to apply this change. Then try your query again and the ut_domain field value should now be "somethin.shop" as desired.
Do you find any error logs if you search the _internal logs for "Sendemail"? (note the e, as in "send email" -> "sendemail", not "sendmail"                                           ) index=_interna... See more...
Do you find any error logs if you search the _internal logs for "Sendemail"? (note the e, as in "send email" -> "sendemail", not "sendmail"                                           ) index=_internal sendemail NOT sourcetype=splunkd_ui_access   If you get a lot of random messages, try filtering the source for python and splunkd: index=_internal sendemail NOT sourcetype=splunkd_ui_access source IN ("/opt/splunk/var/log/splunk/python.log","/opt/splunk/var/log/splunk/splunkd.log")  
Thanks for your reply. Yes, reinstallation works.
If you would just like to make a timechart, then the timechart command should fit your need better than the sitimechart command. The sitimechart is intended for preparing the data to insert into a ... See more...
If you would just like to make a timechart, then the timechart command should fit your need better than the sitimechart command. The sitimechart is intended for preparing the data to insert into a summary index so that later on it can be timecharted from the summary index. Using just timechart:   <your search> | timechart span=1d avg(errorPercentage) as errorPercentage by Process   If you would like to gather data now into a summary index to produce a timechart very quickly in a later search, you can use sitimechart:   <your search> | sitimechart span=1d avg(errorPercentage) as errorPercentage by Process | collect index=yoursummaryindex   Then in a later search:   index = yoursummaryindex <some filter, e.g. for Process=*> | timechart span=1d avg(errorPercentage) as errorPercentage by Process   The docs page describes the sitimechart usage, but does not explain the meaning of the created fields: https://docs.splunk.com/Documentation/Splunk/9.2.0/SearchReference/Sitimechart
Hello, While using sitimechart instead of timechart - The data has been changed. I would like to calculate an error percentage but the system shows 0 or fields count. Thanks!    
Thank you for your reply ...   The email settings are fine  and in internal logs I don't see any error related to "sendemail"...Could you please suggest what else could be done?
Hi @shakti , did you configured the email relay in [Settings > Server Settings > Email Settings] ? did you open all the routes between the Search Head and the eMail host? You can troubleshoot the ... See more...
Hi @shakti , did you configured the email relay in [Settings > Server Settings > Email Settings] ? did you open all the routes between the Search Head and the eMail host? You can troubleshoot the connection searching in _internal the word "Sendmail". Ciao. Giuseppe
Hi @vishwa , if you run your search, have you the table you shared? if yes, using the eval I hinted you sum the values ot the columns in the Total value. You could also use addtotals command that ... See more...
Hi @vishwa , if you run your search, have you the table you shared? if yes, using the eval I hinted you sum the values ot the columns in the Total value. You could also use addtotals command that sums all the values for each row: index=app-index source=application.logs | rex field= _raw "RampData :\s(?<RampdataSet>\w+)" | rex field= _raw "(?<Message>Initial message received with below details|Letter published correctley to ATM subject|Letter published correctley to DMM subject|Letter rejected due to: DOUBLE_KEY|Letter rejected due to: UNVALID_LOG|Letter rejected due to: UNVALID_DATA_APP)" | chart count over RampdataSet by Message | addtotals But also in this case que question is: does your search extract the value for each column? Ciao. Giuseppe  
addtotals is the wrong tool for what you wanted. (It is really bad to use screenshot to illustrate text output.  Always use text unless you are illustrating a visual effect.)  addcoltotals is what yo... See more...
addtotals is the wrong tool for what you wanted. (It is really bad to use screenshot to illustrate text output.  Always use text unless you are illustrating a visual effect.)  addcoltotals is what you need. | addcoltotals labelfield=MOP
hello, We upgraded our red hat 7 to 9 this past monday. and splunk stopped sending emails. We were inexperience and unprepare for this so we upgraded our splunk enterprise from 9.1 to 9.13 to see ... See more...
hello, We upgraded our red hat 7 to 9 this past monday. and splunk stopped sending emails. We were inexperience and unprepare for this so we upgraded our splunk enterprise from 9.1 to 9.13 to see if this would fix it. It did not. then we upgraded to 9.2, that did not fix it. I started adding debug mode to everything and found that splunk would send the emails to postfix and the postfix logs would state the emails were send. however, after looking at it closer, I notice the from field of the splunk sendemail generated emails had the from field like: splunk@prod not splunk@prod.mydomain.com (as they used to before we upgraded to redhat 9 When we use mailx, the fron field from field is constructed correctly such as: splunk@prod.domain.com extra python debugging does not show the from field but only the user and the domain: from': 'splunk', 'hostname': 'prod.mydomain.com', My stanza in /opt/splunk/etc/system/local/alert_action.conf: [email] hostname = prod.mydomain.com Does anyone know how to fix this? Is there a setting in splunk that would make sure the email from field is constructed correctly. It is funny that if you add an incorrect "to" address splunk whines but if splunk create a incorrect to field address in sendemail, it is fine and, just send it to postfix and let it handle it, lol dandy  
Hi @alfredoh14, By default, the From: address is splunk. Are you using default Splunk email settings and a local instance of postfix? Because no domain is specified, postfix likely appends the host ... See more...
Hi @alfredoh14, By default, the From: address is splunk. Are you using default Splunk email settings and a local instance of postfix? Because no domain is specified, postfix likely appends the host name to the user name, i.e. splunk@prod, before forwarding the message to either an upstream relay or the recipient's mail server. You can set the From: address in Splunk Web from Settings > Server settings > Email settings in the "Send emails as" setting. This will update $SPLUNK_HOME/etc/system/local/alert_actions.conf. For example, using no-reply@mydomain.com:  [email] from = no-reply@mydomain.com
Hi @Satyapv, Hi, Here's another alternative. We'll use internal splunkd components to simulation a field named Transaction. To see event counts over [-300,0], [-600,0], and [-900,0] seconds: inde... See more...
Hi @Satyapv, Hi, Here's another alternative. We'll use internal splunkd components to simulation a field named Transaction. To see event counts over [-300,0], [-600,0], and [-900,0] seconds: index=_internal sourcetype=splunkd component=* earliest=-15m latest=now | rename component as Transaction | addinfo ``` assumes a valid latest value ``` | stats count(eval(_time>=info_max_time-300)) as "Last 5min Vol" count(eval(_time>=info_max_time-600)) as "Last 10min Vol" count as "Last 15min Vol" by Transaction To see event counts over [-300,0], [-600,300), and [-900,600) seconds: index=_internal sourcetype=splunkd component=* earliest=-15m latest=now | rename component as Transaction | addinfo ``` assumes a valid latest value ``` | stats count(eval(_time>=info_max_time-300)) as "Last 5min Vol" count(eval(_time>=info_max_time-600 AND _time<info_max_time-300)) as "Last 10min Vol" count(eval(_time<info_max_time-600)) as "Last 15min Vol" by Transaction You can adjust earliest and latest as needed, but note that the last count will always be inclusive of earliest, i.e. the last 15 minutes for -15m. You adjust the count aggregates to disallow counting events more than 900 seconds (15 minutes) prior to latest: count(eval(_time>=info_max_time-900)) as "Last 15min Vol" or count(eval(_time>=info_max_time-900 AND _time<info_max_time-600)) as "Last 15min Vol"  
that was awsome help, it really just solidify the assumption that I had which was that splunk was sending the emails to the postfix server and they were getting dropped by it. My next step was to ad... See more...
that was awsome help, it really just solidify the assumption that I had which was that splunk was sending the emails to the postfix server and they were getting dropped by it. My next step was to add debugging on postfix, and i found out that the "to" field was splunk@prod not splunk@prod.mydomain.com the server name (hostname) on linux is prod but when i access splunk web UI, it is https://prod.mydomain.com:8080/en-US When I added the python debugging it only showed that the user who was sending it was splunk and the domain was (based on debug logs of python sendemail): from': 'splunk', 'hostname': 'prod.mydomain.com', this did not show me that the sendemail.py command would construct the from field to only splunk@prod (I had to look at the postfix logs for that info). My stanza in /opt/splunk/etc/system/local/alert_action.conf: [email] hostname = prod.mydomain.com again, we can send emails from cli using mailx but splunk cannot using sendemail.py because it is not cnstructing the from field correctly so although postfix sents it, the smtp server which receives it drops it. so, do you have any idea of where I have to set a setting so that the from field is constructed correctly by the sendemail.py script?  
Note you need to place source=OMITTED host="SERVER1" OR host="SERVER2" in parentheses; alternatively use IN operator.  Finding difference should not be that complicated. index=_internal earliest=-15... See more...
Note you need to place source=OMITTED host="SERVER1" OR host="SERVER2" in parentheses; alternatively use IN operator.  Finding difference should not be that complicated. index=_internal earliest=-15mindex=OMITTED source=OMITTED host IN ("SERVER1", "SERVER2") | stats max(Value) as Value by host | stats max(Value) as max_of_two min(Value) as min_of_two | where max_of_two / min_of_two > 0.75 However, your OP says you want timechart.  That's why @richgalloway includes _time in groupby in that first stats.  But you can substitute the first stats with timechart to simplify this, then use the same technique in every row to find percent deviation. index=_internal earliest=-15mindex=OMITTED source=OMITTED host IN ("SERVER1", "SERVER2") | timechart span=1d max(Value) as Value by host | eventstats max(Value) as max_of_two min(Value) as min_of_two | where max_of_two / min_of_two > 0.75  
In addition to possible additions such as "Timeout Errors", you case requires an additional consideration.  Using case function (or use timechart command) will count each 5-minute interval separately... See more...
In addition to possible additions such as "Timeout Errors", you case requires an additional consideration.  Using case function (or use timechart command) will count each 5-minute interval separately in disagreement with the semantics of "last 10 min vol" and "last 15min vol".  These terms are cumulative.  Any event in "last 5min" must also be counted in "last 10min" and "last 15min". Here is a semantic implementation; mvappend satisfies both considerations.   | foreach 5min 10min 15min [eval header = mvappend(header, if(_time - relative_time(now(), "-<<FIELD>>") > 0, "Last <<FIELD>> Vol", null()))] | eval header = mvappend(header, if(log_level == "ERROR", "Timeout Errors", null()), ``` This is error emulation. Use real condition(s) ``` if(someother > 0, "Some other count", null())) | chart count OVER Transaction BY header | table Transaction "Last 5min Vol" "Last 10min Vol" "Last 15min Vol" "Timeout Errors"   Note the last mvappend evaluation is emulation.  Use your real condition.  Here is a data emulation I use to test the above code; one emulated error condition gives non-zero output.   index=_internal earliest=-15m | rename sourcetype AS Transaction ``` data emulation above; some events have leg_level "ERROR" ```   This emulation gives the counts like the following: Transaction Last 5min Vol Last 10min Vol Last 15min Vol Timeout Errors dbx_health_metrics 1370 2055 2740 0 dbx_server 0 0 4 0 splunk_python 10 15 20 0 splunk_search_messages 4 2 2 2 splunkd 4736 7390 9833 2779 splunkd_access 388 600 787 0 splunkd_ui_access 244 134 148 0