All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

According to your first search, whether an event is counted as error is solely determined by a field named level, and only when its value is "ERROR" do you consider the event an error.  Is this corre... See more...
According to your first search, whether an event is counted as error is solely determined by a field named level, and only when its value is "ERROR" do you consider the event an error.  Is this correct?  All you need to do is to literally combining this criterion and the filters in the second with OR logic.  Something like index="iss" Environment=PROD Appid IN ("APP-61", "APP-85", "APP-69", "APP-41", "APP-57", "APP-71", "APP-50", "APP-87") ( ("Invalid JS format" OR ":[down and unable to retrieve response" OR "[Unexpected error occurred" OR ": [An unknown error has occurred" OR "exception" OR "IN THE SERVICE" OR "emplateErrorHandler : handleError :" OR "j.SocketException: Connection reset]" OR "Power Error Code" OR "[Couldn't kickstart handshaking]" OR "[Remote host terminated the handshake]" OR "Caused by:[JNObject" OR "processor during S call" OR javx OR "Error while calling" OR level="ERROR" NOT "NOT MATCH THE CTRACT" NOT "prea_too_large" NOT g-500 NOT G-400 NOT "re-submit the request" NOT "yuu is null" NOT "igests data" NOT "characters" NOT "Asset type" NOT "Inputs U" NOT "[null" NOT "Invalid gii" ) OR level!=ERROR) | rex field=_raw " (?<service_name>\w+)-prod" | eval err_flag = if(level="ERROR", 1,0) | eval success_flag = if(level!="ERROR", 1,0) | stats sum(err_flag) as Total_Errors, sum(success_flag) as Total_Successes by service_name | eval Total_Transaction = (Total_Successes+Total_Errors) | fields service_name, Total_Transaction, Total_Errors, Total_Successes Note I removed the "| where" command in the first because it is more efficient to place the simple filter as a search filter.  I also corrected a syntax error in the second from OR OR IN THE SERVICE" to OR "IN THE SERVICE".  If any of these changes semantics, make adjustments.  Hope this helps.
@bharathkumarnec  Hi, Where exactly we can run this btool to check the configurations. I dnt have back end access can we check in ui.
Can you describe your deployment a bit - # search heads, # indexers, are they on VMs or bare metal, maybe share a screenshot where you are seeing this high IOWait vs CPU (e.g. are you seeing this in ... See more...
Can you describe your deployment a bit - # search heads, # indexers, are they on VMs or bare metal, maybe share a screenshot where you are seeing this high IOWait vs CPU (e.g. are you seeing this in a Splunk dashboard or from OS tools).
What type of Splunk instance are you getting this notification on - your Indexer, or are you using a single Splunk instance? I'm also assuming this notification is coming from outside of Splunk - lik... See more...
What type of Splunk instance are you getting this notification on - your Indexer, or are you using a single Splunk instance? I'm also assuming this notification is coming from outside of Splunk - like an OS monitor or something like that. For example, an Indexer is generally going to put lots of stuff into $SPLUNK_HOME/var/lib/splunk because that's where the indexes are stored.   The first thing I could think of is your indexer drive is almost full, and you index more data causing you to dip below a Splunk-configured max index size.  This causes Splunk to then move/delete old buckets of data because it dipped below the configured threshold.   Here's docs on this this concept:  Configure maximum index size - Splunk Documentation Also, these configs would be in one of your indexes.conf files.
When doing a high-availability with multiple cluster managers (CM), you should have a load balancer (LB) in front of them.  The manager_ui configuration then points to the LB - not an individual CM. ... See more...
When doing a high-availability with multiple cluster managers (CM), you should have a load balancer (LB) in front of them.  The manager_ui configuration then points to the LB - not an individual CM.  The CM's will keep their bundles in sync, and the LB will help ensure that your indexers always reach a "good" one that is up.   Here's the docs that explain how to put an LB in front of the CMs in this scenario.
Hello All,  I am setting up a multisite indexer cluster with cluster manager redundancy,  I am setting up 2 clustermanager (site1 and site2) Below is the config e.g. [clustering] mode = manag... See more...
Hello All,  I am setting up a multisite indexer cluster with cluster manager redundancy,  I am setting up 2 clustermanager (site1 and site2) Below is the config e.g. [clustering] mode = manager manager_switchover_mode = auto manager_uri = clustermanager:cm1,clustermanager:cm2 pass4SymmKey = changeme [clustermanager:cm1] manager_uri = https://10.16.88.3:8089 [clustermanager:cm2] manager_uri = https://10.16.88.4:8089 My question is, I have 2 indexers on each site, should I give the manager_uri in the peer (indexer) of site1 to point to cm1 and manager_uri in the peer (indexer) of site2 to  point to cm2. or all should point to the same indexer? indexer 1 / indexer 2 -  manager_uri = https://10.16.88.3:8089 indexer 3 / indexer 4 -  manager_uri = https://10.16.88.4:8089   Also in the SearhHeads what should I define for the manager_uri? please advice.   Thanks, Dhana
...
Hi This is quite long story and there are couple of ways to do it. Before anyone answer it here you could try to look answer from https://splunk-usergroups.slack.com/archives/CD6JNQ03F. Unfortunate... See more...
Hi This is quite long story and there are couple of ways to do it. Before anyone answer it here you could try to look answer from https://splunk-usergroups.slack.com/archives/CD6JNQ03F. Unfortunately there is no only one part of this channel which you need to read, instead of it has spread over all time this channel.  Those i* nodes are good for smart store, but it depends what kind of architecture you have and how you are managing and automate it. I propose that you will contact someone Splunk and AWS specialist who have done this earlier as there are many places where you could fail (read that channel and you see at least some of those). It's not enough that this person is AWS guru or Splunk guru. He/she must understand both environments to get this working. r. Ismo
Use that regex in SEDCMD in props.conf. [mysourcetype] SEDCMD-no_UTF-8 = s/\x1B\[[0-9;]*[mK]//g  
You need to understand the content of issue! Without that understanding those SPLs don't tell you what is the real issue!
Hi One way to look that what @ITWhisperer told is just query like this index=* | eval iDiff=_indextime - _time | bin span=1h _time | stats avg(iDiff) as iDiff by _time host index sourcetype | where... See more...
Hi One way to look that what @ITWhisperer told is just query like this index=* | eval iDiff=_indextime - _time | bin span=1h _time | stats avg(iDiff) as iDiff by _time host index sourcetype | where iDiff > 60*60*5 | fieldformat iDiff = tostring(round(iDiff),"duration") You need just adjust those parameters by your needs and start to dig out is there any issues as lag or is there real issue in onboarding process where your time stampping is not correct. Or could there even be some timezone or other clock issues. r. Ismo
This answer is not relevant to my question.  I am looking for the splunk query to get the index and sourcetype delay. very simple.. 
How would you measure that delay? If it is the difference between the time the event was indexed and the timestamp as stored in _time with the event, you need to look at how that timestamp was derive... See more...
How would you measure that delay? If it is the difference between the time the event was indexed and the timestamp as stored in _time with the event, you need to look at how that timestamp was derived, which will depend on the sourcetype, where the logged event was picked up and its journey into Splunk. For example, if I have a log entry which has a timestamp created in one timezone, and the timezone is not part of the timestamp, when it is parsed into _time, it may not be given the timezone I was expecting, and can therefore appear to be hours late (or even early if the timezones are reversed). Also, depending on the application logging the event, the timestamp in the log may not be the same as the time it was written. For example, Apache HTTPD will log events with a timestamp relating to when the request was received, but will only log it when the response has been sent. Admittedly, 5 hours would be a bit extreme in this case, but the point is that the timestamp that Splunk assigns to the event does not necessarily have to be coincident with the time the log event was written to the log. This is why you should try and determine if there is any pattern to the apparent delay. You could start by comparing the _indextime to the _time field in your events and see where the large differences are (there is always likely to be a difference because the log has to get from where it is written to the index and that always takes some time.
I have given the example of 5 hours span time log delay in last 30 days.. its not specific to particular sourcetype. I want to see if any log delay in splunk. 
How do you know it is 5 hours? Is it always about 5 hours from all hosts? Or for all sourcetypes? Can you isolate a common attribute for all the events which are "delayed"? Which time zone or zones a... See more...
How do you know it is 5 hours? Is it always about 5 hours from all hosts? Or for all sourcetypes? Can you isolate a common attribute for all the events which are "delayed"? Which time zone or zones are you operating in?
Thank you for your assistance 1) Since I am using DS, do you think it's doable if I just display two numbers on two separate "single value" box? 2)  Is this  the alternative solution?    Can  yo... See more...
Thank you for your assistance 1) Since I am using DS, do you think it's doable if I just display two numbers on two separate "single value" box? 2)  Is this  the alternative solution?    Can  you please help translate it with the current case (plus percentile_Inc)?    https://community.splunk.com/t5/Splunk-Search/Is-there-a-way-to-calculate-the-percentile-of-a-value-within-a/m-p/269874 | stats count by value | sort + value | streamstats current=f sum(count) as rank | fillnull rank | eventstats sum(count) as total | eval percentile_rank = rank / total * 100   3) Can I use perc<percentage>(<value>)  or upperperc(<value>,<percentile>) to solve this ? https://docs.splunk.com/Documentation/SCS/current/SearchReference/Aggregatefunctions
Hi,  We have enabled all the default JMX metric collection in the configuration like, kafka, tomcat, weblogic, PMi, cassandra,etc., But when very limited metrics are available under Metric Browser. ... See more...
Hi,  We have enabled all the default JMX metric collection in the configuration like, kafka, tomcat, weblogic, PMi, cassandra,etc., But when very limited metrics are available under Metric Browser.  Only JVM --> classes, garbage collection, memory, threads are visible.  None of the above.  Why is it so? We are more interested in looking at Tomcat related JMX metrics.  Your inputs are much appreciated.  Thanks, Viji
@Cansel.OZCAN  Do you have any comments on my previous message?
we see a delay of over five hours in indexing. Is there a way to find out where these events "got stuck" or please let me know query to get the how much time log delay
I have an index that provides a Date and a row count to populate a line chart on a dashboard using DBConnect.  The data looks like this: Date Submissions 2023-11-13 7 2023-11-14 35 20... See more...
I have an index that provides a Date and a row count to populate a line chart on a dashboard using DBConnect.  The data looks like this: Date Submissions 2023-11-13 7 2023-11-14 35 2023-11-15 19   When the line chart displays the data, the dates show up like this:  2023-11-12T19:00:00-05:00,  2023-11-13T19:00:00-05:00, 2023-11-14T19:00:00-05:00.  Is there some setting/configuration that needs to be updated?