All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

  Hello Community, I'm seeking some guidance with optimizing a Splunk search query that involves multiple table searches and joins. The primary issue I'm encountering is the limitation imposed by... See more...
  Hello Community, I'm seeking some guidance with optimizing a Splunk search query that involves multiple table searches and joins. The primary issue I'm encountering is the limitation imposed by subqueries, restricting the total records to 50,000. Here's the current query structure I'm working with: index="sample" "message.process"="*app-name1" "message.flowName"="*| *" | rex field=message.correlationId "(?<UUID>^[0-9a-z-]{0,36})" | rename "message.flowName" as sapi-outbound-call | stats count by sapi-outbound-call UUID | join type=inner UUID [search index="sample" "message.process"="*app-name2" "message.flowName"="*| *" | rex field=message.correlationId "(?<UUID>^[0-9a-z-]{0,36})" | rename "message.flowName" as exp-inbound-call] | stats count by exp-inbound-call sapi-outbound-call | join left=L right=R where L.exp-inbound-call = R.exp-inbound-call [search index="sample" "message.process"="*app-name2" "message.flowName"="*| *" | rename "message.flowName" as exp-inbound-call | stats count by exp-inbound-call] | stats list(*) AS * by R.exp-inbound-call R.count | table R.exp-inbound-call R.count L.sapi-outbound-call L.count The intention behind this query is to generate statistics based on two query searches or tables while filtering out data based on a common UUID. However, the usage of multiple joins within subqueries is causing limitations due to the 50,000 record cap. I'm looking for alternative approaches or optimizations to achieve the same result without relying heavily on joins within subqueries. Any insights, suggestions, or examples would be incredibly valuable. Thank you in advance for your help and expertise! Regards
I mean return all fields which does not contain firstName or lastName (do not remove them). Like in picture, second field does not contain firstName and lastName.
index=netlogs [| inputlookup baddomains.csv | eval url = "*.domain."*" | fields url] NOT [| inputlookup good_domains.csv | fields domain] I don't think my search is doing what I want it to do. I wou... See more...
index=netlogs [| inputlookup baddomains.csv | eval url = "*.domain."*" | fields url] NOT [| inputlookup good_domains.csv | fields domain] I don't think my search is doing what I want it to do. I would like to take the bad domains from the first lookup table and search the netlogs index to see if there are any hits. however, i would like to remove the good domains from the second lookup table from the search. Anyone know if there is a better way to do this?
It should be done with two SEDCMD statements since the fields may not be next to each other.   SEDCMD-no_cs4 = s/cs4=.*?(\w+=)|$/\1/ SEDCMD-no_cs4Label = s/cs4Label=.*?(\w+=)|$/\1/ SEDCMD-no_cs3 = ... See more...
It should be done with two SEDCMD statements since the fields may not be next to each other.   SEDCMD-no_cs4 = s/cs4=.*?(\w+=)|$/\1/ SEDCMD-no_cs4Label = s/cs4Label=.*?(\w+=)|$/\1/ SEDCMD-no_cs3 = s/cs3=.*// SEDCMD-no_cs3Label = s/cs3Label=.*?(\w+=)|$/\1/   I've also assumed the cs3 field is always last, but that may be erroneous..
Hi @Mo.Abdelrazek, I found some existing info. I hope this helps. Check out this AppD Docs page: https://docs.appdynamics.com/appd/23.x/latest/en/end-user-monitoring/thousandeyes-integration-with... See more...
Hi @Mo.Abdelrazek, I found some existing info. I hope this helps. Check out this AppD Docs page: https://docs.appdynamics.com/appd/23.x/latest/en/end-user-monitoring/thousandeyes-integration-with-browser-real-user-monitoring/thousandeyes-network-metrics-in-browser-rum#id-.ThousandEyesNetworkMetricsinBrowserRUMv23.6-MatchThousandEyesTestsManually    Also, can you regenerate the auth token and reconfigure it in the Appdynamics and let us know if you are facing the same issue or not. https://docs.appdynamics.com/appd/23.x/latest/en/appdynamics-essentials/dashboards-and-reports/dash-studio/thousandeyes-integration-with-appdynamics/enable-the-thousandeyes-token
Again, please illustrate/emulate data in text. Does "exclude firstName, lastName" mean to remove them for display purposes?  For that, you can use rex command's sed mode.  Like | rex field=matching... See more...
Again, please illustrate/emulate data in text. Does "exclude firstName, lastName" mean to remove them for display purposes?  For that, you can use rex command's sed mode.  Like | rex field=matchingField mode=sed "s/\bfirstName\b// s/, *,/,/ s/\[, */[/ s/, *\]/]/" | rex field=matchingField mode=sed "s/\blastName\b// s/, *,/,/ s/\[, */[/ s/, *\]/]/"  
According to your first search, whether an event is counted as error is solely determined by a field named level, and only when its value is "ERROR" do you consider the event an error.  Is this corre... See more...
According to your first search, whether an event is counted as error is solely determined by a field named level, and only when its value is "ERROR" do you consider the event an error.  Is this correct?  All you need to do is to literally combining this criterion and the filters in the second with OR logic.  Something like index="iss" Environment=PROD Appid IN ("APP-61", "APP-85", "APP-69", "APP-41", "APP-57", "APP-71", "APP-50", "APP-87") ( ("Invalid JS format" OR ":[down and unable to retrieve response" OR "[Unexpected error occurred" OR ": [An unknown error has occurred" OR "exception" OR "IN THE SERVICE" OR "emplateErrorHandler : handleError :" OR "j.SocketException: Connection reset]" OR "Power Error Code" OR "[Couldn't kickstart handshaking]" OR "[Remote host terminated the handshake]" OR "Caused by:[JNObject" OR "processor during S call" OR javx OR "Error while calling" OR level="ERROR" NOT "NOT MATCH THE CTRACT" NOT "prea_too_large" NOT g-500 NOT G-400 NOT "re-submit the request" NOT "yuu is null" NOT "igests data" NOT "characters" NOT "Asset type" NOT "Inputs U" NOT "[null" NOT "Invalid gii" ) OR level!=ERROR) | rex field=_raw " (?<service_name>\w+)-prod" | eval err_flag = if(level="ERROR", 1,0) | eval success_flag = if(level!="ERROR", 1,0) | stats sum(err_flag) as Total_Errors, sum(success_flag) as Total_Successes by service_name | eval Total_Transaction = (Total_Successes+Total_Errors) | fields service_name, Total_Transaction, Total_Errors, Total_Successes Note I removed the "| where" command in the first because it is more efficient to place the simple filter as a search filter.  I also corrected a syntax error in the second from OR OR IN THE SERVICE" to OR "IN THE SERVICE".  If any of these changes semantics, make adjustments.  Hope this helps.
@bharathkumarnec  Hi, Where exactly we can run this btool to check the configurations. I dnt have back end access can we check in ui.
Can you describe your deployment a bit - # search heads, # indexers, are they on VMs or bare metal, maybe share a screenshot where you are seeing this high IOWait vs CPU (e.g. are you seeing this in ... See more...
Can you describe your deployment a bit - # search heads, # indexers, are they on VMs or bare metal, maybe share a screenshot where you are seeing this high IOWait vs CPU (e.g. are you seeing this in a Splunk dashboard or from OS tools).
What type of Splunk instance are you getting this notification on - your Indexer, or are you using a single Splunk instance? I'm also assuming this notification is coming from outside of Splunk - lik... See more...
What type of Splunk instance are you getting this notification on - your Indexer, or are you using a single Splunk instance? I'm also assuming this notification is coming from outside of Splunk - like an OS monitor or something like that. For example, an Indexer is generally going to put lots of stuff into $SPLUNK_HOME/var/lib/splunk because that's where the indexes are stored.   The first thing I could think of is your indexer drive is almost full, and you index more data causing you to dip below a Splunk-configured max index size.  This causes Splunk to then move/delete old buckets of data because it dipped below the configured threshold.   Here's docs on this this concept:  Configure maximum index size - Splunk Documentation Also, these configs would be in one of your indexes.conf files.
When doing a high-availability with multiple cluster managers (CM), you should have a load balancer (LB) in front of them.  The manager_ui configuration then points to the LB - not an individual CM. ... See more...
When doing a high-availability with multiple cluster managers (CM), you should have a load balancer (LB) in front of them.  The manager_ui configuration then points to the LB - not an individual CM.  The CM's will keep their bundles in sync, and the LB will help ensure that your indexers always reach a "good" one that is up.   Here's the docs that explain how to put an LB in front of the CMs in this scenario.
Hello All,  I am setting up a multisite indexer cluster with cluster manager redundancy,  I am setting up 2 clustermanager (site1 and site2) Below is the config e.g. [clustering] mode = manag... See more...
Hello All,  I am setting up a multisite indexer cluster with cluster manager redundancy,  I am setting up 2 clustermanager (site1 and site2) Below is the config e.g. [clustering] mode = manager manager_switchover_mode = auto manager_uri = clustermanager:cm1,clustermanager:cm2 pass4SymmKey = changeme [clustermanager:cm1] manager_uri = https://10.16.88.3:8089 [clustermanager:cm2] manager_uri = https://10.16.88.4:8089 My question is, I have 2 indexers on each site, should I give the manager_uri in the peer (indexer) of site1 to point to cm1 and manager_uri in the peer (indexer) of site2 to  point to cm2. or all should point to the same indexer? indexer 1 / indexer 2 -  manager_uri = https://10.16.88.3:8089 indexer 3 / indexer 4 -  manager_uri = https://10.16.88.4:8089   Also in the SearhHeads what should I define for the manager_uri? please advice.   Thanks, Dhana
...
Hi This is quite long story and there are couple of ways to do it. Before anyone answer it here you could try to look answer from https://splunk-usergroups.slack.com/archives/CD6JNQ03F. Unfortunate... See more...
Hi This is quite long story and there are couple of ways to do it. Before anyone answer it here you could try to look answer from https://splunk-usergroups.slack.com/archives/CD6JNQ03F. Unfortunately there is no only one part of this channel which you need to read, instead of it has spread over all time this channel.  Those i* nodes are good for smart store, but it depends what kind of architecture you have and how you are managing and automate it. I propose that you will contact someone Splunk and AWS specialist who have done this earlier as there are many places where you could fail (read that channel and you see at least some of those). It's not enough that this person is AWS guru or Splunk guru. He/she must understand both environments to get this working. r. Ismo
Use that regex in SEDCMD in props.conf. [mysourcetype] SEDCMD-no_UTF-8 = s/\x1B\[[0-9;]*[mK]//g  
You need to understand the content of issue! Without that understanding those SPLs don't tell you what is the real issue!
Hi One way to look that what @ITWhisperer told is just query like this index=* | eval iDiff=_indextime - _time | bin span=1h _time | stats avg(iDiff) as iDiff by _time host index sourcetype | where... See more...
Hi One way to look that what @ITWhisperer told is just query like this index=* | eval iDiff=_indextime - _time | bin span=1h _time | stats avg(iDiff) as iDiff by _time host index sourcetype | where iDiff > 60*60*5 | fieldformat iDiff = tostring(round(iDiff),"duration") You need just adjust those parameters by your needs and start to dig out is there any issues as lag or is there real issue in onboarding process where your time stampping is not correct. Or could there even be some timezone or other clock issues. r. Ismo
This answer is not relevant to my question.  I am looking for the splunk query to get the index and sourcetype delay. very simple.. 
How would you measure that delay? If it is the difference between the time the event was indexed and the timestamp as stored in _time with the event, you need to look at how that timestamp was derive... See more...
How would you measure that delay? If it is the difference between the time the event was indexed and the timestamp as stored in _time with the event, you need to look at how that timestamp was derived, which will depend on the sourcetype, where the logged event was picked up and its journey into Splunk. For example, if I have a log entry which has a timestamp created in one timezone, and the timezone is not part of the timestamp, when it is parsed into _time, it may not be given the timezone I was expecting, and can therefore appear to be hours late (or even early if the timezones are reversed). Also, depending on the application logging the event, the timestamp in the log may not be the same as the time it was written. For example, Apache HTTPD will log events with a timestamp relating to when the request was received, but will only log it when the response has been sent. Admittedly, 5 hours would be a bit extreme in this case, but the point is that the timestamp that Splunk assigns to the event does not necessarily have to be coincident with the time the log event was written to the log. This is why you should try and determine if there is any pattern to the apparent delay. You could start by comparing the _indextime to the _time field in your events and see where the large differences are (there is always likely to be a difference because the log has to get from where it is written to the index and that always takes some time.
I have given the example of 5 hours span time log delay in last 30 days.. its not specific to particular sourcetype. I want to see if any log delay in splunk.