About P_vandereerden

P_vandereerden · ‎05-23-2025

Splunk gives you amazing tools to analyze system data and make business-critical decisions, react to issues, and even proactively address future problems. And since it is so good at those things, it’s not surprising that there are myriad tools to help you reflexively analyze your own Splunk deployment. For example, the Splunk Monitoring Console (CMC), available on Splunk Cloud, allows you to check the health of your deployment, observe ingest and search trends, and dive into details of the usage patterns. But what does it mean if the CMC is telling you your ingest is steady and your system is healthy? How much more load can you put on your system before performance degrades below acceptable levels? What are the hourly patterns in your ingest or search load? What about daily, or monthly, or yearly patterns? What is happening on the system when you see degraded performance? These questions are difficult, if not impossible, to answer with the CMC alone. In many cases, users may resort to opening a tech support case with Splunk to answer them. What if there were a tool that could help you understand the performance characteristics of your particular deployment and tune your system to run optimally? Well, there is! Performance Insights for Splunk (PI) was originally developed for internal use by Performance Engineers at Splunk, who needed a deeper understanding of how usage patterns are tied to system and resource usage. Recognizing the value that this tool provides for both Cloud and On Premise customers, it was added to SplunkBase for anyone to use. This article is an introduction to the tool, giving a high-level overview of what it can do and how you can use it to ensure your system is not only running smoothly, but optimally as well. We’re often tempted, when we see performance issues, to assume that more hardware will solve the problem. While this is often true, it might not be the most cost effective solution. PI is meant to help you identify realized or potential performance issues, understand what parts of the system are involved, and monitor your environment after taking corrective steps. As examples, some of the issues we might see are skipped scheduled searches, search runtimes that are slower than expected, and higher than normal error rates. If you were working inside the CMC alone, and you saw any one of these issues, you would find it difficult to relate any of them to each other or any other events that might have been happening at the same time. The CMC is great for health checks, but not as good for diagnostics; the pages’ layout is to provide health information for a particular area, with its own groupings and time scales. If you know what you’re looking for, you might be able to find the information in there, but it’s likely not going to jump out at you. Figures 1 and 2: CMC pages using very different, often using time periods that are not customizable PI helps by allowing you to set the same time scale and granularity on every page, for every chart. For example, when trying to see why the CPUs are spiking every 5 minutes, you’ll be able to see that scheduled search counts are also higher at those times, as is search concurrency. You’ll be able to see which searches were triggered 12 times in that hour and how much CPU those searches used. Figures 3, 4, and 5: No matter which page, Performance Insights for Spunk can display the same time range and granularity At the top of every page you’ll find the time range and granularity pickers, for zooming in or out on the data. You’ll also find the cluster selector that allows you to filter relevant charts to metrics from the current search head cluster (or search head, if not clustered), or all search head clusters. Figure 6: Time pickers are specific to a page but can be set to the same values on every page. The Performance Trend page gives an overview of ingest rates and search load, as well as indexer and search head CPU and Memory metrics. This is where you can get a high level view of the general system performance over time. Figure 7: Performance Trends The System Environment and Data page provides details of the deployment landscape and installed applications, along with statistics about the distribution of ingest on indexers, indexes, and source types. Knowing how your data is distributed will help you build more efficient searches. You might also choose to filter or redistribute the data to reduce index sizes. Figure 8: System Environment and Data The Search Metrics page is broken into 6 parts: an overview page, and 5 search-type-specific pages. The overview page shows the collective view of all search activity, including concurrency, runtimes, and counts. Here you can check for seasonality in your searches, showing you how to flatten that load. The search-type-specific pages break down the details of each search type, including detailed runtime statistics, resource usage, skipped search details, and long-running search details. These pages can show you which searches to tune first to get better performance from your system. Figures 9 and 10: Search Overview and Details The Resource Monitoring page outlines CPU and Memory statistics for the search heads and indexers, along with search head restarts. This is likely the starting point whenever you are experience sub-optimal performance. Resource contention, especially CPU, is very often at the heart of performance related issues. If you see exhausted resources here, use the other pages to help tune and optimize to lower the burden on that resource. Figure 11: Resource Monitoring The Splunk Features page gives insight into cache, smart store, data model and bucket statistics, along with information about assets and Identities and Notable events for Enterprise Security details. Figure 12: Splunk Features And finally, the Environment Diagnose page exposes the error rates and error details for indexers and search heads. Seeing what errors were happening at the time of an issue is often key to solving the issue. Even if you're not experiencing any performance problems, minimizing warnings and errors on your system is generally a good practice, and can save you some system resources. Figure 13: Error Trends and Reporting Armed with these views into your Splunk deployment, correlating an observed behaviour with other events that were happening at the same time is made easy. PI will not only help you diagnose issues with your system more quickly, but also allow you to find the best adjustments to make to reduce resource contention, allowing you to do more with the same amount of hardware. This can lead to significant savings in the long run. This brief overview of PerfInsights for Splunk just scratches the surface of what you can do with the tool. In coming articles, I will walk through case studies showing how particular problems are solved with this tool, and how you can unlock its potential to get the most out of your Splunk system. So let's get started! Visit SplunkBase and install Performance Insights for Splunk today!

P_vandereerden · ‎07-16-2024

This part suggests that the dropdown selections are values in the COMPONENT ID field: "COMPONENT ID"="$DropdownValue$" If that's the case, you could filter based on "COMPONENT ID" IN ($DropdownValue$) and join the subsearches on COMPONENT ID rather than appending columns. Joins are not particularly efficient, so instead of that suggestion, I would look at pulling all the data from that index back in a single search, and conditionally evaluating the stats. Paul

P_vandereerden · ‎06-20-2024

Yes. You can name multiple capture groups in one rex statement. e.g. | rex field=my_field "foo:\s+\"(?<first_capture>[^\"]+)\",\s+bar:\s+(?<second_capture>[^\"]+)"

P_vandereerden · ‎06-20-2024

Have you tried | table endpointOS * That should sort them in alphabetical order, which might be enough here.

P_vandereerden · ‎06-19-2024

Use the first search as a subsearch: index=myapp [ search index=myapp "some log line" | rex field=_raw "req_id=(?<req_id>[a-zA-Z0-9]*)" | table req_id ] | transaction req_id

P_vandereerden · ‎06-19-2024

The CSV is not structured as a lookup table. The structure should be that, given a value for CPU1 (e.g. "process_a"), what are the (first matching) values for CPU2 ("process_b") and CPU3 ("process_c"). What you seem to be looking for is given a value for some CPU (e.g. "process_a"), to what CPU category does it belong ("CPU1"). Are you able to restructure the test.csv to be more like: Process CPU Class process_a CPU1 process_b CPU2 process_c CPU3 process_d CPU1 process_e CPU2 process_f CPU3 process_g CPU1 process_h CPU2 process_i CPU3 IF you can't restructure that file, something like this would work: | makeresults | eval CPU=mvappend("process_a","process_a","process_b","process_a","process_c","process_a","process_b","process_d","process_a","process_e","process_a","process_b","process_c","process_a","process_a","process_b","process_d","process_a","process_c","process_a","process_b","process_e","process_a") | mvexpand CPU ``` The above is to generate sample data and can be ignored in your SPL ``` ``` uncomment the line below and notice the change from CPU1 to CPU ``` ```index=custom | eval SEP=split(_raw,"|"), eval CPU=trim(mvindex(SEP,1))``` ``` These two lines create aliases to map in the CPU group for each class in turn ``` | eval myCPU1=CPU | eval myCPU2=CPU ``` These next lines assume that a process will only appear once in the test.csv file. ``` ``` If that is the case, then CPU2 and CPU3 will be non-null when CPU1 matches, ``` ``` otherwise that process does not belong to CPU1 (and ditto for the CPU2 case.) ``` | lookup community CPU1 as myCPU1 | eval myCPU1=if(NOT isnull(CPU2),CPU,NULL) | lookup community CPU2 as myCPU2 | eval myCPU2=if(NOT isnull(CPU1),CPU,NULL) ``` Now create your stats on the two CPU classes. ``` | bin _time span=1m | stats count(myCPU1) as CPU1_COUNT count(myCPU2) as CPU2_COUNT by _time

P_vandereerden · ‎06-19-2024

Did you want cids to contain that GUID? Try | rex field=log ".*customers\s(?<cids>.*)" Alternatively, if the GUID is always at the end, following a space, you can even drop the "customers" part: | rex field=log "(?<cids>\S+$)" Your example appears to be creating a capture group named "cids" that captures nothing (the first set of parentheses), and then a second non-capturing group that matches what you want (the second set of parentheses). This document might help explain in more detail: https://docs.splunk.com/Documentation/SCS/current/Search/AboutSplunkregularexpressions#Capture_groups_in_regular_expressions

P_vandereerden · ‎06-19-2024

I believe that your scenario could be accomplished with Ingest Actions: https://docs.splunk.com/Documentation/Splunk/9.2.1/Data/DataIngest This should support cloning data and applying different filtering rules and routing to the two streams.

P_vandereerden · ‎06-18-2024

One possible solution would be to use a lookup (status_lookup) to keep track of the last known state. This solution adds a host field so it can work for more than one host. Step 1: Create a KVStore (or file based) lookup with the fields "host", and "current_status" (Note: the solution below will also add an alert message field, but that 's more of a side effect.) Step 2: Add the "host" group by clause, and lookup commands to your SPL: index=xyz sourcetype=xyz host=* | eval RespTime=time_taken/1000 | eval RespTime = round(RespTime,2) | bucket _time span=2m | stats avg(RespTime) as Average perc80(RespTime) as "Percentile_80" by _time host | eval Current_Server_Status=if(Percentile_80>=5, "Server Down", "Server Up") | lookup status_lookup host | eval alert=case(Current_Server_Status="Server Down",$host$+" is down", (Current_Server_Status="Server Up" AND Server_Status="Server Down"),$host$+" is back up") | rename Current_Server_Status AS Server_Status | table host Server_Status alert | outputlookup status_lookup You'll end up with a serach that outputs something like this (and updates the lookup for the next alert run): +---------------+--------------+------+ | Server_Status | alert | host | +---------------+--------------+------+ | Server Down | a is down | a | | Server Up | b is back up | b | | Server Up | | c | | Server Down | d is down | d | +---------------+--------------+------+ Note that host c has no alert message because it went from "up" to "up" with the sample data I used.

P_vandereerden · ‎06-18-2024

Retracted. @richgalloway 's solution will work.

P_vandereerden · ‎06-18-2024

In conditionals, use "like" instead: | makeresults | eval msg.message=mvappend("Work Flow Passed | for endpoint XYZ","STATUS - FAILED") | mvexpand msg.message ``` SPL above is to create sample data only ``` | rename msg.message as message | eval Status=if(like(message,"%Work Flow Passed | for endpoint XYZ%"),"SUCCESS", "FAIL") | table _time, message, Status It also helps to rename fields with paths to avoid the need for quoting them.

P_vandereerden · ‎06-18-2024

There might be a more fluid way to do this, but one idea would be to make your alert a two-step process: 1) Add " | addinfo " to your search to get the search SID, and have the alert log an event with that SID instead of sending email. 2) Create the alert and make your alert decision by searching for the new event log, and either using the " | rest /services/search/jobs/<SID> ", or searching the _internal or _audit indexes to get metadata about that search.

P_vandereerden · ‎06-18-2024

Better late than never: Sample data would be helpful here. The request is a bit confusing since you seem to want the top 5 urls per status code, but your URL count stops at 10. With 3 status codes, the top 5 could go to 15, right? For the second point, what UserID would that be? Presumably each URL could be hit by multiple users, and the top 5 codes for each URL would differ per user.

P_vandereerden · ‎06-18-2024

Better late than never: This needs more information on what you consider month and week boundaries. Does "January, Week 1" mean the first 7 days of January, January 1st to the last day of the week (e.g. Saturday), or the Sunday before January 1st, to the following Saturday? When you say by week and month together, do you just want a label for the month in front of the 52 weeks?

P_vandereerden · ‎06-18-2024

Better late than never: Assuming your windows host monitoring polls the hosts at regular intervals and logs a success or failure, and if you want a simple line chart with values 1 for up and 0 for down in some interval (say 10 minutes), you could do this: sourcetype=WinHostMon | eval status_num=if(Status="up",1,0) | timechart span=10m min(status_num) by Host

P_vandereerden · ‎06-17-2024

In the "better late than never" category of answers (and I realize this answer might not have been available in previous versions of Splunk)... It's unclear, from the original question, if the "ip:port" belongs to the service, or the client. If it belongs to the service, then every timeout uniquely identifies the service, and all that needs to be done is to count the timeouts, and then map in the service name: | makeresults | eval data="CONNECTION-1.1.1.1:1: connect() timeout,[service_with_2_timeouts] tearing down tcp connection [1.1.1.1.1],CONNECTION-1.1.1.2:2: connect() timeout,[service_with_1_timeout] tearing down tcp connection [1.1.1.2.2],[service_with_no_timeouts] tearing down tcp connection [1.1.1.3.3],CONNECTION-1.1.1.1:1: connect() timeout,[service_with_2_timeouts] tearing down tcp connection [1.1.1.1.1]" | eval mvdata=split(data,",") | mvexpand mvdata ``` Everything above this is to generate sample data ``` | eval is_timeout=if(like(mvdata,"%connect() timeout%"),1,0) | rex field=mvdata "CONNECTION-(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}):(?<port>\d+): connect timeout" | rex field=mvdata "\[(?<service_name>[^\]]+)\] tearing down tcp connection \[(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\.(?<port>\d+)\]" | stats first(service_name) as service_name, sum(is_timeout) as timeout_count by ip, port If, on the other hand, the "ip:port" belong to the client accessing the service, this is a bit more complicated, with too many potential solutions depending on details not available here.

P_vandereerden · ‎06-17-2024

If I'm reading this right, you have data that has events with pods and their phases. In your example query, you appear to be using decimal values to create your ranges, but can we assume that the actual pos states fall on specific integers? Something like this might work: | makeresults count=25 | eval phase=(random()%5)+1 ``` Everything above here is just to create sample data ``` ``` The following statement groups and counts phases. | stats count by phase ``` The following statement maps phases to a string equivalent ``` | eval label=case(phase=1,"A (Pending)", phase=2,"B (Running)", phase=3,"C (Succeeded)", phase=4,"D (Failed)", phase=5,"E (Stopping?)", 1=1,"Unknown") If the phase values are not discreet, and the range you mention is necessary, then you can use a case statement like this: | makeresults count=25 | eval phase=((random()%50)/10)+1 | eval phase_group=case(phase<1.5,1, phase<2.5,2, phase<3.5,3, phase<4.5,4, phase<5.5,5) | stats count by phase_group | eval label=case(phase_group=1,"A (Pending)", phase_group=2,"B (Running)", phase_group=3,"C (Succeeded)", phase_group=4,"D (Failed)", phase_group=5,"E (Stopping?)", 1=1,"Unknown")

P_vandereerden · ‎06-17-2024

Joining the two searches would require some common field to join on. Since none exists in your example, you'll need to either add an identifier to all related logs at the source, or get creative with a solution based on time that could get finicky. For example, in your sample data, you have 3 events. The transaction ID in event 1 occurs 2 seconds before the error log. If there can be more than one concurrent transaction, then there doesn't appear to be a way to be certain that the correct transaction ID will be found that corresponds to error. e.g. 240614 04:35:50 Algorithm: Al10: <=== Recv'd TRN: AAA (TQ_HOST -> TQ_HOST) 240614 04:35:51 Algorithm: Al10: <=== Recv'd TRN: BBB (TQ_HOST -> TQ_HOST) 240614 04:35:52 Algorithm: TSXXX hs_handle_base_rqst_msg: Error Executing CompareRBSrules Procedure. 240614 04:35:52 Algorithm: TSXXX hs_handle_base_rqst_msg: Details of ABC error ReSubResult:-1,FinalStatus:H,ErrorCode:-1,chLogMsg:SQL CODE IS -1 AND SQLERRM IS ORA-00001: unique constraint (INSTANCE.IDX_TS_UAT_ABC_ROW_ID) violated,LogDiscription: In this case, does the error belong to transaction AAA or BBB? The second issue will be how much time can elapse between the "Rec'd TRN" log, and any possible error. Without a field linking these logs, you'll have to use some fixed time range to try to bring logs together. Too short, and you'll fail to find the transaction ID, too long and you might find multiple IDs (leading to the issue mentioned above). IF you can assume that logs are synchronous, and there is no interleaving of transactions, then something like this should work: index=test_index source=/test/instance | sort _time | rex field=_raw "<=== Recv'd TRN:\s+(?<transaction_id>\w+)" | eval failure=if(like(_raw, "%ORA-00001%"), 1, 0) | filldown transaction_id | where failure=1 | table transaction_id, failure, _raw

P_vandereerden · ‎06-15-2024

In my example, I use 3 backslashes when creating the sample data. To get \" in a quoted string, you need escape the backslash \\, and the quote \", resulting in \\\" In the regex, I avoided the need to match on backslashes, so any backslash is just the escape character. However, in my alternative method, you'll notice that there are 5 backslashes in a row. The processing of the escape characters happens once for the string itself, taking \\\\\" down to \\", and then once for the regex, taking \\" down to \".

P_vandereerden · ‎06-14-2024

Ah. I suspect this is more about the rex expression than the table. You could try a join: index=myindex TTY | rex field=_raw "Job Id: (?<jobId>.*?)\." | join left=L right=R where L.jobId=R.jobId [search index=myindex cs2k_transaction_id_in_error="CHG063339403031900" major_code="ERROR" | rex field=_raw "Job Id: (?<jobId>.*?)\." | table jobId ]

P_vandereerden · ‎06-14-2024

Would it be possible to post some sample data. It's a bit too easy to get lost in what is supposed to be an escape character versus a character in your data. Please replace any real phone numbers with dummy values. Escaping backslashes for regex expressions is always fun, but I suspect that's where your issues are coming from. Escaping a backslash in a regex from the search box requires four backslashes as there are two layers of escaping that are happening. I try to construct regexs to avoid that: | makeresults | eval phone_data="[{\"PhoneNumber\":\"123-456-7890\"}]" | append [ | makeresults | eval phone_data="[{\\\"PhoneNumber\\\":\\\"111-111-1111\\\"}]" ] | rex field=phone_data "PhoneNumber[^\d]+(?<my_PhoneNumber>[0-9-]+)" but if I'm making an incorrect assumption about the characters in aphone number, you can try | rex field=phone_data "PhoneNumber[^\d]+(?<my_PhoneNumber>[^\\\\\"]+)"

P_vandereerden · ‎06-14-2024

Have you tried a subsearch? index=myindex "TTY" [ search index=myindex cs2k_transaction_id_in_error="CHG063339403031900 major_code="ERROR" | rex field=_raw "Job Id: (?<jobId>.*?)\." | table jobId ]

P_vandereerden · ‎06-14-2024

I don't know this area well, but the error suggests an issue with "container", and not "update". Within your custom function you are using container, but it would seem it's not defined. How are you passing "container" into your function?

P_vandereerden · ‎06-14-2024

Alternatively, you could use streamstats to build a list of files to match against: index=wealth OR index=transform-file OR index=ace_message earliest=-30m | rex field=_raw "inputFileName: (?<inputFileName>.*?)," | rex field=inputFileName "file\_(?<ID>\d+_\d+)\_" | streamstats values(eval(if(now()-_time<1800,ID,NULL))) as IDS | eval alertable=if((now()-_time>1800) AND (ID IN (IDS)),"True","False") | table _time, ID, IDS, alertable

P_vandereerden · ‎06-14-2024

Besides the approximate time (since the times don't match), there is nothing else to relate those two particular logs together? Will your search be used in the general case to output more than one row's worth of data? If so, how far apart are the various distinct transactions (or can that be arbitrarily short)?

Posts	33
Solutions	8
Karma Given	5
Karma Received	9
Member Since	‎12-06-2022

Online Status	Offline
Date Last Visited	‎05-27-2025 04:28 PM

The All New Performance Insights for Splunk

The All New Performance Insights for Splunk

Re: Loop through splunk search for multiple values

Re: rex - Extracting a string

Re: contingency table column order

Re: How do I search for all events in the same "re...

Re: Combine 2 searches with sub search into One Ti...

Re: rex - Extracting a string

Re: Copying over data to another index for longer ...

Re: Server Down and Up Alert

Re: Alert to run specific time every weekday for s...

Re: Conditional Success/Fail

Re: How to remove results if indexer errors occurr...

Re: How to get multiple search condition on a sing...

Re: How to create a Stacked Bar Chart for the cate...

Re: How do I create a chart of Windows host status

Re: Searching on consecutive lines

Re: Check on in between of a ranger query

Re: How to Join two queries

Re: Splunk regex bug/issue

Re: Search based on response from another search q...

Re: Splunk regex bug/issue

Re: Search based on response from another search q...

Re: Container automation API compatibility with cu...

Re: fetch todays filename and check with last 30 d...

Re: How to Join two queries

Are you a member of the Splunk Community?