Hi geraldcontreras, as I understand your question, your challenge is to combine a query looking at a long historic timespan, basically "All Time" with a very recent view on your network looking for indicators (of compromise). The reason, why your first query is slow is, that for both parts of the query, you are searching through all the events. The terms earliest=, latest= filter the results, but don't affect the search time range. Splunk is great, so there are multiple ways to solve this problem 🙂 ... this is how I would solve it for my customers: I would create a lookup table for known indicators as KV store and have a scheduled report to populate and update it. The trick is to have a unique key that describes your indicator, so that you can easily update the field in the store. In my example I take an md5(indicator) as key : index=... | whatever it takes to calculate your indicators | eval key=md5(indicator_name) | table key, indicator_value | outputlookup indicator_kv key_field=key Have this query run at regular intervals to update the lookup table. Whenever a value changes for a known indicator that's in the table, the key_field option will actually update the existing value. If the key does not yet exist, it is added. Now, your dashboard query (the needs to be fast one) would look like this: index=network src=* | eval key=md5(src) | lookup indicator_kv _key as key output indicator_value as splunk_indicator (Hint: If you don't want the lookup to return NULL values for splunk_indicator, if the key is not found, you can assign a default value in the lookup definition under Advanced Options ) Hope it helps Oliver ... View more

Hi, you need to find the root cause for the error. On the splunk UI, dropdown "Job" and select "Inspect Job" to open the job inspector of a failed query. In the Job inspector window, look for the "search.log" link and click it. In the log, search for the words "WARN" or "ERROR" Once you know the root cause for the problem, we could probably help you to avoid it in the future. Hope it helps, Oliver ... View more

My guess would be that the transaction with mvlist=true does not return NULL as a value but rather "NULL" as a string. You can easily test the assumption by trying len(x). If it returns 4, then the transaction has written the word "NULL" into the result. If len(x) gives you nothing or 0, then it's the NULL value itself. From what you've observed, I think, it's the first. ... View more

Hi poddraj, first, you create an "empty table" with all possible time slots: | gentimes [|makeresults|addinfo|eval start=strftime(info_min_time,"%m/%d/%Y:%H:%M:%S")| return start] [|makeresults|addinfo|eval end=strftime(info_max_time,"%m/%d/%Y:%H:%M:%S")| return end] increment=15min the next will create some demo devices. In your query you could use a | join [|inputlookup] to create one line per time per device | eval FT="deviceA,deviceB,deviceC" | makemv delim="," FT | mvexpand FT the next step is to create one line per possible error code per device per time: | eval Error_Code="8950,2166,7633" | makemv delim="," Error_Code |mvexpand Error_Code and finally initialize the counter with 0 and rename the time: | eval Failure=0, _time=starttime If you concatenate all in sequence, you will get a table like this: '2020-05-06 12:00:00' 'DeviceA' 'Error1' 0 '2020-05-06 12:00:00' 'DeviceA' 'Error2' 0 '2020-05-06 12:00:00' 'DeviceA' 'Error3' 0 '2020-05-06 12:00:00' 'DeviceB' 'Error1' 0 ... '2020-05-06 12:15:00' 'DeviceC' 'Error3' 0 Now, you can append your query: | append [search index=dte_fios sourcetype=dte2_Fios FT=*FT earliest=04/20/2020:11:00:00 latest=04/20/2020:13:00:00 | bin _time span=15m | eval Failure=if(Error_Code!="0000",1,0)] The resulting table has your query results mixed into the "empty frame of no failures". Finally you can run your stats and since you have a result for every possible time value in your query range, you will get a perfectly filled result table | stats count as Total, sum(Failure) AS Failure by FT,Error_Code,_time | eval Failurepercent=round(Failure/Total*100) Hope it helps Oliver ... View more

To me it looks like the user that you use to access the links does not have the debug privileges. Try with admin. ... View more

8.0.2 latest. ... View more

Hi there, first of all, like to4kawa pointed out, the example search was syntactically wrong. second, the implementation of the traffic light that you use may not be suitable for your version of Splunk. The option values additionalClass and classField are deprecated. You may need to look for a newer solution that actually works with v8.x but aside from that, I pasted your code into my dashboard, clicked on the value and it worked like a charm. Please install and take a look at the dashboard examples app from splunk. https://splunkbase.splunk.com/app/1603/ Take a look at the example "Custom Decorations". There they use .css to add a custom character to the single value. Exchange that character with an image and you're pretty much done. Oliver ... View more

Please define "searchable". The expected behaviour is that a click on the number digits of the traffic light single value opens a new browser tab/window with a search "index=_internal | stats count by sourcetype". It works in my lab environment. Please try it in a new dashboard with only one traffic light and post your simple xml code. Oliver ... View more

Hi try setting the option "drilldown" to "all" and insert a drilldown tag like in the example below. Clicking the value will activate the search. <single> <search> <query>index=_internal | timechart count</query> <earliest>-15m</earliest> <latest>now</latest> </search> <option name="drilldown">all</option> <drilldown> <link target="_blank">search?q=index=_internal | stats count by sourcetype</link> </drilldown> <option name="refresh.display">progressbar</option> </single> ... View more

Let's get to the facts: You want to treat every single event differently, drawing your formula from a lookup table. The only way to substitute a formula into a query that I know of is by using a subquery, like you did. Fact: This substitution works globally, because it takes place before your main query starts. If you want to substitute the formula for ALL events alike this works like a charm. If, however, you want to use a different formula based on the P and KPI of every single event, you simply can't because you cannot use a different subquery on every single event. The only way I see to solve this is by using a custom search command: Lookup the formula from your lookup call a streaming command that gets the formula and all the other fields from the event assemble a new query string from the event that splunk passes to you and that looks somewhat like '|makeresults | eval v1="$" ... (assign all the values passed) | eval kpi_value=$formula" ' call splunk through the REST api to execute that single query pass the result back by setting the kpi_value to the result of the REST call Of course you might also try to solve it without implementing some python by using the splunk rest command and assemble the search string using eval. ... View more

Hi, sistats creates the summary index and doesn't output anything. It is an alternative to the collect suggested above. To report from the summaries, you need to use a stats. I have copied this from the documentation of the sistats command: Create a summary index with the statistics about the average, for each hour, of any unique field that ends with the string "lay". For example, delay, xdelay, relay, etc. ... | sistats avg(*lay) BY date_hour To create a report, run a search against the summary index using this search index=summary | stats avg(*lay) BY date_hour Hope it helps Oliver ... View more

Hi, the challenge I see is that you want the subsearch to return exactly one formula for the given KPI and Process based on the parameters in the main search event. The trouble is that the subsearch runs before the main search and hence you simply cannot pass any parameters. You could, however, use a dashboard token in your subsearch, but I don't see how it could help you solve this case. Also, use | return $variable in your subsearch to return the content of your variable: | return $formula My approach to solve this would be by implementing a custom search command. Cheers Oliver ... View more

Hello, could it be that you are running the forwarder as a non-privileged user? If so, you could change the ownership of the splunkhome\var directory tree to the splunk user. Oliver ... View more

Hi, I'm using this approach in my dashboards: index=paloalto sourcetype="pan:log" | transaction user_id keepevicted=true startswith=status="login" endswith=status="logout" | eval session=if(status="logout","inactive",if(status="login","active","unknown")) | search NOT session="unknown" | eval start_time=_time, end_time=if(closed_txn=1,start_time+duration,now()),duration=(end_time-start_time) | eval bucket_time_size=900 | eval start_bucket_time = start_time - (start_time % bucket_time_size) | eval end_bucket_time = end_time - (end_time % bucket_time_size) | eval buckets = mvrange(start_bucket_time,end_bucket_time,bucket_time_size) | eval buckets=if(isnull(buckets),_time,buckets) | mvexpand buckets | eval _time=if(isnull(buckets),_time,buckets) | timechart count by session span=15m Some hints as to what's happening: First, the transaction will group the events based on user_id. A closed transaction will have a multi-value field status=[login,logout] whereas an open transaction will have a status="login". A session that has started before the time searched and closes within will have a status="logout". Any session that starts before and extends after will fall into the "unknown" category. If you'd like to see them and have the user_id in those events, replace "unknown" with "active" and adjust the _time and duration to reflect the search range (hint:addinfo). After the prequel, the binning starts. To make it short, my trick is to define a set of time slices (900s in my example) and for every time slice, determine if the event falls into that time slice or not. Set start_time to the transaction _time which is the time of the first event (login). If the transaction is closed, end_time=start_time+duration, else the transaction is still active, hence end_time=now(). Set a bucket_start_time which is the time the bucket starts that the start event lies in. Same with the end. mvrange creates the list of all buckets (_start_times) that the event covers. mvexpand will create one event per bucket. Set the event _time for every "bucketized" event and you are ready to timechart. Enjoy ... View more

I've amended the answer below. Please always include sample data and an example of the output as it really helps to guess what you are up to 🙂 ... View more

Hi, I assume, that the transaction_id is unique and will identify all events that are connected with the transaction. With this assumption I suggest something like this: <your search to get all the events from one or multiple indexes> | stats earliest(_time) as start_time, latest(_time) as end_time by transaction_id | eval time_used=end_time-start_time This will get you a field time_used that holds the time between the first event and the last event in seconds. If you want to get the time over multiple transaction_id values, you would probably do something like this: <your search to get all the events from one or multiple indexes> | eval key=[<subsearch to produce transaction_id #1>| return $transaction_id].[<subsearch to produce transaction_id #2>| return $transaction_id] | stats values(transaction_id) as transactions, earliest(_time) as start_time, latest(_time) as end_time by key | eval time_used=end_time-start_time | fields - key ... View more

Hi, from a logical data flow point of view: First you need to get a list of all clients, those that have updates and/or users and those that don't. Either you query an index for "client creation events", select the data from another system/database or you use a lookup table/csv. Whatever you choose, you will get a one-column table with all clients like "PPN,CNB,XXD,TYZ...INB...etc.". Try this query to generate some data that show what the result would look like: | makeresults | eval Clients="PPN,INB,CNB,XXA,etc" | table Clients | makemv delim="," Clients | mvexpand Clients Then you join the list with the stats results of the clients and finally, you fill the null values. The result should look something like this: <Client Table generating search> | join type=left Clients [ search <your search generating the user/project events> | stats sum(users) as users, latest(project_id) as project_id by Clients] |fillnull value="N/A" Use this example to generate some sample data for the join search: | makeresults count=10 | eval users=random()%100 | eval Clients=if(users>50,"PPN","CNB") | eval project_id="Project v".users | stats sum(users) as users, latest(project_id) as project_id by Clients The full query looks like this: | makeresults | eval Clients="PPN,INB,CNB,XXA,etc" | table Clients | makemv delim="," Clients | mvexpand Clients | join type=left[| makeresults count=10 | eval users=random()%100 | eval Clients=if(users>50,"PPN","CNB") | eval project_id="Project v".users | stats sum(users) as users, latest(project_id) as project_id by Clients] | fillnull value="N/A" Best Oliver ... View more

Hi, IMHO lookup is the fastest and most efficient way. You could use case statements, which make the code look clumsy and hard to maintain in a non-trivial setting. If you would like to use an index instead of .csv, you could index the map and do a ...| join type=inner process_name [ search index=yourindex process_group=$tok_filterprocess$ | table process_name, process_group] | ... in your search, adding some overhead and complexity. Alas, if you use the "inner" join, you can leave out the additional search that selects the matched processes, as this is implicitly being done in the join. Oliver ... View more

Hi, let's assume you'd like to know all machines that run linux within your network. First, you have to find any active clients and you could use two approaches: Active scans or passive listening in on the traffic. For active scanning, nmap is a good starting point. https://en.wikipedia.org/wiki/Nmap It will deliver some findings and can be configured to give you a probability for the OS that a client runs, based on open ports and probes that the tool uses to discover the version of OS. Whatever nmap delivers, you could index into splunk and do further analysis. For passive scanning, you could always use the stream app https://splunkbase.splunk.com/app/1809/ have it listen to your network and you could do some forensics on the data. For example, you could look for ARP packages, disclosing the MAC to IP translations. If you wait long enough, a rogue linux system will likely contact a ntp time source outside of your own ntp setup. Luckily, linux clients don't use any microsoft.com domains for time resolution, but rather something like ntp.ubuntu.com. Unless you distribute NTP through dhcp... but hey, there are other ways... like testing port 22 as linux usually has ssh running or 5900 for vnc. hih Oliver ... View more

Hi, please take a look at https://docs.splunk.com/Documentation/AddOns/released/AWS/S3 it says that splunk supports these file types for single files : ZIP, GZIP, TAR, or TAR.GZ and for multiple files/directories : ZIP, TAR, or TAR.GZ The way I see it, '.gz' can only be used in single file inputs, not in multiple files. If you want to index multiple files, '.tgz' is supported. hih Oliver ... View more

This answer ist pretty good. I'd go for the python script: https://answers.splunk.com/answers/615685/debug-refresh-or-bump-using-rest-api.html ... View more

Hi, I use this to monitor the health of my sourcetypes: | tstats count where index=* by _time, sourcetype,index span=1h | stats sparkline(sum(count)) as fingerprint, sum(count) as count by sourcetype,index ... View more

Hi Andrew, it all depends. The "normal" indexing processes raw events through the indexing pipeline on the indexers. The summary indexing forwards cooked/processed events from a search head and dumps them into the indexer without going through the indexing pipeline again. This in itself makes a huge difference. On a "normal" index you do expect the events to come in chronologically sorted, but it is not guaranteed. I have multiple indices, where the "natural" order of the events is haywire because of the way the data is received with delays in the processing/transfer resulting in different offsets between _time and _indextime. Thus, you are in a comfortable place, when all your "normal" indices are ordered. Happy splunking Oliver ... View more