All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

What happens when you execute the restart command manually? Do you use the correct user in your ansible script? Maybe you have to set "become: tru"e if splunk runs under root.
Dear @PickleRick  I follow using this step https://www.fortinet.com/content/dam/fortinet/assets/alliances/Fortinet-Splunk-Deployment-Guide.pdf but there's not solve the issue.
Dear @PickleRick  Thankyou for your correction , do you have any sugestion or best practice for me ?   Regards, Dika
OK. This approach is wrong on many levels. 1. Receiving syslog directly on an indexer (or HF or UF) causes data loss whenever you need to restart that Splunk component. 2. When you're receiving sys... See more...
OK. This approach is wrong on many levels. 1. Receiving syslog directly on an indexer (or HF or UF) causes data loss whenever you need to restart that Splunk component. 2. When you're receiving syslog directly on Splunk, you lose at least some of the network-level metadata and you can't use that information to - for example - route events to different indexes or assign them different sourcetypes. Because of that you need to open multiple ports for separate types of sources. Which uses up resources and complicates the setup. 3. In order to receive syslog on a low port (514) Splunk would have to run as root. This is something you should _not_ be doing. Are you sure that input has even opened that port? 4. If you have two indexers (clustered or standalone?) and receive on only one of them, you're asking for data asymmetry.  
I found notfication : File Monitor Input Forwarder Ingestion Latency Ingestion Latency Large and Archive File Reader-0 Large and Archive File Reader-1 Real-time Reader-0 Real-time Reader-1 ... See more...
I found notfication : File Monitor Input Forwarder Ingestion Latency Ingestion Latency Large and Archive File Reader-0 Large and Archive File Reader-1 Real-time Reader-0 Real-time Reader-1 are reds too.  is it because too much logs sent from fortigate?
Did you even try my solution? Here is a runanywhere example showing it working with dummy data | makeresults count=100 | fields - _time | eval department="Department ".mvindex(split("ABCDE",""),ran... See more...
Did you even try my solution? Here is a runanywhere example showing it working with dummy data | makeresults count=100 | fields - _time | eval department="Department ".mvindex(split("ABCDE",""),random()%5) | eval version=round(random()%3,1) | eval thumb_print=random()%10 ``` The lines above create some dummy data and can be replaced by your index search ``` | dedup version thumb_print department | eval version=if(version="2.0","NEW_RUNS","OLD_RUNS") | chart count(thumb_print) by department version | fillnull value=0 | eval total=NEW_RUNS+OLD_RUNS | eval perc=round(100*NEW_RUNS/total,2) | eval department=substr(department, 1, 50) | table department OLD_RUNS NEW_RUNS perc | sort -perc
thanks @PaulPanther  @ITWhisperer   Issue is still there with Output department                     OLD_RUNS    NEW_RUNS   total  PERC -----------------------------------------------------------... See more...
thanks @PaulPanther  @ITWhisperer   Issue is still there with Output department                     OLD_RUNS    NEW_RUNS   total  PERC -------------------------------------------------------------------------- Department1                    10           0                 10  0% Department1                     0            20               20  100% Basically old and new count of same department not in same row so with respect to new runs all percentage comes as 100% as old runs shows as 0.   
If you are using Classic/SimpleXML dashboards, you can do this with CSS. For this you need to give your panel an id (so it gets tagged so CSS can select it), then you need to know the order of the s... See more...
If you are using Classic/SimpleXML dashboards, you can do this with CSS. For this you need to give your panel an id (so it gets tagged so CSS can select it), then you need to know the order of the series in the charts and they are numbered. For example, if you name your panel "panel_one", and your Total was the second series (index 1), you could do something like this <panel id="panel_one"> <html depends="$alwaysHide$"> <style> #panel_one svg g.highcharts-data-labels.highcharts-series-1 { display: none !important; } </style> </html> <chart>
Thank you below is splunkd.log    09-20-2024 06:36:54.626 +0000 INFO Shutdown [2498 Shutdown] - shutting down level="ShutdownLevel_HttpClient" 09-20-2024 06:36:54.626 +0000 INFO Shutdown [2498 Sh... See more...
Thank you below is splunkd.log    09-20-2024 06:36:54.626 +0000 INFO Shutdown [2498 Shutdown] - shutting down level="ShutdownLevel_HttpClient" 09-20-2024 06:36:54.626 +0000 INFO Shutdown [2498 Shutdown] - shutting down name="HttpClient" 09-20-2024 06:36:54.626 +0000 INFO Shutdown [2498 Shutdown] - shutting down level="ShutdownLevel_DmcProxyHttpClient" 09-20-2024 06:36:54.626 +0000 INFO Shutdown [2498 Shutdown] - shutting down level="ShutdownLevel_Duo2FAHttpClient" 09-20-2024 06:36:54.626 +0000 INFO Shutdown [2498 Shutdown] - shutting down level="ShutdownLevel_S3ConnectionPoolManager" 09-20-2024 06:36:54.626 +0000 INFO Shutdown [2498 Shutdown] - shutting down name="S3ConnectionPoolManager" 09-20-2024 06:36:54.626 +0000 INFO Shutdown [2498 Shutdown] - shutting down level="ShutdownLevel_AwsSdk" 09-20-2024 06:36:54.626 +0000 INFO Shutdown [2498 Shutdown] - shutting down name="loader" 09-20-2024 06:36:54.628 +0000 INFO Shutdown [2498 Shutdown] - Shutdown complete in 5.124 seconds 09-20-2024 06:36:54.629 +0000 INFO loader [2296 MainThread] - All pipelines finished.
Sorry, made a mistake with the calculation of totals. I adjusted the search in my previous answer.
Hi Paul, Thanks for help.  But this still has some issues. Output department                     OLD_RUNS    NEW_RUNS   total  PERC ---------------------------------------------------------... See more...
Hi Paul, Thanks for help.  But this still has some issues. Output department                     OLD_RUNS    NEW_RUNS   total  PERC -------------------------------------------------------------------------- Department1                    10           0                 10  0% Department1                     0            20               20  100% Basically old and new count of same department not in same row so with respect to new runs all percentage comes as 100% as old runs shows as 0.   
  Subsearches are limited to 50k events which is one of the issues with using joins. Also, your dedup seems to ignore whether more than one department has the same version and thumb_print (of course... See more...
  Subsearches are limited to 50k events which is one of the issues with using joins. Also, your dedup seems to ignore whether more than one department has the same version and thumb_print (of course, unless thumb_prints or versions are unique to department). Try something like this index=abc | dedup version thumb_print department | eval version=if(version="2.0","NEW_RUNS","OLD_RUNS") | chart count(thumb_print) by department version | fillnull value=0 | eval total=NEW_RUNS+OLD_RUNS | eval perc=round(100*NEW_RUNS/total,2) | eval department=substr(department, 1, 50) | table department OLD_RUNS NEW_RUNS perc | sort -perc
Forget the rest of search.  What do you get from the following?   index="logs" sourceip="x.x.x.x" OR destip="x.x.x.x" | lookup file.csv cidr AS sourceip OUTPUT provider AS sourceprovider, area AS s... See more...
Forget the rest of search.  What do you get from the following?   index="logs" sourceip="x.x.x.x" OR destip="x.x.x.x" | lookup file.csv cidr AS sourceip OUTPUT provider AS sourceprovider, area AS sourcearea, zone AS sourcezone , region AS sourceregion, cidr AS src_cidr | lookup file.csv cidr AS destip OUTPUT provider AS destprovider, area AS destarea, zone AS destzone, region AS destregion, cidr AS dest_cidr | table sourceip sourceprovider sourcearea sourcezone sourceregion src_cidr destip destprovider destarea destzone destregion dest_cidr   Is the output correct? Using your mock lookup data, I made the following emulation   | makeresults format=csv data="sourceip, destip 1.1.1.116,10.5.5.5 10.0.0.5,2.2.2.3 2.2.2.8, 1.1.1.90 192.168.8.1,10.6.0.10" ``` the above emulates index="logs" sourceip="x.x.x.x" OR destip="x.x.x.x" ``` | lookup file.csv cidr AS sourceip OUTPUT provider AS sourceprovider, area AS sourcearea, zone AS sourcezone , region AS sourceregion, cidr AS src_cidr | lookup file.csv cidr AS destip OUTPUT provider AS destprovider, area AS destarea, zone AS destzone, region AS destregion, cidr AS dest_cidr | fields sourceip sourceprovider sourcearea sourcezone sourceregion src_cidr destip destprovider destarea destzone destregion dest_cidr   This is what I get, exactly as expected sourceip sourceprovider sourcearea sourcezone sourceregion src_cidr destip destprovider destarea destzone destregion dest_cidr 1.1.1.116 Unit 1 Finance 2 1.1.1.1/24 10.5.5.5           10.0.0.5           2.2.2.3 Unit 2 HR 16 2.2.2.2/27 2.2.2.8 Unit 2 HR 16 2.2.2.2/27 1.1.1.90 Unit 1 Finance 2 1.1.1.1/24 192.168.8.1           10.6.0.10          
index=abc   | dedup version thumb_print | stats count(eval(if(version!="2.0",thumb_print,null()))) as OLD_RUNS count(eval(if(version="2.0",thumb_print,null()))) as NEW_RUNS by department | fillnu... See more...
index=abc   | dedup version thumb_print | stats count(eval(if(version!="2.0",thumb_print,null()))) as OLD_RUNS count(eval(if(version="2.0",thumb_print,null()))) as NEW_RUNS by department | fillnull value=0 | eval total=NEW_RUNS+OLD_RUNS | eval perc=((NEW_RUNS/total)*100) | eval department=substr(department, 1, 50) | eval perc=round(perc, 2) | sort -perc  
If you don't have GUI access to the remote searchhead you must ask your infra team. They should be able to confirm if the custom fields are configured on the remote searchhead.
SPL can present a steeper learning curve compared with non-streaming languages.  But once you get some basics, it is very rewarding for it gives you so much freedom.  This said, SPL's JSON path notat... See more...
SPL can present a steeper learning curve compared with non-streaming languages.  But once you get some basics, it is very rewarding for it gives you so much freedom.  This said, SPL's JSON path notations need some getting used to.  The JSON functions are actually OK once you understand the notations.  Before I give my suggestions, let's examine your original trial.     | spath input=json.msg output=msg_raw path=json.msg     This will not give you desired output because in the embedded JSON object in json.msg does not contain a path named json.msg.  The object that does contain this path is _raw.  If you try     | spath ``` input=_raw implied ``` output=msg_raw path=json.msg     you would have extracted a field named msg_raw that duplicates the value of json.msg: json.msg msg_raw {"name":"", "connection":22234743, "time":20000, "success":false, "type":"Prepared", "batch":false, "querySize":1, "batchSize":0, "query":["select * from whatever.whatever w where w.whatever in (?,?,?) "], "params":[["1","2","3"]]} {"name":"", "connection":22234743, "time":20000, "success":false, "type":"Prepared", "batch":false, "querySize":1, "batchSize":0, "query":["select * from whatever.whatever w where w.whatever in (?,?,?) "], "params":[["1","2","3"]]} Of course, this is not what you wanted.  What did we learn here?  That path option in spath goes into the JSON object itself. But if you try     | spath input=json.msg     you will get these fields from json.msg: batch batchSize connection name params{}{} querySize query{} success time type false 0 22234743   1 2 3 1 select * from whatever.whatever w where w.whatever in (?,?,?) false 20000 Prepared What did we learn here?  Place that field name whose value is itself a valid JSON object directly in spath's input option to extract from that field.  Additionally, Splunk uses {} to denote fields extracted from JSON array, and turn them into a multivalue field. In your other comment, you said you want the equivalent of `jq '.json.msg|fromjson|.query[0]'`.  Such would be trivial from the above result.  Add     | eval jq_equivalent = mvindex('params{}{}', 0) | fields params* jq_equivalent     you get params{}{} jq_equivalent 1 2 3 1 What did we learn here?  1. mvindex selects value from a multivalue field (params{}{}), using base 0 index; 2. Use single quote to dereference value of field whose name contains special characters. A word of caution: If all you want from params{}{} is a single multivalue field, the above can be sufficient.  But params[[]] is an array of arrays.  To complicate things, your developer doesn't do you the best of service by throwing in query[] array in the same flat structure.  As the JSON array query can have more than one element, my speculation is that your developer intended for each element in top level array of params to represent params to each element of query[]. What if, instead of     {\"name\":\"\", \"connection\":22234743, \"time\":20000, \"success\":false, \"type\":\"Prepared\", \"batch\":false, \"querySize\":1, \"batchSize\":0, \"query\":[\"select * from whatever.whatever w where w.whatever in (?,?,?) \"], \"params\":[[\"1\",\"2\",\"3\"]]}     your raw data contains json.msg of this value?     "{\"name\":\"\", \"connection\":22234743, \"time\":20000, \"success\":false, \"type\":\"Prepared\", \"batch\":false, \"querySize\":2, \"batchSize\":0, \"query\":[\"select * from whatever.whatever w where w.whatever in (?,?,?) \", \"select * from whatever.whatever2 w where w.whatever2 in (?,?) \"], \"params\":[[\"1\",\"2\",\"3\"],[\"4\",\"5\"]]}"     i.e., query[] and params[] each contains two elements? (For convenience, I assume that querySize represents the number of elements in these arrays.  We can live without this external count but why complicate our lives in a tutorial.)  Using the above search, you will find query{} and params{}{} to contain querySize query{} params{}{} 2 select * from whatever.whatever w where w.whatever in (?,?,?) select * from whatever.whatever2 w where w.whatever2 in (?,?) 1 2 3 4 5 This is one of shortcomings of flattening structured data like JSON, not unique to SPL but the shortcoming becomes more obvious.  On top of the flattened structure, the spath command also cannot handle array of arrays correctly.  Now what? Here is what I would use to get past this barrier. (This is not the only way.  But JSON functions introduced in 8.2 works really well while preserving semantic context.)     | spath input=json.msg | eval params_array = json_array_to_mv(json_extract('json.msg', "params")) | eval idx = mvrange(0, querySize) ``` assuming querySize is size of query{} ``` | eval query_params = mvmap(idx, json_object("query", mvindex('query{}', idx), "params", mvindex(params_array, idx))) | fields - json.msg params* query{} idx | mvexpand query_params     With this, the output contains batch batchSize connection name querySize query_params success time type false 0 22234743   2 {"query":"select * from whatever.whatever w where w.whatever in (?,?,?) ","params":"[\"1\",\"2\",\"3\"]"} false 20000 Prepared false 0 22234743   2 {"query":"select * from whatever.whatever2 w where w.whatever2 in (?,?) ","params":"[\"4\",\"5\"]"} false 20000 Prepared I think you know what I am going for by now.  What did we learn here?  To compensate for the unfortunate implied semantics your developer forces on you, first construct an intermediary JSON object that binds each query with each array of params.  Then, use mvexpand to separate the elements. (Admittedly, json_array_to_mv is an oddball function at first glance.  But once you understand how Splunk uses multivalue, you'll get used to the concept.  Hopefully you will find many merits of using a multivalue representation.) From here, you can use spath again to get desired results, but I find JSON functions to be simpler AND more semantic considering there are only two keys in this intermediary JSON.  Add the following to the above     | eval query = json_extract(query_params, "query") | eval params = json_array_to_mv(json_extract(query_params, "params"))     With this, you get the final result batch batchSize connection name params query querySize success time type false 0 22234743   1 2 3 select * from whatever.whatever w where w.whatever in (?,?,?) 2 false 20000 Prepared false 0 22234743   4 5 select * from whatever.whatever2 w where w.whatever2 in (?,?) 2 false 20000 Prepared Hope this is a useful format for your further processing. Below is an emulation of the above 2-query mock data that I adapted from @ITWhisperer's original emulation.  Play with it and compare with real data.     | makeresults | eval _raw="{ \"time\": \"2024-09-19T08:03:02.234663252Z\", \"json\": { \"ts\": \"2024-09-19T15:03:02.234462341+07:00\", \"logger\": \"<anonymized>\", \"level\": \"WARN\", \"class\": \"net.ttddyy.dsproxy.support.SLF4JLogUtils\", \"method\": \"writeLog\", \"file\": \"<anonymized>\", \"line\": 26, \"thread\": \"pool-1-thread-1\", \"arguments\": {}, \"msg\": \"{\\\"name\\\":\\\"\\\", \\\"connection\\\":22234743, \\\"time\\\":20000, \\\"success\\\":false, \\\"type\\\":\\\"Prepared\\\", \\\"batch\\\":false, \\\"querySize\\\":2, \\\"batchSize\\\":0, \\\"query\\\":[\\\"select * from whatever.whatever w where w.whatever in (?,?,?) \\\", \\\"select * from whatever.whatever2 w where w.whatever2 in (?,?) \\\"], \\\"params\\\":[[\\\"1\\\",\\\"2\\\",\\\"3\\\"],[\\\"4\\\",\\\"5\\\"]]}\", \"scope\": \"APP\" }, \"kubernetes\": { \"pod_name\": \"<anonymized>\", \"namespace_name\": \"<anonymized>\", \"labels\": { \"whatever\": \"whatever\" }, \"container_image\": \"<anonymized>\" } }" | spath ``` data emulation ```     Hope this helps.
Hi,   Join is not returning the data with subsearch, I tried many options from other answers but nothing working out. Target is to check how many departments are using latest version of some so... See more...
Hi,   Join is not returning the data with subsearch, I tried many options from other answers but nothing working out. Target is to check how many departments are using latest version of some software compare to all older versions together.    My search query index=abc version!="2.0" | dedup version thumb_print | stats count(thumb_print) as OLD_RUNS by department | join department [search index=abc version="2.0" | dedup version thumb_print | stats count(thumb_print) as NEW_RUNS by department ] | eval total=OLD_RUNS + NEW_RUNS| fillnull value=0 | eval perc=((NEW_RUNS/total)*100) | eval department=substr(department, 1, 50) | eval perc=round(perc, 2) | table department OLD_RUNS NEW_RUNS perc | sort -perc Overall this search over 1 week time period expected to return more than 100k events. 
Is there any step or checklist for me to first step check or tshoot regarding this, I just currious why the logs is stop ingesting to splunk because previously I don’t have any issue using this way. 
I did not write the logs into the file because lack of resource. 
Hi @gcusello  Thankyou for your answer, I did not install any add ons for fortinet.  sure, I have 1 SH and 2 indexer actualy but I only ingest the log to 1 indexer. The others log from another serv... See more...
Hi @gcusello  Thankyou for your answer, I did not install any add ons for fortinet.  sure, I have 1 SH and 2 indexer actualy but I only ingest the log to 1 indexer. The others log from another service are ingest correctly and can be search in SH.