All Posts

Top

All Posts

You recognize that this is a Splunk forum where volunteers offer help related to Splunk, right?  As I said, Splunk does not "color" search results. (The only coloring function in Splunk is provided i... See more...
You recognize that this is a Splunk forum where volunteers offer help related to Splunk, right?  As I said, Splunk does not "color" search results. (The only coloring function in Splunk is provided in dashboard visualization.)  If you want to color text, you will need to develop something external to Splunk.  As you suggested, you can possibly achieve this by modifying sendmail.py (not recommended).  Alternatively, you can develop a custom command for this.  Either way, this is not the right forum.
From your the logs it shows: Splunk HEC connection test successful to index=main for sourcetype=sc4s:events (So run this and check the main index - if you can see this then your the connection is wo... See more...
From your the logs it shows: Splunk HEC connection test successful to index=main for sourcetype=sc4s:events (So run this and check the main index - if you can see this then your the connection is working. In terms of the /opt/sc4s/local/context/splunk_index.csv follow all the steps from the below run time configuration, there are a number of stpes and you need to complete them all.  https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/getting-started-runtime-configuration/ As you can send curl test events to cloud you don't need whitelist (BUT its best practise to have them in place for security reasons.) 
That's great news @Ryan.Paredez, thx for keeping me updated! I'm looking forward to the 24.4 release. Cheers, Jerg
mvzip, mvexpand and mvindex are simply wrong tools for your data structure. (Well, mvexpand will be needed, but only after you properly handle the array in your data.)  As everybody in this post has ... See more...
mvzip, mvexpand and mvindex are simply wrong tools for your data structure. (Well, mvexpand will be needed, but only after you properly handle the array in your data.)  As everybody in this post has pointed out: You need to post sample or precise mock data to reveal the structure. (In text, never screenshot.) This is extremely important when asking question about data analytics in a forum.  When you force volunteers to read your mind, not only will they get FRUSTRATED, but even if they are willing, most of the time their mind reading will be incorrect. This said, based on your code, I kind of picture together a rough structure of your data.  I will use JSON to illustrate.  Something like   { "userActions": [ { "application": "app1", "name": "action1", "targetUrl": "url1", "duration": 1234, "type": "actiontype1", "apdexCategory": "SATISFIED" }, { "application": "app1", "name": "action2", "targetUrl": "url1", "duration": 2345, "type": "actiontype1", "apdexCategory": "DISATISFIED" }, { "application": "app1", "name": "action3", "targetUrl": "url2", "duration": 3456, "type": "actiontype2", "apdexCategory": "FRUSTRATED" } ], "userExperienceScore": "FRUSTRATED", "events": [ {"application": "xxx", "irrelevant": "aaa"}, {"application": "yyy", "irrelevant": "bbb"} ] }     Your event could be in JSON or it could be in XML, but it contains at least two arrays, events[] and userActions[].  Is this correct?  The array events[] is not what frustrates you because its elements and components are no longer needed after initial search.  Your end goal from the above three elements of userActions[] is to pick out   { "application": "app1", "name": "action3", "targetUrl": "url2", "duration": 3456, "type": "actiontype2", "apdexCategory": "FRUSTRATED" }   and display it in this format: _time Application Action Target_URL Duration_in_Mins User_Action_Type useractions_experience_score 2024-04-18 22:45:22 app1 action3 url2 0.06 actiontype2 FRUSTRATED If the above looks close, the first thing you need to do is to forget all about Splunk's flattened fields userActions{}.*; in fact, discard them all.  Use spath to reach elements of this array, then mvexpand over the elements, no funny mvzip business.  After that, everything becomes trivial. Using my speculated data, I can reconstruct your SPL into the following to obtain my illustrated output:     index="xxx" sourcetype="xxx" source=xxx events{}.application="xxx" userExperienceScore=FRUSTRATED | fields - userActions{}.* | spath path=userActions{} | mvexpand userActions{} | spath input=userActions{} | dedup application name targetUrl | search apdexCategory = FRUSTRATED application = * name = * | sort - _time | rename application as Application, name as Action, targetUrl as Target_URL, type as User_Action_Type, apdexCategory as useractions_experience_score | eval Duration_in_Mins = round(duration / 60000, 2) | table _time, Application, Action, Target_URL,Duration_in_Mins,User_Action_Type,useractions_experience_score   Hope this helps. Here is an emulation of my speculated data.  Play with it and compare with real data | makeresults | eval _raw = "{ \"userActions\": [ { \"application\": \"app1\", \"name\": \"action1\", \"targetUrl\": \"url1\", \"duration\": 1234, \"type\": \"actiontype1\", \"apdexCategory\": \"SATISFIED\" }, { \"application\": \"app1\", \"name\": \"action2\", \"targetUrl\": \"url1\", \"duration\": 2345, \"type\": \"actiontype1\", \"apdexCategory\": \"DISATISFIED\" }, { \"application\": \"app1\", \"name\": \"action3\", \"targetUrl\": \"url2\", \"duration\": 3456, \"type\": \"actiontype2\", \"apdexCategory\": \"FRUSTRATED\" } ], \"userExperienceScore\": \"FRUSTRATED\", \"events\": [ {\"application\": \"xxx\", \"irrelevant\": \"aaa\"}, {\"application\": \"yyy\", \"irrelevant\": \"bbb\"} ] }" | spath ``` data speculation for index="xxx" sourcetype="xxx" source=xxx events{}.application="xxx" userExperienceScore=FRUSTRATED ```
Thank you for your kind response @ITWhisperer I have made the correction in the 2nd query: "<===" was referring from a different log event. Updated query:  source=/applications/test/*instance_xyz* ... See more...
Thank you for your kind response @ITWhisperer I have made the correction in the 2nd query: "<===" was referring from a different log event. Updated query:  source=/applications/test/*instance_xyz* ("<--- TRN:" OR "---> TRN:" OR "AP sent to" OR "AH sent to" OR "MP sent to") Refer below inline response to your question: The main issue with your request is that you haven't explained how the events are to be correlated between the two sources and how you would like to count them to give the desired result. Answer: There are basically 2 log files. "testget.log" using search criteria as "<--- TRN:" and Priority field information. "testput.log" using search criteria as "---> TRN:" OR "AP sent to" OR "AH sent to" OR "MP sent to" I need help to co-relate these 2 logs based on TRN. And final count I need to get it using TRN and TestMQ. Select: Low, Medium, High (From the Dashboard dropdown) Output Expected: TestMQ| Low-Testget | Low-Testput | Low-AP | Low-AH | Low-MP | Low-Pending TestMQ | Medium-Testget | Medium-Testput | Medium-AP | Medium-AH | Medium-MP | Medium-Pending TestMQ | High-Testget | High-Testput | High-AP | High-AH | High-MP | High-Pending Please suggest.
Thank you, @Cansel.OZCAN  this information helped me alot.
Have you tried my previous code? | eval route = if(match(request_path, "^/orders/\d+"), "/order/{orderID}", null()) This does exactly what you ask: create a new field named route that has a fixed p... See more...
Have you tried my previous code? | eval route = if(match(request_path, "^/orders/\d+"), "/order/{orderID}", null()) This does exactly what you ask: create a new field named route that has a fixed pattern "/order/{orderID}".  Is there anything wrong with this? In fact, because you really only care about first segment of the path - that fixed string "{orderID}" is just a decoration, the command could be simplified to slightly less expensive | eval route = "/" . mvindex(split(request_path, "/"), 1) . "/{orderID}" You can do whatever analysis against this field.
WHen I have uncommented the line SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no. I get the below logs when I run the command journalctl -b -u sc4s Apr 18 13:53:40 ip-MachineIP systemd[1]: Starting SC4... See more...
WHen I have uncommented the line SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no. I get the below logs when I run the command journalctl -b -u sc4s Apr 18 13:53:40 ip-MachineIP systemd[1]: Starting SC4S Container... Apr 18 13:53:41 ip-MachineIP docker[12242]: latest: Pulling from splunk/splunk-connect-for-syslog/container3 Apr 18 13:53:41 ip-MachineIP docker[12242]: Digest: sha256:f8ff916d9cb6836cb0b03b578f51a3777c7a4c84e580fdad9b768cdc7ef2910e Apr 18 13:53:41 ip-MachineIP docker[12242]: Status: Image is up to date for ghcr.io/splunk/splunk-connect-for-syslog/container3:latest Apr 18 13:53:41 ip-MachineIP docker[12242]: ghcr.io/splunk/splunk-connect-for-syslog/container3:latest Apr 18 13:53:41 ip-MachineIP systemd[1]: Started SC4S Container. Apr 18 13:53:42 ip-MachineIP docker[12254]: SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=main for sourcetype=sc4s:fallback... Apr 18 13:53:43 ip-MachineIP docker[12254]: SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=main for sourcetype=sc4s:events... Apr 18 13:53:47 ip-MachineIP docker[12254]: syslog-ng checking config Apr 18 13:53:47 ip-MachineIP docker[12254]: sc4s version=3.22.3 Apr 18 13:53:48 ip-MachineIP docker[12254]: starting goss Apr 18 13:53:50 ip-MachineIP docker[12254]: starting syslog-ng I have created all the indexes mentioned in the document (https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/getting-started-splunk-setup/) I cannot find the file /opt/sc4s/local/context/splunk_index.csv I am able to curl and send message to splunk using -k flag in my curl command. Do I need to whitelist if I am able to curl?
Run this command to see if you have poor data ingestion balance across the indexers | tstats count where index=* by index splunk_server | stats sum(count) as total dc(splunk_server) as dc_splunk_ser... See more...
Run this command to see if you have poor data ingestion balance across the indexers | tstats count where index=* by index splunk_server | stats sum(count) as total dc(splunk_server) as dc_splunk_server by index  The dc_splunk_server field will show you how many indexers contain the data for a particular index. If you sort by count, check if the largest data counts are across all indexers. You can also go a bit deeper to check the min/max/avg data count per indexer/index and see if the min or max are outside 3*stdev from average. Also checks if the data is not across all indexers. | tstats count where index=* by index splunk_server | stats avg(count) as avg_count min(count) as min_count max(count) as max_count stdev(count) as stdev_count dc(splunk_server) as dc_splunk_server by index | eventstats max(dc_splunk_server) as total_splunk_servers | where dc_splunk_server < total_splunk_servers OR (min_count < (avg_count - 3*stdev_count)) OR (max_count > (avg_count + 3*stdev_count))  
Hi @inventsekar  Thank you for answer! 1) I don't see any warnings in MC. 2) I see only 1 indexer's bucket count is about 50,000. 9 indexer's count is about 140,000 ~150,000. And each bucket size ... See more...
Hi @inventsekar  Thank you for answer! 1) I don't see any warnings in MC. 2) I see only 1 indexer's bucket count is about 50,000. 9 indexer's count is about 140,000 ~150,000. And each bucket size in 1 indexer is three times bigger than other indexers. So I checked bucket in terminal, i found that tsidx file's sizes are large. 3) Every indexer's conf is same. This trouble continues a few months. Is there anything else to check?
Hi @Bisho-Fouad .. on the DMC / license master.. you can find out the license usage of a specific host.  pls suggest us exactly which step/status you are in..    As you are asking GUI.. the SPL gi... See more...
Hi @Bisho-Fouad .. on the DMC / license master.. you can find out the license usage of a specific host.  pls suggest us exactly which step/status you are in..    As you are asking GUI.. the SPL gives more control actually. 
Hi @dongwonn  Maybe more details pls..  1) on Monitoring Console, do you see any errors / warnings 2) on the indexer clustering, do you see the buckets imbalance? 3) may we know how you say -- on... See more...
Hi @dongwonn  Maybe more details pls..  1) on Monitoring Console, do you see any errors / warnings 2) on the indexer clustering, do you see the buckets imbalance? 3) may we know how you say -- only 1 indexer out of 10 is overused.  4) any recent changes to the indexer cluster, .. any upgrades/migrations, any new apps deployed.. etc.. 
HI, I'm working in splunk team. Environment: 3 SH 10 IDX (1 of 10 IDX overused) Replication factor 3 Search factor 3   Could it happen that searches are continuously done only on certain indexe... See more...
HI, I'm working in splunk team. Environment: 3 SH 10 IDX (1 of 10 IDX overused) Replication factor 3 Search factor 3   Could it happen that searches are continuously done only on certain indexer? I've been constantly monitoring them with top and ps -ef, and I'm seeing a lot of search operations on certain indexer. The cpu usage is roughly double... It's been going on for months. Can it be considered normal?
Here is another page that pretty much shows you how to do this https://docs.splunk.com/Documentation/Splunk/9.2.1/Viz/Buildandeditforms  
I can help - I asked a question about whether you had already added the dropdown field. Have you done so? What have you tried before - it's pretty straightforward to add a dropdown input and add val... See more...
I can help - I asked a question about whether you had already added the dropdown field. Have you done so? What have you tried before - it's pretty straightforward to add a dropdown input and add values to the dashboard - you don't need to write XML The XML reference manual is here https://docs.splunk.com/Documentation/Splunk/latest/Viz/PanelreferenceforSimplifiedXML This is a really good app you can install to a Splunk environment that shows many techniques to create powerful dashboards https://splunkbase.splunk.com/app/1603  
Judging from the SPL, where you have two searches for FRUSTRATED, it seems like you have data where multiple userExperienceScores can exist for the same event, hence all the mvexpanding out. As @Pic... See more...
Judging from the SPL, where you have two searches for FRUSTRATED, it seems like you have data where multiple userExperienceScores can exist for the same event, hence all the mvexpanding out. As @PickleRick points out, it's quite tricky to deal with multivalue fields, particularly when you have 6 MV fields that you are zipping up into 2 pairs (x and y). I assume you are using 2 pairs as there is not a 1:1 correlation between the MV's in each of the pairs. What I would suggest is to find a reasonably complex SINGLE (or two) event where you can exhibit the problem. This will make it much easier to diagnose the issue. Then we can help explain what is going on. If you are able to share an example of the raw data (sanitised and preferably NOT a screen image - so we can produce a working example of a solution) that would be good.  
Solution in mycase: Since i was using ArgoCD for deployment, it was overwriting new changes Appd Cluster Agent as part of sync, hence the agents were getting terminating. Also i had to include below ... See more...
Solution in mycase: Since i was using ArgoCD for deployment, it was overwriting new changes Appd Cluster Agent as part of sync, hence the agents were getting terminating. Also i had to include below in my instrumentation rules as my container was running as non-root for appd to work.. runAsUser: 9001 runAsGroup: 9001  
I tried this a few days ago with a 404 - I see the main site is up .. but did the tool get taken down? 
We don't know your raw data but the main question is why do you go to all this trouble of mvzipping and joining all those values into a multivalued field when next you want to do is mvexpand. Why no... See more...
We don't know your raw data but the main question is why do you go to all this trouble of mvzipping and joining all those values into a multivalued field when next you want to do is mvexpand. Why not just filter on raw data in the initial search?
@ITWhisperer Thank you so much, it really saved my time.