All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Since Splunk 6.x is not more available, the new URL's are: Forwarder Manual: https://docs.splunk.com/Documentation/Forwarder/9.1.2/Forwarder/Installanixuniversalforwarder..  Installation on Ma... See more...
Since Splunk 6.x is not more available, the new URL's are: Forwarder Manual: https://docs.splunk.com/Documentation/Forwarder/9.1.2/Forwarder/Installanixuniversalforwarder..  Installation on MacOS: https://docs.splunk.com/Documentation/Splunk/9.1.2/Installation/InstallonMacOS 
Are there lines where "AP sent to" or "AH sent to" or "MP sent to" exist in events without "---> TRN:" also being presents? Similarly, are there events where "---> TRN:" exists and one of "AP sent t... See more...
Are there lines where "AP sent to" or "AH sent to" or "MP sent to" exist in events without "---> TRN:" also being presents? Similarly, are there events where "---> TRN:" exists and one of "AP sent to" or "AH sent to" or "MP sent to" does not exists? Please can you explain the significance of the dropdown and how it determines which events are counted?
Is this TA hosted somewhere so we could have a better picture of what the complete python code looks like? 
Hi Hassan, This is a generic 401 authentication problem. When you send metrics or valid messages from agent to controller there are several things you have to configure properly below host, port, a... See more...
Hi Hassan, This is a generic 401 authentication problem. When you send metrics or valid messages from agent to controller there are several things you have to configure properly below host, port, account name (default "customer1" if you don't use the controller in a multi-tenant mode), and account key. So basically there are 2 reasons that you need to focus.  First can you please control this step below given, and try again?  To create a secret with a Controller access key: $ kubectl -n appdynamics create secret generic cluster-agent-secret --from-literal=controller-key=<access-key> Thanks Cansel
When I run  echo '<14>1 2024-04-19T12:34:56.789Z myhostname myapp 12345 - [exampleSDID@32473 iut="3" eventSource="application" eventID="1011"] Something happened through echoing.' > /dev/udp/127.0.0... See more...
When I run  echo '<14>1 2024-04-19T12:34:56.789Z myhostname myapp 12345 - [exampleSDID@32473 iut="3" eventSource="application" eventID="1011"] Something happened through echoing.' > /dev/udp/127.0.0.1/514 I am able to see it in Splunk. But when my application is sending syslog on port 514, it does not appear on Splunk although the same message is visible when I run TCP dump on port 514.  What would I be missing here? To reply to your question, I believe I have followed the steps in runtime configuration (https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/getting-started-runtime-configuration/)
@richgalloway  I agree to your point. I tried using case statement as well . Unfortunately its not working as expected. Do you know any other way to handle this ? That really helps me. I am also re-s... See more...
@richgalloway  I agree to your point. I tried using case statement as well . Unfortunately its not working as expected. Do you know any other way to handle this ? That really helps me. I am also re-searching.
You recognize that this is a Splunk forum where volunteers offer help related to Splunk, right?  As I said, Splunk does not "color" search results. (The only coloring function in Splunk is provided i... See more...
You recognize that this is a Splunk forum where volunteers offer help related to Splunk, right?  As I said, Splunk does not "color" search results. (The only coloring function in Splunk is provided in dashboard visualization.)  If you want to color text, you will need to develop something external to Splunk.  As you suggested, you can possibly achieve this by modifying sendmail.py (not recommended).  Alternatively, you can develop a custom command for this.  Either way, this is not the right forum.
From your the logs it shows: Splunk HEC connection test successful to index=main for sourcetype=sc4s:events (So run this and check the main index - if you can see this then your the connection is wo... See more...
From your the logs it shows: Splunk HEC connection test successful to index=main for sourcetype=sc4s:events (So run this and check the main index - if you can see this then your the connection is working. In terms of the /opt/sc4s/local/context/splunk_index.csv follow all the steps from the below run time configuration, there are a number of stpes and you need to complete them all.  https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/getting-started-runtime-configuration/ As you can send curl test events to cloud you don't need whitelist (BUT its best practise to have them in place for security reasons.) 
That's great news @Ryan.Paredez, thx for keeping me updated! I'm looking forward to the 24.4 release. Cheers, Jerg
mvzip, mvexpand and mvindex are simply wrong tools for your data structure. (Well, mvexpand will be needed, but only after you properly handle the array in your data.)  As everybody in this post has ... See more...
mvzip, mvexpand and mvindex are simply wrong tools for your data structure. (Well, mvexpand will be needed, but only after you properly handle the array in your data.)  As everybody in this post has pointed out: You need to post sample or precise mock data to reveal the structure. (In text, never screenshot.) This is extremely important when asking question about data analytics in a forum.  When you force volunteers to read your mind, not only will they get FRUSTRATED, but even if they are willing, most of the time their mind reading will be incorrect. This said, based on your code, I kind of picture together a rough structure of your data.  I will use JSON to illustrate.  Something like   { "userActions": [ { "application": "app1", "name": "action1", "targetUrl": "url1", "duration": 1234, "type": "actiontype1", "apdexCategory": "SATISFIED" }, { "application": "app1", "name": "action2", "targetUrl": "url1", "duration": 2345, "type": "actiontype1", "apdexCategory": "DISATISFIED" }, { "application": "app1", "name": "action3", "targetUrl": "url2", "duration": 3456, "type": "actiontype2", "apdexCategory": "FRUSTRATED" } ], "userExperienceScore": "FRUSTRATED", "events": [ {"application": "xxx", "irrelevant": "aaa"}, {"application": "yyy", "irrelevant": "bbb"} ] }     Your event could be in JSON or it could be in XML, but it contains at least two arrays, events[] and userActions[].  Is this correct?  The array events[] is not what frustrates you because its elements and components are no longer needed after initial search.  Your end goal from the above three elements of userActions[] is to pick out   { "application": "app1", "name": "action3", "targetUrl": "url2", "duration": 3456, "type": "actiontype2", "apdexCategory": "FRUSTRATED" }   and display it in this format: _time Application Action Target_URL Duration_in_Mins User_Action_Type useractions_experience_score 2024-04-18 22:45:22 app1 action3 url2 0.06 actiontype2 FRUSTRATED If the above looks close, the first thing you need to do is to forget all about Splunk's flattened fields userActions{}.*; in fact, discard them all.  Use spath to reach elements of this array, then mvexpand over the elements, no funny mvzip business.  After that, everything becomes trivial. Using my speculated data, I can reconstruct your SPL into the following to obtain my illustrated output:     index="xxx" sourcetype="xxx" source=xxx events{}.application="xxx" userExperienceScore=FRUSTRATED | fields - userActions{}.* | spath path=userActions{} | mvexpand userActions{} | spath input=userActions{} | dedup application name targetUrl | search apdexCategory = FRUSTRATED application = * name = * | sort - _time | rename application as Application, name as Action, targetUrl as Target_URL, type as User_Action_Type, apdexCategory as useractions_experience_score | eval Duration_in_Mins = round(duration / 60000, 2) | table _time, Application, Action, Target_URL,Duration_in_Mins,User_Action_Type,useractions_experience_score   Hope this helps. Here is an emulation of my speculated data.  Play with it and compare with real data | makeresults | eval _raw = "{ \"userActions\": [ { \"application\": \"app1\", \"name\": \"action1\", \"targetUrl\": \"url1\", \"duration\": 1234, \"type\": \"actiontype1\", \"apdexCategory\": \"SATISFIED\" }, { \"application\": \"app1\", \"name\": \"action2\", \"targetUrl\": \"url1\", \"duration\": 2345, \"type\": \"actiontype1\", \"apdexCategory\": \"DISATISFIED\" }, { \"application\": \"app1\", \"name\": \"action3\", \"targetUrl\": \"url2\", \"duration\": 3456, \"type\": \"actiontype2\", \"apdexCategory\": \"FRUSTRATED\" } ], \"userExperienceScore\": \"FRUSTRATED\", \"events\": [ {\"application\": \"xxx\", \"irrelevant\": \"aaa\"}, {\"application\": \"yyy\", \"irrelevant\": \"bbb\"} ] }" | spath ``` data speculation for index="xxx" sourcetype="xxx" source=xxx events{}.application="xxx" userExperienceScore=FRUSTRATED ```
Thank you for your kind response @ITWhisperer I have made the correction in the 2nd query: "<===" was referring from a different log event. Updated query:  source=/applications/test/*instance_xyz* ... See more...
Thank you for your kind response @ITWhisperer I have made the correction in the 2nd query: "<===" was referring from a different log event. Updated query:  source=/applications/test/*instance_xyz* ("<--- TRN:" OR "---> TRN:" OR "AP sent to" OR "AH sent to" OR "MP sent to") Refer below inline response to your question: The main issue with your request is that you haven't explained how the events are to be correlated between the two sources and how you would like to count them to give the desired result. Answer: There are basically 2 log files. "testget.log" using search criteria as "<--- TRN:" and Priority field information. "testput.log" using search criteria as "---> TRN:" OR "AP sent to" OR "AH sent to" OR "MP sent to" I need help to co-relate these 2 logs based on TRN. And final count I need to get it using TRN and TestMQ. Select: Low, Medium, High (From the Dashboard dropdown) Output Expected: TestMQ| Low-Testget | Low-Testput | Low-AP | Low-AH | Low-MP | Low-Pending TestMQ | Medium-Testget | Medium-Testput | Medium-AP | Medium-AH | Medium-MP | Medium-Pending TestMQ | High-Testget | High-Testput | High-AP | High-AH | High-MP | High-Pending Please suggest.
Thank you, @Cansel.OZCAN  this information helped me alot.
Have you tried my previous code? | eval route = if(match(request_path, "^/orders/\d+"), "/order/{orderID}", null()) This does exactly what you ask: create a new field named route that has a fixed p... See more...
Have you tried my previous code? | eval route = if(match(request_path, "^/orders/\d+"), "/order/{orderID}", null()) This does exactly what you ask: create a new field named route that has a fixed pattern "/order/{orderID}".  Is there anything wrong with this? In fact, because you really only care about first segment of the path - that fixed string "{orderID}" is just a decoration, the command could be simplified to slightly less expensive | eval route = "/" . mvindex(split(request_path, "/"), 1) . "/{orderID}" You can do whatever analysis against this field.
WHen I have uncommented the line SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no. I get the below logs when I run the command journalctl -b -u sc4s Apr 18 13:53:40 ip-MachineIP systemd[1]: Starting SC4... See more...
WHen I have uncommented the line SC4S_DEST_SPLUNK_HEC_DEFAULT_TLS_VERIFY=no. I get the below logs when I run the command journalctl -b -u sc4s Apr 18 13:53:40 ip-MachineIP systemd[1]: Starting SC4S Container... Apr 18 13:53:41 ip-MachineIP docker[12242]: latest: Pulling from splunk/splunk-connect-for-syslog/container3 Apr 18 13:53:41 ip-MachineIP docker[12242]: Digest: sha256:f8ff916d9cb6836cb0b03b578f51a3777c7a4c84e580fdad9b768cdc7ef2910e Apr 18 13:53:41 ip-MachineIP docker[12242]: Status: Image is up to date for ghcr.io/splunk/splunk-connect-for-syslog/container3:latest Apr 18 13:53:41 ip-MachineIP docker[12242]: ghcr.io/splunk/splunk-connect-for-syslog/container3:latest Apr 18 13:53:41 ip-MachineIP systemd[1]: Started SC4S Container. Apr 18 13:53:42 ip-MachineIP docker[12254]: SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=main for sourcetype=sc4s:fallback... Apr 18 13:53:43 ip-MachineIP docker[12254]: SC4S_ENV_CHECK_HEC: Splunk HEC connection test successful to index=main for sourcetype=sc4s:events... Apr 18 13:53:47 ip-MachineIP docker[12254]: syslog-ng checking config Apr 18 13:53:47 ip-MachineIP docker[12254]: sc4s version=3.22.3 Apr 18 13:53:48 ip-MachineIP docker[12254]: starting goss Apr 18 13:53:50 ip-MachineIP docker[12254]: starting syslog-ng I have created all the indexes mentioned in the document (https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/getting-started-splunk-setup/) I cannot find the file /opt/sc4s/local/context/splunk_index.csv I am able to curl and send message to splunk using -k flag in my curl command. Do I need to whitelist if I am able to curl?
Run this command to see if you have poor data ingestion balance across the indexers | tstats count where index=* by index splunk_server | stats sum(count) as total dc(splunk_server) as dc_splunk_ser... See more...
Run this command to see if you have poor data ingestion balance across the indexers | tstats count where index=* by index splunk_server | stats sum(count) as total dc(splunk_server) as dc_splunk_server by index  The dc_splunk_server field will show you how many indexers contain the data for a particular index. If you sort by count, check if the largest data counts are across all indexers. You can also go a bit deeper to check the min/max/avg data count per indexer/index and see if the min or max are outside 3*stdev from average. Also checks if the data is not across all indexers. | tstats count where index=* by index splunk_server | stats avg(count) as avg_count min(count) as min_count max(count) as max_count stdev(count) as stdev_count dc(splunk_server) as dc_splunk_server by index | eventstats max(dc_splunk_server) as total_splunk_servers | where dc_splunk_server < total_splunk_servers OR (min_count < (avg_count - 3*stdev_count)) OR (max_count > (avg_count + 3*stdev_count))  
Hi @inventsekar  Thank you for answer! 1) I don't see any warnings in MC. 2) I see only 1 indexer's bucket count is about 50,000. 9 indexer's count is about 140,000 ~150,000. And each bucket size ... See more...
Hi @inventsekar  Thank you for answer! 1) I don't see any warnings in MC. 2) I see only 1 indexer's bucket count is about 50,000. 9 indexer's count is about 140,000 ~150,000. And each bucket size in 1 indexer is three times bigger than other indexers. So I checked bucket in terminal, i found that tsidx file's sizes are large. 3) Every indexer's conf is same. This trouble continues a few months. Is there anything else to check?
Hi @Bisho-Fouad .. on the DMC / license master.. you can find out the license usage of a specific host.  pls suggest us exactly which step/status you are in..    As you are asking GUI.. the SPL gi... See more...
Hi @Bisho-Fouad .. on the DMC / license master.. you can find out the license usage of a specific host.  pls suggest us exactly which step/status you are in..    As you are asking GUI.. the SPL gives more control actually. 
Hi @dongwonn  Maybe more details pls..  1) on Monitoring Console, do you see any errors / warnings 2) on the indexer clustering, do you see the buckets imbalance? 3) may we know how you say -- on... See more...
Hi @dongwonn  Maybe more details pls..  1) on Monitoring Console, do you see any errors / warnings 2) on the indexer clustering, do you see the buckets imbalance? 3) may we know how you say -- only 1 indexer out of 10 is overused.  4) any recent changes to the indexer cluster, .. any upgrades/migrations, any new apps deployed.. etc.. 
HI, I'm working in splunk team. Environment: 3 SH 10 IDX (1 of 10 IDX overused) Replication factor 3 Search factor 3   Could it happen that searches are continuously done only on certain indexe... See more...
HI, I'm working in splunk team. Environment: 3 SH 10 IDX (1 of 10 IDX overused) Replication factor 3 Search factor 3   Could it happen that searches are continuously done only on certain indexer? I've been constantly monitoring them with top and ps -ef, and I'm seeing a lot of search operations on certain indexer. The cpu usage is roughly double... It's been going on for months. Can it be considered normal?
Here is another page that pretty much shows you how to do this https://docs.splunk.com/Documentation/Splunk/9.2.1/Viz/Buildandeditforms