All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Hi @Karthikeya  Is the sample data you provided after you have modified it with any SPL, or is that as it arrives into Splunk? It looks like its already a JSON string when it arrives, if so then js... See more...
Hi @Karthikeya  Is the sample data you provided after you have modified it with any SPL, or is that as it arrives into Splunk? It looks like its already a JSON string when it arrives, if so then json functions should work. I will test this further.
 Hi @splunklearner  Try using the following REGEX  \"compression\":\"([^\"]*)\"\, Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.... See more...
 Hi @splunklearner  Try using the following REGEX  \"compression\":\"([^\"]*)\"\, Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped. Regards Will
Thanks for responding. Time range is exactly same.  We ended up opening a support case for this. The cause was found to be duplicate events in index=notable for a particular correlation search . What... See more...
Thanks for responding. Time range is exactly same.  We ended up opening a support case for this. The cause was found to be duplicate events in index=notable for a particular correlation search . What is causing these duplicates is under investigation.
Please help me in extracting only compression values from this raw event -  "response_time_last_byte":5,"compression_percentage":0,"compression":"NO_COMPRESSION_CAN_BE_COMPRESSED","client_insights":... See more...
Please help me in extracting only compression values from this raw event -  "response_time_last_byte":5,"compression_percentage":0,"compression":"NO_COMPRESSION_CAN_BE_COMPRESSED","client_insights":"","request_headers":577,"response_headers":13,"request_state":"AVI_HTTP_REQUEST_STATE_SEND_RESPONSE_BODY_TO_CLIENT", "response_time_last_byte":1,"compression_percentage":0,"compression":"","client_insights":"","request_headers":3,"response_headers":12,"request_state":"AVI_HTTP_REQUEST_STATE_READ_CLIENT_REQ_HDR", Tried this but it is extracting client insights as well. I need to exclude all compression string values by writing SEDCMD   
@kiran_panchavat Thank you for those steps and suggestions: I tried those steps and below are the details: Can you check this on the heavy forwarder?  netstat -tulnp | grep 9997 OR ss -tulnp |... See more...
@kiran_panchavat Thank you for those steps and suggestions: I tried those steps and below are the details: Can you check this on the heavy forwarder?  netstat -tulnp | grep 9997 OR ss -tulnp | grep 9997 Ran the above command in on my HF: 1. First it said: grep: invalid option -- 't' Usage: grep [OPTION]... PATTERN [FILE]... Try 'grep --help' for more information. 2. Then when I tried to only grep for 9997 (netstat -tulnp | grep 9997) I did not see any output. Check the metrics.log if any queues are getting blocked. tail -f /opt/splunk/var/log/splunk/metrics.log | grep -i "blocked=true" Verify that outputs.conf the HF is correctly configured. Ensure there are no typos in the IP addresses or port numbers. I verified that both my indexers IPs mentioned in the HF's outputs.conf file are correct. Can you please confirm if the below stanza name tcpout is correct or if there is any typo with it?  [tcpout:errorGroup] server = indexr_1_ip_addr:9997 [tcpout:successGroup] server = indexer_2_ip_addr:9997 File permission issues could be a possible reason why Splunk HF is not reading test.log. If the Heavy Forwarder (HF) process does not have the required permissions to read the file, it won't be able to forward logs to the indexers. On the HF the file /opt/splunk/var/log/splunk/test.log I changed the user and group ec2-user: -rw-r--r-- 1 ec2-user ec2-user 1133 Feb 19 00:53 test.log I restarted the HF and checked my indexers for logs/events from the HF under main index. But, no luck 
We have a requirement to remove few strings from the events while indexing the data. Here is my raw event sample -    {"adf":true,"significant":0,"udf":false,"virtualservice":"virtualservice-fe4a30... See more...
We have a requirement to remove few strings from the events while indexing the data. Here is my raw event sample -    {"adf":true,"significant":0,"udf":false,"virtualservice":"virtualservice-fe4a30d8-ce53-4427-b920-ec81381cb1f4","report_timestamp":"2025-02-19T06:31:56.065370Z","service_engine":"GB-DRN-AB-Tier2-se-vxeuz","vcpu_id":0,"log_id":20138,"client_ip":"128.12.73.92","client_src_port":39688,"client_dest_port":443,"client_rtt":1,"http_version":"1.1","method":"HEAD","uri_path":"/path/to/monitor/page/","host":"udg1704n01.hc.cloud.uk.sony","response_content_type":"text/html","request_length":93,"response_length":94,"response_code":400,"response_time_first_byte":1,"response_time_last_byte":1,"compression_percentage":0,"compression":"","client_insights":"","request_headers":3,"response_headers":12,"request_state":"AVI_HTTP_REQUEST_STATE_READ_CLIENT_REQ_HDR","significant_log":["ADF_HTTP_BAD_REQUEST_PLAIN_HTTP_REQUEST_SENT_ON_HTTPS_PORT","ADF_RESPONSE_CODE_4XX"],"vs_ip":"128.160.71.14","request_id":"jjc-HmSo-8zb3","max_ingress_latency_fe":0,"avg_ingress_latency_fe":0,"conn_est_time_fe":0,"source_ip":"128.12.73.92","vs_name":"v-atcptest-wdc.hc.cloud.uk.sony-443","tenant_name":"admin"} I need to remove strings like avg_ingress_latency_fe, conn_est_time_fe, client_insights etc. I gone through the google and found giving SEDCMD will help me. Hence giving this in props.conf and giving this in my cluster manager and it is working well. SEDCMD-removeavglatency=s/\"avg_ingress_latency_fe\"\:[\d+]\,//g SEDCMD-removeclientinsights=s/\"client_insights\"\:\"\.*"\,//g But my problem we need to give more lines like this which will not be in readable format in future. I want to keep it in less lines. Tried this but not working and in return this is disturbing the Json format-  == props.conf == [yourSourceType] TRANSFORMS-removeJsonKeys = removeJsonKeys1 == transforms.conf == [removeJsonKeys1] INGEST_EVAL = _raw=json_delete(_raw, "avg_ingress_latency_be", "avg_ingress_latency_fe", "max_ingress_latency_fe", "client_insights" ) because already we removed few lines from this event by giving in props.conf for auto extraction of json fields -  SEDCMD-removeheader=s/^[^\{]*//g   and here is SH props.conf -    [mysourcetype] KV_MODE = json AUTO_KV_JSON = true   Please suggest what can I do now instead of this to keep props.conf neat?  
How can I export the host values in excel for the particular serverclass  Is there is any query for that that will be helpful . Path will be  Deployment server -> forwarder management ->serv... See more...
How can I export the host values in excel for the particular serverclass  Is there is any query for that that will be helpful . Path will be  Deployment server -> forwarder management ->serverclass -> action (edit clients) -> need to export the hostname from the list 
@vksplunk1  The KV store isn’t very reliable, so it's best to back it up regularly. 1. Some apps store their lookups in the kvstore. (collections.conf) 2. Some apps store all their configu... See more...
@vksplunk1  The KV store isn’t very reliable, so it's best to back it up regularly. 1. Some apps store their lookups in the kvstore. (collections.conf) 2. Some apps store all their configuration in the kvstore (ITSI, but they also do daily backups. For Splunk itself 1. It sometimes uses the kvstore to track which summary indexing time range was done. It's wise to back up your KV store regularly since it's vulnerable to data loss. If it gets corrupted, deleted, or runs into issues during an upgrade or restart, you could lose valuable data. Keeping backups helps you recover your data quickly if anything goes wrong. https://community.splunk.com/t5/Knowledge-Management/Is-there-any-way-to-retrieve-kv-store-that-was-accidentally/m-p/408788 
@spy_jrCheck this community link for more details:- https://community.splunk.com/t5/Alerting/Error-Code-3/m-p/689100/highlight/true
@spy_jr  This usually happens when there are 0 results from the preceding search. If the results are more than 0 then you'll not see this error. So its safe to ignore this.
Get rid of that dedup host.  You will see some events with error_msg, some without.  I cannot decipher what that dedup is supposed to accomplish, or what real problem you are trying to solve.  So, I ... See more...
Get rid of that dedup host.  You will see some events with error_msg, some without.  I cannot decipher what that dedup is supposed to accomplish, or what real problem you are trying to solve.  So, I cannot suggest an alternative.  But if you have that dedup and if for each host the last event is NOT a failure or disconnect, you will get no error_msg.  Maybe you mean this? index=kafka-np sourcetype="KCON" connName="CCNGBU_*" ERROR=ERROR OR ERROR=WARN | eval error_msg = case(match(_raw, "Disconnected"), "disconected", match(_raw, "restart failed"), "restart failed", match(_raw, "Failed to start connector"), "failed to start connector") | search error_msg = * | dedup host | table host connName error_msg  
As many here will tell you, map is probably the wrong answer to the problem you are trying to solve.  That aside, you need to clarify what "not working", "problem ... with 'map' command" mean, how do... See more...
As many here will tell you, map is probably the wrong answer to the problem you are trying to solve.  That aside, you need to clarify what "not working", "problem ... with 'map' command" mean, how does "things" grey out, etc. (In which window, for example, do "things" grey out?) It is atypical to use a makeresults subsearch to produce search terms inside a map command.  But I think I get that return search should be interpreted verbatim by the compiler.   My wild guess is that the search inside the map command does not return any result.  Is this the case?  It seems that the map command can be simplified to | map maxsearches=100 search="search $time_token$ $index_token$ $rule$ | eval rule_found=\"$rule$\", rule_id=\"$id$\"" If this is semantically correct, try turning substitute select values of time_token, index_token and rule from earlier search, and perform the mapped search manually for diagnosis.  Hope this helps. (Still, there might be more elegant and less error-prone method than using map command to solve the problem you are attacking.)
@shashank9  Can you check this on the heavy forwarder?  netstat -tulnp | grep 9997 OR ss -tulnp | grep 9997 Check the metrics.log if any queues are getting blocked. tail -f /opt/splunk/var/lo... See more...
@shashank9  Can you check this on the heavy forwarder?  netstat -tulnp | grep 9997 OR ss -tulnp | grep 9997 Check the metrics.log if any queues are getting blocked. tail -f /opt/splunk/var/log/splunk/metrics.log | grep -i "blocked=true" Verify that outputs.conf the HF is correctly configured. Ensure there are no typos in the IP addresses or port numbers. File permission issues could be a possible reason why Splunk HF is not reading test.log. If the Heavy Forwarder (HF) process does not have the required permissions to read the file, it won't be able to forward logs to the indexers.
Hi Everyone, I've installed and configured a Splunk Heavy Forwarder on an EC2 instance in AWS and configured two Splunk Indexers on EC2 instances in AWS. I created a test.log file on my HF with samp... See more...
Hi Everyone, I've installed and configured a Splunk Heavy Forwarder on an EC2 instance in AWS and configured two Splunk Indexers on EC2 instances in AWS. I created a test.log file on my HF with sample log events to forward them to my Splunk indexers. I'm trying to forward the logs/events with keyword "success" to indexer_1 and forward logs/events with keyword "error" to indexer_2. But, for some reason the logs/events from the HF are not visible in both Indexers. Just for the context, I have installed and configured a UF on another EC2 Instance in AWS and sending data to Indexer_1 and I can see the data successfully forwarded with no issues. Below are the .conf files and setup on my HF and two indexers. HF: inputs.conf: [monitor:///opt/splunk/var/log/splunk/test.log] disabled = false sourcetype = test outputs.conf: [tcpout:errorGroup] server = indexr_1_ip_addr:9997 [tcpout:successGroup] server = indexer_2_ip_addr:9997 props.conf: [test] TRANSFORMS-routing=errorRouting,successRouting transforms.conf: [errorRouting] REGEX=error DEST_KEY=_TCP_ROUTING FORMAT=errorGroup [successRouting] REGEX=success DEST_KEY=_TCP_ROUTING FORMAT=successGroup Indexer_1 & Indexer_2: Configured the port 9997 on both indexers. Note: I tried below steps to troubleshoot or identify the issue, but no luck so far: 1. Checked if the forwarder has any inactive forwards or receivers through CLI: Active forwards: indexr_1_ip_addr:9997 indexr_2_ip_addr:9997 Configured but inactive forwards: None 2. Check the splunkd.log on the forwarder to see if there are any errors related to data forwarding: No errors 3. Checked the Security Group rules (Inbound and Outbound) in AWS console: Port 9997 is enabled for both Inbound and Outbound traffic. 4. All EC2 Instances running Splunk are on the same Security Group in AWS. 5. Tried to Ping both Indexers from HF. But, no response. Can someone please help me with this issue as I'm stuck and unable to figure out what is the root cause of the issue. Also, I'm using the same security group for both HF and UF with same Inbound and Outbound rules, but I can only see the logs sent from UF and not seeing the logs/events from my HF. I'm not sure what I am missing here to resolve or fix the issue to see the logs/events from HF in my Indexers. Thank you!
Hi,   I am trying to update IT Essentials Work (ITEW) from v4.13.0 to v4.15.0. There is no much documentation on ITEW so I am using the documentation for IT Service Intelligence (ITSI). My understa... See more...
Hi,   I am trying to update IT Essentials Work (ITEW) from v4.13.0 to v4.15.0. There is no much documentation on ITEW so I am using the documentation for IT Service Intelligence (ITSI). My understanding is ITEW is the free version of ITSI without premium features. I checked the prerequisites, updated as per the documentation  1. Stopped the service (it is a single instance - SH) 2. Extracted the new version into $SPLUNK_HOME/etc/apps 3. Started the service Then opened the app on the search head to proceed with the update, it passed the pre checks, got to    2025-02-19 14:30:56,637+1100 process:654449 thread:MainThread INFO [itsi.migration] [itsi_migration_log:43] [info] UI: Running prechecker: EAPrechecks   I left it for 30 minutes or so, then checked the status by running    curl -k -u admin:changeme -X GET https://localhost:8089/servicesNS/nobody/SA-ITOA/migration/info   and it was is_running: false Cannot see anything alarming when I check the status. Tried several times and every time it is the same. Checked the permissions, Troubleshooting documentation, restarted the service - still could not update. Please, advise
Anyone please help in this 
I just stumbled on this looking for something else,  and wanted to say you can dynamically colour both the background and the major value:    "visualizations": { "viz_UVeH0JP5": { ... See more...
I just stumbled on this looking for something else,  and wanted to say you can dynamically colour both the background and the major value:    "visualizations": { "viz_UVeH0JP5": { "type": "splunk.singlevalue", "dataSources": { "primary": "ds_VyZ1EWbM" }, "options": { "majorColor": "> majorValue | matchValue(majorColorEditorConfig)", "backgroundColor": "> majorValue | matchValue(backgroungColorEditorConfig)" }, "context": { "majorColorEditorConfig": [ { "match": "NotDropped", "value": "#2f8811" }, { "match": "Dropped", "value": "#ffffff" } ], "backgroundColorEditorConfig": [ { "match": "NotDropped", "value": "#000000" }, { "match": "Dropped", "value": "#2f8811" } ] } },   You probably can't do it though the UI, though. I rarely use it, so I'm not sure. 
Has anyone been able to use the "| sendalert risk ..." command from the correlation search query, even when the search returns no results? I currently need to do this, but when there are no result... See more...
Has anyone been able to use the "| sendalert risk ..." command from the correlation search query, even when the search returns no results? I currently need to do this, but when there are no results I get the message "Error in 'sendalert' command: Alert script returned error code 3." Is there a way to truncate (abort) the sendalert command when there are no results?
| stats max(notificationId) by iNumber
I cannot download from the results pane as well. Here Export is greyed out. I am the admin but i cannot download as csv.