All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

Hi Frank, I just found a quick-fix with inline-code enhancements in file but appreciate it using a configfile as mentioned. /opt/splunk/etc/apps/TA_oui-lookup/bin/get-oui-table.py proxy_host = 'lo... See more...
Hi Frank, I just found a quick-fix with inline-code enhancements in file but appreciate it using a configfile as mentioned. /opt/splunk/etc/apps/TA_oui-lookup/bin/get-oui-table.py proxy_host = 'localhost:1234'    # host and port of your proxy OUI_URL = "https://standards-oui.ieee.org" req = urllib.request.Request(OUI_URL) req.set_proxy(proxy_host, 'http') req.add_header("User-agent", USER_AGENT)  Enjoy your vacation! Lothar from Germany
@cdevoe57  Try below, index="server" source="Unix:Service" UNIT IN ("iptables.service", "auditd.service", "chronyd.service") | eval status=if(ACTIVE=="failed" OR ACTIVE=="inactive", "failed", "OK")... See more...
@cdevoe57  Try below, index="server" source="Unix:Service" UNIT IN ("iptables.service", "auditd.service", "chronyd.service") | eval status=if(ACTIVE=="failed" OR ACTIVE=="inactive", "failed", "OK") | eval service=case( UNIT=="iptables.service", "IPTABLES", UNIT=="auditd.service", "AUDITD", UNIT=="chronyd.service", "CHRONYD" ) | stats values(status) as status by host service | xyseries host service status | where IPTABLES="failed" OR AUDITD="failed" OR CHRONYD="failed" | table host IPTABLES AUDITD CHRONYD Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!
@Karthikeya  Change the index name Since its summary index, i would suggest to use collect command to copy your data. 1 - Create new index called opco_yes_summary 2 -Search and use collect to cop... See more...
@Karthikeya  Change the index name Since its summary index, i would suggest to use collect command to copy your data. 1 - Create new index called opco_yes_summary 2 -Search and use collect to copy index=waf_opco_yes_summary | collect index=opco_yes_summary 3 - Verify the data index=opco_yes_summary 4 - Once verified, Delete old index.   Restricting users based on the apps from summary index I would say creating separate indexes per app might be a nightmare. As a workaround, Can we consider creating tagging field for summary events if there is no specific field. For eg: field app_name Then create a role based filter. Eg: [role_appA] srchFilter = index=opco_yes_summary app_name="AppA" This ideally ensures users only see data for their app, even if they run index=* But need to avoid “All non-internal indexes” in Roles.   Anyway this needs to be tested and verified, but it might be a good starting point. Regards, Prewin Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!
@PickleRick so what will be the best approach to do this? Creating new summary index for every application? We have nearly 100 applications on-boaarded and it will be bit painful to write same query ... See more...
@PickleRick so what will be the best approach to do this? Creating new summary index for every application? We have nearly 100 applications on-boaarded and it will be bit painful to write same query for everything and deploy? Any automation can we do? But still I have zero knowledge on coding.
Agree with others that your purpose is better served by knowing your data.  Given what you have revealed, you can simply describe the four events and lay them out with xyseries. index=application_na... See more...
Agree with others that your purpose is better served by knowing your data.  Given what you have revealed, you can simply describe the four events and lay them out with xyseries. index=application_na sourcetype=my_logs:hec appl="*" message="***" | eval event = case(match(message, "Received request"), "DoPayment start", match(message, "Sending result"), "DoPayment end", match(message, "Sending request"), "OtherApp start", match(message, "Received result"), "OtherApp end") | eval _time = strftime(_time, "%F %T.%3N") | xyseries interactionid event _time Obviously regex's used in the match functions are just to illustrate what you can do.  But xyseries can achieve what you want without complex transformations.  Using your mock data, the output is interactionid DoPayment end DoPaymet start OtherApp end OtherApp start 12345 2025-06-26 07:55:58.017 2025-06-26 07:55:56.317 2025-06-26 07:55:57.512 2025-06-26 07:55:56.717 Here is an emulation you can play with and compare with real data | makeresults format=csv data="interactionid,_time,message 12345,2025-06-26 07:55:56.317,TimeMarker: WebService: Received request. (DoPayment - ID:1721 Amount:16 Acc:1234) 12345,2025-06-26 07:55:56.717,OtherApp: -> Sending request with timeout value: 15 12345,2025-06-26 07:55:57.512,TimeMarker: OtherApp: Received result from OtherApp (SALE - ID:1721 Amount:16.00 Acc:1234) 12345,2025-06-26 07:55:58.017,TimeMarker: WebService: Sending result @20234ms. (DoPayment - ID:1721 Amount:16 Acc:1234)" | eval _time = strptime(_time, "%F %T.%N") | sort - _time ``` above emulates index=application_na sourcetype=my_logs:hec appl="*" message="***" ```  
Hello folks, We are doing splunkforwarder upgrade to 9.4.x (from 8.x) recently, we build the splunk sidecar image for our k8s application and i noticed the same procedures which works previous in fw... See more...
Hello folks, We are doing splunkforwarder upgrade to 9.4.x (from 8.x) recently, we build the splunk sidecar image for our k8s application and i noticed the same procedures which works previous in fwd version 8.x don't work anymore in 9.4.x. during the docker image startup, it's clearly to see the process hanging there and wait for interaction. bash-4.4$ ps -ef UID PID PPID C STIME TTY TIME CMD splunkf+ 1 0 0 02:11 ? 00:00:00 /bin/bash /entrypoint.sh splunkf+ 59 1 99 02:11 ? 00:01:25 /opt/splunkforwarder/bin/splunk edit user admin -password XXXXXXXX -role admin -auth admin:xxxxxx --answer-yes --accept-license --no-prompt splunkf+ 61 0 0 02:12 pts/0 00:00:00 /bin/bash splunkf+ 68 61 0 02:12 pts/0 00:00:00 ps -ef bash-4.4$ rpm -qa | grep splunkforwarder splunkforwarder-9.4.3-237ebbd22314.x86_64   there is a workaround to add a "tty: true" to k8s deployment template but this will cause a lot of efforts in our environment.   Any idea if any newer version has the fix? or any splunk command parameter can be used to bypass the tty requirement? Thanks.
This syntax is wrong and will never work | eval IPTABLES = if(UNIT=iptables.service AND (ACTIVE="failed" OR ACTIVE="inactive"), "failed", "OK") UNIT is a string, so must be quoted as you have done... See more...
This syntax is wrong and will never work | eval IPTABLES = if(UNIT=iptables.service AND (ACTIVE="failed" OR ACTIVE="inactive"), "failed", "OK") UNIT is a string, so must be quoted as you have done for the ACTIVE field. | eval IPTABLES = if(UNIT="iptables.service" AND (ACTIVE="failed" OR ACTIVE="inactive"), "failed", "OK") You probably want to use  | fields _time host IPTABLES AUDITD CHRONYD | stats latest(*) as * by host to get you the most recent state  
From those 4 events, which ones do you want to calculate time between, it's not clear to me. If you have multiple messages and only two of them are relevant to your calculation, then can you not just... See more...
From those 4 events, which ones do you want to calculate time between, it's not clear to me. If you have multiple messages and only two of them are relevant to your calculation, then can you not just include search contstraints to only find the 2 you are interested in? If you have only 2 events, then you can use min/max as you are doing. Otherwise, you can use this type of logic | sort 0 _time | streamstats window=2 global=f range(_time) as duration by interactionid which will sort the events into time ascending order and put a new field into each event with the duration (time gap) between that event and the previous event for the same interactionid. You could also use eval and stats (which would be faster than streamstats) to set a field with the start time of the event you want to find - same for the end and then use stats to collect those new fields to then calculate duration. Also, note that you should never sort unless you know you need to. In this case, you don't. Also, sort has a 10,000 result limit and will chop your data to only 10,000 results (maybe not an issue in your case), but get used to using | sort 0 xxx to make sure your entire data set is sorted.    
Following on from @PickleRick suggestion, to avoid the @mt issue, you could do something like this | spath msg.@mt output=mt | rex field=mt max_match=0 "{(?<templates>[^}]+)}" | foreach mode=multiva... See more...
Following on from @PickleRick suggestion, to avoid the @mt issue, you could do something like this | spath msg.@mt output=mt | rex field=mt max_match=0 "{(?<templates>[^}]+)}" | foreach mode=multivalue templates [ | eval mt=replace(mt,"{".<<ITEM>>."}", json_extract(_raw,"msg.".<<ITEM>>)) ]
There are a lot variables in the mix, but this worked for me when I needed to do a quick POC from a syslog-based source.   input { syslog { port => 601 syslog_field => "syslog" } } ... See more...
There are a lot variables in the mix, but this worked for me when I needed to do a quick POC from a syslog-based source.   input { syslog { port => 601 syslog_field => "syslog" } } filter { #Add time for Auto time extraction in Epoch for Splunk HEC ruby { code => "event.set('time', event.get('@timestamp').to_i)" } } output { # stdout { codec => rubydebug } # file { # codec => json_lines # path => "E:\logstash-output\ZPA-records.json" # } http { format => "json" content_type => "application/json" url => "https://http-inputs.XXX.splunkcloud.com/services/collector" http_method => "post" headers => ['Authorization', 'Splunk ${splunk_hec_token}'] mapping => { time => "%{time}" host => "ZPA-HEC" source => "xxx" event => "%{message}" } } }  
Ugh. That is one ugly cookie. On the one hand it's indeed very convenient to search if you know what to look for. It's the opposite of parsing the fields out of your logs - you already have them so ... See more...
Ugh. That is one ugly cookie. On the one hand it's indeed very convenient to search if you know what to look for. It's the opposite of parsing the fields out of your logs - you already have them so you're saving yourself a lot of work and don't have to create extractions for various message formats. But on the other hand - I don't recall any function which gives you this kind of formatting. You could try using foreach in multivalue mode (after extracting the placeholders from the template). Since your original data is json you can try using json_extract to find the value for replacing your placeholders. In a general case you can't just get a value of a field by its name stored in another field. But since your data is a json, you might be able to pull off something like (just a general idea, might have typos and such | rex field='@mt' max_match=0 "{(?<templates>[^{]+)}" | foreach mode=multivalue templates [ | eval '@mt'=replace('@mt',"{".<<ITEM>>."}", json_extract(_raw,"msg.".<<ITEM>>) ] I'm not even sure you can have a field named @MT - you might need to rename it.
The data architecture is kinda self-contradicting here indeed. Restricting access to "source" data and giving free access to summarized data is asking for trouble. You could try to limit your users w... See more...
The data architecture is kinda self-contradicting here indeed. Restricting access to "source" data and giving free access to summarized data is asking for trouble. You could try to limit your users with search filters but that's not a very reliable mechanism. If you cannot create multiple summary indexes, you can try with simply storing the summaries back in the original source indexes. After all summary indexing is just a method of preparing summarized data, there is no "special type" of index called summary index. It doesn't have to be a separate index either. The data would be clearly distinguishable from the "original" data by sourcetype. But you'd probably be violating one of the basic rules of index architecture - don't mix data of different frequency/cardinality characteristics. In your case, since you need summary indexing, you probably have lots of "base" data and would have just s fairly limited set of summarized data points generated every now and then. So it's possible but it's not the best practice.
1. Don't use the "table" command for other things than displaying your results in tabular format at the end of your search.  Most importantly, don't use it early in your search! It moves the processi... See more...
1. Don't use the "table" command for other things than displaying your results in tabular format at the end of your search.  Most importantly, don't use it early in your search! It moves the processing to the search-head tier. In your case it prevents Splunk from effectively running stats in a distributed way across indexers. 2. Well, we don't know your data. You've shown some single example with no additional explanation as to the format of your events. The general idea would be to extract some form of a "transactionID" and use streamstats co copy over the start of the transaction to the ending event (or probably the other way around since Splunk by default returns events in reverse chronological order. Then gather some general aggregations from thus preprocessed data.
I am trying to find the time taken by our processes. I wrote a basic query that fetch a start, end time, and the difference for a particular interaction. This uses the max and min to find the start a... See more...
I am trying to find the time taken by our processes. I wrote a basic query that fetch a start, end time, and the difference for a particular interaction. This uses the max and min to find the start and the end times. But I am not sure how to look for multiple process start and end times by looking at the messages.     index=application_na sourcetype=my_logs:hec appl="*" message="***" interactionid=12345 | table interactionid, seq, _time, host, severity, message, msgsource | sort _time | stats min(_time) as StartTime, max(_time) as EndTime by interactionid | eval Difference=EndTime-StartTime | fieldformat StartTime=strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3N") | fieldformat EndTime=strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3N") | fieldformat Difference=tostring(Difference,"duration") | table interactionid, StartTime, EndTime, Difference   I have messages that look like this: interactionid _time message 12345 2025-06-26 07:55:56.317 TimeMarker: WebService: Received request. (DoPayment - ID:1721 Amount:16 Acc:1234) 12345 2025-06-26 07:55:56.717 OtherApp: -> Sending request with timeout value: 15 12345 2025-06-26 07:55:57.512 TimeMarker: OtherApp: Received result from OtherApp (SALE - ID:1721 Amount:16.00 Acc:1234) 12345 2025-06-26 07:55:58.017 TimeMarker: WebService: Sending result @20234ms. (DoPayment - ID:1721 Amount:16 Acc:1234) So, I want to get an output of time taken by `OtherApp` from when it received a request to when it responded back to my app, and then the total time taken by my service `DoPayment`. Is this something achievable. Output that I am looking for is  interactionid DoPayment Start OtherApp Start OtherApp End DoPayment End          
@livehybrid's advise is good.  I would add that the idea of copying protected data into an unprotected index is unsound.
Splunk enterprise and it is clustered environment. We are already using RBAC to restrict users with their indexes. But this shared summary index is giving headache. By giving access to that, they ca... See more...
Splunk enterprise and it is clustered environment. We are already using RBAC to restrict users with their indexes. But this shared summary index is giving headache. By giving access to that, they can view other app's summary data as well which is not secure
I have a dotnet application logging template formatted log messages with serilog library and since everything is in JSON format they are great to filter my results when I know the fields to use but I... See more...
I have a dotnet application logging template formatted log messages with serilog library and since everything is in JSON format they are great to filter my results when I know the fields to use but I am having a hard time just to read logs when I dont know the fields available. So for example, the application might log things like: Log.Information("Just got a request {request} in endpoint {endpoint} with {httpMethod}", request,endpoint, httpMethod); And in Splunk I will see something like: { "msg": { "@mt": "Just got a request {request} in endpoint {endpoint} with {httpMethod}", "@sp": "11111", "request": "some_data", "endpoint": "some_url". "httpMethod": "POST" } } So this is awesome to create splunk queries using msg.request or msg.endpoint, but since the application logs pretty much everything using these message templates from serilog, when I am just doing investigations, I have a hard time in making readable results because everythig is hidden behind a placeholder. I am trying to achieve something like in Splunk Search: <some_guid> index=some_index | table _time msg.@mt and of course the msg.@mt will just give me the log line with the placeholders, but how can I just bring back the full log line in the table with the actual values?
Hi @Karthikeya  Just a heads up - there are many other ways of triggering a search inside Splunk, so disabling the open in search might not give you as much protection as you think.  Ultimately I t... See more...
Hi @Karthikeya  Just a heads up - there are many other ways of triggering a search inside Splunk, so disabling the open in search might not give you as much protection as you think.  Ultimately I think the main focus here should be on clear segregation of the data. I would focus initially on splitting you raw data into different, well named, indexes based on the following: 1) Retention periods - Data should be split based on different data retention requirements, as retention is set at a per-index basis. 2) Access requirements - in your case you have multiple stakeholders accessing multiple app system data in your indexes. You should split the indexes appropriately such as app_A, app_B etc and then use RBAC to give specific roles access to the appropriate indexes. 3) Technology - We have source/sourcetype for segregation but you can take this a little further and split data in indexes based on technology (e.g. dont put network data in an index with app data...) I wont go into detail on here how to rename an index or move data around as this depends on your architecture etc.  Are you using Splunk Enterprise on-premise? What does your deployment look like?  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @Alex_S  splunklib shouldnt be an issue in TAs - I use this all the time. How are you testing/executing your script? Ive only ever seen this issue when running using the system python rather than... See more...
Hi @Alex_S  splunklib shouldnt be an issue in TAs - I use this all the time. How are you testing/executing your script? Ive only ever seen this issue when running using the system python rather than Splunks python build. I'd highly recommend checking out https://conf.splunk.com/files/2024/slides/DEV1885B.pdf and getting to know UCC which makes it very easy to start building modular inputs. The boilerplate code is pretty much ready to go, just stick in your requests and off you go!  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
@thahir @PickleRick  I am testing something with  /api/search/jobs/$SID$/control?action=save it is working on and off , sometime it work and sometime not.