All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

The data architecture is kinda self-contradicting here indeed. Restricting access to "source" data and giving free access to summarized data is asking for trouble. You could try to limit your users w... See more...
The data architecture is kinda self-contradicting here indeed. Restricting access to "source" data and giving free access to summarized data is asking for trouble. You could try to limit your users with search filters but that's not a very reliable mechanism. If you cannot create multiple summary indexes, you can try with simply storing the summaries back in the original source indexes. After all summary indexing is just a method of preparing summarized data, there is no "special type" of index called summary index. It doesn't have to be a separate index either. The data would be clearly distinguishable from the "original" data by sourcetype. But you'd probably be violating one of the basic rules of index architecture - don't mix data of different frequency/cardinality characteristics. In your case, since you need summary indexing, you probably have lots of "base" data and would have just s fairly limited set of summarized data points generated every now and then. So it's possible but it's not the best practice.
1. Don't use the "table" command for other things than displaying your results in tabular format at the end of your search.  Most importantly, don't use it early in your search! It moves the processi... See more...
1. Don't use the "table" command for other things than displaying your results in tabular format at the end of your search.  Most importantly, don't use it early in your search! It moves the processing to the search-head tier. In your case it prevents Splunk from effectively running stats in a distributed way across indexers. 2. Well, we don't know your data. You've shown some single example with no additional explanation as to the format of your events. The general idea would be to extract some form of a "transactionID" and use streamstats co copy over the start of the transaction to the ending event (or probably the other way around since Splunk by default returns events in reverse chronological order. Then gather some general aggregations from thus preprocessed data.
I am trying to find the time taken by our processes. I wrote a basic query that fetch a start, end time, and the difference for a particular interaction. This uses the max and min to find the start a... See more...
I am trying to find the time taken by our processes. I wrote a basic query that fetch a start, end time, and the difference for a particular interaction. This uses the max and min to find the start and the end times. But I am not sure how to look for multiple process start and end times by looking at the messages.     index=application_na sourcetype=my_logs:hec appl="*" message="***" interactionid=12345 | table interactionid, seq, _time, host, severity, message, msgsource | sort _time | stats min(_time) as StartTime, max(_time) as EndTime by interactionid | eval Difference=EndTime-StartTime | fieldformat StartTime=strftime(StartTime, "%Y-%m-%d %H:%M:%S.%3N") | fieldformat EndTime=strftime(EndTime, "%Y-%m-%d %H:%M:%S.%3N") | fieldformat Difference=tostring(Difference,"duration") | table interactionid, StartTime, EndTime, Difference   I have messages that look like this: interactionid _time message 12345 2025-06-26 07:55:56.317 TimeMarker: WebService: Received request. (DoPayment - ID:1721 Amount:16 Acc:1234) 12345 2025-06-26 07:55:56.717 OtherApp: -> Sending request with timeout value: 15 12345 2025-06-26 07:55:57.512 TimeMarker: OtherApp: Received result from OtherApp (SALE - ID:1721 Amount:16.00 Acc:1234) 12345 2025-06-26 07:55:58.017 TimeMarker: WebService: Sending result @20234ms. (DoPayment - ID:1721 Amount:16 Acc:1234) So, I want to get an output of time taken by `OtherApp` from when it received a request to when it responded back to my app, and then the total time taken by my service `DoPayment`. Is this something achievable. Output that I am looking for is  interactionid DoPayment Start OtherApp Start OtherApp End DoPayment End          
@livehybrid's advise is good.  I would add that the idea of copying protected data into an unprotected index is unsound.
Splunk enterprise and it is clustered environment. We are already using RBAC to restrict users with their indexes. But this shared summary index is giving headache. By giving access to that, they ca... See more...
Splunk enterprise and it is clustered environment. We are already using RBAC to restrict users with their indexes. But this shared summary index is giving headache. By giving access to that, they can view other app's summary data as well which is not secure
I have a dotnet application logging template formatted log messages with serilog library and since everything is in JSON format they are great to filter my results when I know the fields to use but I... See more...
I have a dotnet application logging template formatted log messages with serilog library and since everything is in JSON format they are great to filter my results when I know the fields to use but I am having a hard time just to read logs when I dont know the fields available. So for example, the application might log things like: Log.Information("Just got a request {request} in endpoint {endpoint} with {httpMethod}", request,endpoint, httpMethod); And in Splunk I will see something like: { "msg": { "@mt": "Just got a request {request} in endpoint {endpoint} with {httpMethod}", "@sp": "11111", "request": "some_data", "endpoint": "some_url". "httpMethod": "POST" } } So this is awesome to create splunk queries using msg.request or msg.endpoint, but since the application logs pretty much everything using these message templates from serilog, when I am just doing investigations, I have a hard time in making readable results because everythig is hidden behind a placeholder. I am trying to achieve something like in Splunk Search: <some_guid> index=some_index | table _time msg.@mt and of course the msg.@mt will just give me the log line with the placeholders, but how can I just bring back the full log line in the table with the actual values?
Hi @Karthikeya  Just a heads up - there are many other ways of triggering a search inside Splunk, so disabling the open in search might not give you as much protection as you think.  Ultimately I t... See more...
Hi @Karthikeya  Just a heads up - there are many other ways of triggering a search inside Splunk, so disabling the open in search might not give you as much protection as you think.  Ultimately I think the main focus here should be on clear segregation of the data. I would focus initially on splitting you raw data into different, well named, indexes based on the following: 1) Retention periods - Data should be split based on different data retention requirements, as retention is set at a per-index basis. 2) Access requirements - in your case you have multiple stakeholders accessing multiple app system data in your indexes. You should split the indexes appropriately such as app_A, app_B etc and then use RBAC to give specific roles access to the appropriate indexes. 3) Technology - We have source/sourcetype for segregation but you can take this a little further and split data in indexes based on technology (e.g. dont put network data in an index with app data...) I wont go into detail on here how to rename an index or move data around as this depends on your architecture etc.  Are you using Splunk Enterprise on-premise? What does your deployment look like?  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Hi @Alex_S  splunklib shouldnt be an issue in TAs - I use this all the time. How are you testing/executing your script? Ive only ever seen this issue when running using the system python rather than... See more...
Hi @Alex_S  splunklib shouldnt be an issue in TAs - I use this all the time. How are you testing/executing your script? Ive only ever seen this issue when running using the system python rather than Splunks python build. I'd highly recommend checking out https://conf.splunk.com/files/2024/slides/DEV1885B.pdf and getting to know UCC which makes it very easy to start building modular inputs. The boilerplate code is pretty much ready to go, just stick in your requests and off you go!  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
@thahir @PickleRick  I am testing something with  /api/search/jobs/$SID$/control?action=save it is working on and off , sometime it work and sometime not.
I've developed TA's previously, and when using python2, everything worked just fine. But now, using python3 with splunk 9.x, it seems nothing works. Trying to develop a TA that makes some REST calls ... See more...
I've developed TA's previously, and when using python2, everything worked just fine. But now, using python3 with splunk 9.x, it seems nothing works. Trying to develop a TA that makes some REST calls out to a 3rd-party service, and then uses those values in some local confs. It's been a nightmare to try to make this work. Started with a modular input design, but contrary to the docs, my python code would never receive a splunk token on STDIN. Literally had this working perfectly in a python2 TA. This time? Doesn't matter how or when attempting to read STDIN, the python3 code *NEVER RECEIVES ANYTHING*. Finally I just gave up on this... Next try was with a scripted input; at least this **bleep** thing does receive a token on STDIN. Great, that token can be used w/ the SDK, right?  RIGHT??? Well, no, because 1) splunklib is not installed/included in the splunk python env, and 2) attempting to use the system python causes the whole **bleep** thing to crash, and 3) including splunklib inside the TA, and attempting to import it by manipulating python paths is also horribly broken. If we munge the python system paths thusly, we can in theory import our included libs (not concerned if this is idiomatic python; it works m'kay?): import os, sys modules = sys.argv[0].split('/')[:-2] modules.append('lib') sys.path.append('/'.join(modules)) This inserts our local lib path into python's lib search dirs. And it works to find splunklib. But then splunklib fails to load completely since: ImportError: libssl.so.1.0.0: cannot open shared object file: No such file or directory This is true even if LD_LIBRARY_PATH points to a dir containing libssl.so.1.0.0. I suspect this is due to the fact that Splunk is also doing an LD_PRELOAD="libdlwrapper.so" I don't know what this library is or what it's doing, but I also suspect it's breaking my env preventing anything from running. But it doesn't actually matter. If I remove my "import splunklib" and just leave the REST client to attempt to make its HTTPS request, that too is apparently horribly broken: ...(Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available")) What in the everloving fsck is going on here??!? Best I can tell, these two things are now true: 1) splunklib cannot be used from a TA 2) TA's cannot make HTTPS requests   This is happening in a clean-room environment with a fresh splunk install on a host that is not running selinux or apparmor or any other MAC system that might interfere. This is very much a problem with Splunk and splunklib.  So, how exactly can splunklib be used in TAs? And how exactly can TAs execute HTTPS requests??    
One more thing I'd check would be to call the btool with a user and app context. It seems a bit illogical to treat the command differently per this context additionally to giving a user capability to... See more...
One more thing I'd check would be to call the btool with a user and app context. It seems a bit illogical to treat the command differently per this context additionally to giving a user capability to run such command but it's worth checking. If it still shows that the setting should effectively be false, it might be worth creating a support case.
Hey, I never heard back from anyone since posting the output of btool.  Any suggestions why this setting is not working as documented in the Splunk documentation?
Before one week I created a summary index named waf_opco_yes_summary and it is working fine. Now they asked to change the index name as opco_yes_summary and already existing summary index should be c... See more...
Before one week I created a summary index named waf_opco_yes_summary and it is working fine. Now they asked to change the index name as opco_yes_summary and already existing summary index should be come to this index and that index shouldn't be visible anywhere either in dashboards or searches. That should be deleted and all its data should be moved to new index. What can I do here?  One more problem is we created a single summary index to all applications and afraid of giving access to them because any of them see that there can see other's apps summary data, it will be a security issue right. We have created a dashboard with summary index and disabled open in search. At some point, we need to give them access to summary index and what if they search index=* then their restricted index and this summary index shows up which can be risky. Is there any way we can restrict users running index=*. NOTE - already we are using RBAC to restrict users to their specific indexes. But this summary index will show summarised data of all. Any way to restrict this? Can't create summary index for each application. However in dashboard we are restricting them by a field should be selected then only panel with summary index shows up by filtering. How people handle this type of situations?
Hi @Eshwar  I know you mentioned the firewall is disabled, but I wanted to check - is there any corporate firewall/proxy in place (which is often transparent) between your host and the internet? Ive... See more...
Hi @Eshwar  I know you mentioned the firewall is disabled, but I wanted to check - is there any corporate firewall/proxy in place (which is often transparent) between your host and the internet? Ive seen this error countless times with firewalls blocking the traffic, even when no firewall is on the host itself.  Another issue could possibly be ciphers - What OS are you running? Is it relatively modern and up to date?   Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
Try something like this (essentially, you need to calculate each "top 5" and eliminate the stats events for each level, before calculating the next "top 5" for the next level). action=load OR action... See more...
Try something like this (essentially, you need to calculate each "top 5" and eliminate the stats events for each level, before calculating the next "top 5" for the next level). action=load OR action=Download customer!="" publicationId="*" topic="*" | eval Month=strftime(_time, "%b-%y") | stats count by customer, Month, product, publicationId, topic | eventstats sum(count) as product_count by customer Month product | sort 0 customer Month -product_count | streamstats dc(product) as product_rank by customer, Month | where product_rank <= 5 | eventstats sum(count) as publicationId_count by customer Month product publicationId | sort 0 customer Month product -publicationId_count | streamstats dc(publicationId) as publicationId_rank by customer Month product | where publicationId_rank <= 5 | eventstats sum(count) as topic_count by customer Month product publicationId topic | sort 0 customer Month product publicationId -topic_count | streamstats dc(topic) as topic_rank by customer Month product publicationId | where topic_rank <= 5 | table customer, product, publicationId, topic, count, Month
Excellent Point.   Sadly, I knew that....   Must have been a brain cramp  
1. The host=* condition is completely unnecessary. It doesn't narrow your search and every event must have the host field. It's a purely aesthetic remark but bloating the search makes it less readabl... See more...
1. The host=* condition is completely unnecessary. It doesn't narrow your search and every event must have the host field. It's a purely aesthetic remark but bloating the search makes it less readable. 2. The dedup command works differently than I suppose you think it does. After "dedup host" you will be left with just one event containing data for the first service returned by the initial search. All subsequent services for this host will be discarded. I don't think it's what you want.
actually additional step is just to rolling restart after the config changes. and it will rebalanced
Hello, @LoMueller, and thanks, @livehybrid. My contact information is in the TA's README and default/app.conf. I am leaving for a trip this week, but I will be back on August 18. I can add a config... See more...
Hello, @LoMueller, and thanks, @livehybrid. My contact information is in the TA's README and default/app.conf. I am leaving for a trip this week, but I will be back on August 18. I can add a configuration file to set up a proxy for Python. Please send me a reminder after the 18th, @LoMueller, and I will try to get that done for you.
Yes, this works index="server" host="*"  source="Unix:Service"  UNIT=iptables.service  | eval IPTABLES = if(ACTIVE="failed" OR ACTIVE="inactive", "failed", "OK")  | dedup host  | table host I... See more...
Yes, this works index="server" host="*"  source="Unix:Service"  UNIT=iptables.service  | eval IPTABLES = if(ACTIVE="failed" OR ACTIVE="inactive", "failed", "OK")  | dedup host  | table host IPTABLES