I'd also assume this is splunkd returning a 500 while the TA tries to access configuration files. If so, you should see the 500 pop up in splunkd_access.log too, but that only confirms it's splunkd and won't help you find out what causes it. I'd always check the TA release notes for any changes between the version (and for compatibility with your Splunk version which seems good). Rigt off the bat, the release notes for 2.1.0 list something around a REST API, so it's likely you've either found a bug in a recently introduced feature, or don't meet some prerequisite. I would also always recommend checking out any resources mentioned in the TA description on Splunkbase, which in this case point to documentation which has two dedicated bits for REST calls: https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/troubleshoot.html#add-on-logs-for-first-rest-call-attempts and https://ta-jira-service-desk-simple-addon.readthedocs.io/en/latest/troubleshoot.html#add-on-logs-for-the-internal-rest-api-endpoints. Maybe those logs help you narrow it down further - if not, I'd reach out to the developer listed in the support tab of the TA on Splunkbase. ... View more

For anyone interested, there is an idea for this (current status is "Future Prospect"): https://ideas.splunk.com/ideas/EID-I-486.  ... View more

As I said earlier, I don't think this is possible. You could create an idea on ideas.splunk.com to have a "readonly-admin" role, but I am not sure if this is a very common request that will get many votes/attention. ... View more

This has been asked on the splunk-usergroups slack before. At that time (May 2022) the answer by Lizzy Li was "adding something in the UI to inspect the job is on the roadmap, but unfortunately not available at the exact moment". I'm not sure about the current status but the release notes do not mention anything so I'd assume it's not yet possible. ... View more

There is more than one way to do it. Since you mentioned csv already, let's use that. I'm going to assume it has two columns, uuid and client_id. You need your csv in Splunk for this of course - so either upload it or create it in Splunk if you haven't done so already. Next, I'd suggest you create a lookup definition for your csv file. Then, you'd use your lookup like this: index=this <your search finding UUIDs> | lookup your_lookup_definition uuid OUTPUT client_id | table _time client_id src_ip method (or whatever other fields you want) If this is something you always want for this type of data, you might want to consider an automatic lookup. That would make splunk implicitly run that | lookup command for all searches against e.g. this sourcetype, so you wouldn't need to have it in every SPL explicitly. ... View more

Well, have you read your error message? It says "No such file or directory", which indicates the tar file is either not in the same directory as you are in or the permissions might be off. You can verify that running ls -al to show the files in your current working directory and their permissions. This is not a Splunk issue. ... View more

It should not be a permission issue against the endpoint. You'll be able to run this search as any user: | rest splunk_server=local services/authentication/users But it'll only return what your roles permit you to see, i.e. only your own user or all users. ... View more

I don't think you can do this with permissions alone, as e.g. a call to services/authentication/users with | rest is also limited to your permissions and a "readonly" capability for users exists to my knowledge. Depending on your use case, you could collect the users in a summary index or a lookup and have your role search that instead (or better yet, as rich mentioned while I was typing, use a report running as owner!) ... View more

That's not a valid cron expression. For "every month 1st Saturday at 00:05 AM", I don't think that's possible with a single cron. You could schedule your search to run daily for a week, using this cron: 5 0 1-7 * * And then in your search you check what day of the week it is and only return results on saturday, like this: your existing search | eval day_of_week = strftime(now(), "%a") | where day_of_week = "Sat" It's not as pretty as what you brought, but it should result in your desired behavior. ... View more

Unfortunately, the links at the end of this post point to "depublished" conf talks, as Splunk for some reason does not keep old talks searchable from the .conf pages. There is an idea to fix this (https://ideas.splunk.com/ideas/PORTALSID-I-141) - feel free to upvote if you'd like, and @sloshburch maybe you can you link to the pdf directly instead? ... View more

The easy solution to this is the option "allowCustomValues" on a multiselect. With this, you can input custom values that can include a wildcard. <input type="multiselect" token="multi_tok"> <label>multi with custom values</label> <fieldForLabel>sourcetype</fieldForLabel> <fieldForValue>sourcetype</fieldForValue> <search> <query>| tstats count where index=_internal by sourcetype</query> <earliest>-60m@m</earliest> <latest>now</latest> </search> <allowCustomValues>true</allowCustomValues> </input>   ... View more

I see there are existing answers that handle the logic in a search in SPL. For the question asked, I would prefer to handle the logic on the dashboard. Here's how I'd do it:   <fieldset submitButton="false"> <input type="text" token="raw_tok"> <label>Search for something</label> <change> <condition match="match(value, &quot;^\\*$&quot;)"> <unset token="target"></unset> </condition> <condition value="*"> <set token="target">$value$</set> </condition> </change> </input> </fieldset> <row> <panel rejects="$target$"> <html> <p>Please use wildcards only after at least specifying part of a value, such as "something*"</p> </html> </panel> <panel depends="$target$"> <table> <search> <query>| makeresults | eval foo = "something_123" | search foo="$target$"</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> <option name="drilldown">none</option> </table> </panel> </row> In addition to keeping the logic on the dashboard, this will only create a search job when the condition is met, otherwise it would wait for the unset token to be filled. Using the dashboard eval logic also allows to e.g. check for a minimum input length or other conditions. More details in docs for eval and match. Obvious reminder that this is not a security feature, only a UI limitation on this dashboard - your users can of course still open a working search and change the SPL to search for "*", or not filter at all. ... View more

You should post more details about what "stopped working". Do you see any error messages on the console? Does your code error on loading, or on specific calls? Were you using any jquery methods that have been deprecated in 3.5? ... View more

Thank you for this in-depth write up. I observe our indexers at values slightly below and sometimes above 5k, and we still have that setting on the default, auto, which limits.conf.spec explains as: When set to 'auto', the Splunk platform will tune this setting based on the physical RAM present in the server at startup. Our indexers have enough ram to support increasing this value. Is there a way to see what it's currently effectively set to, to estimate the room for improvement there? Btool says it's "auto", and I couldn't find anything logged in _internal, e.g. on indexer restart in the loader component. btool outputs a setting max_mem_usage_mb under the input_channels stanza, is that used in combination with max_inactive = auto? Should I just increase max_mem_usage_mb, which by default is 200? Furthermore, can you explain what the other stats mean? I understand new_channels means number of new tuples that has been created, either because it hadn't been used at all previously (e.g. new file) or because the channel hadn't been used in a while, the limit for open channels had been reached and an inactive channel was therefore closed and needs to be reopened. I guess I also understand that inactive_channels is the number of channels currently not actively sending data (probably inactive since inactive_eligibility_age_seconds?). But what's the difference between a removed, timed out, reclaimed and abandoned channel? I'm looking for clues about why we have a lot of new channels - is it because we remove channels too early, or are we actually receiving many net new tuples - and if I knew which of these numbers tell me what happens in my environment, I could make an informed decision whether increasing the number of channels is going to help. ... View more

Of course, this is much nicer. It's still only a workaround, but still an obviously much better one - thanks! ... View more

We're on 3.3.1 and I see the exact same error message, no details other than null pointer. Am I missing something? ... View more

Since the comment section mentioned in this answer is gone from docs but this thread comes up first when googling for splunk strptime %Q %N (at least for me), I'll add a link to a different place where I found an explanation by @DalJeanis : https://community.splunk.com/t5/Splunk-Search/How-can-I-find-the-time-duration-between-two-fields/m-p/372494/highlight/true#M109556 Basically, %N and %Q can do the same if you provide them with a length (such as %6N or %3N). Without length specified, %N will default to three and %Q to six digits. ... View more

Even better. This uses file mod time for all events and selectively overwrites that value with the timestamp from the data if available. Nice thinking! ... View more

Well of course I can! Thanks for pointing this out, I don't know how I missed this option. Too focused on props.conf TIME_* settings I think. I've used the following transforms for this: INGEST_EVAL = _time := if(match(_raw, "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}"), _time, now()) Only downside being that this is not as close to the actual event as file modification time would have been, as this happens on the indexer during parsing. ... View more

So you're saying there is no way (you know of) to force splunk to change its behavior and it will always use the timestamp of the previous event? ... View more

Just to quickly leave a note here for anyone googling this: using INGEST_EVAL in transforms.conf and targeting _time with := to overwrite it, this makes it even better than EVAL-searchtime_things because the event will be saved with a correct timestamp, meaning you'll find your events where you expect. ... View more

I struggled with the absence of the mvmap function in Splunk before version 8 as well. This is not exactly pretty, but then again, what is. | makeresults | eval Interval = "0,15,30,60,300,3600", Scores = "5,4,3,2,1,0", MedDelta = "100" | eval Interval=split(Interval,","), Scores=split(Scores,",") | eval Score = 0 | foreach 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 [| eval debug_Interval_<<FIELD>> = mvindex(Interval, <<FIELD>>), debug_Scores_<<FIELD>> = mvindex(Scores, <<FIELD>>) | eval Score = if(MedDelta > debug_Interval_<<FIELD>>, debug_Scores_<<FIELD>>, Score)] | fields - debug_* What this does is basically create a column for each of your possible intervals and scores, and simply compare the value in MedDelta to each one. Depending on the maximum number of Intervals, you might have to add more numbers to the foreach loop.  ... View more