About nickhills

nickhills · ‎10-23-2025

Nothing whatsoever. Not a single error, or mention of the aforementioned script

nickhills · ‎10-23-2025

I have recently deployed this TA, but it is failing to run on our SC Stack. Attempting to call the lookup with: |stats count by http_user_agent|lookup user_agents http_user_agent OUTPUT Search Log reports : 10-23-2025 10:57:21.744 INFO Timeliner [2977630 DownloadRemoteEventLoopRunner] - Sending POST request 'redacted.splunkcloud.com/1761217038.13953/events?offset=2113&count=48' 10-23-2025 10:57:21.749 ERROR ExternalProvider [2914921 phase_1] - Script execution failed for external search command '/opt/splunk/etc/apps/TA-user-agents/bin/user_agents.py'. 10-23-2025 10:57:21.750 ERROR SearchOrchestrator [2905027 searchOrchestrator] - Phase_1 failed due to : Error in 'lookup' command: Script execution failed for external search command '/opt/splunk/etc/apps/TA-user-agents/bin/user_agents.py'. 10-23-2025 10:57:21.750 INFO SearchStatusEnforcer [2914554 StatusEnforcerThread] - sid=1761217038.13953, newState=FAILED, message=Error in 'lookup' command: Script execution failed for external search command '/opt/splunk/etc/apps/TA-user-agents/bin/user_agents.py'. 10-23-2025 10:57:21.750 ERROR SearchStatusEnforcer [2914554 StatusEnforcerThread] - SearchMessage orig_component=SearchStatusEnforcer sid=1761217038.13953 message_key= message=Error in 'lookup' command: Script execution failed for external search command '/opt/splunk/etc/apps/TA-user-agents/bin/user_agents.py'. 10-23-2025 10:57:21.750 INFO SearchStatusEnforcer [2914554 StatusEnforcerThread] - State changed to FAILED: Error in 'lookup' command: Script execution failed for external search command '/opt/splunk/etc/apps/TA-user-agents/bin/user_agents.py'. Splunkd.log contains one more useful detail: 10-23-2025 10:57:21.946 +0000 ERROR SearchProcessRunner [1168347 PreforkedSearchesManager-0] - preforked process=0/9376 with search=0/27967 and cmd=splunkd\x00search\x00--id=1761217038.13953\x00--maxbuckets=300\x00--ttl=600\x00--maxout=500000\x00--maxtime=8640000\x00--lookups=1\x00--reduce_freq=10\x00--rf=*\x00--user=redacted.com\x00--pro\x00--roles=power:sc_admin:tokens_auth:user\x00--sslclientsession=SESSION_CACHE_REDACTED died on exception (exit_code=111): Error in 'lookup' command: Script execution failed for external search command '/opt/splunk/etc/apps/TA-user-agents/bin/user_agents.py'. The app suggests support for SC and versions up to v10, although our stack is currently at 9.3.2411.118 I have asked Cloud-Ops to verify the app is correctly installed and enabled after I SSAI'd it on Victoria, and they have confirmed that in their opinion, there is an issue with the script. Is anyone else running this TA on Splunk Cloud 9.3x ? Or can anyone from @aplura help?

nickhills · ‎07-15-2025

To use a lookup to enrich a search, the lookup needs to exist as a lookup on the Search Head A lookup on a heavy forwarder is not going to be available at search time. What you need to do is get a copy of the lookup on the SH. The easiest (imo) option is to index the lookup file on the HF - simply define it as an input on the HF and have Splunk monitor it for changes. You can send this to any index, but lets assume you create and use one called "lookups_index" and sourcetype "my_hf_lookup" On your search head, you can now create a lookup-generating search: Depending on what your lookup contains (dates, product_ids, error codes) you would create a search like: index=lookups_index soucetype=my_hf_lookup |dedup product_code |table product_code product_description product_price |outputlookup my_sh_lookup.csv I like to name these something like: "LOOKUPGEN-my_sh_lookup.csv" You can then schedule that to run once a day/week/hour (depending on your anticipated lookup change frequency) You can then use: |lookup my_sh_lookup.csv product_code OUTPUT product_name product_price In your searches - Although I find it better practice to actually create a lookup definition and use that

nickhills · ‎07-14-2025

After some reading https://help.splunk.com/en/splunk-enterprise/administer/manage-users-and-security/9.4/manage-splunk-platform-users-and-roles/define-roles-on-the-splunk-platform-with-capabilities Followed by experimenting and testing... The current platform versions provide a capability called edit_upload_and_index which is defined as “Lets the user use the indexing preview feature when creating inputs in Splunk Web” This sounds highly promising, however granting that capability alone does not enable the Add Data Button in the settings menu.. In order to present the option to a standard user, additionally the edit_tcp_stream capability is also required – this is not immediately obvious, because the name of the capability masks the documented definition: “Lets the user send data to the the /services/receivers/stream REST endpoint.” I suspect that this second permission has the side effect of granting access to the relevant rest API which allows the /manager/<app>/adddata button to be added to the settings menu. These two permissions allow the Upload Data option to function, and whilst the option is also presented for other monitor types, the UI throws a (partial) 404 and prevents the user from adding anything more exotic. It is not clear to me if this is an expected combination of permissions Splunk intends you to grant (in which case, I will submit a documentation update suggestion) or if the edit_upload_and_index capability is intended to facilitate the outcome on its own. TLDR: To enable a non-admin user to upload files via the UI (in at least Splunk versions greater than 9.3), grant the edit_upload_and_index AND edit_tcp_stream capability to the users role.

nickhills · ‎07-14-2025

I want to provide a standard Splunk user the ability to upload files via the web UI. Specifically, so that members of our finance team can upload supplier bills for reconciliation with our platform data. In this scenario granting full sc_admin is certainly not appropriate! I had (incorrectly) assumed that Power Users had this ability, but that is not the case. There is an article from 2014 that details what was required 11 years ago, but the cited permissions in that article are no longer relevant in 2025: https://community.splunk.com/t5/Getting-Data-In/Capability-to-upload-data-files-via-the-gui-for-a-user/m-p/190518 What is required in Splunk >9.3 (specifically Splunk Cloud) to enable this feature for a non-admin user?

nickhills · ‎04-05-2023

The important part (and missing from your post) is how much of that allocated space is actually "in-use"? If you are using all 8TB of hot and 160TB of cold, then there is no way around it - you need more disk. However, if your true usage of space falls below this, then you can set your limits to match the space that is available. For example. if your only using 4TB of your 8TB actual hot-device you can safely amend the maxVolumeDataSizeMB to 1TB on each host. You should also understand that if you set the maxVol.. to a value smaller than what you have in current use will only cause that data to roll to cold - not be removed. The limits only tell Splunk what to do at those limits, so as long as you have a small amount of Cold space you can add your new indexers leaving the cold maxVol... set to 40TB each (320 total) Let the cluster replicate and balance, then when that process is complete you can reduce it to 20TB each. But, Ideally your retention should be dictated by dates rather than volume. I tend to think of the maxVol.. setting as a safety net incase you are getting close to the physical limits of the hardware, but what really drives my storage is how long i need to keep that data for.

nickhills · ‎04-05-2023

How are you installing the package - are you using the UI, or copying the package directly? Also are you certain you are using specific Windows version? https://apps.splunk.com/app/2883/

nickhills · ‎04-05-2023

Can you provide a bit more detail on what you are trying to do - are you trying to trigger a webhook as an alert action?

nickhills · ‎04-04-2023

@splunkreal wrote: So sc4s is just a filter, we can't use it as log collector to store data for several months if I understood? That is correct. SC4S is a transient combined syslog receiver and Splunk forwarder. It is not a useful tool without a platform (Splunk) to send the data to. The big advantage with SC4S is the "rule soup" which helps classify and route data into appropriate sourcetypes and indexes without needing any further configuration

nickhills · ‎06-23-2022

If your metrics are coming from the nix or windows TA then changing the DS version would not have any effect as it is the TA providing that data not the Splunk version. With that said, so far, my limited testing of v9 I haven’t seen any differences in any of the internal metrics either, so you *should* be ok. if you need anything further you are better off starting a new topic so more people will see your question!

nickhills · ‎02-04-2022

Thanks to the Splunk Docs team for updating the table I referenced above. Hopefully this makes it clearer for anyone approaching this in the future!

nickhills · ‎02-04-2022

Solution Use the latest version of the Splunk release train. Upgrading to the latest version of the destination version will mitigate this issue. The Splunk documentation (https://docs.splunk.com/Documentation/Splunk/8.2.4/Installation/HowtoupgradeSplunk) Suggests that to upgrade to the latest version, for 7.3.x starting points you need to migrate to a 8.0.x or 8.1.x version. In reality this is slightly more nuanced as there are specific version incompatibilities that need to be considered. As a general rule, when performing updates, you should select the latest version of the release, even if this is an intermediate step to an eventual target version. 7.3.9 > 8.1.72 > 8.2.4 (latest versions at date of this post)

nickhills · ‎02-04-2022

Workaround From 7.3.9 > 8.0.0 Stop the splunk service using the services snapin. Snapshot the VM/Backup the system From the $SPLUNK_HOME/bin folder delete the following files (take backups first) libxml2.dll libeay32.dll ssleay32.dll Run the Splunk 8.0.0 installer (leaving the Splunk service stopped) Validate that the install process completes and check and test the upgrade.

nickhills · ‎02-04-2022

Upgrading from Splunk 7.3.9 to Versions before 8.0.8 or 8.1.1 will fail. During the install process the installer will error with the following messages relating to the files: libxml2.dll libeay32.dll ssleay32.dll After dismissing these messages, the installer rolls back and reverts to the previously installed version 7.3.9 This appears related to the dates on these files that the installer does not handle correctly and does not overwrite. At the end of the failed install (before rollback) these 3 files are missing from the $SPLUNK_HOME/bin folder I presume the installer removes (backs up) the original files and during the install process validates that the files to be installed are newer. In the case of these 3 files, the situation is not handled correctly and the installer fails to correctly remedy the situation.

nickhills · ‎12-22-2020

I think the wording is a little confusing, but "tstatsHomePath" allows you to dictate where this data is stored if you wanted to separate it from the rest of your index data, but importantly you do not have to specify a "tstatsHomePath"! The wording in the spec file says that you should not configure this value for an index with remote storage, so this just means for a Smartstore index do not define "tstatsHomePath" - the cachemanager will handle this for you.

nickhills · ‎12-21-2020

If the syslog receiver is using UDP you can't test it with telnet, instead try netcat. nc -z -v -u <your_IP> <your_port_number> Although, a good start is to use netstat on the recieving host and confirm the host is listening on the right interface/port/proto! netstat -ln|grep <your_port_number>

nickhills · ‎12-21-2020

I made a minor change to the answer above, but I can not reproduce the scenario you describe. https://regex101.com/r/FXD0Q4/2

nickhills · ‎12-20-2020

Splunk will not perform CIDR matches against regular expressions. You will need to construct your regex to match the range of addresses you need (10\.) (172\.1[6-9]\.)|(172\.2[0-9]\.)|(172\.3[0-1]\.) (192\.168\.) But you should be able to do this in one stanza if you wish [internal_IPs] REGEX = dst\=((?:10\.)|(?:172\.1[6-9]\.)|(?:172\.2[0-9]\.)|(?:172\.3[0-1]\.)|(?:192\.168\.)).+ DEST_KEY = queue FORMAT = nullQueue

nickhills · ‎12-18-2020

Where are you seeing this? Inside the web datamodel? In which case, the action field should look like this (see attached) If you really want to include that additional logic into the datamodel (which I am reluctant to advise) you will need to change it to a "case" statement, you cant just layer up additional "if()"s. case(action="File quarantined","blocked", isnull(action) OR action="","unknown", 1=1, action)

nickhills · ‎12-18-2020

try changing it to if(action="File quarantined","blocked",action) That looks to me like the intent is to re-write the action to be "blocked" for a quarantine message, otherwise leave action as it was if (action = quarantine, re-write it as action="blocked", otherwise set action=action( i.e whatever it already was) )

nickhills · ‎12-18-2020

@gozdeyildizz Thats great news! Don't forget to accept an answer and upvote posts that helped you!

nickhills · ‎12-15-2020

I am pleased you have it working, but for the record, chmod'ding to 777 is not a sensible fix for a production system. 🙂

nickhills · ‎12-15-2020

In that case, unless you set up the splunk user by hand, it wont exist. At a guess you were therefore running Splunk as root. If root has permission issues then you probably have bigger problems! Can you access the contents of /opt/splunk?

nickhills · ‎12-15-2020

How was splunk installed? rpm/deb or from the tar.gz? Who owns files in /opt/splunk?

nickhills · ‎12-15-2020

Thats odd. Can you run "sudo id splunk" It should return something like uid=xxx(splunk) gid=xxx(splunk) group=xxx(splunk) or does it report "splunk" no such user

Posts	1786
Solutions	267
Karma Given	147
Karma Received	503
Member Since	‎10-10-2011

Online Status	Offline
Date Last Visited	‎10-23-2025 03:54 AM

TA-user-agents Failing on Splunk Cloud

Standard User – UI File Upload – What Capabilities...

[Windows] Upgrade 7.3.9 to Versions:<8.0.8 or:<8.1...

Setting Splunk Community Signatures

GOTCHA: Upgraded Memory with systemd? Read This!

AWS TA - S3 Generic input. Insane Memory Usage!

PROMOTION: Splunk Ideas - Help shape Splunk's futu...

splunk_ta_o365 DLP event collection fails with: "D...

Has the TA Metricator prevented cluster peer resta...

Notice: Sophos Central App for Splunk is retiring....

Re: TA-user-agents Failing on Splunk Cloud

TA-user-agents Failing on Splunk Cloud

Re: Splunk Distributed Architecture

Re: Standard User – UI File Upload – What Capabili...

Standard User – UI File Upload – What Capabilities...

Re: How to reduce hot and cold volume safely?

Re: Error when installing Python for Scientific Co...

Re: Why am I unable to create webhook for Splunk C...

Re: Splunk sc4s log collector license

Re: Deployment Server Upgrade

Re: [Windows] Upgrade 7.3.9 to Versions:<8.0.8 or:...

Re: [Windows] Upgrade 7.3.9 to Versions:<8.0.8 or:...

Re: [Windows] Upgrade 7.3.9 to Versions:<8.0.8 or:...

[Windows] Upgrade 7.3.9 to Versions:<8.0.8 or:<8.1...

Re: Datamodels with Smartstore

Re: Splunk can't get data from remote machines

Re: How to Exclude private IP range from transform...

Re: How to Exclude private IP range from transform...

Re: edit fields with eval expressions

Re: edit fields with eval expressions

Re: How to Calculate SLA of Different Teams for a ...

Re: Splunk Permission Error

Re: Splunk Permission Error

Re: Splunk Permission Error

Re: Splunk Permission Error

Join the Conversation