All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

@m_zandinia I can confirm that this is a known issue in version 9.4.1 & your report is private report as per the above screenshot.   https://docs.splunk.com/Documentation/Splunk/9.4.1/ReleaseNotes/K... See more...
@m_zandinia I can confirm that this is a known issue in version 9.4.1 & your report is private report as per the above screenshot.   https://docs.splunk.com/Documentation/Splunk/9.4.1/ReleaseNotes/KnownIssues  It got fixed in 9.4.2 https://docs.splunk.com/Documentation/Splunk/9.4.2/ReleaseNotes/Fixedissues   
Let me start with the obvious: Because you are using table in your query, <table /> is the appropriate panel type, not <event/> used in your illustration. You use drilldown type to be "All", set t... See more...
Let me start with the obvious: Because you are using table in your query, <table /> is the appropriate panel type, not <event/> used in your illustration. You use drilldown type to be "All", set token to $click.value$ and expect it to take the value of field Channel.   That is not how drilldown works.  The value should be set to $row.Channel$. (There is another drilldown type "Cell".  But if you want to the token to represent Channel, this is inappropriate.) Here is a complete mock dashboard for you to play with.  Wherever you click, the clicked Channel value will be displayed in the panel's title.  Play with it and adapt it for your use. <dashboard version="1.1" theme="light"> <label>Click to set token</label> <description>https://community.splunk.com/t5/Dashboards-Visualizations/Classic-Dashboard-Drilldown-Click-on-a-Value-and-Set-Token/m-p/746080#M58677</description> <row> <panel> <title>Click on any row</title> <table> <title>Channel in that row should be &gt;$channel_token$&lt;</title> <search> <query>index = _internal component=* thread_id=* | rex "^(?&lt;Timestamp&gt;\S+ \S+ \S+)" | rename component as Channel, log_level as Level, event_message as Details, thread_id as RecordID, thread_name as Ruletitle | table Timestamp Level Channel RecordID Ruletitle Details *</query> <earliest>-24h@h</earliest> <latest>now</latest> <sampleRatio>1</sampleRatio> </search> <option name="count">50</option> <option name="dataOverlayMode">none</option> <option name="drilldown">cell</option> <option name="percentagesRow">false</option> <option name="rowNumbers">false</option> <option name="totalsRow">false</option> <option name="wrap">true</option> <drilldown> <set token="channel_token">$row.Channel$</set> </drilldown> </table> </panel> </row> </dashboard>  
@isoutamo  Thanks. The SHC is in the sync and there is no issue with kvstore.   Yes. They create both the alerts and reports from Web UI. They can delete alerts from the Web UI but they can't... See more...
@isoutamo  Thanks. The SHC is in the sync and there is no issue with kvstore.   Yes. They create both the alerts and reports from Web UI. They can delete alerts from the Web UI but they can't delete their own reports from the Web UI. I'm using Splunk Enterprise Version 9.4.1.
@isoutamoYes, I understand that there isn't a specific capability called "delete." I was referring to the modify/write capability in a broader context.
Have you check that your SHC is in sync and there is no general errors wit it or with kvstore? I suppose that your users have created those alerts with gui and also they try to remove those via gui?... See more...
Have you check that your SHC is in sync and there is no general errors wit it or with kvstore? I suppose that your users have created those alerts with gui and also they try to remove those via gui? Or are they trying to use REST api?
There is no separate delete capability for reports. It works if you have modify/write capability, which you obviously have as you could create those reports under your private scope. That can_delete c... See more...
There is no separate delete capability for reports. It works if you have modify/write capability, which you obviously have as you could create those reports under your private scope. That can_delete capability is totally different stuff and you never want to get it to normal user.
Hi @illuminatedaxis    The XML tags you are using for batchInterval, batchCount, sendMode, and retriesOnError do not match the property names expected by the com.splunk.logging.HttpEventCollectorLo... See more...
Hi @illuminatedaxis    The XML tags you are using for batchInterval, batchCount, sendMode, and retriesOnError do not match the property names expected by the com.splunk.logging.HttpEventCollectorLogbackAppender. You should use the following tags instead: <batch_interval> instead of <batchInterval> <batch_count_size> instead of <batchCount> <send_mode> instead of <sendMode> <retries_on_error> instead of <retriesOnError> Here is your configuration block with the corrected tags:   xml <?xml version="1.0" encoding="UTF-8"?> <configuration> <appender name="SPLUNK_HTTP" class="com.splunk.logging.HttpEventCollectorLogbackAppender"> <url>my-splunk-url</url> <token>my-splunk-token</token> <index>my-index</index> <sourcetype>${USER}_local</sourcetype> <disableCertificateValidation>true</disableCertificateValidation> <batch_interval>1</batch_interval> <!-- Corrected tag --> <batch_count_size>1000</batch_count_size> <!-- Corrected tag --> <send_mode>parallel</send_mode> <!-- Corrected tag --> <retries_on_error>1</retries_on_error> <!-- Corrected tag --> <layout class="my-layout-class"> <!-- some custom layout configs --> </layout> </appender> <logger name="com.myapplication" level="DEBUG" additivity="false"> <appender-ref ref="SPLUNK_HTTP"/> </logger> <root level="DEBUG"> <appender-ref ref="SPLUNK_HTTP"/> </root> </configuration>   Logback configures appenders by mapping XML tags to Java setter methods. For example, an XML tag <exampleProperty> would typically call a method setExampleProperty(...) on the appender class. The Splunk logging library's HttpEventCollectorLogbackAppender defines setters like setBatch_interval(String interval), setBatch_count_size(String count), and setSend_mode(String mode). Therefore, the XML tags must match these names, including the underscores.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
I have the below configuration in my logback.xml. While the url, token, index sourcetype and disableCertificateValidation fields are getting picked up, the batchInterval, batchCount and sendMode are ... See more...
I have the below configuration in my logback.xml. While the url, token, index sourcetype and disableCertificateValidation fields are getting picked up, the batchInterval, batchCount and sendMode are not. I ran my application in debug mode, and I did see that the `ch.qos.logback.core.model.processor.AppenderModelHandler` is picking up the these tags as submodels correctly. Can someone please help me understand if I'm doing anything wrong here? <?xml version="1.0" encoding="UTF-8"?> <configuration> <appender name="SPLUNK_HTTP" class="com.splunk.logging.HttpEventCollectorLogbackAppender"> <url>my-splunk-url</url> <token>my-splunk-token</token> <index>my-index</index> <sourcetype>${USER}_local</sourcetype> <disableCertificateValidation>true</disableCertificateValidation> <batchInterval>1</batchInterval> <batchCount>1000</batchCount> <sendMode>parallel</sendMode> <retriesOnError>1</retriesOnError> <layout class="my-layout-class"> <!-- some custom layout configs --> </layout> </appender> <logger name="com.myapplication" level="DEBUG" additivity="false"> <appender-ref ref="SPLUNK_HTTP"/> </logger> <root level="DEBUG"> <appender-ref ref="SPLUNK_HTTP"/> </root> </configuration>   I'm using the following dependency for splunk, if it matters -  <dependency> <groupId>com.splunk.logging</groupId> <artifactId>splunk-library-javalogging</artifactId> <version>1.11.7</version> </dependency>    
Did you read my question carefully? Users can delete their own alerts, but they cannot delete their own reports. If they didn’t have the "delete_knowledge_objects" capability, how would they even b... See more...
Did you read my question carefully? Users can delete their own alerts, but they cannot delete their own reports. If they didn’t have the "delete_knowledge_objects" capability, how would they even be able to delete their own alerts in the first place? This inconsistency is exactly the issue I’m trying to highlight.
@m_zandinia  To delete a report in Splunk, a user must have the delete capability assigned to their role. This capability allows users to delete knowledge objects such as saved reports, alerts, and ... See more...
@m_zandinia  To delete a report in Splunk, a user must have the delete capability assigned to their role. This capability allows users to delete knowledge objects such as saved reports, alerts, and dashboards that they own or have permission to manage. Create and edit reports - Splunk Documentation By default admin has the capability called "admin_all_objects", so he can delete the report.  admin: Created test user called "test-user" and assigned user & power user role to that, and can see that admin_all_objects has not assigned.       
It's not an orphaned knowledge object. A user creates a report and immediately afterward is unable to delete it. A user creates an alert and can delete it immediately without issue. The report i... See more...
It's not an orphaned knowledge object. A user creates a report and immediately afterward is unable to delete it. A user creates an alert and can delete it immediately without issue. The report is created in the correct location (as shown in the screenshot below).     However, the delete request sent by the Splunk UI is targeting the wrong URL, and I’m not sure why this is happening.   P.S. The admin is able to delete the report without any issue. P.S. For privacy reasons, I manually changed the report name to "test-user-report" in the log samples. The actual report name matches what is shown in the screenshot.  
@clumicao  I haven't worked on Mission Control before, but you can check this documentation – it might be helpful. Apply filters and save filtered views for incidents Triage incidents using incide... See more...
@clumicao  I haven't worked on Mission Control before, but you can check this documentation – it might be helpful. Apply filters and save filtered views for incidents Triage incidents using incident review in Splunk Mission Control - Splunk Documentation
I forgot what it felt like for someone to copy my assignment in homeroom 
Hi @m_zandinia  It sounds like in your scenario that your users can delete their own reports / knowledge objects within the app, but not owned by "nobody" (and probably not able to delete anything o... See more...
Hi @m_zandinia  It sounds like in your scenario that your users can delete their own reports / knowledge objects within the app, but not owned by "nobody" (and probably not able to delete anything owned by any other user either). In order for them to delete App-level shared objects they will need to have write permission to the app - does their role have this? (Or the admin_all_objects capability) Additionally if the knowledge object is globally shared they would need the admin_all_objects capability. @kiran_panchavat Im not convinced those answers from 10 years ago are still valid, or only partially valid - If you delete a user which owns knowledge objects the owner does not get changed to "nobody". It stays owned by the original owner but becomes Orphaned: In the below example I have 2 searches "owned" by "testing1" which I deleted, thus they become orphaned and still owned by the testing1 user. Its not uncommon for things to be owned by "nobody" - whilst I prefer the use of service accounts, a lot of customers use the nobody user for owning artifacts which dont have a specific owner within an app (e.g. no specific named person, thus "nobody"). According to the docs an orphaned search "will not run the scheduled report on its schedule at all" - whereas a search owned by "nobody" will. Its important to know that searches which are owned by "nobody" do get executed and will be essentially run as the system user, so will have access to all indexes, lookups etc, and therefore if someone has write access to a search owned by nobody that they could modify it to search in indexes which they themselves are now allowed to search! This is why I always recommend searches to be owned by a service account following the principals of least-privileged access.  Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing
@m_zandinia  Check this  | rest /services/saved/searches | table title, eai:acl.owner, eai:acl.app, eai:acl.sharing, eai:acl.perms.read, eai:acl.perms.write | rename title as "Report Name", ea... See more...
@m_zandinia  Check this  | rest /services/saved/searches | table title, eai:acl.owner, eai:acl.app, eai:acl.sharing, eai:acl.perms.read, eai:acl.perms.write | rename title as "Report Name", eai:acl.owner as "Owner", eai:acl.app as "App", eai:acl.sharing as "Sharing Level", eai:acl.perms.read as "Read Permissions", eai:acl.perms.write as "Write Permissions" | sort App, "Report Name"    
@m_zandinia  Are they unable to delete all reports or just specific ones? Did you check the permissions for the reports? 
@m_zandinia  It means that the user that created the object is no longer a user in the authenticating system. If you create a local user, then login as that user, then create any knowledge object, t... See more...
@m_zandinia  It means that the user that created the object is no longer a user in the authenticating system. If you create a local user, then login as that user, then create any knowledge object, then delete that user, then all of his KOs will switch to be owned by nobody. What are "splunk-system-user" and nobody"? - Splunk Community Solved: What does 'nobody' (under owner column) signify in... - Splunk Community saved searches become orphaned or are associated with users that no longer exist. In such cases, reassigning the saved search to an existing valid user and then deleting it via the GUI can resolve the issue. https://community.splunk.com/t5/Deployment-Architecture/Orphaned-Scheduled-Search-cannot-delete/m-p/221185 
Hi Splunkers, I’m running a Splunk Search Head Cluster (SHC) with 3 search heads, authenticated via Active Directory (AD). We have several custom apps deployed. Currently, users are able to: Crea... See more...
Hi Splunkers, I’m running a Splunk Search Head Cluster (SHC) with 3 search heads, authenticated via Active Directory (AD). We have several custom apps deployed. Currently, users are able to: Create alerts Delete alerts Create reports However, they are unable to delete reports. Investigation Details From the _internal logs, here’s what I observed: When deleting an alert — the deletion works fine: 192.168.0.1 - user [17/May/2025:11:06:59.687 +0000] "DELETE /en-US/splunkd/__raw/servicesNS/username/SOC/saved/searches/test-user-alert?output_mode=json HTTP/1.1" 200 421 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/20100101 Firefox/138.0" - eac203572253a2bd3db35ee0030c6a76 68ms 192.168.0.1 - user [17/May/2025:11:06:59.690 +0000] "DELETE /servicesNS/username/SOC/saved/searches/test-user-alert HTTP/1.1" 200 421 "-" "Splunk/9.4.1 (Linux 6.8.0-57-generic; arch=x86_64)" - - - 65ms   When deleting a report — it fails with a 404 Not Found: 192.168.0.1 - user [17/May/2025:10:27:51.699 +0000] "DELETE /en-US/splunkd/__raw/servicesNS/nobody/SOC/saved/searches/test-user-report?output_mode=json HTTP/1.1" 404 84 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/20100101 Firefox/138.0" - eac203572253a2bd3db35ee0030c6a76 5ms 192.168.0.1 - user [17/May/2025:10:27:51.702 +0000] "DELETE /servicesNS/nobody/SOC/saved/searches/test-user-report HTTP/1.1" 404 84 "-" "Splunk/9.4.1 (Linux 6.8.0-57-generic; arch=x86_64)" - - - 1ms   Alerts are created under the user’s namespace (servicesNS/username/...) and can be deleted by the user. Reports appear to be created under the nobody namespace (servicesNS/nobody/...), which may be the reason users lack permission to delete them. Has anyone faced a similar issue?
Hi @Na_Kang_Lim  @livehybrid  is 100% right, TrackMe V1 is out of date and unsupported since more than 2 years now, therefore I wouldn't respond on this. Please consider upgraded to TrackMe V2: h... See more...
Hi @Na_Kang_Lim  @livehybrid  is 100% right, TrackMe V1 is out of date and unsupported since more than 2 years now, therefore I wouldn't respond on this. Please consider upgraded to TrackMe V2: https://docs.trackme-solutions.com there are options to migrate from V1 to V2 (https://docs.trackme-solutions.com/latest/migration_trackmev1.html) but I would suggest to consider a fresh configuration instead rather than migrating stuffs.
I've seen this exact issue before with Splunk Universal Forwarders. The "splunkd.pid doesn't exist" error combined with the "tcp_conn_open_afux ossocket_connect failed" messages typically happens whe... See more...
I've seen this exact issue before with Splunk Universal Forwarders. The "splunkd.pid doesn't exist" error combined with the "tcp_conn_open_afux ossocket_connect failed" messages typically happens when there's a conflict between how the Splunk process is started and managed. Based on your description, this is likely one of two issues: a. Duplicate systemd service files causing a "split brain" situation b. Permission problems with the Splunk installation directory For the first issue, check if you have duplicate service definitions: ls -la /usr/lib/systemd/system/SplunkForwarder.service ls -la /etc/systemd/system/SplunkForwarder.service If both exist, that's causing your problem! The one in /etc/systemd/system takes precedence, and they might have different user/permission settings. You can fix this by: sudo rm /etc/systemd/system/SplunkForwarder.service sudo systemctl daemon-reload sudo systemctl restart SplunkForwarder If that doesn't work, check the ownership of your Splunk files: ls -la /opt/splunkforwarder Make sure everything is owned by the correct user (typically splunk:splunk). If permissions are wrong, you can fix with: chown -R splunk:splunk /opt/splunkforwarder As a last resort, the complete reinstall approach works well: sudo systemctl stop SplunkForwarder sudo yum remove splunk* sudo rm -rf /opt/splunkforwarder Then reinstall the forwarder and configure it properly. I've had good success with this approach when dealing with these mysterious pid and socket connection errors. Please give for support happly splunking ....