All Posts

Find Answers
Ask questions. Get answers. Find technical product solutions from passionate members of the Splunk community.

All Posts

IHAC with an SVA C3 (On-Prem) setup running 9.4.0 on the MN, SHC, Deployer but 9.3.2 on the peers (upgrade in the works due to unsupported linux kernel 3.x). They've been running this way OK for abou... See more...
IHAC with an SVA C3 (On-Prem) setup running 9.4.0 on the MN, SHC, Deployer but 9.3.2 on the peers (upgrade in the works due to unsupported linux kernel 3.x). They've been running this way OK for about a month whilst the upgrade is pending. Start of issue The problem that is being seen is that the client wanted to disable the new 'audit_trail' app for platform confidentiality a week ago. They created a local folder for the app on the deployer ($SPLUNK_HOME/etc/shcluster/apps/audit_trail) and disabled it via a .conf file change, no issue worked ok and pushed to the SHC from the deployer. The SHC is all in sync. Symptom The issue now being seen is that they can't delete TA's and apps with pushes from the Deployer. For example they are removing legacy TA's and despite not being on the deployer they remain on the SHC. The cluster is operational and in sync OK and I have temporarily removed the 'audit_trail' workaround which allows the usual command to operate again: ./splunk apply shcluster-bundle -target <https://x.x.x.x:8089> -preserve-lookups true If not you have to include the switch (-push-default-apps true) Next steps: I'm trying to locate the correct component in index _internal to troubleshoot what is happening and why it is not deleting apps and TA's not on the Deployer Example: index="_internal" source="/opt/splunk/var/log/splunkd.log" host IN (SH, SH, SH, Deployer) I can't locate any warnings or relevant errors even when including the relevant TA being intended for removal on the short time period in question Any suggestions welcome        
Hi Will, Thanks! That kind of worked but is there a way to have the html within same panel next to input without creating another panel? Its sort of didn’t align the panels correctly.
@livehybrid Thanks for the quick reply  .  We are trying to avoid manual instrumentation in the code completely. I am trying to do a onetime setup so that whatever the methods that we want to trace... See more...
@livehybrid Thanks for the quick reply  .  We are trying to avoid manual instrumentation in the code completely. I am trying to do a onetime setup so that whatever the methods that we want to trace, I can add them through env variable OTEL_DOTNET_TRACES_METHODS_INCLUDE. Please suggest me if there is any other better approach where I can implement method level tracing in a better way to implement.   Regarding the error All the paths mentioned are correct and the dlls exists in those paths. We have checked the versions and all the versions exists in .nuget package. Could you please help me in resolving the issue.  "%USERPROFILE%\\.nuget\\packages\\opentelemetry.autoinstrumentation.startuphook\\1.10.0\\lib\\netcoreapp3.1\\OpenTelemetry.AutoInstrumentation.StartupHook.dll", "OTEL_DOTNET_AUTO_HOME": "%USERPROFILE%\\.nuget\\packages\\splunk.opentelemetry.autoinstrumentation\\1.9.0", "CORECLR_PROFILER_PATH_64": "%USERPROFILE%\\.nuget\\packages\\opentelemetry.autoinstrumentation.runtime.native\\1.10.0\\runtimes\\win-x64\\native\\OpenTelemetry.AutoInstrumentation.Native.dll", "CORECLR_PROFILER_PATH_32": "%USERPROFILE%\\.nuget\\packages\\opentelemetry.autoinstrumentation.runtime.native\\1.10.0\\runtimes\\win-x86\\native\\OpenTelemetry.AutoInstrumentation.Native.dll"
Hello @ITWhisperer , Thanks for asking! You are right.., It will be like, the next event will be received within 3 days, it wont take more time at wrost cases. I'm using those values in the ... See more...
Hello @ITWhisperer , Thanks for asking! You are right.., It will be like, the next event will be received within 3 days, it wont take more time at wrost cases. I'm using those values in the chart, when we are searching with less time range, I can't see the logs of the timerange in that time range because of the gap in logs,  I have listed two scenarious, As per the scenario1, The perevious value is just a opposite value of the next one.  Scenario 2 is bit hard, having multiple values, which can be generated before 3 days at wrost cases. Thansk!
Just to say this isn't possible (to reference the readme file direct) from the UI. You'd have to do it from AppServer or as a view in the data folder thus duplicating the file and effort. would have ... See more...
Just to say this isn't possible (to reference the readme file direct) from the UI. You'd have to do it from AppServer or as a view in the data folder thus duplicating the file and effort. would have been a nice option.
How does Splunk know what the previous state was unless it is included in the search? For example, if the first state is "System Stop" and the system was reset 3 days, or 3 weeks, or 3 months ago, w... See more...
How does Splunk know what the previous state was unless it is included in the search? For example, if the first state is "System Stop" and the system was reset 3 days, or 3 weeks, or 3 months ago, what do you want Splunk to report?
Hello @ITWhisperer , Thanks for your reply. 17:54:01 - System reset 22:09:04 - System Stop 23:01:01 - System Started 01:01:01 - System Stop In case of from 21:00, I need to take as System... See more...
Hello @ITWhisperer , Thanks for your reply. 17:54:01 - System reset 22:09:04 - System Stop 23:01:01 - System Started 01:01:01 - System Stop In case of from 21:00, I need to take as System reset and followed by other values. Actually I just need to fill the value, even the logs weren't there in teh selcted timerange. Thanks!
Hi @salikovsky  Regarding SU Behaviour - when the expected logs are missing, are you able to confirm that they are present in the /var/log/bash_history.log file and only missing from Splunk. Also, ... See more...
Hi @salikovsky  Regarding SU Behaviour - when the expected logs are missing, are you able to confirm that they are present in the /var/log/bash_history.log file and only missing from Splunk. Also, run the following search and check that each of the UFs is correctly configured to monitor the files: index=_internal TailingProcessor "/var/log/bash_history.log" You should see something like 02-21-2025 13:47:29.836 +0000 INFO TailingProcessor [1354 MainTailingThread] - Parsing configuration stanza: monitor:///var/log/bash_history.log. 02-21-2025 13:47:29.836 +0000 INFO TailingProcessor [1354 MainTailingThread] - Adding watch on path: /var/log/bash_history.log. Please let me know how you get on and consider adding karma to this or any other answer if it has helped. Regards Will  
Please clarify what you want Splunk to assume in the second case, for example, if the search was from 21:00, would you want Splunk to assume the previous state was "System reset" or "System Start"? ... See more...
Please clarify what you want Splunk to assume in the second case, for example, if the search was from 21:00, would you want Splunk to assume the previous state was "System reset" or "System Start"? Do you want to search for a longer period of time to try and find the previous state, and then remove these results from the chart?
Hi @smanojkumar  I may have misunderstood, but If you want the search to include the event at 6AM then you will need to change the earliest time within the search to cover this event.  Feel free to... See more...
Hi @smanojkumar  I may have misunderstood, but If you want the search to include the event at 6AM then you will need to change the earliest time within the search to cover this event.  Feel free to share a screenshot example of what you are seeing to help explain the difference to your expectation/intention. Please let me know how you get on and consider adding karma to this or any other answer if it has helped. Regards Will
Try something like this <dashboard version="1.1" theme="light"> <label>Events</label> <row> <panel id="panel1"> <input type="text" token="tk1" id="tk1id" searchWhenChanged="true"> ... See more...
Try something like this <dashboard version="1.1" theme="light"> <label>Events</label> <row> <panel id="panel1"> <input type="text" token="tk1" id="tk1id" searchWhenChanged="true"> <label>Refine further?</label> <prefix> | where </prefix> </input> <html id="htmlid"> <p>count: $job.resultCount$</p> </html> <event> <search> <query>index=_internal</query> <earliest>-24h@h</earliest> <latest>now</latest> </search> <option name="list.drilldown">none</option> </event> </panel> <panel depends="$alwaysHide$"> <html> <style> #panel1 .dashboard-panel { display: flex; flex-direction: row; flex-wrap: wrap; } </style> </html> </panel> </row> </dashboard>
@salikovsky  Check UF Logs: tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log | grep -i bash_history File truncation, rotations, or buffering delays could be preventing logs from reaching Sp... See more...
@salikovsky  Check UF Logs: tail -f /opt/splunkforwarder/var/log/splunk/splunkd.log | grep -i bash_history File truncation, rotations, or buffering delays could be preventing logs from reaching Splunk. Run on affected hosts: /opt/splunkforwarder/bin/splunk list monitor Ensure /var/log/bash_history.log is listed.  
Hello Splunkers, I'm having a logs which will be generated only where there is change in system, 6:01:01 - System Stop 10:54:01 - System Start 13:09:04 - System Stop 16:01:01 - System Start 1... See more...
Hello Splunkers, I'm having a logs which will be generated only where there is change in system, 6:01:01 - System Stop 10:54:01 - System Start 13:09:04 - System Stop 16:01:01 - System Start 17:01:01 - System Stop These are the logs. Lets say If I'm searchit it in a chart, for the timerange from 7Am - 4Pm the chart from 8Am until 10:54:01 Am is empty since the previous event was generated at 6:01:01, so there is a gap. I would like to fix this. In some cases only 2 values is been repeated, so we can take the one in present, the past can be its opposite. Eg -  At 10:54:01 - System Start, We have received this log, where the system is start, the previous one will be stop.  These are fixed for some cased, I need two best solutions, only for this scenario, other for multiple values, like these 14:01:01 - System Started 17:54:01 - System reset 22:09:04 - System Stop 23:01:01 - System Started 01:01:01 - System Stop wheres here I'm getting three values like Started, Stop and reset. Thanks in Advance!
Hello, I am trying to collect bash_history logs in real-time from multiple Linux hosts using Splunk. I have deployed the following script to append executed commands to /var/log/bash_history.log: #... See more...
Hello, I am trying to collect bash_history logs in real-time from multiple Linux hosts using Splunk. I have deployed the following script to append executed commands to /var/log/bash_history.log: #!/bin/bash LOG_FILE="/var/log/bash_history.log" PROMPT_COMMAND_STR='export PROMPT_COMMAND='\''RECORD_CMD=$(history 1 | sed "s/^[ ]*[0-9]*[ ]*//"); echo "$(date "+%Y-%m-%d %H:%M:%S") $(whoami) $RECORD_CMD" >> /var/log/bash_history.log'\'''   # 1. Create log file if it doesn't exist and set permissions if [ ! -f "$LOG_FILE" ]; then     touch "$LOG_FILE"     echo "[INFO] Log file created: $LOG_FILE" fi chmod 666 "$LOG_FILE" chown root:users "$LOG_FILE" echo "[INFO] Log file permissions set"   # 2. Add PROMPT_COMMAND to /etc/bash.bashrc if ! grep -q "PROMPT_COMMAND" /etc/bash.bashrc; then     echo "$PROMPT_COMMAND_STR" >> /etc/bash.bashrc     echo "[INFO] PROMPT_COMMAND added to /etc/bash.bashrc" fi   # 3. Force loading of ~/.bashrc through /etc/profile if ! grep -q "source ~/.bashrc" /etc/profile; then     echo 'if [ -f ~/.bashrc ]; then source ~/.bashrc; fi' >> /etc/profile     echo "[INFO] ~/.bashrc now loads via /etc/profile" fi   # 4. Add PROMPT_COMMAND to all users' ~/.bashrc and ~/.profile for user in $(ls /home); do     for FILE in "/home/$user/.bashrc" "/home/$user/.profile"; do         if [ -f "$FILE" ] && ! grep -q "PROMPT_COMMAND" "$FILE"; then             echo "$PROMPT_COMMAND_STR" >> "$FILE"             echo "[INFO] PROMPT_COMMAND added to $FILE (user: $user)"         fi     done done   # 5. Add PROMPT_COMMAND for root user for FILE in "/root/.bashrc" "/root/.profile"; do     if [ -f "$FILE" ] && ! grep -q "PROMPT_COMMAND" "$FILE"; then         echo "$PROMPT_COMMAND_STR" >> "$FILE"         echo "[INFO] PROMPT_COMMAND added to $FILE (root)"     fi done   # 6. Ensure ~/.bashrc is sourced in ~/.profile for all users for user in $(ls /home); do     PROFILE_FILE="/home/$user/.profile"     if [ -f "$PROFILE_FILE" ] && ! grep -q ". ~/.bashrc" "$PROFILE_FILE"; then         echo ". ~/.bashrc" >> "$PROFILE_FILE"         echo "[INFO] ~/.bashrc now sources from ~/.profile (user: $user)"     fi done   # 7. Ensure all users use Bash shell while IFS=: read -r username _ _ _ _ home shell; do     if [[ "$home" == /home/* || "$home" == "/root" ]]; then         if [[ "$shell" != "/bin/bash" ]]; then             echo "[WARNING] User $username has shell $shell, changing to Bash..."             usermod --shell /bin/bash "$username"         fi     fi done < /etc/passwd   # 8. Apply changes exec bash echo "[INFO] Configuration applied" The script runs correctly, and /var/log/bash_history.log is created on all hosts. However, Splunk is not collecting logs from all hosts. Some hosts send data properly, while others do not. What I have checked: Permissions on /var/log/bash_history.log → The file is writable by all users (chmod 666 and chown root:users). Presence of PROMPT_COMMAND in user sessions → When running echo $PROMPT_COMMAND, it appears correctly for most users. SU behavior → If users switch with su - username, it works. However, if they switch with su username, sometimes the logs are missing. Splunk Inputs Configuration: [monitor:///var/log/bash_history.log] disabled = false index = os sourcetype = bash_history This is properly deployed to all hosts. Questions: Could there be permission issues with writing to /var/log/bash_history.log under certain circumstances? Would another directory (e.g., /tmp/) be better? How can I ensure that all user sessions (including su username) log commands consistently? Could there be an issue with Splunk Universal Forwarder not properly monitoring /var/log/bash_history.log on some hosts?   Any insights or best practices would be greatly appreciated! Thanks.
@SN1  To resolve the missing indexer in your License Master: Test network connectivity (ping, telnet 8089) between the indexer and License Master. Restart the indexer (and License Master if neede... See more...
@SN1  To resolve the missing indexer in your License Master: Test network connectivity (ping, telnet 8089) between the indexer and License Master. Restart the indexer (and License Master if needed) after fixing config or network issues. Check if the indexer is running. On the indexer, open $SPLUNK_HOME/etc/system/local/server.conf and look for  [license] master_uri = https://<license-master-host>:8089 Replace <license-master-host> with the License Master’s IP or FQDN. If this is missing or incorrect, update it. From the indexer’s host : ping <license-master-host> Test the management port : telnet <license-master-host> 8089 . If it doesn’t connect, troubleshoot firewalls, network routes, or confirm the License Master is listening (netstat -tuln | grep 8089 on the License Master). 
Hi @SN1  Are you able to share the output of this page from the server with an issue, please? Go to https://yourSplunkInstance/en-US/manager/system/licensing  Does the license is showing as valid?... See more...
Hi @SN1  Are you able to share the output of this page from the server with an issue, please? Go to https://yourSplunkInstance/en-US/manager/system/licensing  Does the license is showing as valid? Or does it show connection to license server? Please let me know how you get on and consider adding karma to this or any other answer if it has helped. Regards Will
@SN1   The indexer can’t reach the License Master due to firewall rules, network outages, or DNS resolution problems. Please check with your network team. 
Hi @nnkreddy  Each of the objects you want to be next to each other horizontally need to be in <panel> within the same <row>. This will produce something like below, is this what you are looking to... See more...
Hi @nnkreddy  Each of the objects you want to be next to each other horizontally need to be in <panel> within the same <row>. This will produce something like below, is this what you are looking to achieve?     <form version="1.1" theme="light"> <label>xmltest</label> <fieldset submitButton="false"></fieldset> <row> <panel id="input1"> <input type="text" token="tk1" id="tk1id" searchWhenChanged="true"> <label>Refine further?</label> <prefix> | where </prefix> </input> </panel> <panel id="html1"> <html id="htmlid"> <p>count: $job.resultCount$</p> </html> </panel> </row> <row> <panel id="panel1"> <table> <search> <query>|makeresults | eval msg="test" | stats count by msg</query> </search> <option name="drilldown">cell</option> </table> </panel> </row> </form>   Please let me know how you get on and consider adding karma to this or any other answer if it has helped. Regards Will  
we dont have any UF on the windows machine, we are receiving the logs through UDP
The server count is huge and we are receiving the logs from the LB to our syslog server.