Hi All, we wanted to upgrade Splunk Enterprise clustered environment from 8.2.2  version to 8.2.6 and have this question running in my mind what splunk precedence rules should be followed to upgrade splunk clustered environment.   Can anyone guide me on what order should we need to upgrade the splunk instances. Sequence of order  1) Cluster Master 2) Indexer Peers 3) Search head captain  4) Search peers 5) Deployer  6) Deployment Server  7) Heavy Forwarders 😎 UF  Back-up SPLUNK_HOME/etc.  ... View more

Hi Team, I wanted to wirte query to find the Splunk agent version of specific set of hosts in our environment, I had tired the below link to find out version detail for all UF uisng the below link. https://community.splunk.com/t5/Getting-Data-In/How-can-I-find-a-listing-of-all-universal-forwarders-that-I-have/m-p/324298 But I am unable to segregate to specific set of hosts.  So could anyone let me know how to wirte a query to fetch the version details.   Thanks in Advance.  ... View more

Hi  We are planning to upgrade Splunk to 8.2.6.1 but I am unable to find the release notes in the Splunk site. And what is the difference between 8.2.6.1 version with the latest splunk 9.0 version. Current Splunk version is 8.2.2 (On-Prem)  Please provide me the exact link to find this details.   ... View more

Hi All, I am trying to built the parsing stanza for one of the data, while testing I am getting an pop-up message stating that "could not use the strptime to parse timestamp from “2022-26-05T11:29:57”.   As soon as I apply the Time_Format stanza the Splunk is throwing the message.  I am not sure what I am missing here.  so could you please help me resolving this issue.   Event details: <Event CompactMode="1" sEventType="OpResult" dwBasicEventType="9" dwAppSpecificEventID="5000" sEventID="EVENT_ID_SCHEDULER_STARTED" sOriginatingApplicationName="RED Identity Management Console" sOriginatingApplicationComponent="Scheduler" sOriginatingApplicationVersion="5.5.3.0" sOriginatingSystem="XXXXXXXXXXXXX" sOriginatingAccount="XXXX\XXXXX" dtPostTime="2022-26-05T11:29:57" sMessage="RED Identity Management Console (running as user XXXX\XXXXX) on system XXXXXXXXXXXXX; - background processor started"/> Props stanza SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+)\<Event NO_BINARY_CHECK=true TIME_PREFIX=dtPostTime\=\" TIME_FORMAT=%Y-%m-%dT%H:%M:%S MAX_TIMESTAMP_LOOKAHEAD=20 Event Details: [5/26/2022 4:09:55 PM UTC] Note: Unknown provider type; cannot verify object name 'tbl_BaseJobInfo' valid for data store. Props.conf SHOULD_LINEMERGE=false LINE_BREAKER=([\r\n]+)\[\d+\/\d{2}\/\d{4}\s\d+\:\d{2}\:\d{2}\s[^\]]+\] NO_BINARY_CHECK=true disabled=false TIME_PREFIX=^\[ TIME_FORMAT=%m-%d-%Y %I:%M:%S %p %s MAX_TIMESTAMP_LOOKAHEAD=25 ... View more

Hi All,  We have request from a Cybersecurity team to monitor the Windows Event Viewer logs in Splunk, my question is how to configure the monitoring stanza to get the event data into splunk. Event Viewer (Local) --->Application and Services Logs --> OpenSSH --> Admin Event Viewer (Local) --->Application and Services Logs --> OpenSSH --> Operational  When I check the properties to find the exact Log Path details I could see like this  %SystemRoot%\System32\Winevt\Logs\OpenSSH%4Operational.evtx %SystemRoot%\System32\Winevt\Logs\OpenSSH%4Admin.evtx My question is how to write the monitoring stanza for this path and define the sourcetype for the same. [WinEventLog://Application/OpenSSH/Operational] sourcetype=winEventLog:OpenSSH:Operational index=test disable=0 [WinEventLog://Applicaion/OpenSSH/Adminl] sourcetype=winEventLog:OpenSSH:Admin index=test disable=0 Please guide me on this  ... View more

Hi All,  We have concern raised by one of our application team as they could see incorrect data in their dashboard, When validated the same by looking into the source of the file where the splunk is reading it, we noticed that there is no actual data present in the log source. Problem:  Getting incorrect data ingested into Splunk in the status field value [13/Apr/2022:06:33:03 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8444 status=2 [13/Apr/2022:04:30:01 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=2 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=2 [12/Apr/2022:09:11:37 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8444 status=2 Actual data present in the application server  Path:/var/mware/logs/xxx/localhost_access_log.2022-04-12.11.log [12/Apr/2022:11:10:26 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=219 response=1 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=0 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=219 response=0 [12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1 Monitoring Stanza details: [monitor:///var/mware/logs/*/*localhost*] sourcetype = access_combined index = test disabled = 0 ignoreOlderThan = 1d blacklist=\.(gz)$ Splunkd.log : There is no significant ERROR|WARN|INFO related to this issue found. So could you please guide me what will be the reason why Splunk is ingesting an incorrect information when there is no actual data present in the source and also guide me how to troubleshoot this issue. ... View more

Hi All, One of our Cyber security person facing a strange issue while trying to access the data from the Splunk search portal. Initial level of troubleshooting the issue we found that Roles/Permission are not syncing but later we found that Roles/Permission are auto changing frequently. We could not find any ERROR/WARN in the splunkd.log, so not sure how to troubleshoot this issue Splunk version : 8.2  OS: Linux  Authentication mode: LDAP  Environment: Splunk distributed Production Environment.  Problem statement:  Roles/Permission are not syncing properly its getting auto changed frequently.  Kindly let me know what are steps we should follow to troubleshoot this type of issue. ... View more

HI Team, I am facing an issue with few of the servers which client had request to on-board new set of log data into splunk.  We had deployed the monitoring stanza & Parsing stanza by updating an existing app and app was successfully deployed into their respective servers. But we are unable to see the data ingest happening from the new monitoring stanza in Splunk. When troubleshooting could see this INFO related to the monitoring  stanza in _internal logs. Apart from this is INFO, there is no other messages or Events related to the below source found in the _internal logs.   Monitoring Stanza details [monitor:///usr/local/tet/t12/var/was/log/server.log] sourcetype = usr:genericapp:server index = test_index disabled = 0 ignoreOlderThan = 14d Parsing stanza: [usr:genericapp:wfserver] NO_BINARY_CHECK=true LINE_BREAKER=([\r\n]+)\d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3} TIME_PREFIX=^ TIME_FORMAT=%Y-%m-%d %H:%M:%S.%3N MAX_TIMESTAMP_LOOKAHEAD= 23 SHOULD_LINEMERGE=false internal logs: 1:40:04.292 PM 02-25-2022 13:40:04.292 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor:///usr/local/tet/t12/var/was/log/server.log Kindly guide me to fix this .   ... View more

Hi All, We have request from end user for monitoring a CSV files which are placed in the file share folder and there is no splunk agent running in the file share machine. Example :  Server01  is the actual application which is generating a report and Server02 is the file share machine where the reports are stored and shared with the user.  \\fileshare\power\Powerfile\TO\IAM\Export Files\OSBD - Terminated Users List.csv  --  Location of the file to be monitored in splunk.   Above mentioned path has required permission to access the file from the share drive In Server01 we have splunk UF agent running and inputs.conf configured for monitoring the log files present in the server.   Question:  Can we use the same app which is present in the server01 to monitor the file present in the server02 as it has the required permission to access the file from that server. Stanza in inputs.conf:  [monitor://\fileshare\power\Powerfile\TO\IAM\Export Files\OSBD - Terminated Users List.csv] sourcetype = powerfile:power:osbd_terminateduser index = indexname disabled = 0 ignoreOlderThan = 14d kindly guide me how to get this share folder to be monitored in splunk. ... View more

HI All, I am looking for host mapped to a list of app in the serverclass using the btool command but unfortunately I am unable  to get details.   ./splunk cmd btool serverclass list --app="test-fwd-p" --debug | grep -i "testhost*"     Please correct me if the above syntax is in correct.  ... View more

Hi All, Currently one of the windows user contacted us and informed  that he could notice that Splunk UF agent is failing frequently in his machine. When investigated the issue, we could see the following error details in Splunk _internal logs.  Component= ExecProcessor ========================= 01-17-2022 09:22:11.436 +0000 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" ERROR splunk-winevtlog - WinEventLogChannel::~WinEventLogChannel: Failed to checkpoint for channel='Windows PowerShell' 01-17-2022 09:22:11.436 +0000 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" ERROR splunk-winevtlog - WinEventLogChannel::saveBookMark: Failed to update Windows Event Log bookmark, channel='Windows PowerShell 01-17-2022 09:22:11.436 +0000 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" ERROR splunk-winevtlog - WinEventLogChannel::~WinEventLogChannel: Failed to checkpoint for channel='Security' component=AuthenticationManagerSplunk ========================================= 01-17-2022 09:22:19.839 +0000 ERROR AuthenticationManagerSplunk - Either password or seed file not found! No users configured! component=Metrics ==================== 01-17-2022 09:22:20.245 +0000 ERROR Metrics - Metric with name thruput:idxSummary already registered component=TcpOutputFd ====================== 01-17-2022 04:45:00.175 +0000 ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host. component=PipelineComponent ============================= 01-17-2022 05:21:48.213 +0000 ERROR PipelineComponent - Monotonic time source didn't increase; is it stuck? component=FileClassifierManager ================================== 01-17-2022 09:22:23.780 +0000 WARN FileClassifierManager - The file 'C:\Program Files\SplunkUniversalForwarder\var\log\splunk\C__Program Files_SplunkUniversalForwarder_bin_splunk-winevtlog_exe_crash-2021-08-13-08-22-30.dmp' is invalid. Reason: binary component=TailReader ========================== 01-17-2022 09:22:23.780 +0000 INFO TailReader - Ignoring file 'C:\Program Files\SplunkUniversalForwarder\var\log\splunk\C__Program Files_SplunkUniversalForwarder_bin_splunk-winevtlog_exe_crash-2021-08-13-08-22-30.dmp' due to: binary component=WatchedFile ========================== 01-17-2022 09:22:23.498 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\C__Program Files_SplunkUniversalForwarder_bin_splunk-winevtlog_exe_crash-2021-10-01-08-22-12.log'.   I have checked the truncate value and found all are within the control limit except the below sourcetype  Default Value is set to 10000  WinEventLog:Microsoft-Windows-PowerShell/Operational 21132   Splunk Agent version is 7.0  Splunk Enterprise indexer version is 8.2.2  Please guide me what kind of troubleshooting steps needs to taken in-order to resolve this issue.       ... View more

Hi All, How to search the internal logs of the remote agent (UF) node via Splunk Portal ?  I am trying to troubleshoot why the logs are not ingested into Splunk from the remote agent node, I did simple search query from the search head console. index="_internal"  sourcetype="splunkd.log"  host="test1"  but unable to get any result, so please do let me know how to search the internal log details from the search head portal. When I log into the UF server I can see the following information  Error | Warn | Info details  from the splunkd.log  but my intension is to check the same from the Splunk console . Kindly guide me on the same.   ... View more