We have 1 indexer and 1 search head in our Splunk environment. Since this morning, after every search is run, a 'Server Error' message is seen, both on the search head, as well as the indexer. The search continues to run and even completes without any real error. However, this message always appears when the search completes/ is paused/ finalized. Even during a failed login attempt (incorrect credentials), the 'Server Error' message is seen. This message is also seen when scheduled searches and alerts are run. What could be the reason for the 'Server Error' message? I skimmed through the logs, and there is nothing unusual there. ... View more

I have configured a scheduled search, which runs and triggers an alert if the search returns >0 results. This search is scheduled to run every 30 minutes. When the alert is triggered, it sends an email and runs a script. Once the alert is triggered, I want it to be throttled for the rest of the day. So the next day, from 12:00 AM on-wards, the throttle condition needs to be reset and the alert should be again triggered if and when the alert condition is met. To implement this, I have set alert.suppress = 1 alert.suppress.period = 1d My understanding is, that '1d' should throttle the alert for the remaining of that day. However, I often find that the scheduled search returns a result, say at 4PM, while my alert is only triggered during a later search in the day (say 6:30PM). Is this due to the throttling that may have been introduced during the previous day's triggered alert? When I check scheduler.log, i see the following entry: sid="scheduler__xxx__search__RMD5623ae5e44f926665_at_1435674600_314", suppressed=1, thread_id="AlertNotifierWorker-0" 06-30-2015 07:45:01.593 -0700 INFO SavedSplunker - savedsearch_id="nobody;search;<SearchName>", user="xxx", app="search", savedsearch_name="<Search Name>", status=success, digest_mode=1, scheduled_time=1435675500, dispatch_time=1435675501, run_time=0.250, result_count=1, alert_actions="" I see suppressed=1 even when result_count=1 during the first scheduled search of the day. I know that the scheduled search itself doesn't have any errors, since the alert is triggered (email sent and script run) as expected, at the right time, on some days. Am I understanding it wrong? How do I ensure that the alert is always triggered the first time the scheduled search returns a result, on any day? ... View more

We recently installed the Splunk on Splunk app on our search head and the TA-sos-win app on the Indexer. The SOS panel for Indexer CPU usage is at 100% almost always, while the panel for Search head CPU usage is at 2-5%. When I log into the actual servers, and open the Task Manager, the CPU utilization for both the servers is between 10-30%. Why is there such a big mismatch in the two values? The SOS panel is probably right in identifying the high load, since we have been observing slow indexing speeds on the Indexer. What could be the reason for high cpu usage on the Indexer? Any suggestions to bring it down? ... View more

I have an xml file with the following format: <?xml version="1.0"> <ChangeHistory> <Left_2015_01_01_9_45_00___Right_2015_02_02_9_50_00> <field A status="modified value=123"> <field B xxxxxxx> <value 1> <value 2> <field C xxxx> </field C> </field B> </field A> </Left_2015_01_01_9_45_00___Right_2015_02_02_9_50_00> <Left_2015_02_02_9_50_00___Right_2015_03_03_9_50_00> . . . </Left_2015_02_02_9_50_00___Right_2015_03_03_9_50_00> </ChangeHistory> Every time my batch job runs, I have a new entry of <Left_previous_run_time___Right_curent_run_time> ... </Left_previous_run_time___Right_curent_run_time> When I monitor a directory with such a file, every time a new entry is added, since the addition of the event is within the "ChangeHistory" tags and not at the end of the file, the entire gets re indexed. Is there any way to add just the new event to my index? I have come across the LINE_BREAKER setting. But my understanding is that this just breaks the xml file into different events, which I am able to do even with BREAK_ONLY_BEFORE setting. How do I prevent re-indexing of the entire file in this case? ... View more

I have configured LDAP authentication with Active Directory on Splunk. We are still waiting on the group to role mapping, so currently we have mapped individual users to specific roles. However, 1 of the 5 users we have currently mapped is unable to login. When I add his username to the authentication.conf file, I see his username, Full name and email address under Settings->Access controls->Users When he tries to log in, he gets "Invalid username or password" and immediately after that, his details are no longer visible under Settings->Access controls->Users splunkd.log only shows user=xxx action=login status=failure reason=user-initiated The password can't be invalid since he logs into his local machine with the same credentials. The other 4 users are able to log in successfully. Also, since I can see his 'Full Name' under Settings->Access controls->Users , I don't think its a problem with his display name, either. ... View more

I'm using the collect command to copy a set of frequently queried events to a summary index. When I search for the following: index= original_index | table _raw I see a timestamp prefixing the log information. I run the following query to populate my summary index: index= original_index field=value | table field1, field2, field3 | collect index=summary_index I use the table command to preserve the extracted fields in the summary index. However, when i run the above query, I lose the actual timestamp of the event. Instead, all events in the summary index have the current system time as the timestamp. I know this happens when the _raw field does not have time information, but that is not the case here. How do i preserve the timestamp of the event in the summary index? ... View more

Our environment consists of 1 indexer and 1 search head. Our indexer is currently indexing close to 400GB per day, since we are catching up on historical data. In another week, this should reduce to about 20GB per day. Meanwhile, we are running a few saved searches on the search head, which would normally run for a few hours. However, we always see the error: Timed out waiting for peer xx-xxxx-xxx. If this occurs frequently, receiveTimeout in distsearch.conf may need to be increased. Search results might be incomplete! I have increased receiveTimeout to 900s. I am planning on adding the following stanza to distsearch.conf to reduce the knowledge bundle size: [replicationWhitelist] allConf = *.conf allSpec = *.spec I know that there may be network issues that are causing the problem. Are there any commands I can use to check the network health between the search head and indexer? Any other suggestions to avoid this message would be welcome. ... View more