One of the six indexers we have were unresponsive today. I couldn't login through the web interface and ssh'ing to the server was very slow. I figured it's an OS problem, rebooted the server, and things seem to be clear. While looking at the logs, I noticed a number of splunkd dying. Is that normal? The server OS is RHEL 7x. Dec 28 10:56:21 PRDSRV01 systemd[1]: systemd-journald.service: got WATCHDOG=1 Dec 28 10:56:23 PRDSRV01 systemd[1]: Received SIGCHLD from PID 120270 (splunkd). Dec 28 10:56:23 PRDSRV01 systemd[1]: Child 120270 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:25 PRDSRV01 systemd[1]: Received SIGCHLD from PID 120277 (splunkd). Dec 28 10:56:25 PRDSRV01 systemd[1]: Child 120277 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:27 PRDSRV01 systemd[1]: Received SIGCHLD from PID 121218 (splunkd). Dec 28 10:56:27 PRDSRV01 systemd[1]: Child 121218 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:29 PRDSRV01 systemd[1]: Received SIGCHLD from PID 121211 (splunkd). Dec 28 10:56:29 PRDSRV01 systemd[1]: Child 121211 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:31 PRDSRV01 systemd[1]: Received SIGCHLD from PID 121697 (splunkd). Dec 28 10:56:31 PRDSRV01 systemd[1]: Child 121697 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:32 PRDSRV01 systemd[1]: Received SIGCHLD from PID 121431 (splunkd). Dec 28 10:56:32 PRDSRV01 systemd[1]: Child 121431 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:34 PRDSRV01 systemd[1]: Received SIGCHLD from PID 121228 (splunkd). Dec 28 10:56:34 PRDSRV01 systemd[1]: Child 121228 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:36 PRDSRV01 systemd[1]: Received SIGCHLD from PID 122819 (splunkd). Dec 28 10:56:36 PRDSRV01 systemd[1]: Child 122819 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:38 PRDSRV01 systemd[1]: Received SIGCHLD from PID 121324 (splunkd). Dec 28 10:56:38 PRDSRV01 systemd[1]: Child 121324 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:40 PRDSRV01 systemd[1]: Received SIGCHLD from PID 120159 (splunkd). Dec 28 10:56:40 PRDSRV01 systemd[1]: Child 120159 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:42 PRDSRV01 systemd[1]: Received SIGCHLD from PID 120296 (splunkd). Dec 28 10:56:42 PRDSRV01 systemd[1]: Child 120296 (splunkd) died (code=exited, status=255/n/a) Dec 28 10:56:44 PRDSRV01 systemd[1]: Received SIGCHLD from PID 123182 (splunkd). Dec 28 10:56:44 PRDSRV01 systemd[1]: Child 123182 (splunkd) died (code=exited, status=255/n/a) ... View more

I am trying to build a dashboard with multiple Text inputs that are optional. Say I have 4 Text input boxes: UserID, PID, SPID, and IP, with Default/Load values being "*". The initial search will return all results. But say I want to just search a UserID and want to bring back all data when UserID = "This_is_a_user". If I'm searching only on UserID, I don't want the other fields being searched on. How can I do this? (index=AUTH OR index=EXTERNAL_APP) UserID=$userid$ PID=$pid$ SPID=$spid$ IP=$ip$ | table AppID _time UsreID Session IP SystemID PID SPID Page Function FName MI LName Address Email | sort _time ... View more

For one of our syslog devices, some events that come through only contain the syslog datetime format, while there are others that contain the syslog datetime AND a "timestamp=" field at the end of the event. What would be the best to setup timestamp recognition where it first reads the "timestamp=" field and if the event does not contain that field, then for it to look at the syslog datatime at the beginning of the event. The timestamp field and the syslog datetime are two different formats too. See below for an example. Sep 23 23:59:57 2016 aa_wlc_01 wms[1234]: <123456> <WARN> <aa_wlc_01 192.168.000.000> |ids| AP(aa:aa:aa:aa:aa:aa@aa-aa-aa): Wireless Bridge: An AP detected a wireless bridge between transmitter aa:aa:aa:aa:aa:aa and receiver aa:aa:aa:aa:00:00. SNR value is 25. Additional Info: BSSID:aa:aa:aa:aa:aa:aa; Channel:1. Sep 23 23:59:44 2016-09-23 23: 59:44,5 192.168.111.111 CPPM_RADIUS_Accounting_Detail aaa 1 0 id=aaa,session_id=aaa-01-aaa,acct_session_id=aaa\\aaa-aaa,type=aaa,attr_name=aaa-Location-Id,attr_value=aaa.6-aaa-aaa,timestamp=2016-09-23 23:58:35-07 ... View more

I have the following syslog data and I need help extracting the timestamp field at the end of the event: Sep 6 06:07:20 2016-09-06 06: 07:20,165 192.168.0.0 CPPM_Dashboard_Summary 17000 1 0 session_id=A00001234-01-56bcde1f,req_source=RADIUS,user_name=XXXXXXXXXX,service_name=ABC802.1X,alerts_present=0,nas_ip=192.168.1.1,nas_port=0,conn_status=Unknown,login_status=ACCEPT,error_code=0,mac_address=12a3123456n2,timestamp=2016-09-06 06:05:48-07,write_timestamp=2016-09-06 06:05:50.133529-07 Sep 6 14:24:18 2016-09-06 14: 24:18,179 192.168.109.102 CPPM_System_Stat 11115 1 0 id=11112,swap_size_used=65757,slash_size_used=11166596,swap_memory_avail=6079780,system_memory_avail=3737736,cpu_raw_user=1,cpu_raw_nice=0,cpu_raw_system=13,cpu_raw_idle=87,mgmt_inf_status=up,data_inf_status=down,uptime=8291850,timestamp=2016-09-06 14:24:07.156076-07 I'm using the following props on the indexers and heavy forwarder, but the timestamp is still not parsing out: TIME_PREFIX = ,timestamp TIME_FORMAT = %Y-%d-%m %H:%M:%S.%6N MAX_TIMESTAMP_LOOKAHEAD = 350 ... View more

We're having issues when we delete some data (with |delete) and after an indexer restarts in the clustered environment, some of the data replicated again. I did some research and found that this was a previous bug (SPL-100516). Has it been fixed? ... View more

I have an alert that runs every hour at the half hour mark. So at 1:30, 2:30, etc... When I run the timechart command, "| timechart count span=1h", it brings back the count of events for each hour, but I want the count of event from 1:30 to 2:30. How can I accomplish this? (index=ABC Page="go.aspx" Refer="*signup_pro.aspx" UserName=*) OR (index=ABC Page="signup_pro.aspx" (SID3Type=A1 OR SID3Type=A2) UserName=* SID1=* SID3=*) OR (index=ABC Page="Registration.aspx" UserName=*) | transaction UserName, SessionSID maxspan=60m | table _time, UserName, SID3, SID3Type, SID1, FName1, LName1, Email, Page | sort - _time | search UserName=* SID1=* SID3=* SID3Type=* FName1=* LName1=* Email=* Page=go.aspx | rename UserName AS UserSID, SID1 AS "SID1 (SSN)", FName1 AS FirstName, LName1 AS LastName | fields - Page ... View more

My search is not returning any results.. index=_internal source="/opt/splunk/var/log/splunk/metrics.log*" sourcetype="splunkd" fwdType="*" [|inputlookup Servers.csv | return 9999 $name] | dedup sourceHost | table hostname, sourceHost | search [| ldapsearch basedn="OU=ABC,OU=Servers,OU=SMG,DC=ZZZ,DC=COM" search="(&(objectClass=computer))" attrs="name,distinguishedName" | table name | sort name | outputlookup Servers.csv] Here was my logic which doesn't work (brain is fried for the day): ldapsearch queries ldap for the list of host names and creates a lookup The lookup is then used in the beginning of the search to find all hosts that are reporting to Splunk. Does that seem correct? ... View more