I was having problems with one of my heavy forwarders (splunk 6.6.3) running on Windows 2008, so I noted what inputs I had, uninstalled and then installed version 7.0.1. After adding my configurations and restoring my connections, I started Splunk and got the following: Checking prerequisites... Checking http port [8000]: open Checking mgmt port [8089]: open Checking appserver port [127.0.0.1:8065]: open Checking kvstore port [8191]: open Checking configuration... Done. Checking critical directories... Done Checking indexes... Validated: _audit _internal _introspection _telemetry _thefishbu cket <snip> Done Bypassing local license checks since this instance is configured with a remote license master. Checking filesystem compatibility... Done Checking conf files for problems... Done Checking default conf files for edits... Validating installed files against hashes from 'C:\Program Files\Splunk\ splunk-7.0.1-2b5b15c4ee89-windows-64-manifest' All installed files intact. Done All preliminary checks passed. Starting splunk server daemon (splunkd)... Splunkd: Starting (pid 3540) Done Waiting for web server at https://127.0.0.1:8000 to be available................ ................................................................................ ........................................................................ Done This took about 30 minutes to before I see "Done". But the web server wasn't running. Checking the log files I had three messages that seemed to relate: ERROR UiPythonFallback - Appserver running on port 8065 exited unexpectedly: exited with code 1 ERROR UiHttpListener - An applicaiton server has exited unexpectedly, web UI cannot be used until it is restarted WARN UiHttpListener - Web UI now stopped (yes, application is spelled wrong in the logs) I didn't have anything on that port when I checked with netstat and via the resource monitor. I tried rebooting to no avail. I found the following posts: https://answers.splunk.com/answers/545000/slow-splunkweb-startup-caused-by-splunk-instrument.html https://answers.splunk.com/answers/563807/why-does-splunkweb-in-662-take-so-long-to-start.html https://answers.splunk.com/answers/211525/how-to-troubleshoot-why-we-are-getting-appserver-p.html and tried changing those changes, but I still had problems. After seeing this post: https://answers.splunk.com/answers/616294/unable-to-start-splunk-1.html I looked around but didn't see a repair option for splunk. I turned on debuging (http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Enabledebuglogging) and found the following: 7:00:52.429 PM 03-07-2018 19:00:52.429 -0500 ERROR KVStoreBulletinBoardManager - Failed to start KV Store process. See mongod.log and splunkd.log for details. host = splunk-04 message = Failed to start KV Store process. See mongod.log and splunkd.log for details. 3/7/18 7:00:52.429 PM 03-07-2018 19:00:52.429 -0500 ERROR KVStoreConfigurationProvider - Could not start mongo instance. Initialization failed. host = splunk-04 message = Could not start mongo instance. Initialization failed. 3/7/18 7:00:52.429 PM 03-07-2018 19:00:52.429 -0500 ERROR KVStoreConfigurationProvider - Could not get ping from mongod. host = splunk-04 message = Could not get ping from mongod. 3/7/18 7:00:44.067 PM 03-07-2018 19:00:44.067 -0500 ERROR KVStoreBulletinBoardManager - KV Store changed status to failed. KVStore process terminated. host = splunk-04 message = KV Store changed status to failed. KVStore process terminated. 3/7/18 7:00:44.067 PM 03-07-2018 19:00:44.067 -0500 ERROR KVStoreBulletinBoardManager - KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details. host = splunk-04 message = KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details. 3/7/18 7:00:44.067 PM 03-07-2018 19:00:44.067 -0500 ERROR MongodRunner - mongod exited abnormally (exit code 100, status: exited with code 100) - look at mongod.log to investigate. host = splunk-04 message = mongod exited abnormally (exit code 100, status: exited with code 100) - look at mongod.log to investigate. Which led me to these posts: http://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/StartSplunk#Start_Splunk_Enterprise_on_Windows_in_legacy_mode https://answers.splunk.com/answers/514443/after-editing-the-kv-store-for-my-custom-app-why-d.html#answer-521153 After I changed the permissions on the whole splunk folder, I restarted splunk AGAIN and I have zero ERRORs in my logs, but it STILL takes over 30 minutes to start. And the web interface still won't work. I then saw these errors: WARN HttpListener - Socket error from 127.0.0.1 while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number ERROR UiPythonFallback - Appserver at http://127.0.0.1:8065 never started up! and searched and found these posts, but they don't seem to be of much help. https://answers.splunk.com/answers/7899/splunkweb-fails-to-start-timeout-when-binding-to-port.html https://answers.splunk.com/answers/507379/how-to-resolve-splunk-web-not-starting-after-the-h.html It IS forwarding logs, so the basic functionality is there. The logs are windows server logs being collected by wmi calls. But the logs are about 10 minutes behind. I checked the firewall and for splunkd, I have any any for incoming. But nothing for outgoing. (hopefully that is the simple answer?) So I'm asking if anyone has any other suggestions? Thanks ... View more

I have a small Splunk installation and I'm adding a new HF. I just upgraded my current servers to 7.0.1. When I performed the new install, the first thing I notice was that the installer no longer asked for a deployment server nor did it ask for a license server. OK, so after the install finished I went to Settings -> Licensing and tried to connect the server to my current licensing server, but I got an error message saying the pass4SymmKey under [general] in the server.conf was different than the licensing server and needed to be the same. OK, so I went to my licensing server and copied the key and pasted it into the same place in my new HF. I restarted the HF and tried again, but I got the same error. I checked this page on installing Splunk (http://docs.splunk.com/Documentation/Splunk/7.0.2/Installation/InstallonWindows) but there was no mention of adding a new server to a current installation. After a few googles, I haven't found the Splunk document page that talks about this. So, I was wondering two things: 1. Has anyone else seen this error? and 2. if someone could point me to the document page for joining a new 7.x server to a current installation? Thanks ... View more

I have a file that I am monitoring on a Heavy Forwarder(HF). The file is JSON logs. On the HF I have the following props.conf: [EC-json] KV_MODE=JSON TIME_PREFIX="timestamp":" TIME_FORMAT=%Y-%m-%dT%H:%M:%S SHOULD_LINEMERGE=false TRUNCATE=0 After the file gets to the indexers, from the SH, I am trying to create several search time extractions. I first tested from the search bar using this search: sourcetype=EC-json | rex field =_raw "userid\":.+?,ou=(?<user_org>\w+)," | rex field = _raw "SourceName.+?:.+?\/\/.+\/(?<PDF>.+?\.pdf)" This was successful, I was able to create two new fields user_org and PDF. Then I tried using props.conf in /etc/apps/search/local/ on the SH: [EC-json] EXTRACT-user_org = userid\".+?,ou=(?<user_org>\w+), EXTRACT-PDF = SourceName\".+?:.+?\/\/.+\/(?<PDF>.+?\.pdf) Here is a sample of my data: {"timestamp":"02/16/2018 08:02:23","Accountid":"userj", <snip> ,"SourceName":"https://share.org.com/sites/reports/ORGReports/report1.pdf","userid":"cn= joe user,ou=SOC,ou=org,ou=company,ou=us"} I tried using the suggestions here: https://www.splunk.com/blog/2016/06/28/eureka-extracting-key-value-pairs-from-json-fields.html and added the following to the props.conf on my SH to pull all the information from the userid: EXTRACT-KVPS = (?:\\[rnt]|:")(?<_KEY_1>[^="\\]+)=(?:\\")?(?<_VAL_1>[^="\\]+) But that doesn't seem to pull the info into the right fields. And all I care about is the first OU anyway. Can someone help with my props.conf syntax? Do I need to escape the quote after userid and SourceName or not? Thanks. ... View more

Given a representative sample of my logs: Jan 25 14:19:20 1.1.1.1 64: Jan 25 22:19:19.281: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx Jan 25 14:19:15 1.1.1.1 74: Jan 25 22:19:15.282: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx Jan 25 14:18:56 1.1.1.1 79: Jan 25 22:18:56.285: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx Jan 25 14:18:25 1.1.1.1 66: Jan 25 22:18:25.284: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx Jan 25 14:18:15 1.1.1.1 62: Jan 25 22:18:15.274: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx Jan 25 14:17:22 1.1.1.1 34: Jan 25 22:17:22.287: %LINK-3-UPDOWN: xxxxxxxxxxxxxxxxxxxxxxxxxx These logs are being written to a file on my Heavy Forwarder and the HF is monitoring the file and sending to the indexers. Currently I'm testing a new source on a new HF. The same configuration is being used on another HF and I've just copied the props.conf from the previous HF to the new one. But I'm having some weird behavior when trying to extract the time field. I want to use the second timestamp as the time. Here is my props.conf: TIME_PREFIX = \w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}\s+\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s\d+:\s+ MAX_TIMESTAMP_LOOKAHEAD = 15 TIME_FORMAT = %b\s+%d\s+%H:%M:%S.%3N Splunk shows the time for all events as _time = "Jan 25 2018 5:19:19 PM" Which is the time of the top event and the timestamp of the file it is reading from. Which makes me thing that Splunk could not parse the timestamp. I've tried the following TIME_PREFIX modifications: TIME_PREFIX = ^(?:[^\n]* ) {5} TIME_PREFIX = \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s\d+:\s+ and I've tried TIME_PREFIX = ^ to try to get the first timestamp. Same issue. Did I make a config mistake, syntax? I saw other people had found an error that says "Could not use strptime to parse timestamp from xxxxx" , but I'm not sure where splunk would write that (splunkd.log?).... Thanks for any suggestions ... View more

This error is occurring about every .5 second in splunkd.log on one of my Search Heads: WARN MongoModificationsTracker - Could not load configuration for collection 'acknotescoll' in application 'TA-FireEye_v3'. Collection will be ignored. I just upgraded to 6.6.3, but this error has been going on unnoticed for some time. ... View more