Ingestion does not survive reboot

[ShaunBaker]

I have a CentOS7 Splunk 7.x build using the A3sec pfSense app, snort for splunk app and missile app. the A3sec pfSense app does not resume ingesting logs if the VM has been down/restarted. I ensured:
- not firewalld
- tcpdump shows the syslog is flowing in on UDP 514
- Rebuilt the 514 data input
- Restarted the VM and splunk service (this will cause just a brief grab of a few logs as they flowed in)
- Restarted the pfSense router (this fixed that the snort logs on UDP 1514 were having the same issue)

I even disabled firewalld for trouble shooting sake. What other steps should I take?

[ShaunBaker]

Still having issues, the server just will not ingest syslog into the A3Sec app, though the Snort for Splunk app is pulling events in. So I checked out the splunkd.log and there are two log events that are interesting, and googling them hasn't found a silver bullet yet.

A bunch of:
06-27-2018 19:23:58.543 -0700 WARN DateParserVerbose - A possible timestamp match (Sat Setp 8 18:46:43 2001) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Contex: source=udp:514 | host xxx.xxx.x.x | pfsense_syslog |

Earlier in the log there are a lot of:
06-26-2018 20:51:26.834 -0700 WARN DateParserVerbose - Failed to parse timestamp in the first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. Defaulting to time stamp of previous event (Tue June 26 08:27:00 2018). Context: source=udp:514 | host =xxx.xx.x.x | pfsense_syslog

The CentOS7 VM the server is running on has the correct time, and I checked the pfsense syslog in pfsense's WebUI and they are correct as well.

If this happened randomly I would have assumed the props.conf might had become outdated to a pfsense log output change, but this happened when my pfsense router lost power and the splunk server (and rest of the house) had no connection.

[rteja9]

You should enable boot-start for Splunk service. This will make sure Splunk starts after VM is down/restarted.
Here is the documentation on how to enable it,
https://docs.splunk.com/Documentation/Splunk/7.1.1/Admin/ConfigureSplunktostartatboottime
You need to run $SPLUNK_HOME/bin/splunk enable boot-start.

[ShaunBaker]

Thats a great tip reduce steps when coming back up, but that does not fix the problem of splunk being up and running and it simply does not want to index syslog udp 514 when up and running.