I would like to leave the central loghosts in place and move the data to my new splunk server for analysis.
From what I have read, it seems like my options are:
I started on option #1 but stopped when I couldn't get the syslog-ng port to compile with "--enable-spoof-source" under OpenBSD 4.9.
Any recommendations or other options I should be considering?
Assuming you can get the Splunk forwarder to run on OpenBSD -- which you may not be able to -- I would use approach #2. The cleanest approach by far is to use a Splunk forwarder reading the disk files created by syslogd. This gives you added resiliency during maintenance and certain outage situations.
You'll be able to upgrade the splunk installations without losing any of your syslog data (because it will be destaged to disk by syslogd), and if there is an outage either on the indexer or the network between the forwarder and indexer you won't lose any data there either.
A substantial issue, though, is that Splunk does not provide a native OpenBSD binary. And, no one has reported success using OpenBSD's FreeBSD compatibility ABI.
If you can't get a forwarder to run on the syslog host, you could run Splunk on Linux (or something else) and have it read from the NFS-exported log directory.
Definitely option #2, if there is any way to make it work.