topic Re: How Can I One-Time Index a File with "Normal" Processing? in Getting Data In

How Can I One-Time Index a File with "Normal" Processing?

I_am_Jeff — Tue, 13 Sep 2011 15:39:34 GMT

Short statement: I want to one-time import a file to splunk and have the events processed/indexed/identified/tagged as if it was my normal log file.

Splunk is my enterprise logger and is happily indexing my monitored files, one of which is /opt/apps/splunk-index01/splunk-log. The file is written by syslog-ng. The events are identified by host, actually IPs, and source and sourcetype. Everything you'd expect from Splunk. Many of my users rely on searching by host and sourcetype.

I have a snippet of a log file. (It's a big snippet.) Exact same format as is generated by the exact same syslog-ng daemon. It contains events from multiple other devices. Everything you'd get from the "normal" syslog file.

If I do a one-shot index of the file, every host is identified incorrectly as host=splp01. Every sourcetype is incorrectly identified as the file-name, sourcetype=splunk-log

I want this snippet processed as the normal splunk-log file is processed. It already should go to the default index. It should correctly identify which host the event belongs to. It should correctly determine the sourcetype: broadsoft, cisco-asa, syslog, netscreen-fw, cisco-pix, F5, etc.

As a nice-to-have, but not a need-to-have, I'd like to have the source listed as /opt/apps/splunk-index01/splunk-log.

Can I do this? I wouldn't mind reconfiguring my normal data inputs if necessary.

Splunk 4.1.5. Solaris 10.

Re: How Can I One-Time Index a File with "Normal" Processing?

MuS — Wed, 14 Sep 2011 06:07:04 GMT

Hi 'I am Jeff'

yes, you can do this either with the CLI oneshot command or spool it via the sinkhole directory. The data feed in there will be handled like any other log file.

Find more on that topic here

Re: How Can I One-Time Index a File with "Normal" Processing?

I_am_Jeff — Thu, 22 Sep 2011 13:25:49 GMT

The problem is the oneshot will label all event as being from a single host. I need something that will process the event: assign correct host, assign correct sourcetype.

I have had some luck with reconfiguring the data imput on Splunk to be something like /var/log/messages* and dropping in a file named /var/log/messages.importme. It gets the hostname, but I still don't have the sourcetypes correct.