topic Re: Configure Receiver To Create Events by Source in Deployment Architecture

Configure Receiver To Create Events by Source

cpt12tech — Thu, 25 Apr 2013 00:46:28 GMT

I'm new to Splunk and having some issues with getting logs to create events correctly.
I've installed the universal forwarder, and set to a directory of plain text logs for a specific application.
The logs are created 1 per day, and each line should be an event.
I configured the universal forward to get local data from the logs directory (in which there are multiple sub directories with log files in them).

When it reaches the receiver, Splunk creates events based on the log files and not the line items in the logs.
I was able to connect the Splunk server to the computer and import the files, and can see in the data preview.

However, I'm haven't figured out how to set this up on a receiver. I don't want the client computer to do the processing, and would rather have the Splunk server split the events up correctly at time of indexing.

Is there an app I need to install to configure a receiver?
Should I have multiple receivers for different source types? Or do I modify the config file to look for the source and run an event filter based on the source?

Re: Configure Receiver To Create Events by Source

kristian_kolb — Thu, 25 Apr 2013 06:42:31 GMT

Hm, you might have to rephrase that a little bit ('Splunk creates events based on the log files and not the line items in the logs').

Also it's a bit unclear if you are using the forwarder OR getting the indexer to read the files directly (presumably through a file share?).

With a universal forwarder reading and sending the logs, the indexer will do the work of splitting the stream of incoming data into separate events.

No app needed to enable the receiver. See: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Enableareceiver#Set_up_receiving_with_Splunk_Web

Re: Configure Receiver To Create Events by Source

Ayn — Thu, 25 Apr 2013 06:43:02 GMT

This should already be the case though - Universal Forwarder do not perform parsing, and so do not decide how events are broken up. This all happens on the indexer (or to be more precise, on the first parsing Splunk instance that events arrive to, which usually is the indexer unless you have some kind of multi-layered setup with several chained forwarding instances).

The rules Splunk uses for breaking events are documented here: http://docs.splunk.com/Documentation/Splunk/5.0.2/Data/Indexmulti-lineevents

Basically by default, Splunk will break into a new event when it encounters a line with a valid timestamp, but you can configure this pretty much any way you want.

Re: Configure Receiver To Create Events by Source

cpt12tech — Fri, 26 Apr 2013 16:29:23 GMT

Very Cool.

• I'm assuming the connection_host parameter is only applying the sourcetype for matched hosts and not setting the host for all inputs?

I plan on using Splunk for other servers, data streams, SNMP, WMI, etc. Do I need to create a different receiver for different source files? Or can I use the host field to match different sourcetypes?

Re: Configure Receiver To Create Events by Source

cpt12tech — Fri, 26 Apr 2013 16:29:45 GMT

I grabbed a file from the client, and uploaded it to Splunk using the web
manager : data inputs : files & directories
Changed the Data Preview to Every Line is One Event
Saved the data preview as a new source type
then canceled

I read through the inputs.conf and set up the following:
[tcp://9997]
connection_host = myComputerName
sourcetype = mySourceName

Installed the forwarder, pointed to Splunk and the data is coming in parsed out by the timestamp on each line!

Very Cool.

Re: Configure Receiver To Create Events by Source

cpt12tech — Fri, 26 Apr 2013 16:39:52 GMT

Just realized that now I'm having duplicates reimported. When a new line is written to the end of the log file, the entire file is imported as an event, and not just the new event.