I have read through the many documentation articles but they are all so broken apart that it is hard to piece together all the needed info.
Under - Manager» Data inputs» Add data» Files & directories» Data preview
It allows to me edit using a regex to define my fields. All my record events start with "At 16:38:54 -0500 - " so a regex of [0-9][0-9]:[0-9][0-9]:[0-9][0-9] or something similar will catch the lines.
This works, but only for the local Splunk server. If I try to grab the data off of a remote system, then I need to use the universal forwarder. This is where my problem starts.
First, I configured the forwarder via the GUI and it appears to work, kind of. It never presented me with any options on how to define the data that it found in the log files.
Then, when I thought it wasn't working, I am unable to figure out how to modify the forwarder to change its configuration. It just runs as a service and I'm unable to modify any options.
So, how do I modify the config of the forwarder? and is it possible to define the data via regex?
This is running 5.0.1 running on Windows for both server and clients.
Fields are generally extracted at search-time, so any configuration related to that will be done on the instance you use for searching (typically the indexer in a simple deployment). So, you shouldn't need to worry about forwarder configurations.
Once you have data coming into the indexer from your forwarders, you can use the interactive field extractor to define your fields. More information on field extraction is available here: http://docs.splunk.com/Documentation/Splunk/5.0/knowledge/Addfieldsatsearchtime
I will read that article, but let me expand my question so that we are all on the same page.
Is there no way to pre-define what data types will be coming from a specific host via the Universal Forwarder? I did read in the documentation that the forwarder is just a dumb process, it does not parse, just sends the data blindly, so I understand that it will not do the parsing.
Secondly, how do I reconfigure the forwarder that is installed on my client? Once it was setup and running, I found no way to modify it's configuration.
I'm not sure I understand your first question - what do you mean by "data types" and where would the pre-definition of this be done (and, to what end)?
Splunk configuration, particularly on forwarders where the web interface is generally disabled, is most often done using its configuration files. These reside in various directories under
etc in the directory where Splunk is installed.
the format is defined by the sourcetype.
It can be event linebreaking, timestamp detection, field extraction