topic Field names from file, including source and host in Splunk Search

Field names from file, including source and host

jgauthier — Thu, 02 Jun 2011 19:43:14 GMT

I've written an application that outputs data that I would like to index. Of course, I have a series of requirements.

First, I'd like to extract the host and the sourcetype from the file.
I am reading http://www.splunk.com/base/Documentation/4.1.8/Admin/Advancedsourcetypeoverrides
on how to achieve this, but haven't succeeded.

props.conf

[source::Z:\ServerInput]
TRANSFORMS-changesrchost=SrvMonsource,SrvMonHost

transforms.conf

[SrvMonSource]
DEST_KEY = MetaData:Sourcetype
REGEX = Source: (.+)
FORMAT = sourcetype::$1

[SrvMonHost]
REGEX = Host: (.+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

Secondly, I'd like to extract the field names from the fields in the file. The file format looks like this:

Host: L-JGAUTHIER
Source: OSInfo
BootDevice: \Device\HarddiskVolume2
BuildNumber: 7600
BuildType: Multiprocessor Free
Caption: Microsoft Windows 7 Enterprise 
CodeSet: 1252
CountryCode: 1
CreationClassName: Win32_OperatingSystem
CSCreationClassName: Win32_ComputerSystem
CSDVersion: 
CSName: L-JGAUTHIER

And lastly, I want to delete the file once it's indexed. I can probably use the spool directory for this, but I'm not sure yet if that will work since I am not sure if I can make that a source or not in my application (in props.conf).

Thanks for any pointers.

Re: Field names from file, including source and host

hazekamp — Thu, 02 Jun 2011 21:44:21 GMT

jgauthier,

1.  Having your inputs.conf would be helpful.  Is the source file "ServerInput" or "ServerInput\<somefile>"?
2.  Try:
## props.conf
[source::<source>]
KV_MODE = None
REPORT-auto_kv_for_my_source = auto_kv_for_my_source

## transforms.conf
[auto_kv_for_my_source]
REGEX = ^(\S+):(?:\s+)?(.+)
FORMAT = $1::$2
MV_ADD = True

3.  Use the batch input instead of the monitor input.
[batch://<path>]
* One time, destructive input of files in <path>.
* For continuous, non-destructive inputs of files, use monitor instead.

# Additional attributes:

move_policy = sinkhole
* IMPORTANT: This attribute/value pair is required. You *must* include "move_policy = sinkhole" when defining batch inputs.
* This loads the file destructively.  
* Do not use the batch input type for files you do not want to consume destructively.

host_regex = see MONITOR, above.
host_segment = see MONITOR, above.
crcSalt = see MONITOR, above.

# IMPORTANT: The following attribute is not used by batch:
# source = <string>

Re: Field names from file, including source and host

jgauthier — Fri, 03 Jun 2011 14:23:39 GMT

I was able to achieve all aspects of what I wanted.

First, my inputs.conf:

[batch://Z:\ServerInput]
disabled=0
recursive=false
sourcetype=SrvMon
move_policy = sinkhole

This implemented the sinkhole I wanted.

props.conf

[SrvMon]
TRANSFORMS-metadata=SrvMonHost,SrvMonSource

Transforms.conf:

[SrvMonSource]
DEST_KEY = MetaData:Sourcetype
REGEX = Source=(.+)
FORMAT = sourcetype::$1

[SrvMonHost]
REGEX = Host=(.+)
FORMAT = host::$1
DEST_KEY = MetaData:Host

And lastly, I changed my field format to be "Field=Data" so splunk picked up the key/value pair automatically.

Working great!