I've written an application that outputs data that I would like to index. Of course, I have a series of requirements.
First, I'd like to extract the host and the sourcetype from the file.
I am reading http://www.splunk.com/base/Documentation/4.1.8/Admin/Advancedsourcetypeoverrides
on how to achieve this, but haven't succeeded.
props.conf
[source::Z:\ServerInput]
TRANSFORMS-changesrchost=SrvMonsource,SrvMonHost
transforms.conf
[SrvMonSource]
DEST_KEY = MetaData:Sourcetype
REGEX = Source: (.+)
FORMAT = sourcetype::$1
[SrvMonHost]
REGEX = Host: (.+)
FORMAT = host::$1
DEST_KEY = MetaData:Host
Secondly, I'd like to extract the field names from the fields in the file. The file format looks like this:
Host: L-JGAUTHIER
Source: OSInfo
BootDevice: \Device\HarddiskVolume2
BuildNumber: 7600
BuildType: Multiprocessor Free
Caption: Microsoft Windows 7 Enterprise
CodeSet: 1252
CountryCode: 1
CreationClassName: Win32_OperatingSystem
CSCreationClassName: Win32_ComputerSystem
CSDVersion:
CSName: L-JGAUTHIER
And lastly, I want to delete the file once it's indexed. I can probably use the spool directory for this, but I'm not sure yet if that will work since I am not sure if I can make that a source or not in my application (in props.conf).
Thanks for any pointers.
I was able to achieve all aspects of what I wanted.
First, my inputs.conf:
[batch://Z:\ServerInput]
disabled=0
recursive=false
sourcetype=SrvMon
move_policy = sinkhole
This implemented the sinkhole I wanted.
props.conf
[SrvMon]
TRANSFORMS-metadata=SrvMonHost,SrvMonSource
Transforms.conf:
[SrvMonSource]
DEST_KEY = MetaData:Sourcetype
REGEX = Source=(.+)
FORMAT = sourcetype::$1
[SrvMonHost]
REGEX = Host=(.+)
FORMAT = host::$1
DEST_KEY = MetaData:Host
And lastly, I changed my field format to be "Field=Data" so splunk picked up the key/value pair automatically.
Working great!
I was able to achieve all aspects of what I wanted.
First, my inputs.conf:
[batch://Z:\ServerInput]
disabled=0
recursive=false
sourcetype=SrvMon
move_policy = sinkhole
This implemented the sinkhole I wanted.
props.conf
[SrvMon]
TRANSFORMS-metadata=SrvMonHost,SrvMonSource
Transforms.conf:
[SrvMonSource]
DEST_KEY = MetaData:Sourcetype
REGEX = Source=(.+)
FORMAT = sourcetype::$1
[SrvMonHost]
REGEX = Host=(.+)
FORMAT = host::$1
DEST_KEY = MetaData:Host
And lastly, I changed my field format to be "Field=Data" so splunk picked up the key/value pair automatically.
Working great!
jgauthier,
1. Having your inputs.conf would be helpful. Is the source file "ServerInput" or "ServerInput\<somefile>"?
2. Try:
## props.conf
[source::<source>]
KV_MODE = None
REPORT-auto_kv_for_my_source = auto_kv_for_my_source
## transforms.conf
[auto_kv_for_my_source]
REGEX = ^(\S+):(?:\s+)?(.+)
FORMAT = $1::$2
MV_ADD = True
3. Use the batch input instead of the monitor input.
[batch://<path>]
* One time, destructive input of files in <path>.
* For continuous, non-destructive inputs of files, use monitor instead.
# Additional attributes:
move_policy = sinkhole
* IMPORTANT: This attribute/value pair is required. You *must* include "move_policy = sinkhole" when defining batch inputs.
* This loads the file destructively.
* Do not use the batch input type for files you do not want to consume destructively.
host_regex = see MONITOR, above.
host_segment = see MONITOR, above.
crcSalt = see MONITOR, above.
# IMPORTANT: The following attribute is not used by batch:
# source = <string>