Firstly, I'll give my apologies now as you'll find my attempt to explain my problem will most likely show my inexperience with Splunk.
To start off - I'm running Splunk 6 on RedHat Enterprise Linux 5.
I'm attempting to ingest many application log files into Splunk where part of the filename contains the application subsystem, a date and time string, and a process ID. Suffice to say, these logs are only created once by a job triggered from the application - and never used in any subsequent jobs.
I've based my research on suggestions from blogs and other posts in this forum such as:
http://blogs.splunk.com/2010/02/11/sourcetypes-gone-wild/
http://answers.splunk.com/answers/25560/field-names-from-file-including-source-and-host.html
http://answers.splunk.com/answers/83619/source-sourcetype-defined-by-folder-names.html
A sample of log files I'm wanting to ingest are:
/app/prod/app_1/logs/ABCDE123_20141013163738_24772.log
/app/prod/app_1/logs/XYZABC456_20141013093007_16799.log
/app/prod/app_1/logs/EFGHIJK789_20141013093007_16799.log
/app/prod/app_1/logs/123ABC_20141013093007_16799.log
In my universal forwarder I have an inputs.conf file with the following entry:
[monitor:///app/prod/app_1/logs/*.log]
disabled = false
followTail = 1
index = app_index
In my indexer I have a props.conf file with the following entry:
[source::/app/prod/app_1/logs/*.log]
TRANSFORMS-set_sourcetype_app_logs = set_sourcetype_app_logs
Also in my indexer a transforms.conf file with the following entry:
[set_sourcetype_app_logs]
DEST_KEY=MetaData:Sourcetype
SOURCE_KEY=MetaData:Source
REGEX=\w+(?=_\w+_\w+\.log$)
FORMAT=sourcetype::$1
My expectation is that indexed logs should a source like "/app/prod/app_1/logs/ABCDE123_20141013163738_24772.log" and a sourcetype like "ABCDE123"
However, once the logs are ingested and indexed, a search reveals that all data ingested appeared literally with sourcetype of '$1' instead of the intended filename regex,
Do I have a problem with my transforms.conf regex or is my configuration completely off the mark?
Any help would be greatly appreciated.
Thanks,
Bobby
... View more