We already use a custom CSV formt to report application metrics. The format is very similar to the one introduced in Splunk 7.
But while Splunk extracts dimensions from the CSV lines, we extract some of the dimenstions from the source field.
According to the docs something like that is possible for nearly all other methods of importing metrics, but not for CSV files.
Is there any way I can achieve this without modifying the CSV files (e.g. via search- or index-time field extractions)?
This is en example CSV file (source = X:\LogFiles\MyEnvironment\MyApplication\MyInstance\Values.amf):
metric_name,_value Process.IO,16620.4 Process.ProcessorTime,4.0666666666666664 Process.ThreadCount,40 Process.WorkingSet,258634547.2
We currently use a search time field extraction to extract the following three fields (~dimensions) from the source field:
Environment=MyEnvironment Application=MyApplication Instance=MyInstance
Those fields are extracted via the following line in our props.conf:
EXTRACT-source = (?i)LogFiles\\(?<Environment>[^\\]+)\\(?<Application>[^\\]+)\\(?<Instance>[^\\]+)\\ in source
Using either this or an equivalent transform to extract the fields at index time did not work.
I've tried using the following conf file stanzas:
[metrics_csv] TRANSFORMS-amf2 = amf2
[amf2] SOURCE_KEY = field:source REGEX = (?i)LogFiles\\(?<Environment>[^\\]+)\\(?<Application>[^\\]+)\\(?<Instance>[^\\]+)\\
Can you please post a line of sample data for your currently used CSV format and an explanation how it would have to be split?
This should be possible even with CSV using index-time field extractions. Can you please show the props + transforms you tried?
Hey, try this please for your transforms.conf:
SOURCE_KEY = MetaData:Source REGEX = yourregex WRITE_META = true
You can find more information on how to extract index time fields here:
For metrics data, you can completely ignore anything written that relating to fields.conf.
Thanks for your support, you clearly pointed me in the right direction:
SOURCE_KEY = MetaData:Sourceis required.
WRITE_META = trueis required.
FORMAT = Environment::$1 Application::$2 Instance::$3does work.
This transforms.conf stanza works:
[amf2] SOURCE_KEY = MetaData:Source REGEX = (?i)LogFiles\\([^\\]+)\\([^\\]+)\\([^\\]+) FORMAT = Environment::$1 Application::$2 Instance::$3 WRITE_META = true
Thanks for pointing this out!
Especially 4. is a no-go for uns.
@Splunk: Any news on this?
Support for named capture groups in transforms should also work for metric dimensions.