topic Re: How to extract fields at index time? in Splunk Enterprise

How to extract fields at index time?

paulmilbank — Sun, 01 Oct 2017 22:33:35 GMT

We have .net logs from SeriLog and we would like to break it down into key value pairs at index time and extract some fields.
I have tried to follow the splunk guides and blog posts, but my indexed fields are not available. I can't post links yet unfortunately.
transforms.conf:

[SerilogKVPairs]
DELIMS = "{,}", ":"

[LogLevel]
REGEX = ^(?:[^ \n]* ){3}(?P<LogLevel>[^ ]+)

props.conf:

# Extract fields from Serilog log inputs
TRANSFORMS-KVPairs= SerilogKVPairs
TRANSFORMS-LogLevel= LogLevel

fields.conf:

[SerilogKVPairs]
INDEXED=true

[LogLevel]
INDEXED=true

if I search with a pipe to kv SerilogKVPairs it all works, I have searchable values from my Serilog files.
But the fields are not available in the UI unless I pipe it through kv SerilogKVPairs.
We would like them to be available on all logs without having to pipe through the KV command.
Loglevel does not seem to be extracted either.

Is there a log which shows what is going on here?
Thanks
Paul

Re: How to extract fields at index time?

paulmilbank — Sun, 01 Oct 2017 23:05:02 GMT

I have managed to get loglevel working as a field now. I am still unsure how to get the KV pairs to be extracted and available without needing to pipe search through KV.

Re: How to extract fields at index time?

xisura — Mon, 02 Oct 2017 01:22:13 GMT

Have you try this?

http://docs.splunk.com/Documentation/SplunkCloud/6.6.1/Data/Configureindex-timefieldextraction

Re: How to extract fields at index time?

paulmilbank — Mon, 02 Oct 2017 01:29:39 GMT

This is now resolved from the look of things. I changed TRANSFORMS to REPORTS for the KV Delimiter, Removed it from fields.conf and it seems to be working now.

Re: How to extract fields at index time?

paulmilbank — Mon, 02 Oct 2017 01:31:11 GMT

That is the guide I followed today and it has enabled me to sort this out now. Thanks very much.

Re: How to extract fields at index time?

paulmilbank — Mon, 02 Oct 2017 01:54:34 GMT

Final config looks like this if anyone ever needs it:
transforms.conf:

[SerilogKVPairs]
DELIMS = "{,}", ":"

[LogLevel]
REGEX = ^(?:[^ \n]* ){3}(?P<loglevel>[^ ]+)
WRITE_META = true

fields.conf:

[LogLevel]
INDEXED=true

props.conf:

# Extract fields from Serilog log inputs
REPORT-SerilogKVPairs= SerilogKVPairs
TRANSFORMS-LogLevel= LogLevel