Field extraction with props.conf and transforms.conf

[DrFedtke]

Hi all,

I tried to find a way to extract fields automatically after adding new data.

The input is of the type:

Log^key1=value1^key2=value2^key3=value3^

props.conf:

[LogType]
REPORT-LogType = LogTypeKV

transforms.conf:

[LogTypeKV]
FORMAT = $1::$2
REGEX = (\w+)=([^\^]+)

In the past it somehow worked, but now it doesn't. What am I missing?

Regards,
Caspar

[DrFedtke · Accepted solution]

The problem were read and write permissions of the *.conf files under Windows.
I had to manually assign the right permission to get it working.

[DalJeanis]

@DrFedtke - Thank you for posting the solution. Since your issue appears to be solved, please accept the answer so that others will know the problem is complete.

[Richfez]

To my imperfect eyes, that looks like it should still work. So perhaps it's no longer being applied for some reason? I can think of two broad reasons: one is that the format of the events could have changed slightly so the regex doesn't apply, another is that for some reason the entire stanzas aren't applying because of a change in sourcetype, host IP or something like that.

So, first, if you have older data still around and properly parsed, compare the fields list for it (sourcetype, source, host, etc...) with some of the newer ones. If you can get it to a short timeframe where you only have a couple of entries, about half of which work and half of which don't, that would be perfect. Also compare the _raw entries closely and see if you can spot a change.

If that doesn't turn up anything interesting, try running ./splunk cmd btool props list and ./splunk cmd btool transforms list from your $splunkhome/bin directory to see what's there. You might need to redirect that output to a file so you can peruse it at your leisure. If you are on windows, you can pipe it to the clipboard ... | clip and paste it into Notepad++ or something, too. Here's some minor help on btool.

Report back with findings if that doesn't help you sort it out. You have already included the stanzas above (if they're complete), but if you could post an actual event from before the breakage and one from after, then maybe if appropriate some relevant btool output, we could lend some eyeballs to the problem better.