Getting Data In

Field extraction with props.conf and transforms.conf

DrFedtke
Explorer

Hi all,

I tried to find a way to extract fields automatically after adding new data.

The input is of the type:

Log^key1=value1^key2=value2^key3=value3^

props.conf:

[LogType]
REPORT-LogType = LogTypeKV

transforms.conf:

[LogTypeKV]
FORMAT = $1::$2
REGEX = (\w+)=([^\^]+)

In the past it somehow worked, but now it doesn't. What am I missing?

Regards,
Caspar

0 Karma
1 Solution

DrFedtke
Explorer

The problem were read and write permissions of the *.conf files under Windows.
I had to manually assign the right permission to get it working.

View solution in original post

0 Karma

DrFedtke
Explorer

The problem were read and write permissions of the *.conf files under Windows.
I had to manually assign the right permission to get it working.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

@DrFedtke - Thank you for posting the solution. Since your issue appears to be solved, please accept the answer so that others will know the problem is complete.

0 Karma

Richfez
SplunkTrust
SplunkTrust

To my imperfect eyes, that looks like it should still work. So perhaps it's no longer being applied for some reason? I can think of two broad reasons: one is that the format of the events could have changed slightly so the regex doesn't apply, another is that for some reason the entire stanzas aren't applying because of a change in sourcetype, host IP or something like that.

So, first, if you have older data still around and properly parsed, compare the fields list for it (sourcetype, source, host, etc...) with some of the newer ones. If you can get it to a short timeframe where you only have a couple of entries, about half of which work and half of which don't, that would be perfect. Also compare the _raw entries closely and see if you can spot a change.

If that doesn't turn up anything interesting, try running ./splunk cmd btool props list and ./splunk cmd btool transforms list from your $splunkhome/bin directory to see what's there. You might need to redirect that output to a file so you can peruse it at your leisure. If you are on windows, you can pipe it to the clipboard ... | clip and paste it into Notepad++ or something, too. Here's some minor help on btool.

Report back with findings if that doesn't help you sort it out. You have already included the stanzas above (if they're complete), but if you could post an actual event from before the breakage and one from after, then maybe if appropriate some relevant btool output, we could lend some eyeballs to the problem better.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...