topic Re: Field names in lowercase, transforms.conf in Getting Data In

Field names in lowercase, transforms.conf

gelica — Fri, 19 Jul 2013 07:35:27 GMT

Hi!

I have some different sourcetypes defined by me where I'm extracting some of the fields with stanzas in transforms.conf at search time (I'm using REPORT in props.conf). Here is one example of a stanza I'm using:

REGEX=(?im)[\r\n]+([^\r\n]*name)\: ([^\r\n]+)
FORMAT=$1::$2
MV_ADD=true

This extracts the fields I want, but since I extract the field name like this, the field name may be in uppercase, lowercase or a combination.

Creating new stanzas for each field is not an option since I have a lot of fields and most of my stanzas are of the form shown above, where I just define the ending of the field name, to be able to extract most of them.

I wonder if there is any way to "force" the field names to lowercase?

Re: Field names in lowercase, transforms.conf

Ayn — Fri, 19 Jul 2013 07:58:00 GMT

I guess this is mostly for cosmetic purposes? Because field names are case insensitive.

Re: Field names in lowercase, transforms.conf

gelica — Mon, 28 Sep 2020 14:23:35 GMT

The field values are case insensitive, but the field names are not.

If I have a field called Machine_name, and run a search for Machine_name=* I get the events I'm looking for, but if I search fort machine_name=* instead, I get no results..

Re: Field names in lowercase, transforms.conf

Ayn — Fri, 19 Jul 2013 08:37:56 GMT

My mistake - you're absoutely right, field names are case sensitive. I don't have a good solution right now though, sorry.

Re: Field names in lowercase, transforms.conf

grijhwani — Sun, 21 Jul 2013 19:36:01 GMT

If you look at the answer to a slightly different question (Dealing with key/value pairs with inconsistent key case) the solution is, perhaps, to pre-process the log stream at input time to convert to lower case with sed commands.

Re: Field names in lowercase, transforms.conf

gelica — Mon, 22 Jul 2013 05:13:06 GMT

I tried that, but didn't get it to work, hence my confused comments on that answer.
I don't really understand the regex (s/\([A-Za-z0-9]*)/\1\L\2\g) either, I haven't found any information about it(I'm used to regexes so it is only this one that confuses me 🙂 )

However, I tried this approach again, by adding it to my default-clause in props.conf, restarting Splunk and index new files, but I can't get it to work.. :S

Re: Field names in lowercase, transforms.conf

grijhwani — Wed, 24 Jul 2013 21:03:24 GMT

So - you got it to work in the end?

Re: Field names in lowercase, transforms.conf

landen99 — Wed, 19 Aug 2015 17:39:30 GMT

stats and where both care about case, to name a couple. I suggest using Calculated fields at search time.

Re: Field names in lowercase, transforms.conf

landen99 — Wed, 19 Aug 2015 18:20:06 GMT

It is for SEDCMD in props.conf and it looks like the author got it wrong. Part of the string is regex and the other parts are code: s/regex/text/g
"s" means replace and "g" means global

He should have written:

SEDCMD-contacts = s/([A-Za-z]*)/\L\1/g

Keep in mind that this only changes the raw text at index time and not the field names after extraction at search time, but if you are extracting the field name from the raw text then search time extractions will extract all lower case.

Re: Field names in lowercase, transforms.conf

marcoscala — Fri, 04 Mar 2016 09:02:55 GMT

Fields names ARE case sensitive!!! Values aren't.

Re: Field names in lowercase, transforms.conf

ddrillic — Fri, 04 Mar 2016 18:11:17 GMT

It does seem right to convert all characters to lower case in transforms.conf, in the spirit of -

[syslog-header-stripper-ts]
REGEX = ^[A-Z][a-z]+\s+\d+\s\d+:\d+:\d+\s(.*)$
FORMAT = $1
DEST_KEY = _raw

Re: Field names in lowercase, transforms.conf

marcoscala — Mon, 07 Mar 2016 08:26:16 GMT

Sorry, but if I'm right, this transforms just STRIPS out the syslog header. It doesn't convert it to lowercase.