topic Re: Why is the field extraction transformation not working in conjunction with data filtering transformation? in Getting Data In

Why is the field extraction transformation not working in conjunction with data filtering transformation?

alanzchan — Sat, 16 Mar 2019 21:49:24 GMT

I'm trying to minimize the amount of data from Kubernetes JSON events that are being indexed into my Splunk instance. Rather than having the whole JSON which includes headers and mostly unimportant metadata, I want to only display the raw text payload in my Splunk event viewer. However, this filters out the metadata that I actually find useful so I created a field extraction transformation in an attempt to add the metadata I need to my events before doing my filtering logic.

I've tried the above, but it doesn't work when they are in combination.

Is it possible to add, for example key1=value1 in _meta and have this field extracted for all events even if the events would not have it in _raw?

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

nickhills — Tue, 19 Mar 2019 10:59:13 GMT

Can you share what you have so far?

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

woodcock — Tue, 19 Mar 2019 13:39:44 GMT

Attention @mmodestino_splunk !!!

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

alanzchan — Tue, 29 Sep 2020 23:46:00 GMT

[my_sourcetype]
TIME_PREFIX = "publish_time":
TIME_FORMAT = %s.%Q
TRANSFORMS-assign_sourcetype_access=access_log_to_st
TRANSFORMS-assign_sourcetype_application=application_log_to_st
TRANSFORMS-assign_index = change_index_by_application

TRANSFORMS-system_labels_transform = system_labels_transform

TRANSFORMS-change_format = change_format

Here is my props.conf.

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

FrankVl — Tue, 19 Mar 2019 16:08:56 GMT

And what does your transforms.conf look like? Perhaps also some sample data?

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

woodcock — Tue, 19 Mar 2019 23:31:35 GMT

Sure you can, especially if it is a static value. See here:
https://docs.splunk.com/Documentation/Splunk/latest/Data/Configureindex-timefieldextraction

So like this:

[my_sourcetype]
FORMAT = key1::value1
WRITE_META = true

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

alanzchan — Wed, 20 Mar 2019 01:15:54 GMT

The value in this case would not be static.

The problem here is that I only want the message of the log event to show in my event preview on my search head, not the whole JSON structure which means I then have to click [+] each time I want to see my log message.

Is there anyway that I can “bind” the metadata to their events without having the data in raw?

Hope that make sense, but my gut feeling that’s me there’s no way to do this.

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

FrankVl — Wed, 20 Mar 2019 09:01:18 GMT

If it is really just about how it shows in the event view in Splunk Web, then you could just add an | eval _raw = message to your search query (assuming you have that message extracted as a field called message). And if that is to your liking, then you could also implement that as a calculated field in your props.conf.

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

mattymo — Wed, 20 Mar 2019 14:10:32 GMT

HI @alanzchan!

How are you getting the data into Splunk? Are we talking Docker JSON driver logging here?

Our Splunk Connect for Kubernetes solution solves the problem of docker json driver cruft, and allows you to reshape and enrich the data inflight.

https://github.com/splunk/splunk-connect-for-kubernetes

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

woodcock — Wed, 20 Mar 2019 14:16:48 GMT

wait, you would like to replace all of your event ( _raw ) with just the part that is in message and throw the rest away forever?

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

woodcock — Mon, 25 Mar 2019 05:14:02 GMT

Why not just click on the All Fields, search for your field, click the checkbox, and then see that it is BOTH a Selected field shown in aggregate on the left side of your events and ALSO an in-line callout field at the bottom of each event. I think that this meets your desire without having to change anything.

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

alanzchan — Wed, 27 Mar 2019 00:00:27 GMT

Let’s put it this way, is it possible to just display a portion of _raw and leave the rest out. I don’t want to discard the other information, but I don’t want it clogging up my single event where I have to click [+] three times just to see the text payload. I prefer not to do this during search time. Can I edit a Splunk event viewer configuration or something?

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

alanzchan — Wed, 27 Mar 2019 00:14:52 GMT

I’m using the Google Cloud Platform add-on for Splunk. Kubernetes logs are sent in a JSON format and my goal is to display only the text payload of an event without the other metadata and attributes (not that they’re not important, but for convenience sake). However, If I filter out for just the text payload, I would lost out on all of the metadata which I would like to avoid. Can I modify some Splunk configuration? Any advice?

For another description of my problem, my question is identical to this one: https://answers.splunk.com/answers/659626/modify-raw-but-keep-extracted-fields.html

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

alanzchan — Wed, 27 Mar 2019 00:16:20 GMT

Please check out this question from another Splunk user for a better description of my problem: https://answers.splunk.com/answers/659626/modify-raw-but-keep-extracted-fields.html

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

alanzchan — Wed, 27 Mar 2019 00:31:16 GMT

I would prefer not to do it during search time if possible, but this would achieve a somewhat desirable result.

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

woodcock — Wed, 27 Mar 2019 06:08:29 GMT

See my new answer at the bottom.

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

FrankVl — Wed, 27 Mar 2019 08:33:58 GMT

You prefer not to do it during search time, as in: you don't like doing it manually as part of the query? Or would even moving it to a search time field calculation in props.conf (such that it happens automatically) not be preferred?

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

mattymo — Tue, 02 Apr 2019 20:00:13 GMT

Can you not deploy our collector to your k8s cluster instead??

It solves for all this pain already, ensuring only the log payload shows up in your events, but the key metadata fields are preserved as index time fields.

It is also going to be much more reliable and scalable than having to pull with a TA, and likely will optimize your storage/index volumes.

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

alanzchan — Thu, 04 Apr 2019 15:41:24 GMT

Thanks for you reply. I will work on this solution and provide an update shortly.

Re: Why is the field extraction transformation not working in conjunction with data filtering transformation?

alanzchan — Wed, 26 Jun 2019 01:08:21 GMT

Hi @mmodestino_splunk,

Sorry for the super late reply. I'm testing Splunk Connect for Kubernetes right now, sending Kubernetes cluster container data to Splunk via HEC. The logs are still coming in as JSON format. Is there any way to process this data so it indexes as raw text?

Thanks,
Alan