topic Re: How to do a field extraction on the source field? in Splunk Search

How to do a field extraction on the source field?

a212830 — Tue, 03 Feb 2015 17:48:14 GMT

Hi,

I need to create a field on the source field, but am not sure how to do that. Can someone help me?

Re: How to do a field extraction on the source field?

ppablo — Tue, 03 Feb 2015 17:51:55 GMT

Hi @a212830

Would you be able to provide sample data and what exactly you're trying to extract from the source field so users have more content to work with?

Re: How to do a field extraction on the source field?

a212830 — Tue, 03 Feb 2015 18:10:59 GMT

Sure. We have hosts that report on multiple instance, which have log files and need to be reported on differently. Here are two samples:

/apps/oracle/install/admin/instances/ovdprtp2a/diagnostics/logs/OVD/ovd1/diagnostic.log

/apps/oracle/install/admin/instances/ovdprtp2b/diagnostics/logs/OVD/ovd1/diagnostic.log

We want to report on the ovdrtp2a/2b individually, via a field extraction.

Re: How to do a field extraction on the source field?

aakwah — Mon, 28 Sep 2020 18:49:39 GMT

I think you should have 2 sourcetypes one for instance ovdrtp2a and the other one for instance ovdrtp2b, then apply field extraction as per sourcetype,

For example:

[monitor:///apps/oracle/install/admin/instances/ovdprtp2a]
index = Your_index_name
sourcetype = ovdprtp2a

[monitor:///apps/oracle/install/admin/instances/ovdprtp2b]
index = Your_index_name
sourcetype = ovdprtp2b

Then apply field extraction through props.conf and transforms.conf.

Regards

Re: How to do a field extraction on the source field?

a212830 — Tue, 03 Feb 2015 19:52:18 GMT

That won't work, as we have dozens of these, and I don't want to maintain that across these servers.

Re: How to do a field extraction on the source field?

wpreston — Tue, 03 Feb 2015 19:54:16 GMT

Have you tried using rex?

... search terms here ... | rex field=source "instances\/(?<NewFieldName>[^\/]+)" | stats count by NewFieldName

Re: How to do a field extraction on the source field?

David — Tue, 03 Feb 2015 21:15:11 GMT

You could also apply it in props/transforms.conf. I had one scenario where given a file like /var/log/SystemAOutput.good I wanted to extract "SystemAOutput" and "good." I did this via the props.conf and transforms.conf:

props.conf:

[LogFiles]
TIME_FORMAT = %m/%d/%Y
MAX_EVENTS = 100000
NO_BINARY_CHECK = true
disabled = false
pulldown_type = true
REPORT-reporting = extract_filename

transforms.conf:

[extract_filename]
SOURCE_KEY = source
REGEX = [^/\\]([^\\/\.]*?)(?:_File\d*){0,1}\.(bad|good)$
FORMAT = srcfile::$1 status::$2

Output will then be:

Filename: /var/log/SystemAOutput.good
srcfile: SystemAOutput
status: good

Re: How to do a field extraction on the source field?

masonmorales — Tue, 03 Feb 2015 21:35:48 GMT

This should work.

Re: How to do a field extraction on the source field?

a212830 — Wed, 04 Feb 2015 00:24:33 GMT

Thanks. I'm testing out both methods. Is there a way to put the rex above in an extraction, rather than in a search?

Re: How to do a field extraction on the source field?

David — Wed, 04 Feb 2015 00:29:03 GMT

Yes, the method I posted below.

Re: How to do a field extraction on the source field?

a212830 — Wed, 04 Feb 2015 00:52:34 GMT

So, this?

vds_* : EXTRACT-vdsHost Inline field=source "instances\/(?[^\/]+)"

I need this to work across multiple sources and sourcetypes, can I wildcard a sourcetype when creating a field extraction?

Re: How to do a field extraction on the source field?

a212830 — Wed, 04 Feb 2015 01:32:31 GMT

I decided to go with the props/transforms method. Can someone help me with the regex? I'm not very good with these expressions.

Source = /apps/oracle/install/admin/instances/ovdprtp2a/diagnostics/logs/OVD/ovd1/diagnostic.log

I need to extract the value between instances and diagnostics.

Re: How to do a field extraction on the source field?

_d_ — Wed, 04 Feb 2015 02:25:14 GMT

An even easier props.conf method is to use EXTRACT- without referencing transforms.conf:

[LogFiles]
TIME_FORMAT = %m/%d/%Y
...
...
EXTRACT-myfield = instances/(?<myField>[^/]*)/diagnostics in source

Re: How to do a field extraction on the source field?

David — Wed, 04 Feb 2015 05:29:58 GMT

Awesome, I didn't know that you could do "in source"

Re: How to do a field extraction on the source field?

a212830 — Wed, 04 Feb 2015 12:06:25 GMT

Tried this, but the field is not showing up.

Put this in my props, pushed it via my cluster manager, even did a rolling restart on the indexers, but it's not appearing.

EXTRACT-vdsHost = instances/(?[^/]*)/diagnostics in source

Re: How to do a field extraction on the source field?

_d_ — Wed, 04 Feb 2015 13:18:13 GMT

Field extractions go on the search head, not indexers. Also, your capture group in the regex is missing a name; myField above

Re: How to do a field extraction on the source field?

wpreston — Wed, 04 Feb 2015 13:23:31 GMT

Like @David said, props/transforms.conf is the way to go. From the docs on using props.conf only extractions:

All extraction configurations in props.conf are restricted by a specific source, source type, or host. Start by identifying the source type, source, or host that provide the events that your field should be extracted from

Also from the docs on transforms.conf extractions:

Your search-time field extractions require a field transform component if you need to:
• Reuse the same field-extracting regular expression across multiple sources, source types, or hosts (in other words, configure one field transform for multiple field extractions). If you find yourself using the same regex to extract fields for different sources, source types, and hosts, you may want to set it up as a transform. Then, if you find that you need to update the regex, you only have to do so once, even though it is used more than one field extraction.

So you can't wildcard the sourcetype. To dowhat you want while making maintenance easy, create a field transform in transforms.conf and reference it in props.conf for each host/source/sourcetype to which it applies:

transforms.conf:

[myNewFieldExtract]
REGEX = instances\/(?<NewFieldName>[^\/]+)
SOURCE_KEY = source

props.conf:

[sourcetype::first_sourcetype_this_applies_to]
REPORT-my_class_name = myNewFieldExtract

[sourcetype::second_sourcetype_this_applies_to]
REPORT-my_class_name = myNewFieldExtract

... and so on...

Re: How to do a field extraction on the source field?

wpreston — Wed, 04 Feb 2015 13:30:43 GMT

Also note that the class names in each props.conf report stanza should be unique.

Re: How to do a field extraction on the source field?

masonmorales — Wed, 04 Feb 2015 14:04:58 GMT

Try this:

On your search heads, in props.conf, within the stanzas you want to create this extraction for, add:

EXTRACT-vdsHost = instances\/(?<vdsHost>[^\/]+)/diagnostics in source

After saving, either reload your search head(s), or less intrusively, open the following URL while logged into the search head under an admin account:

https://YOURSPUNKSERVERHERE:8000/en-US/debug/refresh

Lastly, run a search on the data and verify that the new "vdsHost" field appears in the sidebar.

Re: How to do a field extraction on the source field?

a212830 — Wed, 04 Feb 2015 14:58:14 GMT

Not sure what I'm doing wrong here... followed what you have...

props.conf:

[sourcetype::vds_access]
ANNOTATE_PUNCT = false
KV_MODE = auto
LINE_BREAKER = ([\r\n]+).\d{4}-\d{2}-\d{2}
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX = ^.
TRUNCATE = 999999
REPORT-vdsaccessExtract = vdsHost_extract

[sourcetype::vds_diagnostic]
ANNOTATE_PUNCT = false
KV_MODE = auto
LINE_BREAKER = ([\r\n]+).\d{4}-\d{2}-\d{2}
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX = ^.
TRUNCATE = 999999
REPORT-vdsdiagExtract = vdsHost_extract
pulldown_type = 1

transforms.conf:

[vdsHost_extract]
REGEX = instances\/(?[^\/]+)
SOURCE_KEY = source

I pushed these out via the cluster manager, but still don't see the field.