Splunk Search

Field extraction from host field value

michaeljorgense
Path Finder

Hi,

I would like to extract two new fields from the value of the host field at search time. I'd like the first 3 characters of the host field value to be a new field named 'group', and the next 3 characters of the host field value to be a new field named 'site'.

e.g.

if 

host = AAABBBsomestring

then

group = AAA
site = BBB

I believe I have the regex to make this work. I've tested it with rex in a Splunk Search & can see the new fields 'group' & 'site' being correctly populated in the events resulting from that rex modified search.

Here is my search:

index=mjtest | rex field=host "(?<group>\w{3})(?<site>\w{3})\w+"

So I then tried to place this as an inline field extraction in props.conf.

My environment for testing is a single virtual machine hosting all splunk functions including search head & indexers. Additionally the files being monitored are also on this same test server. All config files named below are in my own app in SPLUNK_HOME/etc/apps/app_name/local.

My inputs.conf looks like:

[monitor:///path/to/file/*/child_*.log]
index=mjtest
sourcetype=mjtest
host_regex = /path/to/file/.*/child_(\w+).log

My props.conf looks like:

[mjtest]
EXTRACT-group,site = (?<group>\w{3})(?<site>\w{3})\w+ in host

However these fields are not extracted at search time, well they don't appear in the event data or the list of interesting fields in a Splunk Search.

Any clues on why this might be? Is it because I'm setting the host value via host_regex in inputs.conf? I would have thought that wouldn't matter as that would happen before any props.conf actions took place in the processing pipeline.

For what it's worth I also tried doing a REPORT extraction with a props.conf & transforms.conf combo of the below which also didn't work.

props.conf:

[sourcetype::mjtest]
REPORT-fieldextract = group_site_extract

transforms.conf:

[group_site_extract]
REGEX = (?<group>\w{3})(?<site>\w{3})\w+
SOURCE_KEY = host

Ultimately I don't mind which way my goal is achieved... but it's driving me crazy why it's not working.

Any help would be much appreciated.

Michael.

1 Solution

jajung
Splunk Employee
Splunk Employee

Hi Michael,

I would suggest you check your permissions on the app that is containing the props.conf. Depending on how your app was created, the default behaviour of apps is to apply configuration within it to its OWN app context. That means that if you're trying to access configuration defined in $SPLUNK_HOME/etc/apps/my_app/ within your Search and Reporting app, but the metadata doesn't allow access, you won't have the settings applied.

You can update these settings via the Manage Apps page in the UI or by modifying the default.meta file in $SPLUNK_HOME/etc/apps/my_app/metadata appropriately.

A good way to debug these things is to create a lazy EVAL in your props.conf to see if you get configuration correctly applied, to trace where the issue is occuring.

I tend to do something like the following in props.conf

EVAL-a = my_app/local/props.conf

Good luck!

View solution in original post

jajung
Splunk Employee
Splunk Employee

Hi Michael,

I would suggest you check your permissions on the app that is containing the props.conf. Depending on how your app was created, the default behaviour of apps is to apply configuration within it to its OWN app context. That means that if you're trying to access configuration defined in $SPLUNK_HOME/etc/apps/my_app/ within your Search and Reporting app, but the metadata doesn't allow access, you won't have the settings applied.

You can update these settings via the Manage Apps page in the UI or by modifying the default.meta file in $SPLUNK_HOME/etc/apps/my_app/metadata appropriately.

A good way to debug these things is to create a lazy EVAL in your props.conf to see if you get configuration correctly applied, to trace where the issue is occuring.

I tend to do something like the following in props.conf

EVAL-a = my_app/local/props.conf

Good luck!

michaeljorgense
Path Finder

Thanks @jajung. That was exactly the problem. Once I reconfigured permissions on the app config to "all apps" then the new fields were immediately accessible within Search & Reporting.

0 Karma

michaeljorgense
Path Finder

It looks like this has all come down to a Splunk permissions issue.

I edited the permissions on my app within Splunk & set the permissions for “sharing for config file-only objects” to “all apps” not “this app only (system)” and now Search & Reporting app shows me the fields I've been trying to extract immediately.

So possibly any number of previous attempts/syntax options has been working but permissions weren't allowing me to see it.

0 Karma

woodcock
Esteemed Legend

You keeping using [sourcetype::mjtest] which is wrong; it must be this:

[mjtest]
0 Karma

michaeljorgense
Path Finder

Sorry for any confusion. The only time I used that incorrect stanza name was in my original question, in the REPORT extraction attempt. Once corrected by @FrankVI I changed it & have never posted config since showing me using it incorrectly. I'm not sure why you think I am still using it, but I'm not.

0 Karma

woodcock
Esteemed Legend

Your host_regex is wrong. Try this:

host_regex = \/path\/to\/file\/.*?\/child_AE_(\S+).log
0 Karma

michaeljorgense
Path Finder

Thanks for the suggestion. The host field is being extracted properly using the regex I have posted. I will give your suggestion a try regardless of that fact in case it helps the later props/transforms regex extraction & update you soon.

0 Karma

michaeljorgense
Path Finder

I tried your suggested host_regex config & can confirm that the host field is still being extracted successfully from the file name, i.e. no change there as it was working before.

Unfortunately it hasn't made the props.conf inline extraction work.

inputs.conf:

[monitor:///path/to/file/*/child_AE_*.log]
index=mjtest
sourcetype=mjtest
host_regex = \/path\/to\/file\/.*?\/child_AE_(\S+).log

props.conf:

[mjtest]
EXTRACT-group_and_site = (?<group>\w{3})(?<site>\w{3})\w+ in host
0 Karma

FrankVl
Ultra Champion

I think your first attempt fails because of the EXTRACT name you chose. group,site is not a valid extract name I think. See props.conf spec for details. Just name it EXTRACT-group_and_site or whatever you prefer, the name does not affect which fields are extracted, it is just a label (which I think shouldn't contain ,).

I think your second attempt failed, because you shouldn't add sourcetype:: in the props.conf stanzaname.

michaeljorgense
Path Finder

Looks like I accidentally "commented" on the wrong response. Thanks for your reply @FrankVI, please find my response to it below @woodcock comment.

0 Karma

woodcock
Esteemed Legend

Perfectly read.

0 Karma

michaeljorgense
Path Finder

Thanks very much for your reply Frank.

Unfortunately I tested both suggestions & neither is working, i.e. no 'group' or 'site' field at search time on the data in my index. Not on previously indexed data or fresh data that I add to the file to be ingested.

Please find below my config files for each scenario with the fixes you suggested.

Inline scenario.

inputs.conf:

[monitor:///path/to/file/*/child_AE_*.log]
index=mjtest
sourcetype=mjtest
host_regex = /path/to/file/.*/child_AE_(\w+).log

props.conf:

[mjtest]
EXTRACT-group_and_site = (?<group>\w{3})(?<site>\w{3})\w+ in host

Report scenario.

inputs.conf:

[monitor:///path/to/file/*/child_AE_*.log]
index=mjtest
sourcetype=mjtest
host_regex = /path/to/file/.*/child_AE_(\w+).log

props.conf:

[mjtest]
REPORT-fieldextract = group_site_extract

transforms.conf:

[group_site_extract]
REGEX = (?<group>\w{3})(?<site>\w{3})\w+
SOURCE_KEY = host

Thanks for your help so far! 🙂

0 Karma

FrankVl
Ultra Champion

Do you have any other config in that same props.conf/transforms.conf that does work? Or is it only this bit? I don't see any reason why the config you have wouldn't work, so might be something wrong with the props.conf/transforms.conf file itself.

You can run ./splunk cmd btool props list and ./splunk cmd btool transforms list to get a list of all the configuration that is in use. Check if your config shows up in the results of those commands.

Perhaps check permissions of the file, to ensure Splunk is able to read it. Maybe restart Splunk for good measure.

0 Karma

michaeljorgense
Path Finder

Thanks again for the reply & for confirming that it looks ok to your eyes.

This is a specific app just for testing this new input. Nothing else in inputs.conf, props.conf or transforms.conf that could conflict. In fact no other config files in this app but those files in a local/ sub-dir.

I've used btool to confirm that the props settings were visible previously & also seen them in the 'Field Extractions' dashboard in the Web GUI. So I know they are loaded into splunk successfully.

Permissions are fine (splunk runs as root anyway on this test server) & I have been restarting splunk after every change to props.conf. Trust me when I say that's about 30+ times in the last 2 days before I even logged this question.

Again, thanks for taking the time, I guess I'll take it up with support.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...