topic Re: Extract Multiple Values From A Monitored File in Splunk Search

Extract Multiple Values From A Monitored File

barne_dn — Fri, 22 Feb 2013 21:37:36 GMT

Hey Everyone,

I don't know if I'm doing this correctly.

I have a text file that contains data I want to index. So with the universal forwarder I've setup a monitor statement like this:

[monitor://C:\fileToMonitor.txt]

The contents of the file look something like this:

[Fri Feb 22 11:54:51 2013] Serial Number: <333-333-222/12000000>
[Fri Feb 22 11:54:51 2013] Model Type:
[Fri Feb 22 11:54:51 2013] O/S:

I would like to pull the following data:

Serial Number as Field serialNumber
Model Type as Field modelType
And O/S as Field operatingSystem

I'm doing so with a combination of props.conf and transforms.conf

PROPS.CONF
[source::c:\fileToMonitor.txt]
sourcetype = mystuff
REPORT-myst = mystuff_extractions

TRANSFORMS.CONF
[mystuff_extractions]
REGEX = my regex statement?????
FORMAT = serialNumber::$1 modelNumber::$2 operatingSystem::$3

Luckily all of the information is found within the brackets < >. But I am unsure of how to build a regular a regular expression that will do this and pull all three values at the same time. I've tested a few times but have not been successfull.

Aside from getting the regex right, I would like to perform all of this as efficiently as possible. I simple hacked to gether this stuff from different posts I've read, but if there is a better way to do this, please let me know.

Thanks!

Re: Extract Multiple Values From A Monitored File

martin_mueller — Fri, 22 Feb 2013 22:14:58 GMT

The simplest in my mind is to use inline extractions, you can set these through Manager -> Fields -> Field Extractions. As for your expressions, something like this might work (one per extraction, whole line goes into the Inline field):

Serial Number: <(?<serialNumber>[^>]+)>

Model Type: <(?<modelType>[^>]+)>

O/S: <(?<operatingSystem>[^>]+)>

All three work in the same way - they first look for the prefix of your field, for example "O/S: <", then for the field content, at least one character that's not a closing angle bracket, and then for the suffix ">".

If you prefer editing the config files manually, add an EXTRACT-classname key for each to the appropriate props.conf stanza.

Concerning efficiency, these expressions are extremely efficient because they look for a very specific static prefix - in other words, they fail fast. There are no huge trees of options to walk before the automaton notices that it's in the wrong forest.

Re: Extract Multiple Values From A Monitored File

jonuwz — Fri, 22 Feb 2013 22:15:23 GMT

Before you can do that you're going to have to teach splunk what a record looks like. Your data is spread over serveral lines, and by default splunk sees every line as a record. Can you post a larger example of the log, or identify a consistend start or end to an individual record ?

Re: Extract Multiple Values From A Monitored File

barne_dn — Fri, 22 Feb 2013 23:59:25 GMT

This specific log file is very small. There is no larger example.

Re: Extract Multiple Values From A Monitored File

barne_dn — Mon, 25 Feb 2013 15:17:24 GMT

I tried this out and it worked. But I prefer to manage my configurations through text files. Is there a way to get this working through the transforms.conf?

Re: Extract Multiple Values From A Monitored File

martin_mueller — Mon, 25 Feb 2013 15:36:55 GMT

Do keep in mind, every setting in the manager eventually makes its way into a configuration file. In this case it's an EXTRACT-foo statement in props.conf - you can edit this manually of course.

I'm sure you can do the same with REPORT-foo and a stanza in transforms.conf, but it feels like more configuration effort for no functionality gain.

Re: Extract Multiple Values From A Monitored File

barne_dn — Mon, 25 Feb 2013 18:25:41 GMT

That's the thing. I created this through the GUI and looked for the configuration file change but I don't see it. I'm looking in system/local config files. I looked in both the props.conf and transforms.conf. Should I be looking at another props.conf file in one of the other directories? I was under the impression that all changes should be made in the system/local directory.

Re: Extract Multiple Values From A Monitored File

martin_mueller — Mon, 25 Feb 2013 18:38:30 GMT

Don't forget to check the user-specific directories if you didn't set the extraction permissions to global.

Re: Extract Multiple Values From A Monitored File

barne_dn — Mon, 25 Feb 2013 18:42:00 GMT

It was in my apps/search/local/. Still one more thing bugging me. I'd like to at least test this out by using the transforms.conf. Would you be able to provide me the right syntax so I can compare the results? Thanks!

Re: Extract Multiple Values From A Monitored File

martin_mueller — Mon, 25 Feb 2013 18:44:47 GMT

It's right here: http://docs.splunk.com/Documentation/Splunk/5.0.1/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles#Create_advanced_search-time_field_extractions_with_field_transforms