topic Re: Multiline field extractions in Getting Data In

Multiline field extractions

nickhills — Mon, 15 Apr 2013 18:12:05 GMT

I have an event which looks like this"

USERNAME            HOME_DIR           USER_INFO
root                /root              root
ec2-user            /home/ec2-user     EC2 Default User
test_user1          /home/test_user1   Testing User
test.user2          /home/test.user2   Test User 2
realuser            /home/realuser     A Real Person

I want to build a field extraction to capture each value from the 3 columns, but i cant get the extraction tool to find any more than one occurrence in any event. I presume this is because it is not attempting multiline extractions, but fiddle and try as I might, i cant get multiline (?m) extractions to work.

Can anyone point me in the correct direction?

Re: Multiline field extractions

Ayn — Mon, 15 Apr 2013 18:22:29 GMT

If that's what your event looks like, using multikv seems to be the perfect tool. http://docs.splunk.com/Documentation/Splunk/5.0.2/SearchReference/Multikv

Re: Multiline field extractions

nickhills — Mon, 15 Apr 2013 18:38:09 GMT

Multikv works beautifully as a search modifier, but is there a way to actually perform a field extraction with it?

Re: Multiline field extractions

Ayn — Mon, 15 Apr 2013 18:40:17 GMT

search modifier? How? From the docs page's description of what multikv does: "Extracts field-values from table-formatted events."

Tell me more about how you're using it as a search modifier?

Re: Multiline field extractions

nickhills — Mon, 15 Apr 2013 18:45:51 GMT

...sorry, I hit comment before I had finished typing...

What I'm trying to achieve is to collect each username into an extracted field, so that i can run reports like "most common username" "host with most users" "rarest username" "which hosts can x login to" etc.

Is there a way to use multikv to extract these in this way?

Re: Multiline field extractions

nickhills — Mon, 28 Sep 2020 13:44:03 GMT

ha, you got a response in before i finished 🙂

at search time, I can run this:

sourcetype="blah"|multikv fields USERNAME HOME_DIR USER_INFO|table USERNAME HOME_DIR USER_INFO

which gives me a nicely formatted table of my events - what I ideally would want to be able to do is simply:

sourcetype=blah |table USERNAME HOME_DIR USER_INFO

Re: Multiline field extractions

Ayn — Mon, 15 Apr 2013 18:56:51 GMT

Look into delimited extraction using REPORT stuff in props.conf / transforms.conf. This page has lots of info on it: http://docs.splunk.com/Documentation/Splunk/5.0.2/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles#Second.2C_configure_a_field_extraction_and_associate_it_with_the_field_transform

In your case it would be something like:

props.conf:

[yoursourcetype]
REPORT-grabfields = grabfields

transforms.conf:

[grabfields]
DELIMS = "\t"
FIELDS = USERNAME,HOME_DIR,USER_INFO

Re: Multiline field extractions

nickhills — Sat, 27 Apr 2013 23:32:01 GMT

I didn't get this to work, but worked around it in another way.

Whilst I am very appreciative of your help, I don't want to mark this as answered, because (for me at least) it isn't 🙂

I may come back to this in a few weeks, so perhaps we can pick this up again.

Thanks again Ayn.

Re: Multiline field extractions

Ayn — Mon, 29 Apr 2013 07:16:57 GMT

Great that you got it working! If you have the time the best thing would be to write an answer to your own question detailing how you solved the problem in the end, then accept your own answer so people can see what worked.

Re: Multiline field extractions

landen99 — Tue, 01 Jul 2014 12:27:05 GMT

Your example is very similar to the solution I posted at:

http://answers.splunk.com/answers/143107/field-extraction-from-space-aligned-fields-in-multi-line-events

Modify the code a little with max_match=3 and a perhaps a few tweaks in the regex. I know the answer to this question is a little late, but it could help others with similar questions.