Extract field from event with repeated information

[attgjh1]

Hi, ive asked my qn below after my event logs shown:

Example logs:

part of event A:

... ... (other details of event not shown)
Socket connect|10.107.3.157:5000|Retry 1 - starting
Socket connect|10.107.3.157:5000|Retry 1 - connecting
Socket connect|10.107.3.157:5000|Time taken is 9798ms
Socket connect|10.107.3.157:5000|Retry 1 - connected

part of event B:

... ... (other details of event not shown)
Socket connect|10.107.3.157:5000|Retry 1 - starting
Socket connect|10.107.3.157:5000|Retry 1 - connecting
Socket connect|10.107.3.157:5000|Retry 1 - failed
Socket connect|10.107.3.157:5000|Retry 2 - starting
Socket connect|10.107.3.157:5000|Retry 2 - connecting
Socket connect|10.107.3.157:5000|Retry 2 - failed

this is my regex:

(?im)Socket\sconnect\|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{2,5}\|Retry\s(?P<FIELDNAME>\d)\s\-\sstarting

Help!
im trying to extract the number of tries. (i.e. retry # ), # is the value
my regex is only able to identify the '1' but not '2' or '3' or any subsequent numbers.

using "failed|connected" in place of "starting" in my regex also gives an error from the field extractor. Hope someone can teach me why this an error too.

Thanks alot. regex is pretty confusing 😞

[kristian_kolb · Accepted solution]

What you need to do is to make your field multivalued. And your regex need not be that complicated. The following settings should be made on your Search Head, or the Indexer if you do not have a dedicated Search Head.

in props.conf

[your_sourcetype]
REPORT-retries = retry_mv

in transforms.conf

[retry_mv]
REGEX = Retry\s(\d+)
FORMAT = Retry::$1
MV_ADD = True

Then you can find the highest number of retries through the max() function of stats, e.g.

sourcetype=your_sourcetype | stats max(Retry) AS "Number of Retries" | ...

---

UPDATE:

Of course there are just 2 events. And 'Retry' occurs more than once in each event. And you want to find the highest value of Retry in each event. Thats pretty much the idea of multivalued fields.

Like mail, where a single message may have more than one recipient, you could say that the 'To:' field is multivalued.

As for the config files, and how to edit them:

1. Go to /opt/splunk/etc/system/local
   
2. Open/create a file named props.conf.
3. Enter the settings provided above (but use the real sourcetype name).
   
4. Save.
   
5. Create/open transforms.conf
6. Copy settings from above.
   
7. Save.

8. Make sure that file ownership and access permissions are correct for any file you created, i.e. same as the other .conf files.

9. Go to the main Search app, and run the following search (yes it starts with a pipe)

| extract reload=t

Your configuration changes should now be loaded, and the Retry field should be multivalued.

For more information, see:

http://docs.splunk.com/Documentation/Splunk/4.3.1/Admin/Aboutconfigurationfiles

Hope this helps,

Kristian

[attgjh1]

thanks alot. haven implemented it. gonna try.
ok. weird. cant access my etc folder T_T

[kristian_kolb]

se update above.

[attgjh1]

thanks. But currently, the entire example u see are 2 separate events. each line != a single event. They have additional info like their source at the header and more stuffs like message at the btm of the event. (mainly the socket connection shows whether the msg could eventually be sent)

i dont quite understand .conf files as im using splunk web. =/ do u mind explaining further?