Grouping of similar events

[ChhayaV]

Hi,

How can we associate log entries that lead to a particular issue.
I mean suppose there is button click event(Say some form submission), if it fails then the log entries will be different from the entries of successful submission. This activity(form submission) is recorded as 10 log entries. In case of failure some log entries say 6 (if not all) will indicate what went wrong. When we load this data into splunk, it creates 10 events corresponding to 10 log entries.
So question is how to group these 6 entries which lead us to the issue?
I read that this can be done using "Transaction" search command? but I'm not sure.

After reading documentation, i still dont know how to start?
Please if anyone has done similar thing or know about transaction search command, help me a bit

Sample log entries

Timestamp               Process                                     TID     Area                            Category                        EventID Level       Message     Correlation
06/14/2013 04:56:06.18  OWSTIMER.EXE (0x30E0)                       0x2A18  SharePoint Foundation           Monitoring                      nasq    Medium      Entering monitored scope (Timer Job job-email-delivery) e2e3ff09-aefe-47e7-960c-b350b0655f96

06/14/2013 04:56:06.19  OWSTIMER.EXE (0x30E0)                       0x2A18  SharePoint Foundation           E-Mail                          6871    Information The Incoming E-Mail service has completed a batch.  The elapsed time was 00:00:00.  The service processed 0 message(s) in total.    e2e3ff09-aefe-47e7-960c-b350b0655f96

06/14/2013 04:56:06.19  w3wp.exe (0x14C4)                           0x2474  SharePoint Foundation           Topology                        e5mb    Medium      WcfReceiveRequest: LocalAddress: 'http://insprodsp.puneodc.lntinfotech.com:32843/f2edf5b451ae473a9f0c189f38392426/MetadataWebService.svc' Channel: 'System.ServiceModel.Channels.ServiceChannel' Action: 'http://schemas.microsoft.com/sharepoint/taxonomy/soap/IDataAccessReadOnly/GetChanges' MessageId: 'urn:uuid:da3e299a-c06d-4c14-bd4f-9eb7ab101483'  441c5da4-3330-4d9c-bee7-6d1210f228bc

06/14/2013 04:56:06.19  w3wp.exe (0x14C4)                           0x2474  SharePoint Foundation           Monitoring                      nasq    Medium      Entering monitored scope (ExecuteWcfServerOperation)    441c5da4-3330-4d9c-bee7-6d1210f228bc

[starcher · Accepted solution]

You have to have a common field to match on for the transaction command. In this case you need to extract that ID to a field. maybe call it correlation_id. Then you just add | transaction correlation_id to the end of your search and that will group them. If your own in house developers control those logs I would just get them to modify the logging to do a key value pair in the data so splunk auto extracts it. like "correleation_id=e2e3ff09-aefe-47e7-960c-b350b0655f96"

[Ayn]

If you have something to correlate on the rest is just a matter of syntax. Or for that matter any other kind of rule, though it might be trickier to implement. When you look at these log files, how do you know that these events belong together (ie how do you sort out other log events that are also happening at the same time)?

[ChhayaV]

the last field i.e. correlation id (441c5da4-3330-4d9c-bee7-6d1210f228bc).

Suppose if there is no such field as common ID then
how to do this?

[Ayn]

Is there any common ID for these requests? How can you tell that these specific events belong together?

[ChhayaV]

Hi Ayn,

I've added sample entries of logs. I didn't see upload option to load one entire log file.

[Ayn]

Very likely you could do this using the transaction command, but it's hard to give you more specific information without log samples.