topic Re: How to make multiple field extractions from my sample data? in Splunk Search

How to make multiple field extractions from my sample data?

sunnyparmar — Tue, 29 Sep 2015 11:23:28 GMT

Hi,

I have my logs with multiple events for which I have to make field extractions. From the first timestamp, I have to make an extraction for System.Net.WebException, from the second timestamp, I have to make for Exception:System.AggregateException, and from third timestamp, Exception: Error in DB-Logging. With this I have to show total number of counts for each exception event on an hourly basis. Can anybody guide me here on how to do this?

TIMESTAMP - [2015-09-20 21:04:25.484] THREAD ID - [44] CONTENT - Error in IsDirectoryExists/CreateDirectoryIfNotExists for directory:segro/extracts_upload/ Server: ftp://serverip Excecption: System.Net.WebException: The remote server returned an error: (550) File unavailable (e.g., file not found, no access).
TIMESTAMP - [2015-09-18 04:32:41.068] THREAD ID - [7] CONTENT - Non-recoverable Error occurred, Service will run again after 5 minutes. Exception:System.AggregateException: One or more errors occurred. ---> System.UnauthorizedAccessException: Access to the path 'File path name' is denied.
TIMESTAMP - [2015-09-16 09:15:43.508] THREAD ID - [8] CONTENT - Database Update Error for File: CurrencyExtract.txt DatabaseServer: serverip Exception: Error in DB-Logging(UpsertFtpTransfer): A network-related or instance-specific error occurred while establishing a connection to SQL Server.

Thanks

Re: How to make multiple field extractions from my sample data?

muebel — Tue, 29 Sep 2015 11:57:45 GMT

presuming that the first event's "Excecption" is a typo somehow, you could use that pattern to setup transforms to extract that field for this sourcetype

EXTRACT-extract_ip = \sException:(?.*):

You'd want to test this out to make sure it covers the sourcetype as you'd want, but this is the general process.

Re: How to make multiple field extractions from my sample data?

sunnyparmar — Tue, 29 Sep 2015 12:22:02 GMT

Thanks for the reply but getting the following error if try to changed with my current extraction -

Encountered the following error while trying to update: In handler 'props-extract': Regex: unrecognized character after (? or (?-

Re: How to make multiple field extractions from my sample data?

woodcock — Tue, 29 Sep 2015 19:19:18 GMT

Like this:

... | rex "\s+Exception:\s*(?<Exception>\S+)" | timechart span=1h count by Exception

Re: How to make multiple field extractions from my sample data?

sunnyparmar — Wed, 30 Sep 2015 08:11:52 GMT

Hi,

Thanks for replying and the query given. it is giving result but with that it is fetching some extra fields from column like Could, NULL, Unable so could you please suggest why extraction fetching these fields? Do we need to modify more.
Thanks & Regards

Re: How to make multiple field extractions from my sample data?

woodcock — Wed, 30 Sep 2015 14:24:23 GMT

It is extracting those fields because you have some events that look like that. This search will show you an example of each:

 ... | rex "\s+Exception:\s*(?<Exception>\S+)" | dedup Exception | table Exception _raw

To get rid of them just tack on this:

... | fields <list of fields to keep>

Re: How to make multiple field extractions from my sample data?

sunnyparmar — Wed, 30 Sep 2015 14:32:32 GMT

With your first query still it is fetching that columns and yes currently i am doing in this way as you previous also told me in my some another question but i want to avoid these columns without taking help of "fields" . With this I think extraction is not giving the exact result because it is fetching the next word "could" after the "Exception" event. In my logs file in some lines after exception word there is line "could not find the file" so it is fetching "could" word in output from this line. So do you have any suggestion regarding this?
Regards

Re: How to make multiple field extractions from my sample data?

woodcock — Wed, 30 Sep 2015 14:40:59 GMT

If you give a sample of each log variation, we can work on updating the regex command to properly accommodate all variations.

Re: How to make multiple field extractions from my sample data?

sunnyparmar — Thu, 01 Oct 2015 05:29:32 GMT

I have my logs with multiple events for which I have to make field extractions. From the first timestamp, I have to make an extraction for System.Net.WebException, from the second timestamp, I have to make for Exception:System.AggregateException, and from third timestamp, Exception: Error in DB-Logging. In fourth time stamp if you will see that is also related to DB-logging error but it has word "could" also which is fetching in your previous regex command. With this I have to show total number of counts for each exception event on an hourly basis.

 TIMESTAMP - [2015-09-20 21:04:25.484] THREAD ID - [44] CONTENT - Error in IsDirectoryExists/CreateDirectoryIfNotExists for directory:segro/extracts_upload/ Server: ftp://serverip Excecption: System.Net.WebException: The remote server returned an error: (550) File unavailable (e.g., file not found, no access).
 TIMESTAMP - [2015-09-18 04:32:41.068] THREAD ID - [7] CONTENT - Non-recoverable Error occurred, Service will run again after 5 minutes. Exception:System.AggregateException: One or more errors occurred. ---> System.UnauthorizedAccessException: Access to the path 'File path name' is denied.
 TIMESTAMP - [2015-09-16 09:15:43.508] THREAD ID - [8] CONTENT - Database Update Error for File: CurrencyExtract.txt DatabaseServer: serverip Exception: Error in DB-Logging(UpsertFtpTransfer): A network-related or instance-specific error occurred while establishing a connection to SQL Server.

TIMESTAMP - [2015-09-16 09:13:40.558] THREAD ID - [20] CONTENT - Database Update Error for File: 003701466427_SupplierExtract.txt DatabaseServer: 10.0.22.52 Exception: Error in DB-Logging(UpsertFtpTransfer): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

Regards

Re: How to make multiple field extractions from my sample data?

sunnyparmar — Sun, 04 Oct 2015 08:56:16 GMT

Hi, any update? Thanks

Re: How to make multiple field extractions from my sample data?

woodcock — Sun, 04 Oct 2015 14:29:05 GMT

Switch to this:

... | rex "\s+Exception:\s*(?<Exception>[^:]+)" | dedup Exception | table Exception _raw