Extract specific fields from unstructured logs

Nidd — Mon, 03 May 2021 04:40:20 GMT

I have a list of unstructured logs like below for which I have to extract certain fields. Tried using "Extract fields" option to pull these fields but not receiving the results as expected.

Can someone please help on a way to achieving this through Splunk search query itself?

Fields to be extracted:

1. myemail@site.com

2. my-pipeline

3. JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: ORA-01017: invalid username/password; logon denied

Logs:

2021-05-02 11:21:13,663 [user:*myemail@site.com] [pipeline:my-pipeline (SCH Test Run)/testRun__12a-23b-34c-45d-56d_site.com__myemail@site.com] [runner:] [thread:ProductionPipelineRunnable-testRun__12a-23b-34c-45d-56d_site.com__myemail@site.com-my-pipeline (SCH Test Run)] [stage:] ERROR JdbcSource - Cannot connect to specified database: com.streamsets.pipeline.api.StageException: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: ORA-01017: invalid username/password; logon denied com.streamsets.pipeline.api.StageException: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: ORA-01017: invalid username/password; logon denied 2021-03-01 04:18:26,910 [user:*myemail@site.com] [pipeline:my-pipeline (SCH Test Run)/testRun__12a-23b-34c-45d-56d_site.com__myemail@site.com] [runner:] [thread:ProductionPipelineRunnable-testRun__12a-23b-34c-45d-56d_site.com__myemail@site.com-my-pipeline (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested in connect descriptor 2021-03-01 04:43:12,985 [user:*myemail@site.com] [pipeline:my-pipeline (SCH Test Run)/testRun__12a-23b-34c-45d-56d_site.com__myemail@site.com] [runner:] [thread:ProductionPipelineRunnable-testRun__12a-23b-34c-45d-56d_site.com__myemail@site.com-my-pipeline (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: The Network Adapter could not establish the connection com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: The Network Adapter could not establish the connection 2021-03-01 05:02:13,113 [user:*myemail@site.com] [pipeline:my-pipeline (SCH Test Run)/testRun__12a-23b-34c-45d-56d_site.com__myemail@site.com] [runner:] [thread:ProductionPipelineRunnable-testRun__12a-23b-34c-45d-56d_site.com__myemail@site.com-my-pipeline (SCH Test Run)] [stage:] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: Unknown host specified com.streamsets.datacollector.runner.PipelineRuntimeException: CONTAINER_0800 - Can't start pipeline due 1 validation error(s). First one: JDBC_06 - Failed to initialize connection pool: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: IO Error: Unknown host specified

Thank you in advance.

Re: Extract specific fields from unstructured logs

ITWhisperer — Mon, 03 May 2021 06:14:59 GMT

| rex "user:\*(?<user>[^\]]+)\]\s\[pipeline:(?<pipeline>[^\s]+).*:\s(?<error>[^:]+)"

Re: Extract specific fields from unstructured logs

Nidd — Tue, 04 May 2021 14:31:32 GMT

Thank you very much ITWhisperer !! 🙂 Worked !!

topic Re: Extract specific fields from unstructured logs in Splunk Search

Extract specific fields from unstructured logs

Re: Extract specific fields from unstructured logs

Re: Extract specific fields from unstructured logs