Getting Data In

Can you help me configure props.conf to linebreak the following SQL statement?

damucka
Builder

Hello,

I have the following log lines (SQL statements) coming from one of the source files into my index:

#TRUNCATE TABLE "/BI0/0600000300"
208028;797;0;893476090372217;8fd4bddbec78f21b5e80a22756f1f082;SAPBWP;7:(D/C/I)######;1541415299457229;11306;;0;;B9D44D5CE0E411E8864E00000D7B145E;25b492e2e0e911e8afd900000d7b145e;5BDFCC8F6D947536E10000000A433AF3;1;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWP;0;6:SAPBWP;
#TRUNCATE TABLE "/BI0/0600000134"
204559;228;0;878577980540115;18718668973202396e3f9760d92a6ad0;SAPBWP;7:(D/C/I)######;1541415299540417;6549;;0;;F00A371EE0E811E88B1200000D7B14C2;2bd1a5abe0e911e8cf6700000d7b14c2;23EEBB50E0E911E8C538F8790A433AF7;72;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWP;0;6:SAPBWP;
#TRUNCATE TABLE "/BI0/0600000368"
239305;1813;0;1027807627168023;2169a1f7b0a4a7a41201ce02a8128bc6;SAPBWP;7:(D/C/I)######;1541415299665464;10308;;0;;B9D44D5CE0E411E8864E00000D7B145E;25b492e2e0e911e8afd900000d7b145e;5BDFAF6F6C63701BE10000000A433AF3;1;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWP;0;6:SAPBWP;

Splunk puts it all into one event.
How would I configure the props.conf to tell Splunk to create separate events for each statement, which would be here a line beginning with # sign?

It's important that this is only for one file (source). The rest of the files should get recognized properly. Is there a way to set the event boundaries per file type ingested?

Kind Regards,
Kamil

0 Karma
1 Solution

damucka
Builder

Hello @ddrillic

Thank you for your answer.
In the meantime I realized that my logfile looks a bit different and the line breaker should be the empty line and not a hash sign, the line with the hash is actually the last one.

218330;987;0;937722769408999;e4cffe25f6e83e37671d5edf961d0cd5;SAPBWP;8:BWREMOTE;1541498512374808;2002;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTS"

218330;987;0;937722455015181;b8acdc65f2da6ecc147a5d7457a24714;SAPBWP;8:BWREMOTE;1541498513021804;1944;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTT"

218330;987;0;937722603342999;f5ede1d4fc60153431c09208e3d2b854;SAPBWP;8:BWREMOTE;1541498513110544;2044;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTU"

So, like first all the parmeters of the SQL sttement and then the SQL itself followed by the hash.
For that I found the following line breaker pattern in one of the Splunk Answers:

[ISP_statements]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((?:\r?\n){2,})

I am going to try it and let you know if it worked. I am bit dependant here on my Splunk admin, cannot do it myself so it an take a while.
Thank you for your support.

BR, Kamil

View solution in original post

0 Karma

damucka
Builder

Hello @ddrillic

Thank you for your answer.
In the meantime I realized that my logfile looks a bit different and the line breaker should be the empty line and not a hash sign, the line with the hash is actually the last one.

218330;987;0;937722769408999;e4cffe25f6e83e37671d5edf961d0cd5;SAPBWP;8:BWREMOTE;1541498512374808;2002;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTS"

218330;987;0;937722455015181;b8acdc65f2da6ecc147a5d7457a24714;SAPBWP;8:BWREMOTE;1541498513021804;1944;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTT"

218330;987;0;937722603342999;f5ede1d4fc60153431c09208e3d2b854;SAPBWP;8:BWREMOTE;1541498513110544;2044;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTU"

So, like first all the parmeters of the SQL sttement and then the SQL itself followed by the hash.
For that I found the following line breaker pattern in one of the Splunk Answers:

[ISP_statements]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((?:\r?\n){2,})

I am going to try it and let you know if it worked. I am bit dependant here on my Splunk admin, cannot do it myself so it an take a while.
Thank you for your support.

BR, Kamil

0 Karma

ddrillic
Ultra Champion

Please try -

[<your sourcetype>] 
disabled=false 
LINE_BREAKER=^#
SHOULD_LINEMERGE=false 

alt text

0 Karma

ddrillic
Ultra Champion

@damucka - have you tried it by any chance?

0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! &#x1f44f; Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...