Getting Data In

Can you help me configure props.conf to linebreak the following SQL statement?

damucka
Builder

Hello,

I have the following log lines (SQL statements) coming from one of the source files into my index:

#TRUNCATE TABLE "/BI0/0600000300"
208028;797;0;893476090372217;8fd4bddbec78f21b5e80a22756f1f082;SAPBWP;7:(D/C/I)######;1541415299457229;11306;;0;;B9D44D5CE0E411E8864E00000D7B145E;25b492e2e0e911e8afd900000d7b145e;5BDFCC8F6D947536E10000000A433AF3;1;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWP;0;6:SAPBWP;
#TRUNCATE TABLE "/BI0/0600000134"
204559;228;0;878577980540115;18718668973202396e3f9760d92a6ad0;SAPBWP;7:(D/C/I)######;1541415299540417;6549;;0;;F00A371EE0E811E88B1200000D7B14C2;2bd1a5abe0e911e8cf6700000d7b14c2;23EEBB50E0E911E8C538F8790A433AF7;72;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWP;0;6:SAPBWP;
#TRUNCATE TABLE "/BI0/0600000368"
239305;1813;0;1027807627168023;2169a1f7b0a4a7a41201ce02a8128bc6;SAPBWP;7:(D/C/I)######;1541415299665464;10308;;0;;B9D44D5CE0E411E8864E00000D7B145E;25b492e2e0e911e8afd900000d7b145e;5BDFAF6F6C63701BE10000000A433AF3;1;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWP;0;6:SAPBWP;

Splunk puts it all into one event.
How would I configure the props.conf to tell Splunk to create separate events for each statement, which would be here a line beginning with # sign?

It's important that this is only for one file (source). The rest of the files should get recognized properly. Is there a way to set the event boundaries per file type ingested?

Kind Regards,
Kamil

0 Karma
1 Solution

damucka
Builder

Hello @ddrillic

Thank you for your answer.
In the meantime I realized that my logfile looks a bit different and the line breaker should be the empty line and not a hash sign, the line with the hash is actually the last one.

218330;987;0;937722769408999;e4cffe25f6e83e37671d5edf961d0cd5;SAPBWP;8:BWREMOTE;1541498512374808;2002;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTS"

218330;987;0;937722455015181;b8acdc65f2da6ecc147a5d7457a24714;SAPBWP;8:BWREMOTE;1541498513021804;1944;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTT"

218330;987;0;937722603342999;f5ede1d4fc60153431c09208e3d2b854;SAPBWP;8:BWREMOTE;1541498513110544;2044;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTU"

So, like first all the parmeters of the SQL sttement and then the SQL itself followed by the hash.
For that I found the following line breaker pattern in one of the Splunk Answers:

[ISP_statements]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((?:\r?\n){2,})

I am going to try it and let you know if it worked. I am bit dependant here on my Splunk admin, cannot do it myself so it an take a while.
Thank you for your support.

BR, Kamil

View solution in original post

0 Karma

damucka
Builder

Hello @ddrillic

Thank you for your answer.
In the meantime I realized that my logfile looks a bit different and the line breaker should be the empty line and not a hash sign, the line with the hash is actually the last one.

218330;987;0;937722769408999;e4cffe25f6e83e37671d5edf961d0cd5;SAPBWP;8:BWREMOTE;1541498512374808;2002;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTS"

218330;987;0;937722455015181;b8acdc65f2da6ecc147a5d7457a24714;SAPBWP;8:BWREMOTE;1541498513021804;1944;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTT"

218330;987;0;937722603342999;f5ede1d4fc60153431c09208e3d2b854;SAPBWP;8:BWREMOTE;1541498513110544;2044;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTU"

So, like first all the parmeters of the SQL sttement and then the SQL itself followed by the hash.
For that I found the following line breaker pattern in one of the Splunk Answers:

[ISP_statements]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((?:\r?\n){2,})

I am going to try it and let you know if it worked. I am bit dependant here on my Splunk admin, cannot do it myself so it an take a while.
Thank you for your support.

BR, Kamil

0 Karma

ddrillic
Ultra Champion

Please try -

[<your sourcetype>] 
disabled=false 
LINE_BREAKER=^#
SHOULD_LINEMERGE=false 

alt text

0 Karma

ddrillic
Ultra Champion

@damucka - have you tried it by any chance?

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...