Getting Data In

Can you help me configure props.conf to linebreak the following SQL statement?

damucka
Builder

Hello,

I have the following log lines (SQL statements) coming from one of the source files into my index:

#TRUNCATE TABLE "/BI0/0600000300"
208028;797;0;893476090372217;8fd4bddbec78f21b5e80a22756f1f082;SAPBWP;7:(D/C/I)######;1541415299457229;11306;;0;;B9D44D5CE0E411E8864E00000D7B145E;25b492e2e0e911e8afd900000d7b145e;5BDFCC8F6D947536E10000000A433AF3;1;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWP;0;6:SAPBWP;
#TRUNCATE TABLE "/BI0/0600000134"
204559;228;0;878577980540115;18718668973202396e3f9760d92a6ad0;SAPBWP;7:(D/C/I)######;1541415299540417;6549;;0;;F00A371EE0E811E88B1200000D7B14C2;2bd1a5abe0e911e8cf6700000d7b14c2;23EEBB50E0E911E8C538F8790A433AF7;72;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWP;0;6:SAPBWP;
#TRUNCATE TABLE "/BI0/0600000368"
239305;1813;0;1027807627168023;2169a1f7b0a4a7a41201ce02a8128bc6;SAPBWP;7:(D/C/I)######;1541415299665464;10308;;0;;B9D44D5CE0E411E8864E00000D7B145E;25b492e2e0e911e8afd900000d7b145e;5BDFAF6F6C63701BE10000000A433AF3;1;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWP;0;6:SAPBWP;

Splunk puts it all into one event.
How would I configure the props.conf to tell Splunk to create separate events for each statement, which would be here a line beginning with # sign?

It's important that this is only for one file (source). The rest of the files should get recognized properly. Is there a way to set the event boundaries per file type ingested?

Kind Regards,
Kamil

0 Karma
1 Solution

damucka
Builder

Hello @ddrillic

Thank you for your answer.
In the meantime I realized that my logfile looks a bit different and the line breaker should be the empty line and not a hash sign, the line with the hash is actually the last one.

218330;987;0;937722769408999;e4cffe25f6e83e37671d5edf961d0cd5;SAPBWP;8:BWREMOTE;1541498512374808;2002;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTS"

218330;987;0;937722455015181;b8acdc65f2da6ecc147a5d7457a24714;SAPBWP;8:BWREMOTE;1541498513021804;1944;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTT"

218330;987;0;937722603342999;f5ede1d4fc60153431c09208e3d2b854;SAPBWP;8:BWREMOTE;1541498513110544;2044;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTU"

So, like first all the parmeters of the SQL sttement and then the SQL itself followed by the hash.
For that I found the following line breaker pattern in one of the Splunk Answers:

[ISP_statements]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((?:\r?\n){2,})

I am going to try it and let you know if it worked. I am bit dependant here on my Splunk admin, cannot do it myself so it an take a while.
Thank you for your support.

BR, Kamil

View solution in original post

0 Karma

damucka
Builder

Hello @ddrillic

Thank you for your answer.
In the meantime I realized that my logfile looks a bit different and the line breaker should be the empty line and not a hash sign, the line with the hash is actually the last one.

218330;987;0;937722769408999;e4cffe25f6e83e37671d5edf961d0cd5;SAPBWP;8:BWREMOTE;1541498512374808;2002;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTS"

218330;987;0;937722455015181;b8acdc65f2da6ecc147a5d7457a24714;SAPBWP;8:BWREMOTE;1541498513021804;1944;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTT"

218330;987;0;937722603342999;f5ede1d4fc60153431c09208e3d2b854;SAPBWP;8:BWREMOTE;1541498513110544;2044;;0;;40F2E99714821ED8B69FFF6A240E24E3;60B0935984750310E005BE144DBB7E18;00000000000000
000000000000000000;0;37:CL_SQL_STATEMENT==============CP:1163;8:ABAP:BWT;0;6:SAPBWP;
#TRUNCATE TABLE "TESTDATRNRPARTU"

So, like first all the parmeters of the SQL sttement and then the SQL itself followed by the hash.
For that I found the following line breaker pattern in one of the Splunk Answers:

[ISP_statements]
SHOULD_LINEMERGE = false
LINE_BREAKER = ((?:\r?\n){2,})

I am going to try it and let you know if it worked. I am bit dependant here on my Splunk admin, cannot do it myself so it an take a while.
Thank you for your support.

BR, Kamil

0 Karma

ddrillic
Ultra Champion

Please try -

[<your sourcetype>] 
disabled=false 
LINE_BREAKER=^#
SHOULD_LINEMERGE=false 

alt text

0 Karma

ddrillic
Ultra Champion

@damucka - have you tried it by any chance?

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...