Splunk Search

ingest line from log file match with multiple regular expression to splunk indexer

Abhineet
Loves-to-Learn Everything

Hi,

Below red highlighted is sample log file.

Sample LogFile

12:08:32.797 [6] (null) DEBUG Bastian.Exacta.AMAT.ImportAdapter.Wcf.AMATWcfImport - JSON received for product import: {"records":[{"lgnum":"407","entitled":"4070","owner":"4070","product":"0205-02304","prd_descr":"PACKAGING, RUNNING BEAM GRIPPERS, REFLEX","base_uom":"EA","gross_weight":"0.000","net_weight":"1.000","weight_uom":"KG","volume":"6480.000","volume_uom":"CCM","length":"40.000","width":"18.000","height":"9.000","dimension_uom":"CM","serial_profile":null,"batch_req":null,"cycle_count_ind":"C","alternative_uom":"EA","shelf_life_flag":null,"shelf_life":null,"req_min_shelf_life":null,"req_max_shelf_life":null,"std_cost":"10.61","matnr":"0205-02304","suffix":null,"rev_level":"01","extension":null}]}
12:08:32.797 [6] (null) DEBUG Bastian.Exacta.Business.Xml.XmlEntity - Started saving XML entity of type 'ProductImportData'
12:08:32.844 [6] (null) DEBUG Bastian.Exacta.Business.Xml.XmlEntity - Finished XML entity of type 'ProductImportData'. Result:
<?xml version="1.0" encoding="utf-16" standalone="yes"?>
<PROD NAME="0205-02304">

14:54:00.242 [8] (null) DEBUG Bastian.Exacta.AMAT.ImportAdapter.Wcf.AMATWcfImport - JSON received for order line cancel import: {"records":[{"Header":{"lgnum":"407","who":"47708597","canrq":"X"},"Detail":[{"tanum":"97908517"}]}]}
14:54:00.242 [8] (null) DEBUG Bastian.Exacta.Business.Persistance.SessionFactory - Opening NHibernate session using the production factory...
14:54:00.258 [8] (null) DEBUG NHibernate.SQL - select order0_.ORDER_TYPE as col_0_0_ from ORDER_HEADER order0_ where order0_.ORDER_NAME=@p0 ORDER BY CURRENT_TIMESTAMP OFFSET 0 ROWS FETCH FIRST 1 ROWS ONLY;@p0 = '47708597' [Type: String (4000:0:0)]
14:54:00.273 [8] (null) DEBUG Bastian.Exacta.Business.Persistance.SessionFactory - Closing NHibernate session...
14:54:00.273 [8] (null) INFO Bastian.Exacta.AMAT.ImportAdapter.Wcf.AMATWcfImport - Creating order cancellation transaction for order 47708597, OrderType : 0
14:54:00.289 [8] (null) DEBUG Bastian.Exacta.Business.Persistance.SessionFactory - Opening NHibernate session using the production factory...
14:54:00.320 [8] (null) DEBUG NHibernate.SQL - select orderline1_.ORDER_LINE_ID as order1_236_, orderline1_.ORDER_LINE_TYPE as order2_236_, orderline1_.LINE_NUM as line3_236_, orderline1_.LOT_NUM_REQUESTED as lot4_236_, orderline1_.QTY_REQUESTED as qty5_236_, orderline1_.UOM_SPECIFIED as uom6_236_, orderline1_.SERIAL_NUM_REQUESTED as serial7_236_, orderline1_.SINGLE_LOT as single8_236_, orderline1_.DAYS_TO_EXPIRE as days9_236_, orderline1_.VAS as vas10_236_, orderline1_.KITTING as kitting11_236_, orderline1_.DEST_ZONE as dest12_236_, orderline1_.SOURCE_ZONE as source13_236_, orderline1_.SEQ_NUM as seq14_236_, orderline1_.RETURNED_INV as returned15_236_, orderline1_.WGT_REQUESTED as wgt16_236_, orderline1_.INVENTORY_GROUP as inventory17_236_, orderline1_.TOTAL_RECEIPT_QUANTITY as total18_236_, orderline1_.LOT_REVISION as lot19_236_, orderline1_.SERIAL_NUM_REQUIRED as serial20_236_, orderline1_.CAPTURE_COUNTRY_OF_ORIGIN as capture21_236_, orderline1_.SECONDARY_SCAN_TYPE as secondary22_236_, orderline1_.SUPPRESS_SCANS_AT_PICK as suppress23_236_, orderline1_.SHOULD_PICK_RESERVED_INVENTORY as should24_236_, orderline1_.QUAR_REASON as quar25_236_, orderline1_.INVOICE_NUMBER as invoice26_236_, orderline1_.INVENTORY_RESERVATION_KEY as inventory27_236_, orderline1_.SSU_VALUE_PER_ITEM as ssu28_236_, orderline1_.PROD_ID as prod29_236_, orderline1_.UOM_TYPE_REQUESTED as uom30_236_, orderline1_.ORDER_ID as order31_236_, orderline1_.WAVE_ID as wave32_236_, orderline1_.ROUTE_ID as route33_236_, orderline1_.DOCK_ID as dock34_236_, orderline1_.DEST_WAREHOUSE_ID as dest35_236_, orderline1_.SOURCE_WAREHOUSE_ID as source36_236_, orderline1_.DOCUMENT_ID as document37_236_, orderline1_.ADJUSTMENT_ORDER_ID as adjustment38_236_, orderline1_.BOM_ID as bom39_236_, orderline1_.BOM_LINE_ID as bom40_236_, orderline1_.BOM_PARENT_LINE_ID as bom41_236_, orderline1_.PREFERRED_CNTNR_PATTERN_ID as preferred42_236_, orderline1_.COUNTRY_OF_ORIGIN as country43_236_ from ORDER_LINE_DETAIL orderlined0_ inner join ORDER_LINE orderline1_ on orderlined0_.ORDER_LINE_ID=orderline1_.ORDER_LINE_ID inner join ORDER_HEADER order2_ on orderline1_.ORDER_ID=order2_.ORDER_ID where order2_.ORDER_NAME=@p0 and orderlined0_.DETAIL_TYPE=@p1 and (orderlined0_.DETAIL_VALUE in (@p2));@p0 = '47708597' [Type: String (4000:0:0)], @p1 = 1000 [Type: Decimal (0:10:29)], @p2 = '97908517' [Type: String (4000:0:0)]
14:54:00.336 [8] (null) DEBUG Bastian.Exacta.Business.Persistance.SessionFactory - Closing NHibernate session...
14:54:00.336 [8] (null) INFO Bastian.Exacta.AMAT.ImportAdapter.Wcf.AMATWcfImport - No order lines found for order 47708597 for order line cancellation request, cannot proceed with cancellation transaction.
14:54:00.352 [8] (null) WARN Bastian.Exacta.AMAT.ImportAdapter.Wcf.AMATWcfImport - Exacta Event

<ORDER CANCEL="N" ORDER_NAME="47708600" TYPE="2">
<DETAIL TYPE="1005" />
<TRAILER_STOP>0</TRAILER_STOP>
<ORDER_PRIORITY>1</ORDER_PRIORITY>
<ORDER_LINE CANCEL="N" LINE_NUM="1">
<PROD_NAME>0010-01283</PROD_NAME>
<PROD_COMPANY_NAME>4070</PROD_COMPANY_NAME>
<PROD_VENDOR_NAME>4070</PROD_VENDOR_NAME>
<QTY_REQUESTED>1</QTY_REQUESTED>
<DETAIL TYPE="1000" VALUE="97908520" />
<DETAIL TYPE="1001" VALUE="1" />

<?xml version="1.0" encoding="utf-16" standalone="yes"?>
<ORDER CANCEL="N" ORDER_NAME="47708563" TYPE="1">
<DETAIL TYPE="1000" VALUE="" />
<DETAIL TYPE="1001" VALUE="90000086570010-01283" />
<DETAIL TYPE="1002" VALUE="1" />
<DETAIL TYPE="1003" VALUE="1" />
<DETAIL TYPE="1004" VALUE="ZCON" />
<TRAILER_STOP>0</TRAILER_STOP>

 

we want to ingest only those line to splunk indexer which matches with below mentioned four green highlighted lines.

  • <ORDER CANCEL="N" ORDER_NAME="XXXXXXXX" TYPE="1">
  • <ORDER CANCEL="N" ORDER_NAME="XXXXXXXX" TYPE="2">
  • Creating order cancellation transaction for order XXXXXXXX,
  • JSON received for product import: {"records":[{"lgnum":"407","entitled":"XXXX","owner":"XXXX","product":"XXXX-XXXXX",

Let me know how we can ingest only green highlighted matched lines to splunk indexer as single event.

 

Thanks

Abhineet Kumar

 

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The green lines make for a good regular expression, once special characters are escaped and wildcards applied.

\<ORDER CANCEL="." ORDER_NAME="[^"]+" TYPE="[12]">|Creating order cancellation transaction for order [^,]+,|JSON received for product import: {"records":\[{"lgnum":"407","entitled":"[^"]+","owner":"[^"]+","product":"[^"]+"

There are two ways to filter events.  The first uses a transform to find events that match a regex and send them either to an index or to nullQueue (equivalent to /dev/null). 

Add the following stanzas to transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \<ORDER CANCEL="." ORDER_NAME="[^"]+" TYPE="[12]">|Creating order cancellation transaction for order [^,]+,|JSON received for product import: {"records":\[{"lgnum":"407","entitled":"[^"]+","owner":"[^"]+","product":"[^"]+"
DEST_KEY = queue
FORMAT = indexQueue

Then reference them in props.conf:

[mysourcetype]
TRANSFORMS-set= setnull,setparsing

See https://docs.splunk.com/Documentation/Splunk/9.1.0/Forwarding/Routeandfilterdatad#Keep_specific_even... for the docs.

The other method uses the newer INGEST_EVAL feature, also in transforms.conf.

INGEST_EVAL = queue=if(match(_raw, "\<ORDER CANCEL=\".\" ORDER_NAME=\"[^\"]+\" TYPE=\"[12]\">|Creating order cancellation transaction for order [^,]+,|JSON received for product import: {\"records\":\[{\"lgnum\":\"407\",\"entitled\":\"[^\"]+\",\"owner\":\"[^\"]+\",\"product\":\"[^\"]+\""), "nullQueue", "indexQueue")

See https://docs.splunk.com/Documentation/ITSI/4.17.0/Configure/transforms.conf for more.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...