ingest line from log file match with multiple regular expression to splunk indexer

[Abhineet]

Hi,

Below red highlighted is sample log file.

Sample LogFile

12:08:32.797 [6] (null) DEBUG Bastian.Exacta.AMAT.ImportAdapter.Wcf.AMATWcfImport - JSON received for product import: {"records":[{"lgnum":"407","entitled":"4070","owner":"4070","product":"0205-02304","prd_descr":"PACKAGING, RUNNING BEAM GRIPPERS, REFLEX","base_uom":"EA","gross_weight":"0.000","net_weight":"1.000","weight_uom":"KG","volume":"6480.000","volume_uom":"CCM","length":"40.000","width":"18.000","height":"9.000","dimension_uom":"CM","serial_profile":null,"batch_req":null,"cycle_count_ind":"C","alternative_uom":"EA","shelf_life_flag":null,"shelf_life":null,"req_min_shelf_life":null,"req_max_shelf_life":null,"std_cost":"10.61","matnr":"0205-02304","suffix":null,"rev_level":"01","extension":null}]}
12:08:32.797 [6] (null) DEBUG Bastian.Exacta.Business.Xml.XmlEntity - Started saving XML entity of type 'ProductImportData'
12:08:32.844 [6] (null) DEBUG Bastian.Exacta.Business.Xml.XmlEntity - Finished XML entity of type 'ProductImportData'. Result:
<?xml version="1.0" encoding="utf-16" standalone="yes"?>
<PROD NAME="0205-02304">

14:54:00.242 [8] (null) DEBUG Bastian.Exacta.AMAT.ImportAdapter.Wcf.AMATWcfImport - JSON received for order line cancel import: {"records":[{"Header":{"lgnum":"407","who":"47708597","canrq":"X"},"Detail":[{"tanum":"97908517"}]}]}
14:54:00.242 [8] (null) DEBUG Bastian.Exacta.Business.Persistance.SessionFactory - Opening NHibernate session using the production factory...
14:54:00.258 [8] (null) DEBUG NHibernate.SQL - select order0_.ORDER_TYPE as col_0_0_ from ORDER_HEADER order0_ where order0_.ORDER_NAME=@p0 ORDER BY CURRENT_TIMESTAMP OFFSET 0 ROWS FETCH FIRST 1 ROWS ONLY;@p0 = '47708597' [Type: String (4000:0:0)]
14:54:00.273 [8] (null) DEBUG Bastian.Exacta.Business.Persistance.SessionFactory - Closing NHibernate session...
14:54:00.273 [8] (null) INFO Bastian.Exacta.AMAT.ImportAdapter.Wcf.AMATWcfImport - Creating order cancellation transaction for order 47708597, OrderType : 0
14:54:00.289 [8] (null) DEBUG Bastian.Exacta.Business.Persistance.SessionFactory - Opening NHibernate session using the production factory...
14:54:00.320 [8] (null) DEBUG NHibernate.SQL - select orderline1_.ORDER_LINE_ID as order1_236_, orderline1_.ORDER_LINE_TYPE as order2_236_, orderline1_.LINE_NUM as line3_236_, orderline1_.LOT_NUM_REQUESTED as lot4_236_, orderline1_.QTY_REQUESTED as qty5_236_, orderline1_.UOM_SPECIFIED as uom6_236_, orderline1_.SERIAL_NUM_REQUESTED as serial7_236_, orderline1_.SINGLE_LOT as single8_236_, orderline1_.DAYS_TO_EXPIRE as days9_236_, orderline1_.VAS as vas10_236_, orderline1_.KITTING as kitting11_236_, orderline1_.DEST_ZONE as dest12_236_, orderline1_.SOURCE_ZONE as source13_236_, orderline1_.SEQ_NUM as seq14_236_, orderline1_.RETURNED_INV as returned15_236_, orderline1_.WGT_REQUESTED as wgt16_236_, orderline1_.INVENTORY_GROUP as inventory17_236_, orderline1_.TOTAL_RECEIPT_QUANTITY as total18_236_, orderline1_.LOT_REVISION as lot19_236_, orderline1_.SERIAL_NUM_REQUIRED as serial20_236_, orderline1_.CAPTURE_COUNTRY_OF_ORIGIN as capture21_236_, orderline1_.SECONDARY_SCAN_TYPE as secondary22_236_, orderline1_.SUPPRESS_SCANS_AT_PICK as suppress23_236_, orderline1_.SHOULD_PICK_RESERVED_INVENTORY as should24_236_, orderline1_.QUAR_REASON as quar25_236_, orderline1_.INVOICE_NUMBER as invoice26_236_, orderline1_.INVENTORY_RESERVATION_KEY as inventory27_236_, orderline1_.SSU_VALUE_PER_ITEM as ssu28_236_, orderline1_.PROD_ID as prod29_236_, orderline1_.UOM_TYPE_REQUESTED as uom30_236_, orderline1_.ORDER_ID as order31_236_, orderline1_.WAVE_ID as wave32_236_, orderline1_.ROUTE_ID as route33_236_, orderline1_.DOCK_ID as dock34_236_, orderline1_.DEST_WAREHOUSE_ID as dest35_236_, orderline1_.SOURCE_WAREHOUSE_ID as source36_236_, orderline1_.DOCUMENT_ID as document37_236_, orderline1_.ADJUSTMENT_ORDER_ID as adjustment38_236_, orderline1_.BOM_ID as bom39_236_, orderline1_.BOM_LINE_ID as bom40_236_, orderline1_.BOM_PARENT_LINE_ID as bom41_236_, orderline1_.PREFERRED_CNTNR_PATTERN_ID as preferred42_236_, orderline1_.COUNTRY_OF_ORIGIN as country43_236_ from ORDER_LINE_DETAIL orderlined0_ inner join ORDER_LINE orderline1_ on orderlined0_.ORDER_LINE_ID=orderline1_.ORDER_LINE_ID inner join ORDER_HEADER order2_ on orderline1_.ORDER_ID=order2_.ORDER_ID where order2_.ORDER_NAME=@p0 and orderlined0_.DETAIL_TYPE=@p1 and (orderlined0_.DETAIL_VALUE in (@p2));@p0 = '47708597' [Type: String (4000:0:0)], @p1 = 1000 [Type: Decimal (0:10:29)], @Anonymous = '97908517' [Type: String (4000:0:0)]
14:54:00.336 [8] (null) DEBUG Bastian.Exacta.Business.Persistance.SessionFactory - Closing NHibernate session...
14:54:00.336 [8] (null) INFO Bastian.Exacta.AMAT.ImportAdapter.Wcf.AMATWcfImport - No order lines found for order 47708597 for order line cancellation request, cannot proceed with cancellation transaction.
14:54:00.352 [8] (null) WARN Bastian.Exacta.AMAT.ImportAdapter.Wcf.AMATWcfImport - Exacta Event

<ORDER CANCEL="N" ORDER_NAME="47708600" TYPE="2">
<DETAIL TYPE="1005" />
<TRAILER_STOP>0</TRAILER_STOP>
<ORDER_PRIORITY>1</ORDER_PRIORITY>
<ORDER_LINE CANCEL="N" LINE_NUM="1">
<PROD_NAME>0010-01283</PROD_NAME>
<PROD_COMPANY_NAME>4070</PROD_COMPANY_NAME>
<PROD_VENDOR_NAME>4070</PROD_VENDOR_NAME>
<QTY_REQUESTED>1</QTY_REQUESTED>
<DETAIL TYPE="1000" VALUE="97908520" />
<DETAIL TYPE="1001" VALUE="1" />

<?xml version="1.0" encoding="utf-16" standalone="yes"?>
<ORDER CANCEL="N" ORDER_NAME="47708563" TYPE="1">
<DETAIL TYPE="1000" VALUE="" />
<DETAIL TYPE="1001" VALUE="90000086570010-01283" />
<DETAIL TYPE="1002" VALUE="1" />
<DETAIL TYPE="1003" VALUE="1" />
<DETAIL TYPE="1004" VALUE="ZCON" />
<TRAILER_STOP>0</TRAILER_STOP>

we want to ingest only those line to splunk indexer which matches with below mentioned four green highlighted lines.

• <ORDER CANCEL="N" ORDER_NAME="XXXXXXXX" TYPE="1">
• <ORDER CANCEL="N" ORDER_NAME="XXXXXXXX" TYPE="2">
• Creating order cancellation transaction for order XXXXXXXX,
• JSON received for product import: {"records":[{"lgnum":"407","entitled":"XXXX","owner":"XXXX","product":"XXXX-XXXXX",

Let me know how we can ingest only green highlighted matched lines to splunk indexer as single event.

Thanks

Abhineet Kumar

[richgalloway]

The green lines make for a good regular expression, once special characters are escaped and wildcards applied.

\<ORDER CANCEL="." ORDER_NAME="[^"]+" TYPE="[12]">|Creating order cancellation transaction for order [^,]+,|JSON received for product import: {"records":\[{"lgnum":"407","entitled":"[^"]+","owner":"[^"]+","product":"[^"]+"

There are two ways to filter events.  The first uses a transform to find events that match a regex and send them either to an index or to nullQueue (equivalent to /dev/null). 

Add the following stanzas to transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \<ORDER CANCEL="." ORDER_NAME="[^"]+" TYPE="[12]">|Creating order cancellation transaction for order [^,]+,|JSON received for product import: {"records":\[{"lgnum":"407","entitled":"[^"]+","owner":"[^"]+","product":"[^"]+"
DEST_KEY = queue
FORMAT = indexQueue

Then reference them in props.conf:

[mysourcetype]
TRANSFORMS-set= setnull,setparsing

See https://docs.splunk.com/Documentation/Splunk/9.1.0/Forwarding/Routeandfilterdatad#Keep_specific_even... for the docs.

The other method uses the newer INGEST_EVAL feature, also in transforms.conf.

INGEST_EVAL = queue=if(match(_raw, "\<ORDER CANCEL=\".\" ORDER_NAME=\"[^\"]+\" TYPE=\"[12]\">|Creating order cancellation transaction for order [^,]+,|JSON received for product import: {\"records\":\[{\"lgnum\":\"407\",\"entitled\":\"[^\"]+\",\"owner\":\"[^\"]+\",\"product\":\"[^\"]+\""), "nullQueue", "indexQueue")

See https://docs.splunk.com/Documentation/ITSI/4.17.0/Configure/transforms.conf for more.