Hello,
I have a text source file with header. Some sample events (first line is a header) and props that I wrote given below.
My props is working ok, except it breaks the events at TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, at Obj.BasePage.Page, TEST\m69xcb, at Obj.BasePage.Page, and TEST\7yxccd|Employee instead of breaking events at TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, TEST\m69xcb, and TEST\7yxccd|Employee . So from following sample events, I should have 5 events , but getting 7 events. Any help will be highly appreciated. Thank you.
UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
TEST\2qw123|Employee|COM|TESTUSER|NTINCheckKCase|089524234|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217122935|Case Information request: (Case-170) - 201612-30|mct0ma01ma4352855|10.219.174.222
TEST\3eraa2|Employee|COM|TESTUSER|NTINCheckKCase|046453942|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217123142|Case Information request: (Case -85) - 201912-30|mct0ma01ma4352855|10.219.174.222
TEST\87xaqw|Employee|COM|SYSTEM|SystemMsg||zsod0mvomcelp3hvln5smm1u|10.216.22.17|01|20220217124743|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced <br>Source: App_Web_pc<br>Message: Object reference not set to an instance of an object.<br> /Case/CaseInventory.aspx<br>Trace: at Case.CaseInventory()
at Obj.BasePage.Page_Load(Object sender, EventArgs e)<br><br>Please try to login again.|mct0ma01ma4382154|10.210.174.221
TEST\m69xcb|Employee|COM|SYSTEM|SystemMsg||z0ae3c25zggbzx5p|10.215.173.231|01|20220217130933|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced a error:<br><br>Source: App_Web_pcf3kniw<br>Message: Object reference not set to an instance of an object.<br> /Case/CaseInventory.aspx<br>Trace: at Case.CaseInventory.page_load3()
at Obj.BasePage.Page_Load(Object sender, EventArgs e)<br><br>Please try to login again.|mct0ma01ma4353159|10.210.174.221
TEST\7yxccd|Employee|COM|TESTUSER|NTINCheckKCase|008422123|zggbzx5pzgnw1nih|10.215.173.231|00|20220217131108|Case Information request: (Case -24) - 202112-30|mct0ma1ma4353159|10.210.174.221
[sourcename]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)
INDEXED_EXTRACTIONS=psv
MAX_TIMESTAMP_LOOKAHEAD=14
HEADER_FIELD_LINE_NUMBER=1
TIME_FORMAT=%Y%m%d%H%M%S
TIMESTAMP_FIELDS=TimeStamp
TRUNCATE=2000
No, it (changing LINE_BREAKER) shouldn't make any difference as you are using INDEXED_EXTRACTION.
------
Upvote would be appreciated!!!
Your data is inconsistent with the definition. You have header specifying some fields and then you have two events with not enough data to fill those fields,
Try using search-time field extraction instead of Index time (INDEXED_EXTRACTIONS) with below configurations:
props.conf
[sourcename]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 2000
TRANSFORMS-filter_events = data_filter_headers
TIME_PREFIX = [^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %Y%m%d%H%M%S
REPORT-headers = data_headers
transforms.conf
[data_headers]
CLEAN_KEYS = 0
DELIMS = "|"
FIELDS = UserID,UserType,System,EventType,EventID,Subject,SessionID,SrcAddr,EventStatus,TimeStamp,AdditionalData,DeviceID,DestSrcAddr
[data_filter_headers]
REGEX = ^UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
DEST_KEY = queue
FORMAT = nullQueue
I hope this helps!!!
Hello,
Thank you so much for your quick response. Are there any ways we can fix it using indexed time field extraction or without using Transform.conf file?
Your configuration for that seems correct. Try checking the splunkd error and warning logs.
If that doesn't help open a case with Splunk and see if they can help!!
Hello,
Thank you so much you all. Just wonder, is it possible to use the pattern of like TEST\3eraa2|Employee| as an event breaking clause? Thank you again.
You can, but you don't need it.
Each of your events is in the new line, so you can just use simply:
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
This is easier and better.
Hello,
Thank you again. Agree and I used that way as you mentioned. But, thought, if I use like TEST\3eraa2|Employee, then it may give be 5 events instead of 7.
No, it (changing LINE_BREAKER) shouldn't make any difference as you are using INDEXED_EXTRACTION.
------
Upvote would be appreciated!!!
Hello,
Do you think following props is a good approach, as I am getting exactly 5 events using this props. Any feedback on it will be highly appreciated. Thank you
[sourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)DS\\
CHARSET=UTF-8
TIME_PREFIX=\|\d{2}\|
TIME_FORMAT=%Y%m%d%H%M%S
MAXIMUM_TIMESTAMP_LOOKAHEAD=14
HEADER_FIELD_LINE_NUMBER=1
TRUNCATE=2000
LINE_BREAKER=([\r\n]+)DS\\
Oh Sorry, you are right, it's TEST\ ....thank you, and should be ...start of each event, is it now makes sense to use this props instead.
LINE_BREAKER=([\r\n]+)TEST\\