topic Re: PROPS Configuration for text file with header in Splunk Search

PROPS Configuration for text file with header

SplunkDash — Wed, 13 Apr 2022 04:30:15 GMT

Hello,

I have a text source file with header. Some sample events (first line is a header) and props that I wrote given below.

My props is working ok, except it breaks the events at TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, at Obj.BasePage.Page, TEST\m69xcb, at Obj.BasePage.Page, and TEST\7yxccd|Employee instead of breaking events at TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, TEST\m69xcb, and TEST\7yxccd|Employee . So from following sample events, I should have 5 events , but getting 7 events. Any help will be highly appreciated. Thank you.

UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
TEST\2qw123|Employee|COM|TESTUSER|NTINCheckKCase|089524234|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217122935|Case Information request: (Case-170) - 201612-30|mct0ma01ma4352855|10.219.174.222
TEST\3eraa2|Employee|COM|TESTUSER|NTINCheckKCase|046453942|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217123142|Case Information request: (Case -85) - 201912-30|mct0ma01ma4352855|10.219.174.222
TEST\87xaqw|Employee|COM|SYSTEM|SystemMsg||zsod0mvomcelp3hvln5smm1u|10.216.22.17|01|20220217124743|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced Source: App_Web_pc Message: Object reference not set to an instance of an object. /Case/CaseInventory.aspx Trace: at Case.CaseInventory()
at Obj.BasePage.Page_Load(Object sender, EventArgs e) Please try to login again.|mct0ma01ma4382154|10.210.174.221
TEST\m69xcb|Employee|COM|SYSTEM|SystemMsg||z0ae3c25zggbzx5p|10.215.173.231|01|20220217130933|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced a error: Source: App_Web_pcf3kniw Message: Object reference not set to an instance of an object. /Case/CaseInventory.aspx Trace: at Case.CaseInventory.page_load3()
at Obj.BasePage.Page_Load(Object sender, EventArgs e) Please try to login again.|mct0ma01ma4353159|10.210.174.221
TEST\7yxccd|Employee|COM|TESTUSER|NTINCheckKCase|008422123|zggbzx5pzgnw1nih|10.215.173.231|00|20220217131108|Case Information request: (Case -24) - 202112-30|mct0ma1ma4353159|10.210.174.221

[sourcename]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)

INDEXED_EXTRACTIONS=psv

MAX_TIMESTAMP_LOOKAHEAD=14

HEADER_FIELD_LINE_NUMBER=1

TIME_FORMAT=%Y%m%d%H%M%S

TIMESTAMP_FIELDS=TimeStamp

TRUNCATE=2000

Re: PROPS Configuration for text file with header

VatsalJagani — Wed, 13 Apr 2022 04:48:22 GMT

Try using search-time field extraction instead of Index time (INDEXED_EXTRACTIONS) with below configurations:

props.conf

[sourcename] SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+) TRUNCATE = 2000 TRANSFORMS-filter_events = data_filter_headers TIME_PREFIX = [^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*| MAX_TIMESTAMP_LOOKAHEAD = 20 TIME_FORMAT = %Y%m%d%H%M%S REPORT-headers = data_headers

transforms.conf

I hope this helps!!!

Re: PROPS Configuration for text file with header

SplunkDash — Wed, 13 Apr 2022 04:53:38 GMT

Hello,

Thank you so much for your quick response. Are there any ways we can fix it using indexed time field extraction or without using Transform.conf file?

Re: PROPS Configuration for text file with header

VatsalJagani — Wed, 13 Apr 2022 05:32:39 GMT

Your configuration for that seems correct. Try checking the splunkd error and warning logs.
If that doesn't help open a case with Splunk and see if they can help!!

Re: PROPS Configuration for text file with header

PickleRick — Wed, 13 Apr 2022 05:38:47 GMT

Your data is inconsistent with the definition. You have header specifying some fields and then you have two events with not enough data to fill those fields,

Re: PROPS Configuration for text file with header

SplunkDash — Wed, 13 Apr 2022 12:29:41 GMT

Hello,

Thank you so much you all. Just wonder, is it possible to use the pattern of like TEST\3eraa2|Employee| as an event breaking clause? Thank you again.

Re: PROPS Configuration for text file with header

VatsalJagani — Wed, 13 Apr 2022 13:04:19 GMT

You can, but you don't need it.

Each of your events is in the new line, so you can just use simply:

SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)

This is easier and better.

Re: PROPS Configuration for text file with header

SplunkDash — Wed, 13 Apr 2022 13:40:49 GMT

Hello,

Thank you again. Agree and I used that way as you mentioned. But, thought, if I use like TEST\3eraa2|Employee, then it may give be 5 events instead of 7.

Re: PROPS Configuration for text file with header

VatsalJagani — Wed, 13 Apr 2022 14:00:42 GMT

No, it (changing LINE_BREAKER) shouldn't make any difference as you are using INDEXED_EXTRACTION.

------
Upvote would be appreciated!!!

Re: PROPS Configuration for text file with header

SplunkDash — Wed, 13 Apr 2022 16:45:27 GMT

Hello,

Do you think following props is a good approach, as I am getting exactly 5 events using this props. Any feedback on it will be highly appreciated. Thank you

[sourcetype]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)DS\\

CHARSET=UTF-8

TIME_PREFIX=\|\d{2}\|

TIME_FORMAT=%Y%m%d%H%M%S

MAXIMUM_TIMESTAMP_LOOKAHEAD=14

HEADER_FIELD_LINE_NUMBER=1

TRUNCATE=2000

Re: PROPS Configuration for text file with header

VatsalJagani — Wed, 13 Apr 2022 16:54:56 GMT

LINE_BREAKER=([\r\n]+)DS\\

Why DS?
Are you sure all lines will start with DS?

Re: PROPS Configuration for text file with header

SplunkDash — Wed, 13 Apr 2022 17:00:24 GMT

Oh Sorry, you are right, it's TEST\ ....thank you, and should be ...start of each event, is it now makes sense to use this props instead.

LINE_BREAKER=([\r\n]+)TEST\\