Splunk Search

PROPS Configuration for text file with header

SplunkDash
Motivator

Hello,

I have a text source file with header. Some sample events (first line is a header) and props that I wrote given below.

My props is working ok, except it breaks the events at TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, at Obj.BasePage.Page, TEST\m69xcb, at Obj.BasePage.Page, and TEST\7yxccd|Employee  instead of breaking events at  TEST\2qw123|Employee, TEST\3eraa2|Employee, TEST\87xaqw|Employee, TEST\m69xcb, and TEST\7yxccd|Employee .  So from following sample events, I should have 5 events , but getting 7 events. Any help will be highly appreciated. Thank you. 

UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
TEST\2qw123|Employee|COM|TESTUSER|NTINCheckKCase|089524234|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217122935|Case Information request: (Case-170) - 201612-30|mct0ma01ma4352855|10.219.174.222
TEST\3eraa2|Employee|COM|TESTUSER|NTINCheckKCase|046453942|ybzjlie3d4ayr1i2|10.212.48.121|00|20220217123142|Case Information request: (Case -85) - 201912-30|mct0ma01ma4352855|10.219.174.222
TEST\87xaqw|Employee|COM|SYSTEM|SystemMsg||zsod0mvomcelp3hvln5smm1u|10.216.22.17|01|20220217124743|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced <br>Source: App_Web_pc<br>Message: Object reference not set to an instance of an object.<br> /Case/CaseInventory.aspx<br>Trace: at Case.CaseInventory()
at Obj.BasePage.Page_Load(Object sender, EventArgs e)<br><br>Please try to login again.|mct0ma01ma4382154|10.210.174.221
TEST\m69xcb|Employee|COM|SYSTEM|SystemMsg||z0ae3c25zggbzx5p|10.215.173.231|01|20220217130933|Type:'error'; Ref:'Case/CaseInventory.aspx?Query=true&Scope=ServiceWide'; Msg: experienced a error:<br><br>Source: App_Web_pcf3kniw<br>Message: Object reference not set to an instance of an object.<br> /Case/CaseInventory.aspx<br>Trace: at Case.CaseInventory.page_load3()
at Obj.BasePage.Page_Load(Object sender, EventArgs e)<br><br>Please try to login again.|mct0ma01ma4353159|10.210.174.221
TEST\7yxccd|Employee|COM|TESTUSER|NTINCheckKCase|008422123|zggbzx5pzgnw1nih|10.215.173.231|00|20220217131108|Case Information request: (Case -24) - 202112-30|mct0ma1ma4353159|10.210.174.221

 

[sourcename]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)

INDEXED_EXTRACTIONS=psv

MAX_TIMESTAMP_LOOKAHEAD=14

HEADER_FIELD_LINE_NUMBER=1

TIME_FORMAT=%Y%m%d%H%M%S

TIMESTAMP_FIELDS=TimeStamp

TRUNCATE=2000

Labels (1)
Tags (1)
0 Karma
1 Solution

VatsalJagani
Super Champion

No, it (changing LINE_BREAKER) shouldn't make any difference as you are using INDEXED_EXTRACTION.

 

------
Upvote would be appreciated!!!

View solution in original post

0 Karma

PickleRick
Ultra Champion

Your data is inconsistent with the definition. You have header specifying some fields and then you have two events with not enough data to fill those fields,

VatsalJagani
Super Champion

Try using search-time field extraction instead of Index time (INDEXED_EXTRACTIONS) with below configurations:

 

props.conf

[sourcename]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 2000
TRANSFORMS-filter_events = data_filter_headers
TIME_PREFIX = [^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|[^|]*|
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %Y%m%d%H%M%S
REPORT-headers = data_headers

 

transforms.conf

[data_headers]
CLEAN_KEYS = 0
DELIMS = "|"
FIELDS = UserID,UserType,System,EventType,EventID,Subject,SessionID,SrcAddr,EventStatus,TimeStamp,AdditionalData,DeviceID,DestSrcAddr

[data_filter_headers]
REGEX = ^UserID|UserType|System|EventType|EventID|Subject|SessionID|SrcAddr|EventStatus|TimeStamp|AdditionalData|DeviceID|DestSrcAddr
DEST_KEY = queue
FORMAT = nullQueue

 

I hope this helps!!!

SplunkDash
Motivator

Hello,

Thank you so much for your quick response. Are there any ways we can fix it using indexed time field extraction or without using Transform.conf file?

0 Karma

VatsalJagani
Super Champion

Your configuration for that seems correct. Try checking the splunkd error and warning logs.
If that doesn't help open a case with Splunk and see if they can help!!

0 Karma

SplunkDash
Motivator

Hello,

Thank you so much you all.  Just wonder, is it possible to use the pattern of like TEST\3eraa2|Employee|  as an event breaking clause? Thank you again.

0 Karma

VatsalJagani
Super Champion

You can, but you don't need it.

Each of your events is in the new line, so you can just use simply: 

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)

This is easier and better. 

SplunkDash
Motivator

Hello,

Thank you again. Agree and I used that way as you mentioned. But, thought, if I use like TEST\3eraa2|Employee, then it may give be 5 events instead of 7.

 

0 Karma

VatsalJagani
Super Champion

No, it (changing LINE_BREAKER) shouldn't make any difference as you are using INDEXED_EXTRACTION.

 

------
Upvote would be appreciated!!!

0 Karma

SplunkDash
Motivator

Hello,

Do you think following props is a good approach, as I am getting exactly 5 events using this props. Any feedback on it will be highly appreciated. Thank you

 

[sourcetype]

SHOULD_LINEMERGE=false

LINE_BREAKER=([\r\n]+)DS\\

CHARSET=UTF-8

TIME_PREFIX=\|\d{2}\|

TIME_FORMAT=%Y%m%d%H%M%S

MAXIMUM_TIMESTAMP_LOOKAHEAD=14

HEADER_FIELD_LINE_NUMBER=1

TRUNCATE=2000

 

0 Karma

VatsalJagani
Super Champion

LINE_BREAKER=([\r\n]+)DS\\

  • Why DS?
  • Are you sure all lines will start with DS?
0 Karma

SplunkDash
Motivator

Oh Sorry, you are right, it's TEST\ ....thank you and should be  ...start of each event, is it now makes sense to use this props instead.

 

LINE_BREAKER=([\r\n]+)TEST\\

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...