https://community.splunk.com/t5/Getting-Data-In/File-monitoring-questions-top-item-change/m-p/72237#M14733 <P>This is not simple but can be achieved.</P> <UL> <LI> first, index your events as multiline events by creating a specific sourcetype the sourcetype has to be in props.conf or defined using the preview. and specify how you want the events to be broken.</LI> </UL> <P>Here we want each <CODE>000000 SIZE = 099999 LOOP = 000000 WDTH = 001536 NWNO = 0004</CODE><BR /> to be the beginning of a new event, so we will break on the line with SIZE</P> <PRE> [test] NO_BINARY_CHECK=1 BREAK_ONLY_BEFORE=\d+ SIZE = SHOULD_LINEMERGE=true pulldown_type=1 MAX_EVENTS=256 # we do not expect more than 256 lines per event. </PRE> <UL> <LI><P>second index your events using the correct sourcetype</P></LI> <LI><P>third during the search do the field extraction for the second part of the events.<BR /> using multikv, each line will be considered as a different events (but the fields from the first line will be common)<BR /> then you can extract the fields from the line (using | as a separator)<BR /> and finally, remove the first line to avoid confusion.</P></LI> </UL> <PRE> `source="*test.log" sourcetype=test |multikv noheader=t | rex "(?<ID>\d+)[^|]*" | rex "(?<FIELDA>\d+)\s\|\s(?<FIELDB>\w+)\s\|\s(?<FIELDC>[^\|]*)\s\|\s(?<FIELDD>\w+)\s\|\s(?<FIELDE>\w+)" | search NOT "LOOP" | table ID SIZE LOOP WDTH NWNO fieldA fieldB fieldC fieldD fieldE _raw </FIELDE></FIELDD></FIELDC></FIELDB></FIELDA></ID></PRE> Tue, 25 Dec 2012 20:39:14 GMT yannK 2012-12-25T20:39:14Z https://community.splunk.com/t5/Getting-Data-In/File-monitoring-questions-top-item-change/m-p/72236#M14732 <P>File monitoring questions</P> <P>Monitoring Point is, the log file<BR /> The peculiar form of the log file to the log record.</P> <H1>Log format</H1> <P>000000 SIZE = 099999 LOOP = 000000 WDTH = 001536 NWNO = 0004<BR /> 00001 | AIX | 6.1 | LCID | xxx<BR /> 00002 | AIX | 6.1 | LCID | xxx<BR /> 00003 | AIX | 6.1 | LCID | xxx<BR /> 00004 | AIX | 6.1 | LCID | xxx</P> <P>Log record while NWNO item changes occur in the above log.<BR /> To collect duplicate indexing problems occur.</P> <P>For example, the search results</P> <P>index = temp 00001</P> <P>00001 | AIX | 6.1 | LCID | xxx<BR /> 00001 | AIX | 6.1 | LCID | xxx<BR /> 00001 | AIX | 6.1 | LCID | xxx<BR /> 00001 | AIX | 6.1 | LCID | xxx</P> <P>Is indexing event</P> <P>Could there be a way to solve the problem?</P> Mon, 24 Dec 2012 05:25:28 GMT https://community.splunk.com/t5/Getting-Data-In/File-monitoring-questions-top-item-change/m-p/72236#M14732 jcisha 2012-12-24T05:25:28Z https://community.splunk.com/t5/Getting-Data-In/File-monitoring-questions-top-item-change/m-p/72237#M14733 <P>This is not simple but can be achieved.</P> <UL> <LI> first, index your events as multiline events by creating a specific sourcetype the sourcetype has to be in props.conf or defined using the preview. and specify how you want the events to be broken.</LI> </UL> <P>Here we want each <CODE>000000 SIZE = 099999 LOOP = 000000 WDTH = 001536 NWNO = 0004</CODE><BR /> to be the beginning of a new event, so we will break on the line with SIZE</P> <PRE> [test] NO_BINARY_CHECK=1 BREAK_ONLY_BEFORE=\d+ SIZE = SHOULD_LINEMERGE=true pulldown_type=1 MAX_EVENTS=256 # we do not expect more than 256 lines per event. </PRE> <UL> <LI><P>second index your events using the correct sourcetype</P></LI> <LI><P>third during the search do the field extraction for the second part of the events.<BR /> using multikv, each line will be considered as a different events (but the fields from the first line will be common)<BR /> then you can extract the fields from the line (using | as a separator)<BR /> and finally, remove the first line to avoid confusion.</P></LI> </UL> <PRE> `source="*test.log" sourcetype=test |multikv noheader=t | rex "(?<ID>\d+)[^|]*" | rex "(?<FIELDA>\d+)\s\|\s(?<FIELDB>\w+)\s\|\s(?<FIELDC>[^\|]*)\s\|\s(?<FIELDD>\w+)\s\|\s(?<FIELDE>\w+)" | search NOT "LOOP" | table ID SIZE LOOP WDTH NWNO fieldA fieldB fieldC fieldD fieldE _raw </FIELDE></FIELDD></FIELDC></FIELDB></FIELDA></ID></PRE> Tue, 25 Dec 2012 20:39:14 GMT https://community.splunk.com/t5/Getting-Data-In/File-monitoring-questions-top-item-change/m-p/72237#M14733 yannK 2012-12-25T20:39:14Z https://community.splunk.com/t5/Getting-Data-In/File-monitoring-questions-top-item-change/m-p/72238#M14734 <P>Thank you Answer by yannK.</P> <P>After all, Is there any way to avoid duplicate is collected?<BR /> Whether the only way to solve collected through search results?</P> <P>Uniqueness, licensing issues</P> Wed, 26 Dec 2012 06:14:18 GMT https://community.splunk.com/t5/Getting-Data-In/File-monitoring-questions-top-item-change/m-p/72238#M14734 jcisha 2012-12-26T06:14:18Z