https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127138#M26159 <P>THANK YOU!<BR /> I have read some of related answers, and thought SEDCMD was just able to drop the content after a tag, nullQueue was "at event level" <BR /> Than i have to dig into "sed" tool. is there a awkcmd?</P> Mon, 04 Nov 2013 09:07:23 GMT crazyeva 2013-11-04T09:07:23Z https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127135#M26156 <P>I have got very large orginal data, with events strictly formatted as "f1,f2,f3,..."<BR /> most of the fields are meaningless: "0,f2,0,0,0,0,f7,0,f9,..." i only want f2,f7,f9<BR /> Can I filter fields before indexing, drop unnecessary data, avoid reach license limit?</P> Mon, 04 Nov 2013 04:32:58 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127135#M26156 crazyeva 2013-11-04T04:32:58Z https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127136#M26157 <P>Yes, as long as you can formulate a regular expression that defines how Splunk should include or exclude data. You can either to nullQueue routing (= drop events altogether) or </P> <P>Docs on how to use each: nullQueue routing - <A href="http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Routeandfilterdatad">http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Routeandfilterdatad</A><BR /> SEDCMD - <A href="http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf">http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf</A> - see specification on SEDCMD at the middle of the page.</P> Mon, 04 Nov 2013 07:43:38 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127136#M26157 Ayn 2013-11-04T07:43:38Z https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127137#M26158 <P>source file</P> <PRE><CODE>2013-11-01 11:11:11 f1 f2 f3 f4 f5 f6 2013-11-02 13:15:11 d1 d2 d3 d4 d5 d6 2013-11-02 14:23:22 e1 e2 e3 e4 e5 e6 2013-11-03 12:23:21 g1 g2 g3 g4 g5 g6 </CODE></PRE> <P>props.conf</P> <PRE><CODE>[your_sourcetype] TRANSFORMS-blah = keep235 </CODE></PRE> <P>transforms.conf</P> <PRE><CODE>[keep235] DEST_KEY = _raw REGEX = ^(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+) FORMAT = $1 $2 $4 $5 $7 </CODE></PRE> <P>Result:</P> <PRE><CODE>03/11/2013 12:23:21.000 2013-11-03 12:23:21 g2 g3 g5 02/11/2013 14:23:22.000 2013-11-02 14:23:22 e2 e3 e5 ^Splunk parsed timestamp ^event timestamp ^less columns/fields </CODE></PRE> <P>Hope this helps,</P> <P>/K</P> Mon, 04 Nov 2013 08:07:12 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127137#M26158 kristian_kolb 2013-11-04T08:07:12Z https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127138#M26159 <P>THANK YOU!<BR /> I have read some of related answers, and thought SEDCMD was just able to drop the content after a tag, nullQueue was "at event level" <BR /> Than i have to dig into "sed" tool. is there a awkcmd?</P> Mon, 04 Nov 2013 09:07:23 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127138#M26159 crazyeva 2013-11-04T09:07:23Z https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127139#M26160 <P>Thank you, that's sweet!</P> Mon, 04 Nov 2013 09:07:47 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-filter-or-extract-fields-before-indexing-time/m-p/127139#M26160 crazyeva 2013-11-04T09:07:47Z