Did you upgrade your local python install, or are you just using splunk's bundled python distro.... it wasn't clear from your comment... BTW, I don't have access to update the docs. However, you could simply email [email protected] with your request and add a link to this page. The docs team has been very responsive to all my nagging about various little doc issues here and there. 😉 ... View more

You may also find some of this helpful: Regex’s and Windows paths in inputs.conf and props.conf ... View more

This should be done automatically. You can get more info about a custom setup on this blog entry: http://blogs.splunk.com/2009/12/02/configure-splunk-to-pull-a-date-out-of-a-non-standard-filename/ However, this is a pretty popular date format, so you shouldn't need a custom setup. This date should match the _isodate named format. (As seen in the default datetime.xml file.) ... View more

Yes, splunk will replace the unprintable character with their C-style hex notation before indexing. That can be quite annoying, but then again, so is trying to search for unprintable characters. If your curious, you can see a table of these conversions on the Wikipedia ASCII page, search down the page for the "Start of Header" character. It seems like you have a fields inside of a field thing going on here, right? You have fields delimited by a pipe ( | ), and then the 8th field (at least in your given example) has and additional delimited field. I'm not sure how splunk handles that exactly. If you simply setup your delimiter as the ^A (or \x1 ) then your first field would contain: M|219620|0|i|I|20100506-16:15:53.443|463|8=FIX.4.4 , when you probably only want it to contain 8=FIX.4.4 . So simply getting your delimiter set properly isn't going to fully work. I'm guessing it would make the most sense to first extract the outer set of fields first using DELIMS="|" and then, setup a secondary field extract to pull out your embedded fields. So, perhaps you would end up with something like this: props.conf : [FIX] SHOULD_LINEMERGE = false KV_MODE = none REPORT-outer_fields = get_outer_fields, get_inner_fields transforms.conf : [get_outer_fields] DELIMS="|" FIELDS = "f1", "f2", "f3", "f4", "f5", "_f6", "f7", "inner_fields" [get_inner_fields] REGEX = (?:^|\\x1) (?<a>.+)\\x1(?<b>.+)\\x1(?<c>.+)\\x1(?<d>.+)$ SOURCE_KEY = inner_fields I think this should work. This does seem like a complicated scenario. If the number of subfields is not constant (4), then you could use a multi-value field extraction like this: (That regex should work, it took me a few tries, but it seems to be best solution I could come up with) [get_inner_fields] REGEX= (?=^|\\x1)(?:\\x1)?(?<my_fields>.+?)(?:\\x1)?(?=$|\\x1) SOURCE_KEY = inner_fields MV_ADD = True Another possible option (and I don't know the FIX format at all, so this may not work). If the 8 in 8=FIX.4.4 means something like 'fix_version_number', you could just write a bunch of extracts that use the leading number of map to different field names. So for example of "8", you could add something like this to your props file: EXTRACT-fix_field_8 = (?:\||\\x1|^)8=(?<fix_version_number>.*?)(?:\||\\x1|$) Another thought (which may make all of the above options simpler) would be to add a SEDCMD to your soucetype to change all of the ^A characters into something more useful at index time. Maybe something like a comma? (You would probably want to find a character or sequence of characters not already being used in your events) Also, using a punctuation character like a comma also has the advantage of improving the way terms are segmented in your index which will let your search on more of these embedded fields more efficiently. For example, in your example event, you can search for "8=FIX.4.4" , but you can't search for "50=FXSpot" because it's would be stored in the index as "150=FXSpot" , you would have to search with "*50=FXSpot" instead. Using a better punctuation character works around this problem. One more option. Email Glenn and take a look at a custom search command he is using to handle FIX log processing. See his post here: Has anyone got a method for decoding FIX financial format logs? ... View more

Is the name (full path) of the log file stored within the log file itself? I didn't think you could use a TIME_PREFIX to match against source. ... View more

Your issues is with event breaking not line breaking. (You shouldn't have to mess with the LINE_BREAKER setting.) I'm not sure if you assigned line numbers to your events of they are in the raw event data. If they are in the raw data, then it's possible that your it is confusing the date parser. So probably the best place to start is by setting up an explicit date format and telling splunk what junk to expect before the date. (The example given will match with or without the leading digit, so it should work either way.) I would try setting the following in your props.conf file: (Note that I'm using the sourcetype of your events instead of creating a host:: matching stanza because that's generally a better approach, at least from my experience.) [IntADOsNotConsolidated] TIME_PREFIX = ^(?:\d+ +)? # Date look like: 5/26/10 9:19:21.000 AM TIME_FORMAT = %m/%d/%y %I:%M:%S.%3N %P BREAK_ONLY_BEFORE_DATE = True If your event are always exactly one line, then you can remove the BREAK_ONLY_BEFORE_DATE entry, and replace it with the following: SHOULD_LINEMERGE = False As with all settings like this, these changes to your props.conf file will only effect newly indexed data after a splunk restart. ... View more

Have you tried adding rt to your earliest/lastest times? ... View more

For anyone interested in a database lookup script example, I have posted one here: I want sample code to connect to Oracle database, and lookup a table at search time ... View more

Never mind the REPEAT_MATCH=true comment. You don't need it. It appears that using the format $1::$2 is what triggers the multiple-matching behavior. ... View more

Are you sure that your file is gziped? I find the unix file utility very helpful for these types of things. Try file app.spl ... View more

See this answer: How can I use tar to unpack app bundles manually? I see you pretty new to the answers.splunk.com site, so here is a hint. Check out the related questions on the right-side of the screen. A number of them look applicable to your question. ... View more

Also, keep in mind that anytime you change crcSalt you are likely to re-index all your files. You should also be aware that setting crcSalt = <SOURCE> means that if you rotate your log files they may be indexed again. Normally splunk detects and handles this automatically, but using <SOURCE> will effectively disable this feature. So you may need to update your rotation scheme to move log files to a different folder, or make sure your monitor is only looking at your non-rotated files. Just a heads up. ... View more

No, you want the literal value " <SOURCE> ". That is a special value with a special meaning. Take a look at http://www.splunk.com/base/Documentation/4.1.2/Admin/Inputsconf ... View more

Hmm. I think you either want just dedup to get a single list of values. Or use the chart command. Using both, as shown, the count will always be 1 for each value of scrip ... View more

Keep in mind that using dedup will probably not work as you would expect when dealing with multi-valued fields. The stats command will multi-valued fields properly. So | stats values(ip) is probably preferable. ... View more