As an addition , you should be aware of a few things: 1) NOT isn't the same as != . Consider the following statements NOT bob - will drop all events containing the string 'bob'. Simple. NOT user=bob - will drop all events where the field 'user' has the value 'bob'. Does not have to contain the literal string. NOT "user=bob" - will drop all events matching the whole string. Will keep events where 'user' gets the value 'bob' from some other type of field extraction. user != bob - will only return events that contain the field 'user', and where the value isn't 'bob'. So events that don't have a 'user' field at all will also be dropped.. 2) asterisks cannot be escaped in the search langauge, it will be parsed as a wildcard char. Unless you're looking for strings like "BIGWARNING UPS AUTO VOLT REG ONTOP", just leave them out. 3) searching for _raw != "*BLAH BLEH*" is probably not as good as NOT "BLAH BLEH" /K ... View more

Hi, please provide real data, describe what you are trying to achieve, use the code markup where appropriate, since this help with formatting and special characters If you got a good answer, vote up a/o mark as answered. /K ... View more

perhaps dc is better than c for Hostname and Access ... View more

Or perhaps, to do it in the config files so that it won't clutter up your search query: props.conf [your_sourcetype] REPORT-blah = pipedash transforms.conf [pipedash] DELIMS = "|", "-" See the docs: http://docs.splunk.com/Documentation/Splunk/6.2.0/Admin/Transformsconf /K The really long version on is also in the docs: http://docs.splunk.com/Documentation/Splunk/6.2.0/Knowledge/Createandmaintainsearch-timefieldextractionsthroughconfigurationfiles#Create_advanced_search-time_field_extractions_with_field_transforms ... View more

First) Splunk sometimes picks the 'wrong' place to split the stream of data into separate events. And no, don't do this on the UF. It will not make a difference. You'll have to do it on an indexer or heavy forwarder, since that is where the parsing phase takes place. Second) There are a few pieces of metadata that have to be correct at index-time, as opposed to at search-time. Almost all other field extractions/manipulations can (and should) be done at search time. These are the ones you need to get right: index timestamp event breaks host source sourcetype So, once an event has been indexed with the wrong information for any of these parameters, you're stuck, so-to-speak. Third) Splunk keeps track of which files it has already seen in a special index called the "fishbucket". Cleaning out this index will let splunk re-index data. /K ... View more

And it might be worth to mention that since both of these settings have a default value, the discarding of data will be triggered by whichever of these limits gets hit first. So if you want to use time as a limiting factor, ensure that you set the maxTotalDataSizeMB so high that you can be certain that frozenTimePeriodInSecs will trigger before the size constraint. And vice versa. /k ... View more

These logs (metrics, splunkd etc) go into the _internal index, and are not counted towards your license. /k ... View more

Well, there can be quite a few events in the _internal index, but they are only retained for 30 days by default. However, these are not counted towards your license allowance. A separate thing could be that you have regular log files that are being indexed in your system which seem to come from your indexer. I.e. if you mount a remote share from your indexer, and read the files from there Depending on how you do your configs, these events may be labeled as coming from your indexer (and will count towards your license). ... View more

If you're just trying this out on your local workstation, you have a lot of local files to play around with. This is a good idea, even if you're working in a project to evaluate or implement Splunk in your organization. On Windows systems, you will have the event logs, and on *nix, there are a lot of stuff in /var/log. This will give you a chance of getting to know the search commands, building dashboards, scheduling searches etc etc, without making alterations to a test/PoC environment. /K ... View more

You can use the 'code sample' formatting option when you have 'special' characters in your posts. It's the little button with "101010". Also, I updated my answer above. /k ... View more

No, it does not work that way. A transaction is a method for grouping separate events together, based on some characteristic, such as a common field value. ... View more

That extracts three separate strings, and it does not put them into a field. ... View more

With rex (searchtime) it's as easy as; your search | rex "<(?<myemail>[^>]+)" | blah blah In props.conf (also searchtime) [your sourcetype] EXTRACT-blah = <(?<myemail>[^>]+) Don't try to do it at index-time. That is not what you want. UPDATE: To extract email addresses into field names based on their context inside an event, you might want to try something like; props.conf [your sourcetype] EXTRACT-to = \sTo:\s(?<to_addr>\S+) EXTRACT-from = \From:\s(?<from_addr>\S+) This should work for the to/from email addresses of the formats below. Sep 28 20:59:57 10.123.78.14/10.123.78.14 Mailrcc1_Splunk_Syslog_Push: Info: MID 20842705 ICID 57360165 RID 0 To: [email protected] Sep 28 20:59:57 10.123.78.14/10.123.78.14 Mailrcc1_Splunk_Syslog_Push: Info: MID 20842705 ICID 57360165 From: [email protected] Sep 28 20:59:56 10.123.78.14/10.123.78.14 Mailrcc1_Splunk_Syslog_Push: Info: MID 20842704 Message-ID '<[email protected]>' Sep 28 20:59:57 10.123.78.15/10.123.78.15 Mailrcc2_Splunk_Syslog_Push: Info: MID 20248631 ICID 58528527 RID 0 To: [email protected] Sep 28 20:59:56 10.123.78.14/10.123.78.14 Mailrcc1_Splunk_Syslog_Push: Info: MID 20842703 ICID 57360164 RID 0 To: [email protected] Sep 28 20:59:56 10.123.78.14/10.123.78.14 Mailrcc1_Splunk_Syslog_Push: Info: MID 20842703 ready 1581 bytes from [email protected] If you also want to extract the other addresses, you could add the following; EXTRACT-other = [=<](?<other_addr>[a-zA-Z0-9-.]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]) Should work /k ... View more

Do it in the form of a REPORT in props/transforms. props.conf [your_sourcetype] REPORT-blah = get_my_fields transforms.conf [get_my_fields] DELIMS = "," FIELDS = S_ID, type, B_ID, mode, B_date, name, c_ID, c_name /K ... View more

Hmm, since that would mean that _raw is rewritten, it could mess up timestamps, which regularly use colons. /k ... View more

Eeh, I think that the metadata regarding which sourcetypes exist where is on a more granular level - in the index buckets. That is where you find files like Hosts.data, Sources.data and SourceTypes.data. From that I assume that you'd need to open every bucket in every index in order to determine if the sourcetype is present in that index. Not efficient. However, if you can instruct your users to type index=blaha instead of sourcetype=blaha , you would get a performance boost. How much will depend on other factors - one of the most important will be the time range for the query. Still I would no recommend you to do this, since you'd have to micromanage all the disk usage for those indexes. Unless you have a really strong use case regarding the access restrictions. /K ... View more

Since we still don't know how the other events look like, it'll be hard to give very precise advice. However, if the keyword you're after is either "PERMIT", "DENY" or "NA" (without the quotes), and these strings do not appear otherwise in your events, you could try; props.conf [your sourcetype] EXTRACT-blah = \s(?(PERMIT|DENY|NA))\s or if the position of the keyword is fixed, you can try to anchor your regex towards the end of the line; props.conf [your sourcetype] EXTRACT-bleh = (?\S+)\s+\S+\s+\"[^\"]+\"\s+[\d.]+$ These are just some of the options available. With more information and sample events, you'll be able to get better help. /K ... View more

It depends on how you want to chart it, and what other events look like. Your original request can be satisfied with something like; sourcetype=your_sourcetype " DENY " | timechart span=1h count I.e. not extracting the field at all, but just searching for events that have the matching string, and then use a timechart to show them over.. time. /k ... View more

have you looked at the wildcard characters? Either of the following should work - take a look at the docs for inputs.conf in the Search Reference manual. [monitor:///var/www/webapp/logs/.../*.log] [monitor:///var/www/webapp/logs/2014/*/*/*.log] ... View more

And also, it perfectly normal to set up a Splunk server and then import archived log files (which could easily be several years old), so looking at the earliest timestamp of an event is not a 100% certain solution. /k ... View more

Well, there is perhaps a far more attractive option; Drop the stuff you don't want (the type) by setting it to the same value everywhere, then make use of the stats max() function. You may first need to replace the string 'null' with a real NULL value, if that is what you have. Or perhaps not. At least in my test you don't your search | eval type = "Combined" | stats max(*) by _time This should look something like; 2014-09-09 05:57:58, KQ25B6P, 7.7.1, CommandPost+, 4, 70701, Combined, 1401, 6432, 1234, 3424, 7663, 2342, null, null, 8787, 1461, 6735, 8101, 3869, 20166, null, null, null, null, null, null, null, 0, 0, 0, 0, null, null, null, null, null, null, null, null, null, 5435, 123, 0, 6676, null, null, null, null, null, 0 i.e. if there is a field that does not have a value in either of the events, the combined event will still have 'null'. Otherwise, the highest value will take that place. For presentation purposes you might then want play with fields, replace, table, or rename etc. Hope this helps, K ... View more

Well, if it is all in one event, you should be able to use a multivalued field (see the docs on rex ) and use/retrieve the individual values with the mvindex() function for eval . Best of luck. /k ... View more

Actually agree with you there. Normally, different format of the logs = different sourcetype. But perhaps these sourcetypes are already in place. ... View more