Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- About dottom
dottom
Path Finder
Member since:
05-13-2010
06-05-2020
Community Statistics
| Posts | 11 |
| Solutions | 0 |
| Karma Given | 0 |
| Karma Received | 4 |
| Member Since | 05-13-2010 |
Activity Feed
- Got Karma for Why is index-time field extraction not searchable?. 06-05-2020 12:45 AM
- Got Karma for Why is index-time field extraction not searchable?. 06-05-2020 12:45 AM
- Got Karma for Re: How to selectively index and forward with filtering?. 06-05-2020 12:45 AM
- Got Karma for Re: How to selectively index and forward with filtering?. 06-05-2020 12:45 AM
- Posted Re: How to selectively index and forward with filtering? on Getting Data In. 06-24-2011 10:18 AM
- Posted Re: How to selectively index and forward with filtering? on Getting Data In. 06-14-2011 11:32 PM
- Posted How to selectively index and forward with filtering? on Getting Data In. 06-14-2011 10:49 PM
- Tagged How to selectively index and forward with filtering? on Getting Data In. 06-14-2011 10:49 PM
- Tagged How to selectively index and forward with filtering? on Getting Data In. 06-14-2011 10:49 PM
- Tagged How to selectively index and forward with filtering? on Getting Data In. 06-14-2011 10:49 PM
- Tagged How to selectively index and forward with filtering? on Getting Data In. 06-14-2011 10:49 PM
- Posted Re: 'unicode' object has no attribute 'get' on Dashboards & Visualizations. 05-30-2011 06:14 PM
- Posted Re: Why is index-time field extraction not searchable? on Splunk Search. 06-15-2010 10:51 PM
- Posted Re: Why is index-time field extraction not searchable? on Splunk Search. 06-15-2010 09:32 PM
- Posted Re: Why is index-time field extraction not searchable? on Splunk Search. 06-15-2010 09:27 PM
- Posted Re: Why is index-time field extraction not searchable? on Splunk Search. 06-15-2010 08:49 PM
- Posted Re: Why is index-time field extraction not searchable? on Splunk Search. 06-15-2010 06:59 AM
- Posted Re: Why is index-time field extraction not searchable? on Splunk Search. 06-15-2010 06:59 AM
- Posted Why is index-time field extraction not searchable? on Splunk Search. 06-12-2010 07:13 PM
- Tagged Why is index-time field extraction not searchable? on Splunk Search. 06-12-2010 07:13 PM
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 0 | |||
| 2 |
06-24-2011
10:18 AM
2 Karma
That solution doesn't work for this environment because I cannot configure the forwarders to send to different indexers. I have a single indexer that is the central hub for many different data inputs. What I want to do is selectively forward and index from a single indexer
Right now, with Splunk, an indexer can selective forward without indexing.
I am using "indexAndForward" and _INDEX_AND_FORWARD_ROUTING in each input stanza. So at the moment, my forwarding granularity is limited to a stanza in inputs.conf. What I want to do is be able to use props.conf and transforms.conf to selective decide:
What to index only, and not forward.
What to index and forward.
... View more
06-14-2011
11:32 PM
My scenario is different in that I don't want to filter out events from being indexed. What I want to do is filter events to be forwarded, i.e. do not forward some events (only index it), forward specific sourcetype to remoteHostA, forward specific REGEX string to remoteHostB, etc.
The scenario:
A single inputs.conf stanza receives logs from 100 different systems.
I want to index all of them (using "indexAndForward" and _INDEX_AND_FORWARD_ROUTING in each input stanza).
But I want to selectively forward some logs to some other log consumer devices (using props.conf and transforms.conf, which does not work for "indexAndForward").
I don't want to just forward using LWF/HF/UF which is very flexible to customize using props.conf and transforms.conf. This is a "index and selectively forward" approach.
As a kludge, I've considered running both a forwarder and index instance (two Splunk instances) and have the forwarder forward locally what I want indexed, and forward remotely what I want sent off to other log collection devices. But I really don't want to run two Splunk instances just to have flexible filtering capability for a "index and forward" design.
... View more
06-14-2011
10:49 PM
Is there a way to selectively index and forward by using filtering criteria such as hostname, sourcetype, or REGEX in transforms.conf? Currently, I can selectively index and forward on a per input stanza basis in inputs.conf, but I don't want to forward everything coming into an input.
If I were to only forward (and not index locally), I would use a REGEX in transforms.conf with a [stanza] in props.conf to filter what to forward. But it looks like using transforms.conf and props.conf is not supported for selective index and forwarding.
... View more
05-30-2011
06:14 PM
This is still broken in 4.2.1.
... View more
06-15-2010
10:51 PM
Changing fields.conf to indexed = false fixes it for the second example which solves my problem. In my particular case, since I use a transform to perform data masking, I am not sure if the actual key=value pairs are written to the index or not. I will run an export to verify. The behavior of the first example in my original post is interesting though, because in theory it should not work if indexed=true for fields.conf?
... View more
06-15-2010
09:32 PM
To answer your other question "why are you using indexed fields instead of extracted fields". I need to inject values to the original syslog message and have those meta-values searchable via key/value pairs. When a user runs a query, the user needs to see only the original syslog message, not the meta-data I've injected. For example, I want a user to be able to run a query "event=logon result=failure user=lowell" and it will pull up all failed logons across 100 different platforms. Note this works fine if I don't use quotes in transforms.conf, but only for values that don't have spaces.
... View more
06-15-2010
09:27 PM
Question edited. I couldn't add the image tag directly but have link to the screenshot.
As for search-time field extractions, by definition, I think if you search time extracted the field you would always be able to search for it. The bug here I believe is with index-time extraction only where the values are written and shows up in search results (and on left hand side blue fields panel), they just aren't searchable.
... View more
06-15-2010
08:49 PM
I do get search results if I do not put the backreferenced values in quotes. The problem I have is I want to use quotes because that's the correct way to capture the values (with spaces in them), but then I have the other problem I listed in the original post in that those fields extracted at index-time are not searchable.
... View more
06-15-2010
06:59 AM
That doesn't work either. Any combination of the following queries all fail when using double-quotes in transforms.conf to define the value of the index-time extraction:
key1=AA
key1="AA"
key1::AA
key1::"AA"
The values are clearly extracted as you can see the meta-data defined in the fields that show right below the text message.
... View more
06-15-2010
06:59 AM
Yes, all the fields are defined in fields.conf
You need the double-quotes in transforms.conf when the regular expression backreference captures a value with a space in it.
... View more
06-12-2010
07:13 PM
2 Karma
I'm double posting, original issue posted here: http://www.splunk.com/support/forum:SplunkGeneral/4378
When I use double-quotes in my index-time field extractions, the meta-data is not searchable. I've seen this problem on 4.0.11 and 4.1.3.
Sample text:
results=AA,BB,CC CC,DD
Transforms.conf without double-quotes:
REGEX = ^results=(.*?),(.*?),(.*?),(.+)$
FORMAT = key1::$1 key2::$2 key3::$3 key4::$4
WRITE_META = true
Transform.conf with double-quotes:
REGEX = ^results=(.*?),(.*?),(.*?),(.+)$
FORMAT = key1::"$1" key2::"$2" key3::"$3" key4::"$4"
WRITE_META = true
Results:
If you use the first transforms.conf without the double-quotes, there are two problems:
The value for key3 (with a space) is
not captured correctly. This is in
the documentation which says to use
double-quotes.
The fields extracted on 4.1.3 are
incorrect for key4. Instead of having
a field "key4" it has "CC key4". I
don't recall seeing this behavior in
4.0.x.
However, if you use the second transforms.conf with the double-quotes:
The meta-data is not searchable, i.e.
search for "key1=AA" fails.
UPDATE 6/15/2010
Here are my conf files so you can replicate this issue. I also have a screenshot below.
inputs.conf:
[monitor:///var/log/test]
disabled = 0
sourcetype = mytest
props.conf:
[mytest]
TRANSFORMS-test = extract-fields
fields.conf:
[key1]
INDEXED = true
[key2]
INDEXED = true
[key3]
INDEXED = true
[key4]
INDEXED = true
transforms.conf:
[extract-fields]
REGEX = ^results=(.*?),(.*?),(.*?),(.+)$
FORMAT = key1::"$1" key2::"$2" key3::"$3" key4::"$4"
WRITE_META = true
screenshot:
In this screenshot, notice that the values are indeed extracted and show up in the search result. However, searching for "key1=AA" (or any other key=value) returns no results.
http://dottom.com/public/images/screenshot_8jd49x4d.png
... View more
- Tags:
- index-time
- metadata
