I am trying to add new app to do some field extraction at index time. I've used a working app to get my folders and files created. I've tested my regex in online regex tester tools. Restarted service but no luck.
Can somebody direct me to which processor I need to set to debug logging mode to troubleshoot my issue?
Thanks
= ==============================
we have multiple servers performing different roles in splunk. So to make search heads life easier, I was thinking on doing this particular extraction on heavy forwarder.
Here is the props file:
[someDevice]
MAX_TIMESTAMP_LOOKAHEAD = 15
TIME_FORMAT = %b %d %H:%M:%S
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRANSFORMS-getMyFields = get_type
Here is transforms:
[get_type]
REGEX = ]\:\s(.+?\s[\d]?[\d]?[\d]?)
FORMAT = event_type::$1
My regex is a bitty cumbersome but that is due to various data coming.
Sample of data is as follows:
Sep 11 15:47:20 111.111.111.111 blabla[123]: sometype on bla bla bla the rest doesnt matter
Sep 11 15:47:20 111.111.111.111 blabla[123]: othertype at bla bla bla the rest doesnt matter
Sep 11 15:47:20 111.111.111.111 blabla[123]: newtype 11 for bla bla bla the rest doesnt matter
I want to extract "sometype", "othertype", "newtype 11".
Last one makes my extraction tricky.
Like lguinn says, do it search-time on the SH instead;
props.conf
[your_sourcetype/source/host]
EXTRACT-blah = \]:\s(?<event_type>\S+(\s\d+)?)
Should work.
/K
zislin,
I think you should escape the initial closing square bracket. And you don't need to put your \d
's in character classes ([]
).
If this regex works in pcregextest - fine - but does it work in Splunk? I think you should test that first, before committing it to your production environment.
At this point, the only thing I am trying to do, is to extract one field from data coming from one existing sourcetype. App is not visible. That's it
The answer depends greatly upon the application.
I've tested my regex with pcregextest command line tool and it was happy.
My regex is
]\:\s(?<event_type>.+?\s[\d]?[\d]?[\d]?)
Does anybody know if I have to restart Splunk services when I am adding a new app via command line on search head?
hi lguinn, that could be good to know, though I tested it and it did not complain on the nested, so-to-speak, capturing groups.
/K
I would use the following regex, though
[your_sourcetype/source/host]
EXTRACT-blah = \]:\s(?<event_type>\S+(?:\s\d+)?)
As I have found that sometimes Splunk is picky about capturing vs. non-capturing groups. Or even
[your_sourcetype/source/host]
EXTRACT-blah = \]:\s(?<event_type>.*?)\s(?:on|at|for)\s
The example I provided does indeed work. Tested on the sample events you provided. /K
just what does your regex look like. What you have in your original question has errors.
Alright. I moved my extraction to SH and did a search-time extraction via props.conf. No luck.
WHen I search for my source type I dont see that new field appear. I have only done debug/refresh not service restart. In addition, I've tested my regex via pcregextest command and it was successful.
I need to find out which debugger to turn on, to see errors. thx
No idea. However, if you post props.conf and transforms.conf stanzas here, someone will probably be able to help you.
Also, I might as well be the first to ask: why an index time field extraction? A search time field extraction is just as fast, easier to create, and much more flexible.