topic Re: Troubleshooting props.cof and transforms.conf in Getting Data In

Troubleshooting props.cof and transforms.conf

zislin — Wed, 11 Sep 2013 19:54:16 GMT

I am trying to add new app to do some field extraction at index time. I've used a working app to get my folders and files created. I've tested my regex in online regex tester tools. Restarted service but no luck.

Can somebody direct me to which processor I need to set to debug logging mode to troubleshoot my issue?

Thanks
= ==============================
we have multiple servers performing different roles in splunk. So to make search heads life easier, I was thinking on doing this particular extraction on heavy forwarder.

Here is the props file:

[someDevice]
MAX_TIMESTAMP_LOOKAHEAD = 15
TIME_FORMAT = %b %d %H:%M:%S
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRANSFORMS-getMyFields = get_type

Here is transforms:

[get_type]
REGEX = ]\:\s(.+?\s[\d]?[\d]?[\d]?)
FORMAT = event_type::$1

My regex is a bitty cumbersome but that is due to various data coming.
Sample of data is as follows:

Sep 11 15:47:20 111.111.111.111 blabla[123]: sometype on bla bla bla the rest doesnt matter
Sep 11 15:47:20 111.111.111.111 blabla[123]: othertype at bla bla bla the rest doesnt matter
Sep 11 15:47:20 111.111.111.111 blabla[123]: newtype 11 for bla bla bla the rest doesnt matter

I want to extract "sometype", "othertype", "newtype 11".
Last one makes my extraction tricky.

Re: Troubleshooting props.cof and transforms.conf

lguinn2 — Wed, 11 Sep 2013 19:59:21 GMT

No idea. However, if you post props.conf and transforms.conf stanzas here, someone will probably be able to help you.

Also, I might as well be the first to ask: why an index time field extraction? A search time field extraction is just as fast, easier to create, and much more flexible.

Re: Troubleshooting props.cof and transforms.conf

kristian_kolb — Wed, 11 Sep 2013 20:35:51 GMT

Like lguinn says, do it search-time on the SH instead;

props.conf

[your_sourcetype/source/host]
EXTRACT-blah = \]:\s(?<event_type>\S+(\s\d+)?)

Should work.

Re: Troubleshooting props.cof and transforms.conf

zislin — Wed, 11 Sep 2013 20:52:32 GMT

Alright. I moved my extraction to SH and did a search-time extraction via props.conf. No luck.
WHen I search for my source type I dont see that new field appear. I have only done debug/refresh not service restart. In addition, I've tested my regex via pcregextest command and it was successful.

I need to find out which debugger to turn on, to see errors. thx

Re: Troubleshooting props.cof and transforms.conf

kristian_kolb — Wed, 11 Sep 2013 21:06:19 GMT

just what does your regex look like. What you have in your original question has errors.

Re: Troubleshooting props.cof and transforms.conf

kristian_kolb — Wed, 11 Sep 2013 21:17:30 GMT

The example I provided does indeed work. Tested on the sample events you provided. /K

Re: Troubleshooting props.cof and transforms.conf

lguinn2 — Wed, 11 Sep 2013 23:51:18 GMT

I would use the following regex, though

[your_sourcetype/source/host] EXTRACT-blah = \]:\s(?<event_type>\S+(?:\s\d+)?)

As I have found that sometimes Splunk is picky about capturing vs. non-capturing groups. Or even

[your_sourcetype/source/host] EXTRACT-blah = \]:\s(?<event_type>.*?)\s(?:on|at|for)\s

Re: Troubleshooting props.cof and transforms.conf

kristian_kolb — Thu, 12 Sep 2013 08:00:17 GMT

hi lguinn, that could be good to know, though I tested it and it did not complain on the nested, so-to-speak, capturing groups.

Re: Troubleshooting props.cof and transforms.conf

zislin — Thu, 12 Sep 2013 13:53:11 GMT

I've tested my regex with pcregextest command line tool and it was happy.

My regex is
]\:\s(?<event_type>.+?\s[\d]?[\d]?[\d]?)

Does anybody know if I have to restart Splunk services when I am adding a new app via command line on search head?

Re: Troubleshooting props.cof and transforms.conf

sowings — Thu, 12 Sep 2013 13:56:38 GMT

The answer depends greatly upon the application.

Re: Troubleshooting props.cof and transforms.conf

zislin — Thu, 12 Sep 2013 14:05:07 GMT

At this point, the only thing I am trying to do, is to extract one field from data coming from one existing sourcetype. App is not visible. That's it

Re: Troubleshooting props.cof and transforms.conf

kristian_kolb — Thu, 12 Sep 2013 19:02:07 GMT

zislin,

I think you should escape the initial closing square bracket. And you don't need to put your \d's in character classes ([]).

If this regex works in pcregextest - fine - but does it work in Splunk? I think you should test that first, before committing it to your production environment.