So I've been using CHECK_FOR_HEADER=true for various csv data in some apps I'm building. I've learned a great deal about it recently, but I still have a lot to learn and I wonder if anyone can help...
See more...
So I've been using CHECK_FOR_HEADER=true for various csv data in some apps I'm building. I've learned a great deal about it recently, but I still have a lot to learn and I wonder if anyone can help me with advice about the following problem.
I'm using guided setup so that the user setting up the app can tell splunk up front which column to use as the timestamp. Specifically, the guided setup writes a value to TIME_PREFIX, and all is well. (I cant really let splunk figure it out because there are a couple other epochTime values in there and I cant allow the ambiguity)
Now the data comes in, and CHECK_FOR_HEADER now does it's really weird thing where it looks at the props stanza [foo], looks at the data, writes another stanza to etc/apps/learned, and calls the sourcetype, [foo-2].
( http://www.splunk.com/base/Documentation/4.1.7/Admin/Extractfieldsfromfileheadersatindextime#props.conf )
Another key ingredient is that I leave links back to the setup page -- the user can always run the app setup again later. The problem is that the CHECK_FOR_HEADER magic has meant that the real config is now hidden in etc/apps/learned. My guided setup's custom handler can write to the main props stanza to its heart's content, but it'll never effect the behavior of this 'learned' sourcetype.
This would maybe be OK if there was any way for the user to go edit etc/apps/learned/props.conf stanzas in Manager, but it looks like there isnt (That is question #1).
So I'm facing a choice of various evils, and I dont know much about any of them:
1) try to make a custom manager that can actually dredge up the learned stanzas. OK the custom manager XML side of this is fine, but the fact that etc/apps/learned is totally invisible in the normal manager pages makes me think that EAI wont even give the stanzas back to me or that it might not let me edit them, or that there might be evil consequences thereof (That is question #2).
2) In my custom rest endpoint, pull out any and all 'learned' stanzas and push config changes to them too as necessary (possibly same problem as above)
3) Tell the user that they have to go dig around in etc/apps/learned and hand-edit props.conf. Sadness.
4) abandon CHECK_FOR_HEADER, switch to setting up the app after the data is indexed, and have some crazy system on setup where I retreive the first events, and turn that text into an extraction. (doable, but nasty. Any paths-less-taken out there? )
advice, EAI lore, and/or cautionary tales?
tia