topic Re: Fields that were once extracted aren't being extracted anymore... why? in Splunk Search

Fields that were once extracted aren't being extracted anymore... why?

_dave_b — Thu, 28 Jan 2016 22:25:12 GMT

Hey there,

I made an app. It worked good and extracted data exactly the way I wanted it to. I am now trying to duplicate the app, using the same custom field extractions, but some key fields aren't being extracted, despite using the exact same regex as before!

The permissions are global for all objects in the app. Anybody have any ideas why my fields won't get extracted?

Thanks

Re: Fields that were once extracted aren't being extracted anymore... why?

skoelpin — Thu, 28 Jan 2016 22:34:02 GMT

Your fields are relative to the sourcetype. Can you verify that your sourcetype has not changed?

Re: Fields that were once extracted aren't being extracted anymore... why?

_dave_b — Tue, 29 Sep 2020 08:37:18 GMT

Thanks for your reply. I just did a global search checked the sourcetype. The sourcetype specified in the search has changed. Originally it was 'sourcetype-ts', Splunk appended 'too_small', so now it's being reported as 'sourcetype-too_small'.

Why does Splunk do this and how can I get around it? A little searching tells me Splunk does this to CSV files when it gets confused by them, but my log files aren't CSV, they just plain text, formatted with date-time stamps. I edited props.conf in the $SPLUNK_HOME\etc\system\local to like this:

[source::c:\Program Files\OS\Logs\*\LogType-TS*.txt]
sourcetype=OS_LogType-TS

where the first * wildcard is there to consume a YYYYMMDD value and the second * wildcard consumes anything that may get appended by Splunk, like too-small. I was hoping that all the sourcetypes with 'too-small' appended to the name would be read

Re: Fields that were once extracted aren't being extracted anymore... why?

somesoni2 — Fri, 29 Jan 2016 18:29:25 GMT

Why not assign the sourcetype in the data inputs (inputs.conf)?

Re: Fields that were once extracted aren't being extracted anymore... why?

skoelpin — Fri, 29 Jan 2016 18:54:13 GMT

As @somesoni2 pointed out, you should assign the sourcetype in your inputs.conf (On the forwarder) so it won't change. This will prevent the problems in the future, or in your case sourcetypes changing

Re: Fields that were once extracted aren't being extracted anymore... why?

_dave_b — Fri, 29 Jan 2016 22:18:17 GMT

That's a good idea. I've tried that, so my inputs.conf now looks like

[monitor://c:\Program Files\Blackberry\BES\Logs\*\BES12_BSCP-TS*.txt]

Re: Fields that were once extracted aren't being extracted anymore... why?

_dave_b — Fri, 29 Jan 2016 22:19:49 GMT

Thanks. It still doesn't work, though 😞

Re: Fields that were once extracted aren't being extracted anymore... why?

skoelpin — Sat, 30 Jan 2016 02:20:23 GMT

That's incorrect. Your stanza should look like this.. Don't forget to restart the Splunk service

[Source::YOUR SOURCE]
host = HOSTNAME
sourcetype = OS_LogType-TS

Re: Fields that were once extracted aren't being extracted anymore... why?

_dave_b — Mon, 01 Feb 2016 17:02:50 GMT

According to the docs for props.conf, sourcetype can be used as a spec for a stanza, so that's what I did, in hopes that the sourcetype would be more closely associated with my custom field extractions.

Re: Fields that were once extracted aren't being extracted anymore... why?

_dave_b — Tue, 29 Sep 2020 08:37:45 GMT

What about when the sourcetype is named OS_LogType-TS1, or OS_LogType-TS2?

I changed the inputs.conf in my app's local directory to look like what you said it should look like, but I am still getting no results. My sourcetypes keep changing with numbers appended to their names.

I briefly had some success, but that was by changing the the props.conf file in the system directory. But since I plan on exporting this, app I need to keep all of my custom conf files in local, so that change couldn't apply.

Re: Fields that were once extracted aren't being extracted anymore... why?

skoelpin — Mon, 01 Feb 2016 20:52:30 GMT

Did you restart the Splunk service after making those changes?

Also, did you change the inputs.conf on the indexer or the forwarder?

Re: Fields that were once extracted aren't being extracted anymore... why?

_dave_b — Mon, 01 Feb 2016 21:41:23 GMT

I restarted the Splunk service, _bumped my version number, my indexer is the same machine my forwarder.

My props.conf looks like

[source::c:\\Program Files\\OS_Log\\*\\OS_LogType-TS*.txt]
[OS_LogType-TS]
EXTRACT-deviceIdAndCorrelationId = \[\S+\-(?P[0-9A-Ha-h]{8})\/(?P[\S]{32})\]

and my inputs.conf looks like

`[source::c:\\Program Files\\OS_Logs\\*\\BES12_BSCP-TS*.txt]
disabled = false
host = $decideOnStartup
sourcetype = BES12_BSCP-TS`

And I thought the deviceIdAndCorrelationId field should show up in the web interface under extracted fields for my app, but it isn't there.

Re: Fields that were once extracted aren't being extracted anymore... why?

skoelpin — Tue, 02 Feb 2016 14:45:46 GMT

Ah I found the problem! This stanza should be in your props.conf not the inputs.conf, this should have been obvious LOL, doh!

Add this stanza to your props.conf on the indexer and restart the Splunk service

[source::c:\\Program Files\\OS_Logs\\*\\BES12_BSCP-TS*.txt]
 disabled = false
 host = $decideOnStartup
 sourcetype = BES12_BSCP-TS

http://docs.splunk.com/Documentation/Splunk/6.1/Data/Bypassautomaticsourcetypeassignment

Re: Fields that were once extracted aren't being extracted anymore... why?

skoelpin — Tue, 02 Feb 2016 18:24:22 GMT

Did this solve your sourcetype issue?

Re: Fields that were once extracted aren't being extracted anymore... why?

_dave_b — Tue, 02 Feb 2016 19:03:24 GMT

I'm still having the same problems. Different sourcetypes are being included in the events, so that's at least some progress, but the fields still aren't being extracted.

I guess I will try using a different command, like report or transform...

Re: Fields that were once extracted aren't being extracted anymore... why?

skoelpin — Tue, 02 Feb 2016 19:18:58 GMT

Your asking a different question.. Let's solve the static sourcetype issue first, then create a second question to solve your transform.conf issue. This doc shows multiple ways of assigning a static sourcetype to your events

http://docs.splunk.com/Documentation/Splunk/6.1/Data/Bypassautomaticsourcetypeassignment

Re: Fields that were once extracted aren't being extracted anymore... why?

skoelpin — Tue, 02 Feb 2016 21:24:43 GMT

You're asking a separate question.. Let's address the static sourcetype issue. If you included the sourcetype in your props.conf and restarted the service, then any log files in c:\\Program Files\\OS_Logs\\*\\BES12_BSCP-TS*.txt path will be assigned the sourcetype of BES12_BSCP-TS

Read this doc which explains overriding sourcetypes

http://docs.splunk.com/Documentation/Splunk/6.1/Data/Bypassautomaticsourcetypeassignment

Re: Fields that were once extracted aren't being extracted anymore... why?

_dave_b — Tue, 02 Feb 2016 21:49:52 GMT

Hey skoelpin, my sourcetype issue appears to have been solved! I'm a little tenuous on putting this issue to rest, because I've tried several different methods, but the last thing I tried was setting it via Splunkweb. I also changed the inputs and props.conf files in $SPLUNK_HOME/etc/system. They didn't seem to take effect at first, but maybe there was a delayed reaction.