Getting Data In

Oneshot and Inputs.conf

mgherman
Explorer

Hi,

I recently had to go back and add some historical data that was generated prior to splunk being installed on some servers.

To do this I used "./splunk add oneshot /var/log/nginx/error.log.1" on a server that is functioning as a Forwarder.

The data has been indexed and is available on Splunk server, however it appears that 2 additional indexed fields that are defined in $SPLUNK/etc/system/local/inputs.conf have not been added.

Data that has been generated and indexed in realtime has these fields defined.

==inputs.conf==

[default]
host = hostname.example.com
_meta = role::examplerole farm::examplefarm

Is this the correct behaviour and how can I define those indexed fields when using oneshot in future?

Regards,

Michael Herman

Tags (2)
1 Solution

Lowell
Super Champion

I think you would have to rewrite these explicit index field additions as a index-time transformer associated with the sourcetype that these events get assigned. From there it's a simple matter of making sure oneshot gets the sourcetype right; which can be down with a [source:....] pattern or a number of other ways.

The oneshot command would have no way of knowing that you have an inputs.conf entry to pick up a similarity named file; therefore it would have no way of knowing that you want to add some indexed fields to that input. Does that make sense? My understanding is that oneshot is like temporarily adding a completely new inputs.conf entry which therefore wouldn't have have a "_meta" key.

View solution in original post

Lowell
Super Champion

I think you would have to rewrite these explicit index field additions as a index-time transformer associated with the sourcetype that these events get assigned. From there it's a simple matter of making sure oneshot gets the sourcetype right; which can be down with a [source:....] pattern or a number of other ways.

The oneshot command would have no way of knowing that you have an inputs.conf entry to pick up a similarity named file; therefore it would have no way of knowing that you want to add some indexed fields to that input. Does that make sense? My understanding is that oneshot is like temporarily adding a completely new inputs.conf entry which therefore wouldn't have have a "_meta" key.

View solution in original post

Genti
Splunk Employee
Splunk Employee

Im not sure where you have found the above option, it might work, but is definitely not documented..

Anyhow, a while back i submitted a bug about .csv files and oneshot not respecting the props/transforms but i have not tried it with regular log files (note csv use headers)

Question is: Does this have to happen at index time? Can you not use props/transforms in order to make a search time field extraction instead? That would be a lot cheaper (resource wise) for the indexer and faster. Also, you would not have to worry about if it works with oneshot or not.

Check the docs for more info on search time field extractions

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!