topic Re: Index past logs under one source in Getting Data In

Index past logs under one source

gnovak — Thu, 21 Mar 2013 14:03:34 GMT

I was wondering: Is there a way to index past logs and still have them show up as just one source?

Example:

I have a directory with a bunch of logs in it. They look like:

BEK02132013.log
BEK02142013.log
BEK02152013.log
BEK02162013.log
BEK02172013.log
BEK02182013.log
....
....
etc., etc.,

So a new log is made every day with the date in it. This means if I setup a monitor inputs for this directory, all the files are indexed. They all show up as a source and this makes my source list huge!

Considering there are timestamps in the logs, I was wondering is there a way for all these logs to just be under one source? Example: All this data is under the source BEK.log.

Re: Index past logs under one source

kristian_kolb — Thu, 21 Mar 2013 14:44:13 GMT

Yes, that can be done, but it will not alter already indexed data, just new stuff coming in.

Assuming you have an inputs.conf that looks like this;

[monitor:///var/logs/BEKLOGS]
index = blah
sourcetype = bek

you would want to have a props.conf entry like this;

[bek]
TRANSFORMS-foo = set_bek_source

and a transforms.conf like this

[set_bek_source]
REGEX = .
DEST_KEY = MetaData:Source
FORMAT = source::BEK.log

For more examples, see:

http://splunk-base.splunk.com/answers/5544/override-source-tcpxxxx-of-a-tcp-input-using-transforms
http://docs.splunk.com/Documentation/Splunk/5.0.2/Admin/Transformsconf

Hope this helps,

Kristian

Re: Index past logs under one source

gnovak — Thu, 21 Mar 2013 15:23:10 GMT

Ok i will try this. I assumed it used transforms but wasn't sure the exact way to go about it. Let me test this.

Re: Index past logs under one source

gnovak — Thu, 21 Mar 2013 15:52:58 GMT

I have the inputs on the forwarder and made entries in props.conf and transforms.conf on the indexer. So far don't have the logs showing up but will look at things. I had a crc salt error so i added crcsalt = and now it's working! Thanks so much for your assistance.

Re: Index past logs under one source

kristian_kolb — Thu, 21 Mar 2013 15:59:16 GMT

just to clarify; if you use a heavy forwarder, the props and transforms should go there and not on the indexer.

For universal or lightweight forwarder, the settings should be on the indexer.

Re: Index past logs under one source

gkanapathy — Thu, 21 Mar 2013 16:17:26 GMT

If you're just setting the source to a static value, you can do it via transforms, as kristian.kolb suggested, but it might be simpler to just do:

[monitor:///my/log/path/BEK*.log]
sourcetype=BEKlogs
source=BEK.log

However, if you're going to do that, I might just suggest you ignore "source" and use "sourcetype" anyway. If on the other hand, you want to preserve part of the source path, e.g., you are monitoring files like:

[monitor:///my/path/*/logs/BEK*.log]

and you want the source to read like /my/path/group1/logs/BEK.log, you would use kristian's method of a transform, but you would need a more complex REGEX and FORMAT to extract and use the appropriate parts of the source you want.

Re: Index past logs under one source

kristian_kolb — Thu, 21 Mar 2013 16:29:34 GMT

aah, I knew there was a simpler way... just never done much of source overriding, just index, host etc.