Archive
Highlighted

How I can index specific part of log ?

Explorer

Hello, I'm noob in this and I don't know still work with .conf files, I hope you can help me

I have a universal forwarder that forward big log file. In the indexer, how can I index only specific part of log and the rest skip? I don't know still work with .conf files

Thank you in advance

0 Karma
Highlighted

Re: How I can index specific part of log ?

SplunkTrust
SplunkTrust

Hi @rjfv8205

You can filter data in splunk. Start from here http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad . If you need to filter a huge portion of the file, it might be useful to have a script to extract the log file entries you need to index and forward to splunk

0 Karma
Highlighted

Re: How I can index specific part of log ?

Explorer

Thank you !! I'll do test, but i have a question about this:

Following example in the topic "Filter event data and send to queues"

Edit props.conf and transforms.conf in universal forwarder to send specific data?

0 Karma
Highlighted

Re: How I can index specific part of log ?

Esteemed Legend

That is if you are dumping the entire events; are you dumping "some events in the log" or "some data in each event"? See my answer for SEDCMD example for the latter. In any case, the settings need to go on the HF or Indexers, not on the UF.

0 Karma
Highlighted

Re: How I can index specific part of log ?

Explorer

What does dumping mean? Sorry I'm from Chile and i try write english best possible jajaja.

If the configuration is in indexer. How I write correctly in props.conf and transfrom.conf for that specific inputs (not all inputs) from UF indexer keep specific entries?

0 Karma
Highlighted

Re: How I can index specific part of log ?

Explorer

Hi again @woodcock I have tested this and results is not expected.

For example I have this event in log:

18-05-30;15:38:06.282 \hola.1,237 aaaaaa bbb
ccccccc ddd

With configuration below index all events that cointain ddd in log

props.conf

[tef]
TRANSFORMS-set= setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ddd
DEST_KEY = queue
FORMAT = indexQueue

But I want only index ddd

0 Karma
Highlighted

Re: How I can index specific part of log ?

Esteemed Legend

You use SEDCMD; see this Q&A for an example:

https://answers.splunk.com/answers/668196/eliminate-unnecessary-values-when-indexing.html

Also @coccyx has an alpha/beta of a new tool that might help you. Clint, what do you say?

0 Karma
Highlighted

Re: How I can index specific part of log ?

Path Finder

Thanks @woodcock! This is possible with SEDCMD and works out of the box with Splunk, so certainly go down that route first. Helping transform data after the forwarder has picked it up off disk but before it gets written to indexer is one area we're looking to make better. Feel free to reach out to me clint@diag.ai if you find you need a better solution in this area!

0 Karma
Highlighted

Re: How I can index specific part of log ?

Esteemed Legend

Considering an event like this in sourcetype tef:

18-05-30;15:38:06.282 \hola.1,237 aaaaaa bbb ccccccc ddd

To index only events that cointain ddd put:

In props.conf:

[tef]
TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ddd
DEST_KEY = queue
FORMAT = indexQueue

Deploy this to your HF and Indexers and restart all Splunk instances and only check events that are indexed after the restart.

0 Karma
Highlighted

Re: How I can index specific part of log ?

Explorer

Yes, it work but I want index only "ddd" match, no event complete. How can I do?

0 Karma