topic Re: How I can index specific part of log ? in Getting Data In

How I can index specific part of log ?

rjfv8205 — Wed, 04 Jul 2018 16:15:51 GMT

Hello, I'm noob in this and I don't know still work with .conf files, I hope you can help me

I have a universal forwarder that forward big log file. In the indexer, how can I index only specific part of log and the rest skip? I don't know still work with .conf files

Thank you in advance

Re: How I can index specific part of log ?

renjith_nair — Thu, 05 Jul 2018 00:54:05 GMT

Hi @rjfv8205

You can filter data in splunk. Start from here http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad . If you need to filter a huge portion of the file, it might be useful to have a script to extract the log file entries you need to index and forward to splunk

Re: How I can index specific part of log ?

woodcock — Thu, 05 Jul 2018 01:18:49 GMT

You use SEDCMD; see this Q&A for an example:

https://answers.splunk.com/answers/668196/eliminate-unnecessary-values-when-indexing.html

Also @coccyx has an alpha/beta of a new tool that might help you. Clint, what do you say?

Re: How I can index specific part of log ?

coccyx — Thu, 05 Jul 2018 03:06:55 GMT

Thanks @woodcock! This is possible with SEDCMD and works out of the box with Splunk, so certainly go down that route first. Helping transform data after the forwarder has picked it up off disk but before it gets written to indexer is one area we're looking to make better. Feel free to reach out to me clint@diag.ai if you find you need a better solution in this area!

Re: How I can index specific part of log ?

rjfv8205 — Thu, 05 Jul 2018 16:08:24 GMT

Thank you !! I'll do test, but i have a question about this:

Following example in the topic "Filter event data and send to queues"

Edit props.conf and transforms.conf in universal forwarder to send specific data?

Re: How I can index specific part of log ?

woodcock — Thu, 05 Jul 2018 16:12:41 GMT

That is if you are dumping the entire events; are you dumping "some events in the log" or "some data in each event"? See my answer for SEDCMD example for the latter. In any case, the settings need to go on the HF or Indexers, not on the UF.

Re: How I can index specific part of log ?

rjfv8205 — Thu, 05 Jul 2018 16:24:18 GMT

What does dumping mean? Sorry I'm from Chile and i try write english best possible jajaja.

If the configuration is in indexer. How I write correctly in props.conf and transfrom.conf for that specific inputs (not all inputs) from UF indexer keep specific entries?

Re: How I can index specific part of log ?

rjfv8205 — Thu, 05 Jul 2018 19:29:45 GMT

Hi again @woodcock I have tested this and results is not expected.

For example I have this event in log:

18-05-30;15:38:06.282 \hola.1,237 aaaaaa bbb
ccccccc ddd

With configuration below index all events that cointain ddd in log

props.conf

[tef]
TRANSFORMS-set= setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ddd
DEST_KEY = queue
FORMAT = indexQueue

But I want only index ddd

Re: How I can index specific part of log ?

woodcock — Thu, 05 Jul 2018 20:29:37 GMT

Considering an event like this in sourcetype tef:

18-05-30;15:38:06.282 \hola.1,237 aaaaaa bbb ccccccc ddd

To index only events that cointain ddd put:

In props.conf:

[tef]
TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ddd
DEST_KEY = queue
FORMAT = indexQueue

Deploy this to your HF and Indexers and restart all Splunk instances and only check events that are indexed after the restart.

Re: How I can index specific part of log ?

rjfv8205 — Thu, 05 Jul 2018 21:00:33 GMT

Yes, it work but I want index only "ddd" match, no event complete. How can I do?

Re: How I can index specific part of log ?

woodcock — Fri, 06 Jul 2018 00:08:04 GMT

OK, then you also need something like this (do try to write a better RegEx but this one will work) in your props.conf in the same section:

SEDCMD-keep_only_ddd = s/^.*?\(ddd).*$/\1/