How I can index specific part of log ?

[rjfv8205]

Hello, I'm noob in this and I don't know still work with .conf files, I hope you can help me

I have a universal forwarder that forward big log file. In the indexer, how can I index only specific part of log and the rest skip? I don't know still work with .conf files

Thank you in advance

[woodcock]

Considering an event like this in sourcetype tef:

18-05-30;15:38:06.282 \hola.1,237 aaaaaa bbb ccccccc ddd

To index only events that cointain ddd put:

In props.conf:

[tef]
TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ddd
DEST_KEY = queue
FORMAT = indexQueue

Deploy this to your HF and Indexers and restart all Splunk instances and only check events that are indexed after the restart.

[rjfv8205]

Yes, it work but I want index only "ddd" match, no event complete. How can I do?

[woodcock]

OK, then you also need something like this (do try to write a better RegEx but this one will work) in your props.conf in the same section:

SEDCMD-keep_only_ddd = s/^.*?\(ddd).*$/\1/

[woodcock]

You use SEDCMD; see this Q&A for an example:

https://answers.splunk.com/answers/668196/eliminate-unnecessary-values-when-indexing.html

Also @coccyx has an alpha/beta of a new tool that might help you. Clint, what do you say?

[coccyx]

Thanks @woodcock! This is possible with SEDCMD and works out of the box with Splunk, so certainly go down that route first. Helping transform data after the forwarder has picked it up off disk but before it gets written to indexer is one area we're looking to make better. Feel free to reach out to me clint@diag.ai if you find you need a better solution in this area!

[renjith_nair]

Hi @rjfv8205

You can filter data in splunk. Start from here http://docs.splunk.com/Documentation/Splunk/7.1.1/Forwarding/Routeandfilterdatad . If you need to filter a huge portion of the file, it might be useful to have a script to extract the log file entries you need to index and forward to splunk

[rjfv8205]

Thank you !! I'll do test, but i have a question about this:

Following example in the topic "Filter event data and send to queues"

Edit props.conf and transforms.conf in universal forwarder to send specific data?

[woodcock]

That is if you are dumping the entire events; are you dumping "some events in the log" or "some data in each event"? See my answer for SEDCMD example for the latter. In any case, the settings need to go on the HF or Indexers, not on the UF.

[rjfv8205]

What does dumping mean? Sorry I'm from Chile and i try write english best possible jajaja.

If the configuration is in indexer. How I write correctly in props.conf and transfrom.conf for that specific inputs (not all inputs) from UF indexer keep specific entries?

[rjfv8205]

Hi again @woodcock I have tested this and results is not expected.

For example I have this event in log:

18-05-30;15:38:06.282 \hola.1,237 aaaaaa bbb
ccccccc ddd

With configuration below index all events that cointain ddd in log

props.conf

[tef]
TRANSFORMS-set= setnull,setparsing

transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ddd
DEST_KEY = queue
FORMAT = indexQueue

But I want only index ddd