Where to find the raw logs coming from my Universal Forwarder to the Heavy Forwarder?

[DanAlexander]

Hello Community,

Now that I have managed to map up the logs from my UF forwarding logs to the HF and then seeing it all landing well on the IDx.

My question is where I can see the passing raw logs on the HF? The main idea is that there will be no indexing done on the HF. The raw logs will be parsed and then send to the IDx for indexing.

I do not have an index where I can apply rules on logs. How to use the HF UI or anything like props and transforms? If the latter how can I know the format of the raw logs on the HF to be able to apply proper filter on?

Thanks All.

[DanAlexander]

Hi @PickleRick and @gcusello 

Thank you for your feedback.

I am aware that indexing on the HF will be reindexed on the IDx, I was thinking about temporarily enabling indexing in order to see variety of logs coming from the UFs. After applying all rules on logs to them disable indexing just enable parsing on the HF.

I most probably do the changes via props and transforms and .will see how it goes. If I am successful I will share back.

Thank you both.

[gcusello]

Hi @DanAlexander,

let us know if we can still help you, otherwise, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors 😉

[gcusello]

Hi @DanAlexander,

if you have to temporary index a log to analyze it, you can also do it on Indexers, eventually in a test index, you don't need to do this on HF, it's an unuseful complication.

In my previous message I described the solutions for your need, anyway I shouldn't index logs on HF.

Ciao.

Giuseppe

[gcusello]

Hi @DanAlexander,

on HF you only apply parsing rules, but you cannot search data.

To understand in which index data are stored you have to run a simple search like the following:

index=* host=<your_host>

Anyway, you don't apply rules on indexes, you apply parsing rules based on sourcetype on HF.

Ciao.

Giuseppe

[DanAlexander]

Thanks @gcusello,

I understand that unless we do not index on the HF we won't be able to see any logs passing the HF.

As a work around can't we not temporarily enable indexAndParse on the outputs config to be able to make logs available on the HF for creating parsing rules and then disable it?

Thanks 

[gcusello]

Hi @DanAlexander ,

as @PickleRick said, you could enable local indexing, but at first there's no advantage to do this, then you pay twice the license consuption.

To find the correct parsing rules yu have two choices:

• make a copy of one of your file data and manually add using GUI;
• send logs to a test index.

I usually use the first.

Ciao.

Giuseppe

[PickleRick · Accepted solution]

You can do indexAndForward but remember that this way your data wil be indexed twice and will incur double license usage (and you can't use forwarder license if you want to index data). Also, the webui extractor is not as powerful and useful as writing files by hand.