How to ingest Data from forwarders with a single file that contains consolidated file and will be separated via sourcetype

[ejmin]

The format of the data is like this,
ColumnA ColumnB ColumnC.....
1. Data A DataB DataC
2. Data A DataB DataC
3. Data A DataB DataC
ColumnD ColumnE ColumnF ColumnG ColumnH
4. DataD DataE DataF DataG DataH
5. DataD DataE DataF DataG DataH
6. DataD DataE DataF DataG DataH
7. ColumnI ColumnJ
8. Datai Dataj

The problem is not the filtration of data, Ive already done that. What the real problem is I can't ingest multiple folders in a single indexers and I checked the inputs.conf the monitor is there but the data when searching is not found The only thing that the forwarder ingest is the first monitor

this is the code of my batch file

msiexec.exe /i splunkforwarder_x64.msi AGREETOLICENSE=yes SPLUNKUSER=admin SPLUNKPASSWORD=changeme DEPLOYMENT_SERVER="xxx.xxx.xxx.xxx:8089" RECEIVING_INDEXER="xxx.xxx.xxx.xxx:9997" WINEVENTLOG_SEC_ENABLE=1 WINEVENTLOG_SYS_ENABLE=1 /quiet /passive
cd "C:\Program Files\SplunkUniversalForwarder\bin"
splunk add monitor C:\auto-forward\folder1\file1 -index aforward -sourcetype st1
splunk add monitor C:\auto-forward\folder2\file1 -index aforward -sourcetype st2

This is what I guess the forwarder consider the file the same and it only separated via sourcetype so it applies the rule of the forwarder that it cant do a double ingestion. But actually the data is not doubled because its filtered via transforms.conf.

So any suggestion guys? Hope you help me.

[lakshman239 · Accepted solution]

I do not think you can monitor the same directory/file with 2 different sourcetypes. You would need a way to extract the events for 2 different sourcetypes before defining them OR have one sourcetype and using transforms explore options to filter events for st1 and override sourcetype using Metadata.

[ejmin]

Ok Thanks I found a solution based on your idea but not exactly the same but I will accept your answer because it helps me to solve my problem......I used transforms.conf and props.conf to solved it by creating 1 sourcetype to have my forwarder monitors only single file and used transforms.conf to separate it to different sourcetype.

[skalliger]

Hi,

This is what I guess the forwarder consider the file the same and it only separated via sourcetype so it applies the rule of the forwarder that it cant do a double ingestion. But actually the data is not doubled

So, the path is different, but the content is the same? If so, try setting crcSalt = <SOURCE> (literally) in your inputs.conf for both monitor stanzas.

Skalli

[ejmin]

Hmmm... Thank you for your response I think your solution is correct but i change my strategy instead of ingesting multiple files separated by sourcetype I used 1 file and then use transforms.conf to extract it.