Why am I losing data during transmission to Splunk

[igor04653]

Hello. Community help please. I can't figure out the problem with the data transfer to splunk. I have an index and data sources from servers. The problem is that some of the data is lost during transfers. There are files on the server that are updated with a new name after a certain time. For example there are files N2-1.out01324, N2-1.out01325 they are searchable and Splunk can see them. But then files are updated with new name for example N2-1.out01326, N2-1.out01327 and these files are not available Splunk can't see them. Then the list is updated and files N2-1.out01328-1329 are visible again

[gcusello]

Hi @igor04653,

as @PickleRick said, Splunk doesn't index a content twice even if the file has a different name, but the same content.

If you want to index all files, also duplicating logs, you should use 

crcSal = <SOURCE>

in this way Splunk index all files with a different filename even if they have the same content.

Ciao.

Giuseppe

[PickleRick]

It's not clear what you're talking about. If the file is called file1.log and is ingested into splunk, events from that file will have their source field set to,file1.log and it will never change no matter what you do with the file on the source server. If you later rename the file on the source computer, splunk will still know it's the same file (unless you configure it to include source filename in crc calculation) and will not re-read it again.

It's not clear what you're doing and what you're expecting.