How to monitor an existing directory and upload all files in one shot?

[kkossery]

I have directed my S3 storage logs to Splunk and its humming along nicely. Problem is, I have a lot of old logs on S3 which have not gone to Splunk which I have downloaded to my system and need to manually index this. The directory structure is named according to the date of the month,

6/1/lots of files
6/2/some files
6/3/lots and lots of files
..

etc..

How can I manually add all these logs in one shot? I'm able to add them one by one by Add Data --> Upload Files from My Computer, but it's for a single file. I'm looking at a directory upload.

Thanks for any help.

[ChrisG · Accepted solution]

In the Add Data screen, choose Monitor. For source, choose Files & Directories. Browse to the directory you want and click Index Once.

That should do the trick.

[kkossery]

Do we need to refresh something? I'm not able to see the data I've uploaded once I've followed your steps.

[ChrisG]

The time it takes the indexing process to complete will vary based on the amount of data. But you should be able to search for events pretty quickly. Are your searches still returning nothing from the log files in those directories? Check the time range on your search - it might be that you don't have any events in those directories for the time range you have specified.

[kkossery]

I uploaded a directory named /usr/local/splunk/temp/06/04 that has about 20 files of 1 KB each.
After clicking Index once and taking the defaults and doing the search, I get no results,

This is on my search bar,

source="/usr/local/splunk/temp/06/04/*" HOST="Hostname"

I don't see any output.

[ChrisG]

You don't want to search the file path, you want to search the index where those log files went. Try index=main unless you specified something else.

[kkossery]

Thank you Chris. I'm sure this is the right way to do it although I wasn't able to see my logs i uploaded. The workaround I was to generate a one line script to direct all the log files and collate to one big file and upload it. I'm able to see the data now.
Thank you!