Getting Data In

How to monitor an existing directory and upload all files in one shot?

kkossery
Communicator

I have directed my S3 storage logs to Splunk and its humming along nicely. Problem is, I have a lot of old logs on S3 which have not gone to Splunk which I have downloaded to my system and need to manually index this. The directory structure is named according to the date of the month,

6/1/lots of files
6/2/some files
6/3/lots and lots of files
..

etc..

How can I manually add all these logs in one shot? I'm able to add them one by one by Add Data --> Upload Files from My Computer, but it's for a single file. I'm looking at a directory upload.

Thanks for any help.

Tags (3)
0 Karma
1 Solution

ChrisG
Splunk Employee
Splunk Employee

In the Add Data screen, choose Monitor. For source, choose Files & Directories. Browse to the directory you want and click Index Once.

That should do the trick.

View solution in original post

ChrisG
Splunk Employee
Splunk Employee

In the Add Data screen, choose Monitor. For source, choose Files & Directories. Browse to the directory you want and click Index Once.

That should do the trick.

kkossery
Communicator

Do we need to refresh something? I'm not able to see the data I've uploaded once I've followed your steps.

0 Karma

ChrisG
Splunk Employee
Splunk Employee

The time it takes the indexing process to complete will vary based on the amount of data. But you should be able to search for events pretty quickly. Are your searches still returning nothing from the log files in those directories? Check the time range on your search - it might be that you don't have any events in those directories for the time range you have specified.

0 Karma

kkossery
Communicator

I uploaded a directory named /usr/local/splunk/temp/06/04 that has about 20 files of 1 KB each.
After clicking Index once and taking the defaults and doing the search, I get no results,

This is on my search bar,

source="/usr/local/splunk/temp/06/04/*" HOST="Hostname"

I don't see any output.

0 Karma

ChrisG
Splunk Employee
Splunk Employee

You don't want to search the file path, you want to search the index where those log files went. Try index=main unless you specified something else.

0 Karma

kkossery
Communicator

Thank you Chris. I'm sure this is the right way to do it although I wasn't able to see my logs i uploaded. The workaround I was to generate a one line script to direct all the log files and collate to one big file and upload it. I'm able to see the data now.
Thank you!

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...