About splunk_eval

splunk_eval · ‎11-23-2012

"| transaction ..." would be perfect but it's very slow. Each data source is 100Ks if not 1Ms+ lines. Is there a better/more efficient way to run this command: (sourcetype="my_source1" OR sourcetype="my_source2") | transaction FILENAME | table FILESIZE,FILENAME,DATETIME,DESCRIPTION This pulls FILESIZE, DATETIME from one file (using the timestamp in the file, not _time), and DESCRIPTION from the other file, using FILENAME as the shared ID. Then, running the table command through a "| selfjoin FILENAME" seems to work, and the selfjoin is pretty quick. I'll do some more testing with this and make sure it's merging the way I want.

splunk_eval · ‎11-23-2012

The data will be changing and also be run on a number of different files, so this could be tricky. "| transaction ..." would be perfect but it's very slow. Each data source is 100Ks if not 1Ms+ lines. Is there a better/more efficient way to run this command: (sourcetype="my_source1" OR sourcetype="my_source2") | transaction FILENAME | table FILESIZE,FILENAME,DATETIME,DESCRIPTION This pulls FILESIZE, DATETIME from one file (using the timestamp in the file, not _time), and DESCRIPTION from the other file, using FILENAME as the shared ID.

splunk_eval · ‎11-23-2012

I have two data sources, one that is a very large file listing with *nix timestamps, and one that has a text description of the file. source 1: ls -lR -> permissions ... owner ... filesize timestamp filename example: -rw-r--r-- 1 ownername 123 1234 Jan 01 23:45 file name with spaces.txt source 2: text file -> filename (category)= one-line description example: file name with spaces.txt = this is the text description here for the file I would like to merge these to get: ownername category timestamp filesize filename description source 2 will not have a useful timestamp, it is just the modified date of the text file. A join or a [subsearch] fails because of the large size of both data sources (> 50,000 entries). I would like to be able to chart the data. For example, I'd like to chart the occurrence of files over time with ownername=johndoe and description is not empty. ownername comes from source 1, description comes from source 2. I would also like to stat/summarize the data to get the number of files in 2012 of category "asdf". The category comes from source 2, filename is common to both, and the date comes from source 1.

Posts	3
Solutions	0
Karma Given	1
Karma Received	1
Member Since	‎11-22-2012

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

Joining two large datasets without a Join

Re: Joining two large datasets without a Join

Re: Joining two large datasets without a Join

Joining two large datasets without a Join

Join the Conversation

Joining two large datasets without a Join

Re: Joining two large datasets without a Join

Re: Joining two large datasets without a Join

Joining two large datasets without a Join