Splunk Search
Highlighted

Load url via SPL at search line

Motivator

Is it possible to load data from a url using SPL at the search line? Three uses cases, specifically:
1) Load https://server.domain.com:8000/en-US/search/inspector?sid=[sid]&namespace=search for job inspect data for a particular sid,
2) Load https://server.domain.com:8000/en-US/api/search/jobs/[sid]/search.log?outputMode=raw for search.log for a particular sid,
3) Load https://mypage.mysite.com/myfile.csv

I know this can be done with scripts somehow, and while I am interested in that process as well, this question focuses on using the search bar (even with the assistance of an app if necessary) to load the data.

The concept is extremely simple in general. At the SPL search bar, a query something like the following:

| loadurl https://mypage.mysite.com/myfile.csv

Loads the file from the webpage specified just like a normal lookup file would be loaded. If the urls from 1 or 2 were used, the data from those pages would be displayed in the Splunk search head client web UI.

0 Karma
Highlighted

Re: Load url via SPL at search line

Splunk Employee
Splunk Employee

There is a load job command.

| loadjob 1447695249.272

where 1447695249.272 is the SID.

Additionally, you can specify that you'd like to load the events with events=t.

0 Karma
Highlighted

Re: Load url via SPL at search line

Motivator

See my post above for clarification (posted under the OP). I want to load more than just the events from a search.

0 Karma
Highlighted

Re: Load url via SPL at search line

Esteemed Legend

If you are on a Search Head's CLI and you need to get search results back from another search head and then SPL on that data, you will have to save the output from the other search head (using curl or similar to run the job and pull the output back) into a directory where you can then use inputcsv on your search head to pull it in as data for a search.

Or am I misunderstanding what you are trying to do?

0 Karma
Highlighted

Re: Load url via SPL at search line

Motivator

See my post above for clarification (posted under the OP). I want to load more than just the events from a search.

0 Karma
Highlighted

Re: Load url via SPL at search line

Motivator

It seems like people are thinking that I am asking about loading the results from a previously run job, but I am not. Regarding the first two use cases, I am asking how to load the results if the Job Inspect from a previous job and how to load the results of the search.log results from a previous job.
Procedure for #1 in the OP (original post) above:
1) Run job, 2) Click Job Inspect, 3) Right click the new window top bar and choose show as tab (Chrome), 4) Compare url of new window to #1 in the OP above.

Procedure for #2 in the OP above:
1) Run job, 2) Click Job Inspect, 3) Right click the new window top bar and choose show as tab (Chrome),
4) Click search.log, 5) Compare url of window to #2 in the OP above.

Procedure for #3 in the OP above:
1) Go to any file summary page on Virustotal, 2) Look at the url.

For any of the three procedures above, load the results via the Splunk search bar (SPL).

0 Karma
Highlighted

Re: Load url via SPL at search line

Motivator
0 Karma
Highlighted

Re: Load url via SPL at search line

Path Finder

We're using the Getwatchlist add-on and it's working very well.

https://splunkbase.splunk.com/app/635/

View solution in original post

Highlighted

Re: Load url via SPL at search line

Motivator

When I try the following SPL example (found at http://blogs.splunk.com/2011/08/16/getwatchlist-getting-watchlists-into-splunk-quickly-and-easily-wi...), I receive an error:

| getwatchlist http://data.phishtank.com/data/online-valid.csv delimiter=”,” relevantFieldName=url relevantFieldCol=2 referenceCol=3 dateCol=4 categoryCol=8 ignoreFirstLine=true isbad=true | outputlookup phishtank.csv

error:

command="getwatchlist", Error getting settings: need more than 1 value to unpack

What does that mean?

0 Karma
Highlighted

Re: Load url via SPL at search line

Motivator

Issue resolved:
It turns out that copying text from an online source sometimes brings the wrong double quotes. In this case, the double quotes around the comma were wrong and this error is resolved by replacing them with normal double quotes.

Resolution:

 | getwatchlist http://data.phishtank.com/data/online-valid.csv delimiter="," relevantFieldName=url relevantFieldCol=2 referenceCol=3 dateCol=4 categoryCol=8 ignoreFirstLine=true isbad=true | outputlookup phishtank.csv
0 Karma