All Apps and Add-ons

Splunk Stream app: Uploading a large pcap file fails without error

sebastianstruwe
Explorer

Hi Splunkers,

I'm trying to uploading a large pcap file (3,5 GB). After entering a name and choosing the file I select next. I can see that the file is being uploaded, but after this step nothing happened.
Is someone facing the same issue?

Best regards,
Sebastian

Tags (2)
0 Karma

dcavuto_splunk
Splunk Employee
Splunk Employee

As noted here http://docs.splunk.com/Documentation/StreamApp/7.1.2/DeployStreamApp/UseStreamtoparsePCAPfiles
there are three ways to use Stream to parse PCAP files;

  • Upload pcaps in Splunk Web.
  • Ingest pcaps using command line options.
  • Ingest pcaps using streamfwd.conf.

Splunk Web has a limit in terms of the file size it can accept via HTTP POST. This seems to be a function of Splunk's implementation of CherryPy in Splunk Web, and likely not to be fixed in the near future.

Using the command line to ingest PCAPs into Stream is the most reliable way to accomplish this task for PCAPs of arbitrary size. From the above page:

Read pcap files

Use the -r option to read individual pcap files. For example:

./streamfwd -r my.pcap

Ingest pcap files from a directory

Use the --pcapdir DIR option to monitor and index pcap files in a directory. For example:

./streamfwd --pcapdir ~/test_pcap_dir --afteringest repeat

You can run these on the search head (if the App for Stream is installed there), or on an arbitrary machine hosting the Stream Forwarder.

0 Karma

pdaigle_splunk
Splunk Employee
Splunk Employee

Have a look here:

https://answers.splunk.com/answers/155769/app-for-stream-pcap-replay.html

and here:

http://docs.splunk.com/Documentation/StreamApp/7.1.2/DeployStreamApp/ConfigureStreamForwarder#stream...

Specifically, Example 4. I think that may hold the key to resolving things and allow you to replay after you upload the pcap to the correct location.

damianpadden
New Member

has it made your inputs.conf file really big?

I am having the same issue. Each time i add a pcap it seems to complain and if i look on my search head under etc\system\local the inputs file will contain the pcap data. Not sure that's right as it could potentially fill up the search heads drive

Damain

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.