All Apps and Add-ons
Highlighted

SA-SPLICE: Is there a MongoDB schema/setup that can be used to force the structure into a mongoDB instance that the install didn't complete?

Explorer

The documentation doesn't appear to contain a schema reference for the app and is stuck reinstalling the app on a search head that now has a second MongoDB instance to use Splice.

Is there a Mongodb schema/setup that can be used to force the structure into a mongoDB instance that the app install didn't complete?

From a working instance I know the collections should be 'atomic_indicators', 'raw', and 'system.indexes' however, this install only hsa the last two 'tables'.

Next steps
- Remove completely the Splice app again

- Stop and remove the existing mongodb data directories
- start clean mongodb with new data directories
- start splunk then reinstall the Splice app
- Hope that works

0 Karma
Highlighted

Re: SA-SPLICE: Is there a MongoDB schema/setup that can be used to force the structure into a mongoDB instance that the install didn't complete?

Splunk Employee
Splunk Employee

The tables, or collections in the NoSQL world, are automatically created by Splice. IF it can create the raw collection like you suggest it but it cannot create the atomic_indicators one, it could be a right permission issue, or most likely that the ingested IOCs do not contains technical indicators. Have you tried with some samples from iocbucket.org for example?

0 Karma
Highlighted

Re: SA-SPLICE: Is there a MongoDB schema/setup that can be used to force the structure into a mongoDB instance that the install didn't complete?

Explorer

Thank you for the quick response; however I understand that the supporting scripts inside of Splice create the collections. But what I was looking for is there a specific script (bin/splice/database.py for example) that could be invoked directly to force creation of the collections?

In this instance case the MongoDB is on the same search head as a new ES app deployment so I created an entirely new MongoDB instance for Splice with a new local port and database paths.

Example,
Splice MongoDB instance
Listening port: 27018/tcp
DBPath=/data/db (taken from MongoDB documentation directly for binary builds)
Execution command: /usr/local/mongodb/bin/mongo --port=27018
Invoked as user: root (just to show that the permissions aren't the issue)

On the Splunk side the Splice app has been deleted/removed using (splunk app remove SA-Splice) the approved Splunk commands.
The Splunk daemon is then restarted and the SA-Splice application re-installed and configured using the application mongodb path.
'mongodb://localhost:27018/splice'

This is the stage that should be creating the collections for the application but without a specific script to force install into mongo or for a log file to check what's erroring out it's impossible to determine where the app or permissions might be failing.

Also my example IOC's are from iocbucket.com; I found that to be a great example to show the power of Splice. My example IOC's are pulled down via the iocbucket RSS feed then decode and store the IOC files on the search head for ingestion into Splice.

0 Karma
Highlighted

Re: SA-SPLICE: Is there a MongoDB schema/setup that can be used to force the structure into a mongoDB instance that the install didn't complete?

Splunk Employee
Splunk Employee

The collections are created "on the fly", that's the whole purpose of nosql, to avoid table schemas. The collections are created when they are "needed", and not at startup as you describe. In short, the atomic_indicators collection is created only when Splice has to store such indicators, meaning it has previously read an IOC file where extractable indicators have been found (like an IPv4 for example). It is possible that the RSS feed include unexpected characters (I don't know, just guessing here). So, what I would recommend is to manually download an IOC file and store it manually in the directory you configured as modular input. Be sure to store a valid OpenIOC file or a STIX file.

For the log part, have a look in the splunkd.log.

Out of curiosity, why do you use Splice and not ES?

0 Karma
Highlighted

Re: SA-SPLICE: Is there a MongoDB schema/setup that can be used to force the structure into a mongoDB instance that the install didn't complete?

Explorer

I think I at least forced the collections to function by dumping a working splice example instance into this search head.
Using the mongoDB from another instance I was able to 'mongorestore' the entire collection with a number of different types of IOCs from iocbucket and others. Performing this action allowed me to have the full splice mongodb instance functioning at least from the backend.

At this point Splice appears to be functioning with around 50 sample IOC's inside of the mongoDB instance. As well the UI finally has the data needed for the tables and no errors are being logged into the splunkd log file anymore.
So the addition of a functioning mongodb with data inside appears to have fixed my initial problem.

As for why I'm using Splice vs ES; I'm using Splice for IOC collection from 3rd parties and the native TAXII/STYX/CYBOX support for quick dashboard design and alerting/reporting functions. This site also has deployed ES and will be leveraging some of the Splice additions into ES for added Endpoint integration. However, I can also use Splice and monitor for IOCs outside of ES, which can be a jacked up deployment that takes time away from monitoring for IOCs while being setup.

Splice (without mongoDB issues) - up and running with sample IOCs < 4 hours to production
ES (without any hiccups) - up and running within about a week often longer

Thanks again for the mongo help

0 Karma