Are you a member of the Splunk Community?
- Find Answers
- :
- Using Splunk
- :
- Splunk Search
- :
- Re: Ingesting different data types into one index
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I am being asked if we can ingest two different data formats into one index. Specifically the primary data type is Key Value pairs however from one source the data format is JSON.
A quick search of this site yielded: https://answers.splunk.com/answers/357887/is-there-a-way-to-configure-splunk-to-parse-a-sour.html
So it seems as thought it is possible.
A second related question would be - If that is possible then it this a bad idea... My experience has been that every time I have introduced strange one-offs into any technological solution the super-cool specialized permutation becomes more hassle than it is worth...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Consider index as your Database and sourcetypes as your tables (schema on the fly) within Database (index). Several types of schemas (sourcetypes) can be defined within your Database (index).
An advantage of such sourcetypes (provided you have followed proper naming conventions would be during event correlation where you can use AND/OR/NOT, =, != and also wildcard (*) within you base search involving your sourcetypes. Adding index and sourcetype and filtering required records upfront in your base query will boost your Splunk search performance.
If you create only one index per schema, you will end up creating too many indexes. However, only advantage is that data from similar sources having same sourcetypes can be corrected/deleted/re-indexed within their own separate indexes, without impacting other sourcetypes.
| makeresults | eval message= "Happy Splunking!!!"
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Consider index as your Database and sourcetypes as your tables (schema on the fly) within Database (index). Several types of schemas (sourcetypes) can be defined within your Database (index).
An advantage of such sourcetypes (provided you have followed proper naming conventions would be during event correlation where you can use AND/OR/NOT, =, != and also wildcard (*) within you base search involving your sourcetypes. Adding index and sourcetype and filtering required records upfront in your base query will boost your Splunk search performance.
If you create only one index per schema, you will end up creating too many indexes. However, only advantage is that data from similar sources having same sourcetypes can be corrected/deleted/re-indexed within their own separate indexes, without impacting other sourcetypes.
| makeresults | eval message= "Happy Splunking!!!"
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ok this really makes sense... how the question is wether or not more than one source type will meet the needs of the customer... I will check with them and ask a follow on question if required.
Thank you.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You see, a Splunk index doesn't enforce any schema at any point, so you can have completely different types of data within one index, hopefully with different sourcetypes.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Of course very logical... thank you for the quick answer...
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
An index can get data from any number of sourcetypes. It's not a strange one-off at all.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Also very logical and helpful... thank you for the quick answer...
Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain
Splunk Lantern’s Guide to The Most Popular .conf25 Sessions
Unlock What’s Next: The Splunk Cloud Platform at .conf25
