Delete existing summary index and move its data to new summary index

[Karthikeya]

Before one week I created a summary index named waf_opco_yes_summary and it is working fine. Now they asked to change the index name as opco_yes_summary and already existing summary index should be come to this index and that index shouldn't be visible anywhere either in dashboards or searches. That should be deleted and all its data should be moved to new index. What can I do here? 

One more problem is we created a single summary index to all applications and afraid of giving access to them because any of them see that there can see other's apps summary data, it will be a security issue right. We have created a dashboard with summary index and disabled open in search. At some point, we need to give them access to summary index and what if they search index=* then their restricted index and this summary index shows up which can be risky. Is there any way we can restrict users running index=*.

NOTE - already we are using RBAC to restrict users to their specific indexes. But this summary index will show summarised data of all. Any way to restrict this? Can't create summary index for each application.

However in dashboard we are restricting them by a field should be selected then only panel with summary index shows up by filtering. How people handle this type of situations?

[PrewinThomas]

@Karthikeya 

Change the index name
Since its summary index, i would suggest to use collect command to copy your data.
1 - Create new index called opco_yes_summary
2 -Search and use collect to copy

index=waf_opco_yes_summary | collect index=opco_yes_summary

3 - Verify the data

index=opco_yes_summary

4 - Once verified, Delete old index.

Restricting users based on the apps from summary index
I would say creating separate indexes per app might be a nightmare.
As a workaround, Can we consider creating tagging field for summary events if there is no specific field. For eg: field app_name

Then create a role based filter.
Eg:

[role_appA]
srchFilter = index=opco_yes_summary app_name="AppA"

This ideally ensures users only see data for their app, even if they run index=*

But need to avoid “All non-internal indexes” in Roles.

Anyway this needs to be tested and verified, but it might be a good starting point.

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[Karthikeya]

extracted service field from raw data and ingested it into summary index so that it will pick service field values. Then I will use this field in srchFilter to restrict users.

Below is authorise.conf what I have given.

[role_opco_yes_123456_prod]
importRoles = user
srchIndexesAllowed = waf_123456_prod, opco_yes_summary
srchIndexesDefault = waf_123456_prod
srchFilter = (index::waf_123456_prod) OR (index::opco_yes_summary service::JUNIPER-HBEU-ACCESS)

Will this help me or any issue or any changes need to do? Sometimes need to add 1 or 2 services. Is it possible? 

Note - Try to give = in srchFilter while testing in UI, but it thrown error like we can't give = can only give :: can I still give = in backend ultimately I need to write and push it from the backend not from UI.

[PrewinThomas]

Yes it should allow full access to waf_123456_prod and restrict access to opco_yes_summary only where service=JUNIPER-HBEU-ACCESS.

Yes, you can add more services.

Make sure service is an indexed field in opco_yes_summary. If it’s extracted at search time, srchFilter may not work.

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[Karthikeya]

But service field is not an indexed field. I am writing rex to extract that field from original index and then giving collect command to feed it in summary index. Still srchFilter fails in this case? It should be indexed field from the original index as well? Please confirm

[Karthikeya]

[role_appA]

srchFilter = index=opco_yes_summary app_name="AppA"

Just confirming, user already had access to this app index=A assigned. If I give this in srchFilter can he still access the normal index=A data normally right? 

[Karthikeya]

@PickleRick so what will be the best approach to do this? Creating new summary index for every application? We have nearly 100 applications on-boaarded and it will be bit painful to write same query for everything and deploy? Any automation can we do? But still I have zero knowledge on coding.

[PickleRick]

The data architecture is kinda self-contradicting here indeed. Restricting access to "source" data and giving free access to summarized data is asking for trouble. You could try to limit your users with search filters but that's not a very reliable mechanism. If you cannot create multiple summary indexes, you can try with simply storing the summaries back in the original source indexes.

After all summary indexing is just a method of preparing summarized data, there is no "special type" of index called summary index. It doesn't have to be a separate index either. The data would be clearly distinguishable from the "original" data by sourcetype.

But you'd probably be violating one of the basic rules of index architecture - don't mix data of different frequency/cardinality characteristics. In your case, since you need summary indexing, you probably have lots of "base" data and would have just s fairly limited set of summarized data points generated every now and then. So it's possible but it's not the best practice.

[richgalloway]

@livehybrid's advise is good. 

I would add that the idea of copying protected data into an unprotected index is unsound.

[Karthikeya]

Splunk enterprise and it is clustered environment.

We are already using RBAC to restrict users with their indexes. But this shared summary index is giving headache. By giving access to that, they can view other app's summary data as well which is not secure

[livehybrid]

Hi @Karthikeya 

Just a heads up - there are many other ways of triggering a search inside Splunk, so disabling the open in search might not give you as much protection as you think. 

Ultimately I think the main focus here should be on clear segregation of the data. I would focus initially on splitting you raw data into different, well named, indexes based on the following:

1) Retention periods - Data should be split based on different data retention requirements, as retention is set at a per-index basis.

2) Access requirements - in your case you have multiple stakeholders accessing multiple app system data in your indexes. You should split the indexes appropriately such as app_A, app_B etc and then use RBAC to give specific roles access to the appropriate indexes.

3) Technology - We have source/sourcetype for segregation but you can take this a little further and split data in indexes based on technology (e.g. dont put network data in an index with app data...)

I wont go into detail on here how to rename an index or move data around as this depends on your architecture etc. 

Are you using Splunk Enterprise on-premise? What does your deployment look like?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing