Using lookup table to circumvent search time constraints

[michaeler]

I just want a sanity check to see if this is possible before I go through the effort. I am currently restricted to searching back <=90 days in Splunk but have access to the >90 days data in the source database.

To circumvent this restriction I figure I can convert the old data into a lookup table file, set the time range at "all-time", and append to the lookup table.

Has anyone tried this before or is this theoretically possible?

[gcusello]

Hi @michaeler,

don't use a lookup, but a summary index:

you can schedule a daily search that extract the fields you need and stores them in a summary index using the collect command (https://docs.splunk.com/Documentation/SplunkCloud/9.0.2303/SearchReference/collect) giving to this summary index a greater retention, then you can run your searches on the summary index with also better performances.

For more infos see at https://docs.splunk.com/Documentation/Splunk/9.0.5/Knowledge/Usesummaryindexing#:~:text=From%20the%2.... 

Ciao.

Giuseppe

[michaeler]

@gcusello Thank you for the response.

I think I'll do that but with data right at the -90d limit to retain it before I lose access. But this won't solve my issue of accessing data that is already outside of my -90d restriction (i.e. -1y to -91d).

[gcusello]

Hi @michaeler,

this solution permits to bypass retentio issue, obviously it cannot solve access limitations for your role.

Ciao.

Giuseppe