Current ThreatQ Version Filter
 

Saved Searches

The Splunk App uses saved searches for discovering sightings and reporting them back to ThreatQ. The App is preconfigured with saved searches, which are periodic processes (registered to the crontab) designed to map indicators to specific Splunk indices and match these indicators to events. Saved search processes also move older indicators out of the main lookup tables and for ES customers, move indicators to specific ES lookup tables according to the mapping described in this document.

The table below describes some of the saved searches with which this App is preconfigured. This table displays two searches applicable only for Raw Matching Mode. Equivalent searches are available for each data model in the Datamodel Matching Mode.

ThreatQuotient does not recommend setting the frequency to less than 30 minutes, the application default for threatq_match_ indicator saved searches, if using the configuration option for creating multiple events for each sighted indicator.

Saved Search Description Default Period
threatq_consume_ indicators_new Post matched indicators to the consume endpoint of ThreatQ and create atomic events. This search will only be enabled if using the "Create multiple events for each sighted indicator" configuration. 30 minutes
threatq_match_ indicators (Raw Matching Mode only) Finds evidence of sightings for all indicators in the master lookup table. If sightings are detected, indicators are moved to the match lookup table. 30 minutes
threatq_match indicators Finds evidence of sightings for all indicators in the match lookup table. 30 minutes
threatq_update_ matched_indicators Finds evidence of sightings for all indicators in the match lookup table. 30 minutes
threatq_consume_ indicators Creates events in ThreatQ for all newly detected sightings. 15 minutes
threatq_update_ retired_indicators Clean up indicators that haven't been updated in the last 90 days from both master lookup table and match lookup table. 1,440 minutes
Edibility Rules: Because of the way sightings are found in Splunk using two saved searches (threatq_match_indicators and threatq_update_ matched_indicators), their frequency must be the same if edited. The default frequency for both saved searches is 30 minutes.

Saved Search Macros

The following table documents the macros for saved searches as configured by default on the ThreatQuotient App.

Saved Search

Default Macro
threatq_consume_indicators_new | inputlookup threatq_matched_indicators | eval start_time=relative_time(now(), "-35m") | where match_time > start_time | sort 10000 -num(score), -num(match_count) | threatqconsumeindicatorsnew
threatq_cleanup_indicators_
on_indicators_ change		 | inputlookup master_lookup | search NOT [search `threatq_index` sourcetype="threatq:indicators" | dedup value |
search [| inputlookup master_lookup | table ioc_value | rename ioc_value as value |
format] NOT (`threatq_score_filter` `threatq_status_filter`) | table value | rename value as ioc_value |
format] | outputlookup master_lookup | join ioc_value [| inputlookup threatq_matched_indicators |
table ioc_value, match_time, first_seen, last_seen, match_count, sid] | outputlookup threatq_matched_indicators
threatq_match_indicators (only Raw Matching Mode) `threatq_match_indices` `threatq_match_sourcetypes` sourcetype!="threatq:indicators" | threatqmatchiocs
threatq_update_matched_ indicators (only Raw Matching Mode) `threatq_match_indices` `threatq_match_sourcetypes` sourcetype!="threatq:indicators" | threatqmatchiocs is_update=true
threatq_consume_indicators | inputlookup threatq_matched_indicators | eval start_time=relative_time(now(), "-16m") | where last_seen > start_time | threatqconsumeindicators
threatq_update_retired_ indicators | inputlookup master_lookup | search NOT [| inputlookup master_lookup | search NOT [| inputlookup threatq_matched_indicators | search NOT [| inputlookup threatq_matched_indicators | eval threshold_time=now()-7776000, value=ioc_value | where last_seen < threshold_time | outputlookup key_field=value threatq_retired_matched_indicators | table ioc_value | format] | outputlookup threatq_matched_indicators | table ioc_value | format] | eval threshold_time=now()-7776000, updated_at_epoch=`threatq_parse_updated_at(updated_at)`, value=ioc_value | where updated_at_epoch < threshold_time | outputlookup key_field=value threatq_retired_indicators | table ioc_value | format] | outputlookup master_lookup

As described above, two of the saved searches are applicable only for the Raw Matching Mode. If you select Datamodel Matching Mode from the configuration as described in the Configuration section in the Installing the App Component topic, the above two saved searches for Raw Matching Mode will disable automatically, and the equivalent saved searches for the Datamodel Matching Mode will be enabled.

Separation of Data

ThreatQ indicator data is separated from the rest of the data in this App using a specific sourcetype. You can use the following Splunk search query to discover all indicators exported from ThreatQuotient.

Splunk Search for Listing TQ Indicators

sourcetype=”threatq:indicators”

The same indicator can be exported multiple times if it experienced a change of status and/or score.

Chunking

You can apply chunking to your datamodel searches using the following option:

chunk_size=<value>

Default chunk size is 50,000.

Example

Chunking - Edit Search Example