Scaling the App
The primary objective of this App is to find evidence of sightings and report those sightings back to ThreatQ. The sightings are discovered using the matching algorithm that works either in the Raw Matching or Datamodel Matching mode, which will take a set of indicators from ThreatQ, a set of events from Splunk, and find which indicators (and how many times) appear in the events.
The best way to scale the App is to run multiple saved searches for matching. This section details performance testing results that supports this recommendation and how to implement this scaling based on your matching preferences.
Performance Testing
Performance testing was conducted to determine how many Splunk Events the application can handle with the default configuration.
The matching algorithm, by default, runs every 30 minutes in a saved search, so it is important that it completes in under 30 minutes on average just to keep up with incoming load.
Datamodel Matching Mode
Test Environment: Dedicated Box with 16 cores and 32GB of ram.
Test Summary Result: The limit is 15 million events per 30 minutes per single saved search.
Recommendation: Users should utilize multiple searches using the datamodel matching mode. If your data is mapped to multiple Splunk data models from the CIM Support table, each data model is handled by a separate saved search. In such an instance, you would need to deploy your search head in a cluster and ensure that these saved searches are distributed in that cluster. You can run up to five of them, thus potentially scaling your App to handle five times the traffic.
Test Details: The table provided below illustrates that testing determined that the algorithm runtime started to exceed the 30 minute mark with 15 million Splunk events. Thus, for a single saved search, this represents the upper limit of how much data this algorithm can handle every 30 minutes.
| Total Indicators from TQ | Total Raw Events in Splunk | Total Indicators Matched | Time to Complete (s) Machine Specs:(16 Core, 32GB RAM ) |
|---|---|---|---|
| 100,000 | 500,000 | 0 | 29.282 |
| 100,000 | 500,000 | 10,000 | 36.92 |
| 100,000 | 1,000,000 | 20,000 | 77.649 |
| 500,000 | 1,000,000 | 0 | 99.473 |
| 500,000 | 1,000,000 | 10,000 | 130.991 |
| 1,000,000 | 1,000,000 | 0 | 166.517 |
| 1,000,000 | 1,000,000 | 25,000 | 261.362 |
| 1,000,000 | 5,000,000 | 0 | 420.111 |
| 100,000 | 5,000,000 | 10,000 | 619.047 |
| 1,000,000 | 10,000,000 | 10,000 | 1,316.541 |
| 1,000,000 | 15,000,000 | 10,000 | 1,866.059 |
| 1,000,000 | 50,000,000 | 25,000 | 6,554.610 |
Raw Matching Mode
Test Environment: Dedicated Box with 16 cores and 32GB of ram.
Test Summary Result: The limit is 1 million events per 30 minutes per separate saved search.
Recommendation: For the raw matching mode, the App by default will only be able to run one saved search. In order to extend it to multiple searches, you will have to break apart this one saved search into multiple, and then, distribute these saved searches in the Splunk cluster of search heads. You can do this by running a separate saved search for:
- Splunk index for events
- ThreatQuotient indicator types
For using a fixed Splunk index for the saved search, you can modify the default saved searches for matching as shown below.
Splunk Search for Listing TQ Indicators
index=<my_index> `threatq_match_sourcetypes` source- type!="threatq:indicators" | threatqmatchiocs indicator_types- ='IP Address, FQDN'(threatq_match_indicators saved search) index=<my_index> `threatq_match_sourcetypes` source- type!="threatq:indicators" | threatqmatchiocs is_update=true (threatq_update_matched_indicators saved search)
Compare the above saved searches with the defaults as shown in the Save Search Documentation table. The macro threatq_match_indices is replaced by passing an actual index to the saved search. Now, you can make multiple copies of the default saved search, run them on the same schedule, and have each saved search get events from a different Splunk index.
To use the similar technique for ThreatQuotient indicator types, you can pass an additional argument to the threatqmatchiocs module as shown below. This allows you to make the saved search use only a specific indicator type. Again, as before, you can then make multiple copies of the saved searches and have each one handle only specific ThreatQuotient indicator types. You are free to pass a single indicator type, or a comma separated list as shown below.
Splunk Search for Listing TQ Indicators
index=<my_index> `threatq_match_sourcetypes` source- type!="threatq:indicators" | threatqmatchiocs indicator_types- ='IP Address, FQDN'(threatq_match_indicators saved search) index=<my_index> `threatq_match_sourcetypes` source- type!="threatq:indicators" | threatqmatchiocs is_update=true indicator_types='IP Address, FQDN'(threatq_update_matched_ indicators saved search)
Test Details: The table provided below illustrates that testing determined that the algorithm runtime started to exceed the 30 minute mark with 1 million Splunk events. Thus, for a single saved search, this represents the upper limit of how much data this algorithm can handle every 30 minutes.
| Total Indicators from TQ | Total Raw Events in Splunk | Total Indicators Matched | Time to Complete (s) Machine Specs:(16 Core, 32GB RAM ) |
|---|---|---|---|
| 100,000 | 500,000 | 0 | 885.36 |
| 100,000 | 500,000 | 10,000 | 899.75 |
| 100,000 | 1,000,000 | 20,000 | 1,932.04 |
| 500,000 | 1,000,000 | 0 | 1,926.62 |
| 500,000 | 1,000,000 | 10,000 | 2,020.56 |
| 1,000,000 | 1,000,000 | 0 | 2,174.18 |
| 1,000,000 | 1,000,000 | 25,000 | 2,294.39 |
| 1,000,000 | 5,000,000 | 0 | 11,354.64 |
| 10,000 | 50,000,000 | 0 | 35,233.185 (9 hr 47 min) |