Relationship Limits
Please contact ThreatQuotient Support, if you have questions or concerns about the impact of relationship limits on your ThreatQ instance.
ThreatQ provides a relationship limit system to assist you in managing the object relationship volume that ThreatQ ingests.
For a smaller subset of ThreatQ objects, some feed providers may include a very large number of relationships in their feeds. In some cases, these large relationship counts may have performance impacts on Threat Library queries and other ThreatQ capabilities. This feature allows you to balance the impact of excluding data that may be of limited intelligence value with the benefits of improved system performance.
Functionally, relationship limits prevent a source from adding more than a configured number of relationships to an object via an integration, bulk update, manual at the object level, import, or parser. The default limit setting is 10,000 relationships, and generally applies to a small percentage of the overall objects in most ThreatQ systems.
Example: Exceeding the Relationship Limit
If you ingest the UNC5174 adversary object from Malpedia and the feed data for this object includes 10,001 related objects, the source (Malpedia) has exceeded the 10,000 related object limit for the system object (UNC517). As a result, ThreatQ will prevent the ingestion of any additional relationships from Malpedia for UNC5174 and flag the object and integration as follows:
In this example, Malpedia has only exceeded the relationship limit for UNC5174. As such, ThreatQ will no longer ingest data from Malpedia for UNC5174 but will continue to ingest data for other system objects.
- The UNC5174 object details page and preview panel display a Relationship Limit Reached banner.
- The Sources pane in the UNC5174 object details page and object preview pane displays a relationship limit badge
with a tooltip listing the source(s) that exceeded the limit. Within the tooltip, you can click the offending source(s) to access a Threat Library view filtered by the Related To (object name) and Relationship Criteria (offending source).
If the offending source is not a feed, the Threat Library view does not list related objects. For example, if the offending source is a user (such as threatq@threatq.com) that added the relationships, when you click it, the Threat Library view does not list any related objects.
- The new Flagged tab within the My Integrations tab displays a flagged integration banner.
- The Malapedia integration card displays a relationship limit badge .
- The Malpedia configuration page displays an Integration Flagged banner as well as a relationship limit badge above the integration logo.
Any banner that displays an X in the upper right corner can be closed so that it does not display for the user in the current or future sessions. In addition, the above banners and badges are automatically removed when you resolve the relationship limit issue.
You can use the Flagged Objects filter in the Threat Library to review a list of objects that have exceeded the relationship limit.
What to Do When an Object Exceeds the Relationship Limit
Once an object and source have been flagged as reaching the relationship limit, ThreatQ prevents the ingestion/addition of additional related objects on that object from the feed. You can use the following methods to reduce the number of object relationships:
- Object details page or object preview pane - From the object details page/object preview panel, use the Unlink option to unlink related objects from the offending source.
- Bulk action - Filter your Threat Library view using the Flagged Objects filter. Then, click the Bulk Actions button and select the Bulk Delete option to delete the objects.
- Data retention policy - Use the Flagged Objects filter to create a data collection you can add to your data retention policy.
Once the relationship limit is no longer exceeded, the offending objects and feed no longer display relationship limit banners or badges.
Tips and Tricks
- Relationship limits are applied based on object and source. For example, an object can have 10,000 related objects from Malpedia and 10,000 related objects from NVD. Although the total number of related objects exceeds 10,000, the total per source does not so the object/source is not flagged as exceeding the relationship limit.
- If a source exceeds the relationship limit for one object, ThreatQ halts the ingestion of data for that specific object but continues to ingest data from this source for other objects.
- Since object relationships from the same source can be added by multiple processes, you may encounter situations in which an object’s relationship count exceeds 10,000.
For example, if your instance ingests 9,999 object relationships from a feed in the first run and then it ingests an equal number of additional relationships in the second run, ThreatQ may allow the object relationships to exceed 10,000 due to a slight delay in incrementing the relationship count. However, once the relationship count is updated, the object and source are flagged and you can no longer add relationships to the flagged object from the flagged source.