Current ThreatQ Version Filter

Export Data

AGDS exports can include the following data:

Data Details
Threat Intelligence Objects ThreatQ default object types include:
  • Adversaries
  • Assets
  • Attachments (Files)

    Physical files for all attachments included in the date range
    are copied into the attachments/files directory of the data tarball.

  • Attack Patterns
  • Campaigns
  • Courses of Actions
  • Events
  • Exploit Targets
  • Identity
  • Incidents
  • Indicators
  • Intrusion Set
  • Malware
  • Notes
  • Report
  • Signatures
  • Tasks
  • Tool
  • TTPs
  • Vulnerability

Custom object schemas must already exist on the target system. If an object type is included in the export data, but is not found on the target, it will be ignored.
Object Context Tables covered for each object type include:
  • <object type>_attributes
  • <object type>_attribute_sources
  • <object type>_comments
  • <object type>_sources

Not all Objects have all Object Contexts (Attributes, Attribute Sources, Comments, and Sources). Tables are only polled if they exist.
Object relationships and links

Tables Covered (Object Links and Object Link Context):

  • object_links
  • object_link_attributes
  • object_link_attribute_sources
  • object_link_comments
  • object_link_sources

Sample Object Link File List:

  • object_links/object_links_0.csv
  • object_links/object_link_attributes_0.csv
  • object_links/object_link_attribute_sources_0.csv
  • object_links/object_link_comments_0.csv
  • object_links/object_link_sources_0.csv
Metadata Metadata included in the export:
  • Attributes
  • Clients
  • Connectors
  • Connector Categories
  • Connector Definitions
  • Content Types
  • Groups
  • Investigation Priorities
  • <Object Type> Statuses
  • <Object Type> Types
  • Other Sources
  • Operations
  • Sources
  • Tags
  • TLP
  • Users
Investigations and Tags Optional and can be included in the export via the addition of a flag parameter.

When Investigations are imported, ownership is automatically assigned to the most recently created admin or super user on the target system.

Excluded or Limited Data

The following data is excluded or limited in ADGS exports:

  • Dashboards and Data Collections are not transferred.
  • Connectors and Operations are not installed on the target system (metadata only).
  • Custom object schemas must already exist on the target system.
  • Permissions and roles are not transferred.

Storage and Other Data Information

The follow provides additional details on object handling, transfer, and storage.  

Storage (Exports)

The data for each object is copied as a dump file in CSV format using "SELECT * INTO OUTFILE..." MariaDB syntax. The full query for the data is built up using the options you provided (start date, end date, etc).

Dump files contain a maximum object limit of 50,000 (set in the Synchronization base class). Dump files are created (with a counter appended to the file name) until the entire object result has been covered.

To ensure that any Objects present in Object Context (Attributes, Comments, and Sources), Object Links, Tagged Objects, or Investigation Timeline Objects are also included in the base Object data, CSV dump files for each Object type are also created from queries against each of these tables. This is necessary because of the differing date columns used in each query (an object may appear in an Object Link in the specified date range according to the Object Link's updated_at date, even though the Objects themselves saw no change to their touched_at date in that date range). When the data from all of these object files is transferred to the target ThreatQ installation, any duplicates across dump files will be consolidated. Files that contain Object data will always include "_obj_" in the file title.

Sample Object File List (all of these files will contain Adversary records):

  • adversaries/adversaries_obj_0.csv
  • adversaries/adversaries_obj_attributes_0.csv
  • adversaries/adversaries_obj_comments_0.csv
  • adversaries/adversaries_obj_investigation_timelines_0.csv
  • adversaries/adversaries_obj_object_links_dest_0.csv
  • adversaries/adversaries_obj_object_links_src_0.csv
  • adversaries/adversaries_obj_sources_0.csv
  • adversaries/adversaries_obj_tags_0.csv

Tags

The date range for queries on Tagged Objects uses the updated_at date column.

Tables Covered (Tags themselves are covered in the Meta Data):

  • tagged_objects
  • Sample Tagged Objects File List:
  • tagged_objects/tagged_objects_0.csv

Spearphish

The date range for queries on Spearphish uses the updated_at date column.

Tables Covered:

  • spearphish
  • Sample Spearphish File List (Spearphish files are stored with Event data):
  • events/spearphish_0.csv

ThreatQ Investigations

The date range for queries on additional Investigation context tables uses the updated_at column.

Tables Covered:

  • investigation_nodes
  • investigation_node_properties
  • investigation_timelines
  • investigation_timeline_objects
  • investigation_viewpoints

Sample Investigation additional context File List:

  • investigations/investigation_node_properties_0.csv
  • investigations/investigation_nodes_0.csv
  • investigations/investigation_timeline_objects_0.csv
  • investigations/investigation_timelines_0.csv
  • investigations/investigation_viewpoints_0.csv