Integration Details

ThreatQuotient provides the following details for this integration:

Introduction

The MISP threat sharing platform is free and open source software that enables sharing of threat intelligence represented in the MISP data model format. The MISP Import integration ingests published MISP events from a user-provided, self-hosted MISP server instance via its POST {{user_fields.domain_name}}/events/restSearch endpoint (note: {{user_fields.domain_name}} must contain the protocol, like https://).

The integration will fetch MIPS events and return the following system objects:

• Events
• Indicators
• Attachments
• Signatures
• Adversaries
• Attack Patterns
• Course of actions
• Malware
• Tools

Prerequisites

If the user intends to ingest MISP events that are related to any MITRE MISP galaxies, make sure the following feeds successfully run prior to running the MISP Import feed:

• MITRE Enterprise ATT&CK
• MITRE Mobile ATT&CK
• MITRE PRE-ATT&CK

Installation

Perform the following steps to install the integration:

The same steps can be used to upgrade the integration to a new version.

1. Log into https://marketplace.threatq.com/.
2. Locate and download the integration file.
3. Navigate to the integrations management page on your ThreatQ instance.
4. Click on the Add New Integration button.
5. Upload the integration file using one of the following methods:
   • Drag and drop the file into the dialog box
   • Select Click to Browse to locate the integration file on your local machine

   ThreatQ will inform you if the feed already exists on the platform and will require user confirmation before proceeding. ThreatQ will also inform you if the new version of the feed contains changes to the user configuration. The new user configurations will overwrite the existing ones for the feed and will require user confirmation before proceeding.

You will still need to configure and then enable the feed.

Configuration

ThreatQuotient does not issue API keys for third-party vendors. Contact the specific vendor to obtain API keys and other integration-related credentials.

To configure the integration:

1. Navigate to your integrations management page in ThreatQ.
2. Select the OSINT option from the Category dropdown (optional).

   If you are installing the integration for the first time, it will be located under the Disabled tab.

3. Click on the integration entry to open its details page.
4. Enter the following parameters under the Configuration tab:

   Parameter	Description
   MISP Domain Name	The MISP server instance domain name (or IP address) preceded by the protocol it uses, such as https://my-misp-server.org. The provided domain name or IP address must be reachable from the ThreatQ instance.
   Disable Proxies	If enabled, specifies that this feed should not honor any proxies setup in ThreatQuotient. The default setting is disabled.
   Enable SSL Verification	If enabled, specifies that this feed should verify SSL connections with the provider. The default setting is enabled.
   Rate Limit Delay	The number of seconds to wait between API requests. This is useful if you are hitting rate limits on the MISP API. Rate limits are configurable on the MISP server, and can vary. Increase this value if you are running into 429 errors.
   API Key	The MISP account API key.
   MISP Client Certificate	Enter your MISP Client Certificate if the MISP server requires certificate-based authentication.  Otherwise, leave this field blank.
   MISP Client Private Key	Enter your MISP Client Private Key if the MISP server requires certificate-based authentication.  Otherwise, leave this field blank.
   Query Filter Characteristics	Set which parameters that decide which events to retrieve at run-time.  Options include: Modified Events -  ingest contents of any modified event since the last run. New Events - ingest events having a newer date than configured start date together with their full context.
   Threat Level Filter	Select the Threat Levels to filter events.  Option include: High (default) Medium (default) Low (default) Undefined (default)
   Organization Filter	Optional - enter a line-separated list of organization names to filter events by. If left blank, all events will be imported. If populated, only events from the specified organizations will be imported. If both Organization and Community filters are populated, events that match either filter will be imported.
   Community Filter	Optional - enter a line-separated list of community names to filter events by. If left blank, all events will be imported. If populated, only events from the specified communities will be imported. If both Organization and Community filters are populated, events that match either filter will be imported.
   Only Import Published Events	Enable this option to only import events that are marked as published in MISP.
   Inherited Context	Select the pieces of context to inherit from the event to the related IOCs.  Options include: Related Malware Related Adversaries Source Organization Member Organization ID MISP Threat Level Analysis Tags
   Save Tags As	Select how to add Tags to the ThreatQ platform.  Options include Tags and Attributes.

   
   
5. Review any additional settings, make any changes if needed, and click on Save.
6. Click on the toggle switch, located above the Additional Information section, to enable it.

ThreatQ Mapping

{
   "response": [
      {
         "Event": {
            "id": "1",
            "orgc_id": "1",
            "org_id": "1",
            "date": "2018-12-14",
            "threat_level_id": "2",
            "info": "EVENT1",
            "published": false,
            "uuid": "5c142f52-5ad0-4c04-8069-03c8ac107221",
            "attribute_count": "4",
            "analysis": "1",
            "timestamp": "1545256410",
            "distribution": "1",
            "proposal_email_lock": false,
            "locked": false,
            "publish_timestamp": "1544827221",
            "sharing_group_id": "0",
            "disable_correlation": false,
            "extends_uuid": "",
            "event_creator_email": "admin@admin.test",
            "Org": {
               "id": "1",
               "name": "ORGNAME",
               "uuid": "5bd7a775-1d18-4fd7-b2f4-08b52dc69e54"
            },
            "Orgc": {
               "id": "1",
               "name": "ORGNAME",
               "uuid": "5bd7a775-1d18-4fd7-b2f4-08b52dc69e54"
            },
            "Attribute": [
                {
                  "id": "1",
                  "type": "link",
                  "category": "Antivirus detection",
                  "to_ids": false,
                  "uuid": "5c17ccfe-3c1c-4f47-9a9f-38f6ac107221",
                  "event_id": "1",
                  "distribution": "3",
                  "timestamp": "1545063678",
                  "comment": "",
                  "sharing_group_id": "0",
                  "deleted": false,
                  "disable_correlation": false,
                  "object_id": "0",
                  "object_relation": null,
                  "value": "https:\/\/www.virustotal.com\/#\/file\/17a0d59255046ed2cff22cd5980fcc86c69e059839fec07d705051ac2e178693\/details",
                  "Galaxy": [],
                  "ShadowAttribute": []
               },
               {
                  "id": "1259319",
                  "type": "filename|md5",
                  "category": "Payload installation",
                  "to_ids": false,
                  "uuid": "5ffc9a4f-7ef0-4077-b278-30a5ac107221",
                  "event_id": "104",
                  "distribution": "5",
                  "timestamp": "1610390095",
                  "comment": "",
                  "sharing_group_id": "0",
                  "deleted": false,
                  "disable_correlation": false,
                  "object_id": "0",
                  "object_relation": null,
                  "value": "bunnyhop.exe|31f3720bef6bb3e2953d9ea2238e0580",
                  "Galaxy": [],
                  "ShadowAttribute": []
               },
               {
                  "id": "477506",
                  "type": "attachment",
                  "category": "Payload delivery",
                  "to_ids": false,
                  "uuid": "5dde5554-6320-4647-baa8-26d3ac107221",
                  "event_id": "75",
                  "distribution": "5",
                  "timestamp": "1574851924",
                  "comment": "sample.pdf",
                  "sharing_group_id": "0",
                  "deleted": false,
                  "disable_correlation": false,
                  "object_id": "0",
                  "object_relation": null,
                  "value": "sample.pdf",
                  "Galaxy": [],
                  "data": "JVBERi0xLjMNCiXi48TDQoNCjEgMCBvYmo8DQovVHlwZS..."
               },
               {
                   "id": "1",
                   "type": "comment",
                   "category": "Payload delivery",
                   "to_ids": false,
                   "uuid": "5e81aec6-5af0-498c-9826-7a63ac107122",
                   "event_id": "1",
                   "distribution": "5",
                   "timestamp": "1585562438",
                   "comment": "not applicable",
                   "sharing_group_id": "0",
                   "deleted": false,
                   "disable_correlation": false,
                   "object_id": "0",
                   "object_relation": null,
                   "first_seen": null,
                   "last_seen": null,
                   "value": "sample comment",
                   "Galaxy": [],
                   "ShadowAttribute": []
               },
               {
                   "id": "3",
                   "type": "snort",
                   "category": "Network activity",
                   "to_ids": false,
                   "uuid": "5e81c3d7-d310-4344-bfe9-7805ac107122",
                   "event_id": "1",
                   "distribution": "5",
                   "timestamp": "1585562583",
                   "comment": "",
                   "sharing_group_id": "0",
                   "deleted": false,
                   "disable_correlation": false,
                   "object_id": "0",
                   "object_relation": null,
                   "first_seen": null,
                   "last_seen": null,
                   "value": "alert tcp $HOME_NET any -> any 3306 (msg: \"mysql general_log write file\"; ...)"
               }
            ],
            "Object": [
               {
                  "id": "1",
                  "name": "file",
                  "meta-category": "file",
                  "description": "File object describing a file with meta-information",
                  "template_uuid": "688c46fb-5edb-40a3-8273-1af7923e2215",
                  "template_version": "15",
                  "event_id": "1",
                  "uuid": "5c1abdda-4cb8-427c-97d5-71c9ac107221",
                  "timestamp": "1545256410",
                  "distribution": "5",
                  "sharing_group_id": "0",
                  "comment": "dnsrslvr.dll",
                  "deleted": false,
                  "ObjectReference": [],
                  "Attribute": [
                     {
                        "id": "26131",
                        "type": "md5",
                        "category": "Payload delivery",
                        "to_ids": true,
                        "uuid": "5c1abdda-0960-4530-a4e4-71c9ac107221",
                        "event_id": "1",
                        "distribution": "5",
                        "timestamp": "1545256410",
                        "comment": "",
                        "sharing_group_id": "0",
                        "deleted": false,
                        "disable_correlation": false,
                        "object_id": "1",
                        "object_relation": "md5",
                        "value": "44d88612fea8a8f36de82e1278abb02f"
                     }
                  ]
               },
               {
                   "id": "1",
                   "name": "yara",
                   "meta-category": "misc",
                   "description": "An object describing a YARA rule (or a YARA rule name) along with its version.",
                   "template_uuid": "b5acf82e-ecca-4868-82fe-9dbdf4d808c3",
                   "template_version": "4",
                   "event_id": "1",
                   "uuid": "5e81cab1-5f2c-4350-8ed0-7b28ac107122",
                   "timestamp": "1585564479",
                   "distribution": "5",
                   "sharing_group_id": "0",
                   "comment": "",
                   "deleted": false,
                   "first_seen": null,
                   "last_seen": null,
                   "ObjectReference": [],
                   "Attribute": [
                      {
                         "id": "4",
                         "type": "yara",
                         "category": "Payload installation",
                         "to_ids": true,
                         "uuid": "5e81cab1-58d4-4155-a691-7b28ac107122",
                         "event_id": "1",
                         "distribution": "5",
                         "timestamp": "1585564470",
                         "comment": "",
                         "sharing_group_id": "0",
                         "deleted": false,
                         "disable_correlation": false,
                         "object_id": "1",
                         "object_relation": "yara",
                         "first_seen": null,
                         "last_seen": null,
                         "value": "rule Contains_VBA_macro_code\r\n{\r\n\tmeta:\r\n\t\tauthor = ..."
                      }
                   ]
               }
            ],
            "Galaxy": [
               {
                  "id": "3",
                  "uuid": "698774c7-8022-42c4-917f-8d6e4f06ada3",
                  "name": "Threat Actor",
                  "type": "threat-actor",
                  "description": "Threat actors are characteristics of malicious actors (or adversaries) representing a cyber attack threat including presumed intent and historically observed behaviour.",
                  "version": "3",
                  "icon": "user-secret",
                  "namespace": "misp",
                  "GalaxyCluster": [
                     {
                        "id": "5401",
                        "collection_uuid": "7cdff317-a673-4474-84ec-4f1754947823",
                        "type": "threat-actor",
                        "value": "Keyhole Panda",
                        "tag_name": "misp-galaxy: threat-actor=\"Keyhole Panda\"",
                        "description": "no description",
                        "galaxy_id": "3",
                        "source": "MISP Project",
                        "authors": [
                           "Alexandre Dulaunoy",
                           "Florian Roth",
                           "Thomas Schreck",
                           "Timo Steffens",
                           "Various"
                        ],
                        "version": "75",
                        "uuid": "ad022538-b457-4839-8ebd-3fdcc807a820",
                        "tag_id": "77",
                        "meta": {
                           "country": [
                              "CN"
                           ],
                           "synonyms": [
                              "temp.bottle"
                           ]
                        }
                     }
                  ]
               },
               {
                  "id": "4",
                  "uuid": "1fb6d5b4-1708-11e8-9836-8bbc8ce6866e",
                  "name": "Pre Attack - Intrusion Set",
                  "type": "mitre-pre-attack-intrusion-set",
                  "description": "Name of ATT&CK Group",
                  "version": "4",
                  "icon": "user-secret",
                  "namespace": "mitre-attack",
                  "GalaxyCluster": [
                     {
                        "id": "5614",
                        "collection_uuid": "1fdc8fa2-1708-11e8-99a3-67b4efc13c4f",
                        "type": "mitre-pre-attack-intrusion-set",
                        "value": "APT16 - G0023",
                        "tag_name": "misp-galaxy:mitre-pre-attack-intrusion-set=\"APT16 - G0023\"",
                        "description": "APT16 is a China-based threat group that has launched spearphishing campaigns targeting Japanese and Taiwanese organizations. (Citation: FireEye EPS Awakens Part 2)",
                        "galaxy_id": "4",
                        "source": "https:\/\/github.com\/mitre\/cti",
                        "authors": [
                           "MITRE"
                        ],
                        "version": "6",
                        "uuid": "d6e88e18-81e8-4709-82d8-973095da1e70",
                        "tag_id": "97",
                        "meta": {
                           "external_id": [
                              "G0023"
                           ],
                           "refs": [
                              "https:\/\/attack.mitre.org\/wiki\/Group\/G0023",
                              "https:\/\/www.fireeye.com\/blog\/threat-research\/2015\/12\/the-eps-awakens-part-two.html"
                           ],
                           "synonyms": [
                              "APT16"
                           ]
                        }
                     }
                  ]
               }
            ],
            "Tag": [
               {
                  "id": "11",
                  "name": "tlp:red",
                  "colour": "#CC0033",
                  "exportable": true,
                  "user_id": "0",
                  "hide_tag": false,
                  "numerical_value": null
               },
               {
                  "id": "394",
                  "name": "ms-caro-malware:malware-type=\"DDoS\"",
                  "colour": "#32003e",
                  "exportable": true,
                  "user_id": "0",
                  "hide_tag": false,
                  "numerical_value": null
               }
            ]
         }
      }
   ]
}

Average Feed Run

MISP server instances vary widely in their setup and the data stored within them. Due to this, average feed run results cannot be confidently provided.

Known Issues / Limitations

• Authentication - users should not mix authentication methods when configuring the integration.  The MISP Client Certificate and MISP Client Private Key configuration fields should only be used if required by the MISP server.    
• MISP does not verify whether a Snort or YARA rule entered into it is valid. However, the Snort and YARA parsers used by this feed depend on the Snort or YARA rules being well-formed. Please refer to the following non-comprehensive list to aid in making sure that the Snort or YARA rules stored in your MISP server instance are valid so that they can be properly ingested by this feed.
• Snort:
  • Each rule option must be terminated with a semicolon (;).
  • Offending Snort rule: alert tcp $HOME_NET any -> $EXTERNAL_NET [80,443,8080,7080,21,50000,995](msg:"BDS MALICIOUS Emotet Worming Traffic Likely";content:"d29ybSBzdGFydGVk";content:"POST";http_method;classtype:spreader;sid:7;rev:1)
    • Correction needed: rev:1 should be rev:1;
  • Rule option values must be valid. For instance, options like rev, sid, and gid must have base 10 integers as their value.
  • Offending Snort option: sid:#########;
    • Correction needed: Either remove sid if the value is not known or replace the value with a valid ID, like sid:7;.
  • The value of the snort MISP attribute must contain at least one entire Snort rule. Multiple Snort rules can be provided in a single value if separated by newlines. The value must not contain any excess text, such as a header like Snort rule:.
• YARA:
  • The value of the yara MISP attribute must contain at least one entire YARA rule. Multiple YARA rules can be provided in a single value if separated by newlines. The value must not contain any excess text, such as a header like YARA rule:.
  • Make sure that any text that is not valid YARA syntax is either removed or commented out.

Change Log

• Version 2.4.1
  • Resolved a Threat Level ingestion issue caused by a provider API update in which this data was returned as an integer opposed to a string.    
• Version 2.4.0
  • Added new configuration parameters:
    • Organization Filter - filter out events based on the organization that published it.
    • Community Filter - filter out events based on the community that published it.
    • Only Import Published Events - filter out events based on whether or not they are published.
    • Threat Level Filter - filter out events based on their threat level.
  • Reorganized user field interface order to improve consistency. 
• Version 2.3.2
  • Added the ability to select how Tags are ingested via a new configuration parameter: Save Tags As.
  • Added two new certificate configuration parameters:
    • MISP Client Certificate
    • MISP Client Private Key
  • Added events pagination improvements.
  • Added new Known Issue / Limitation entry in the user guide regarding authentication - users should only use the MISP Client Certificate and Private Key fields if required by the MISP server.  
• Version 2.3.1
  • Resolved an issue where uploading attachments with empty attributes would cause an error.  
  • Updated minimum ThreatQ version to 5.20.0
• Version 2.3.0
  • Added the ability to inherit meaningful event attributes and related indicators.  See the Inherited Context parameter in the Configuration chapter for more details.   
  • Fixed parsing of YARA and Snort rules that contain unicode special space separators.  
  • Removed inter-relation of domains from the domain-ip object types. 
  • Removed the Save Intrusion Sets As parameter.
  • Updated MISP Galaxy Cluster Type to ThreatQ Object Type Mapping table.
  • Updated minimum ThreatQ version to 5.19.0
• Version 2.2.0
  • Added new configuration option, Inherited Context, to inherit context from top-level events to related IOCs.
  • Added new configuration option, Rate Limit Delay, to set a delay to prevent rate limiting (429 errors).  
• Version 2.1.5
  • Updated the python implementation of the MISP filter to improve feed efficiency.  
  • Updated the minimum ThreatQ version to 4.50.0.
• Version 2.1.4
  • Removed the Use the Event Data as Query Parameter UI configuration parameter.  
  • Added new UI configuration parameter: Query Filter Characteristics.  Time constrained data fetching for both schedule and manual runs are now user configurable via this new field.
• Version 2.1.3
  • Added new configuration parameter - Use Event Date as Query Parameter.
  • Performed updates on time constrained data fetching for both scheduled and manual runs.  Scheduled Runs retrieve MISP events and event attributes from events that have received a modification using the start date only.  Manual Runs can retrieve MISP events and attributes using the start date only or the start and end dates depending on configuration options.  See the Use the Event Date as Query Parameter option in the Configuration chapter of this guide.  
• Version 2.1.2
  • Updated the timing query parameters used for pulling data.  
• Version 2.1.1
  • Added the ability to Normalize more Unicode quotations and hyphens
  • Add the ability to parse the YARA Rule Name ThreatQ Event Attribute from MISP Object Attributes that contain the key-value pair
    object_relation: "yara-rule-name" in addition to the parsing logic added in 2.1.0, in which the YARA Rule Name
    ThreatQ Attribute is parsed from MISP Object Attributes that contain the key-value pair type: "yara-rule-name"
• Version 2.1.0
  • Normalized Unicode double quotations to ASCII double quotations in YARA and Snort signature values
  • Normalized Unicode hyphens to ASCII hyphens in YARA and Snort signature values
  • Adjusted logic used to derive a YARA rule from MISP Object Attributes. Previously, the logic depended on a MISP YARA Object having both yara and yara-rule-name attributes, in which it derived a YARA rule based on the template rule {{yara_rule_name}} {{yara}} before passing the derived rule through the YARA parser. This was not often seen in the wild; the MISP yara attribute usually contains the entire YARA rule. Even the documentation for the MISP YARA Object suggests that these attributes be used in a mutually exclusive manner: use the yara attribute if the user has the entire YARA rule to provide to MISP; else, reference a YARA rule using the yara-rule-name attribute. As a result, the following changes are in this version of the CDF:
    • Ingest the yara-rule-name MISP Object Attribute as a ThreatQ Attribute of the MISP Event object
    • Ingest the yara MISP Object Attribute as a ThreatQ YARA Signature
  • Parse MISP Suricata Objects in order to ingest its Snort Object Attributes as ThreatQ Snort Signatures.
  • Although well-formed Snort rules should contain a msg option, which is used to derive the ThreatQ Snort Signature's Name, account for a Snort rule not containing a msg option by defaulting the ThreatQ Snort Signature's Name to "Snort Rule".
• Version 2.0.0
  • Add configuration parameter "Enable SSL Verification"
  • Change behavior of scheduled feed runs such that it only fetches data published within the last day
  • Add support for parsing MISP YARA Objects
  • Add support for parsing MISP Snort Attributes
  • Add additional key-value pairs to the MISP Galaxy Cluster Type to ThreatQ Object Type Mapping
  • Change the value of happened_at for ingested events by using date instead of timestamp
• Version 1.0.4
  • Ingest MISP Attachment Attributes as ThreatQ Attachments
• Version 1.0.3
  • Add configuration parameter "Disable Proxies"
• Version 1.0.0
  • Initial release