Indicator URL Normalization
Remove Quotes from the Beginning and/or End of an Indicator
Single and double quote characters are removed if they are the first or last character of an indicator.
Remove Unneeded Spaces found within an Indicator
All spaces irrelevant of their position in the Indicator value are removed (when applicable).
Adjust Leading Protocol from Indicators
Indicators with a leading protocol [http://, https://, ftp://, or ftps://] are extracted and included as an attribute. When applicable, this indicator adjustment could change the indicator type from URL to FQDN.
Original URL indicator of http://evilsubdomain.no-ip.biz/ would convert to a FQDN = evildomain.no-ip.biz.
Adjust the Port from an IP Address
An IP address with a port [ex. 199.7.136.88:8143] will be truncated to the IP address and the port assignment will be added as an attribute.
Using the previous example the following indicator/attribute will be created:
| Field | Value |
|---|---|
| URL | 199.7.136.88 |
| Attribute > Port | 8143 |
Adjust Defanged/Neutered Indicators
Indicators that have been defanged/neutered in order to “safely” share them (i.e. www [dot] 3322 [dot] org or badguy [at] gmail.com) need to be adjusted during import in order to ensure the indicators are properly deployed.
Create an IP Address from a URL (when applicable)
Using the previous example the following indicators will be created:
| Field | Value |
|---|---|
| URL | 51.255.131.66/civis/viewforum.php |
| IP Address | 51.255.131.66 |
Create a FQDN from a URL (when applicable)
When a URL contains a domain [ex. bat99-11611.co/gate777.php] a second indicator will be created for the domain [bat99-11611.co].
Using the previous example, the following indicators will be created:
| Field | Value |
|---|---|
| URL | bat99-11611.co/gate777.php |
| FQDN | bat99-11611.co |
Query Strings in a URL Indicator
URL query strings are important but can significantly limit pattern-matching detection capabilities due to the likelihood of deviations, as well as, hamper the volume of URL indicators being deployed.
As such, the normalization process discards query strings.
Maintain “WWW” on FQDN Indicators
When parsing or importing an FQDN the “www” will be maintained.
Replace and/or Remove Special Characters
| Character | Replacement |
|---|---|
|
ASCII Values < 32 |
<space> |
| Ascii 96 | - |
| Ascii145 | ' |
| Ascii146 | ' |
| Ascii147 | " |
| Ascii148 | " |
| Ascii151 | - |
| carriage return and line feed | <space> |
| Control Characters | Remove |
| Convert to UTF8 | Remove leading and trailing space, tab, newline, carriage return, vertical tabs and null characters. |