¶ Where To Find Advanced Search?
Any SOC.OS search bar supports advanced search using the query language. Press the Advanced button to the right of the query to convert a basic query mode into an advanced query. It is then possible to edit the raw text of the query.
A basic 'chip' style query can be converted to an advanced query. Not all advanced queries can be displayed in the basic mode. Keep this in mind that if make changes to a query in advanced mode, it might not be able to convert back to a basic query.
¶ The Query Language
The query language uses a Lucene like syntax. A query is a collection of field expressions joined with operators.
¶ Field Expressions
A field can by searched on the data type using a field expression.
A field expression is the field to query against, and term that specifies the search. These are separated by a :, such as field:term or for numeric field expressions separated with <, <=, > or >=, such as field<20.
For example, to search cluster data and find the cluster entitled "My cluster", the query would be:
title:"My cluster"
The field is only valid for the term that it directly precedes.
¶ Fields
Fields specify which of the properties on the item that is being searched. Any of the fields listed in Available fields can be used to search.
The advanced query input can help find a particular field. It will show and filter the available fields for any particular query as text is entered:
¶ Field Types
There are a number of field types that can be searched in different ways.
| Type | Description | Example | Example query |
|---|---|---|---|
text |
Text are optimised for fuzzy text searches. They're similar to a google search. The terms search for a set of words, and the order can be different | title, alertTypes |
title:"My Cluster" returns clusters titled both "My important cluster" and "My Cluster". |
keyword |
Keyword data is used for exact matches, for example IDs. This is used for searching specific and exact values or phrases. | clusterStatus.status, title.keyword, alertTypes.keyword |
title.keyword:"My Cluster" returns only the clusters titled exactly "My Cluster" |
numeric |
Numeric fields support ranged term searches (see Terms) | score, alertCount |
score>200 Returns clusters with a score larger than 200 |
date |
Date fields support ranged searches using ISO, epoch milliseconds , or the relative date format (see Date Terms) | created, lastModified |
lastModified:["2021-01-01T00:00:00Z" to "now"] Returns clusters between January 2021 and now |
Fields are often broken down using dot notation into sub fields. Dot notation is used in two cases in the data structure. It can specify data in a different type format, such as title (text) and title.keyword (keyword). It can specify nested information (see Nested fields).
¶ Nested Fields
The data can have a nested structure to allow drill down into related objects. The cluster search allow the search of child alerts and entities as nested fields. Those fields themselves have nested attributes. There are two ways to search nested fields. They can be addressed with the dot notation, for example:
entities.attributes.value:"malicious"
Searches for any clusters with entities that have any attributes that has a value of "malicious".
For more control, multiple sub-fields can be used in a nested expression. This uses curly brackets { and } to specify a child query:
alerts:{ baseScore>5 }
This can be used in combination with dot notation:
entities.attributes:{ kind:"Abuseipdb Abuse Confidence Score" and value>50 }
Or nested within other nested expressions:
entities:{ kind:ipV4 and attributes:{ kind:"Abuseipdb Abuse Confidence Score" and value>50 } }
Note: Searching nested fields is more computationally expensive. This means these queries can take longer to run. We 'flatten' some fields into the parent cluster data to make searching more efficient. Use these flattened fields where possible. For example the query:
alertTypes:"Brute Force"Is much more efficient than the nested query:
alerts.alertType:"Brute Force"But can offer less flexibility. See Available fields for a full list of flattened fields
¶ Terms
*which means 'anything', these can be useful for placeholders in default queries- A term can be a single word, for example
title:cluster - A term can be a series of words separated by spaces, for example
title:cluster title. The term will end when the next field expression or operator is occurs. Operator words such asand,not,orcannot be used. - A term can be text surrounded by double quotes. This allows more complex strings (e.g words with numbers), for example:
title.keyword:"Cluster 2" - A term can be a number or decimal number. For example:
score:22.3orscore<100 - A term can be a group of terms using
(and), separated by an operator, for exampleclusterStatus.status:(New or Investigation) - A term can search a range using
[x to y], for examplescore:[10 to 100] - A term can search nested fields within the specified field using a 'sub query' using
{and}, for example:alerts:{ baseScore>2 }
¶ Date Terms
Dates are stored as UTC. They can be expressed in terms using the following formats:
- Millisecond epoch timestamps, for example:
lastModified>1627907000000. - ISO notation where the date must include the year, and the time (separated by T), is optional. Such as:
yyyy-MM-dd'T'HH:mm:ss.SSSSSSZoryyyy-MM-dd. An example would be:lastModified>"2021-08-02", orlastModified>"2021-08-03T12:00:00". - Relative terms using a simple interval notation relative to
now. For example any cluster modified within the last 7 days can be found withlastModified>"now-7d".
The following intervals are supported:
| Interval | Example | Description |
|---|---|---|
y |
now-1y |
Years |
M |
now-2M |
Months |
w |
now-3w |
Weeks |
d |
now-3d |
Days |
h |
now-12h |
Hours |
m |
now-30m |
Minutes |
s |
now-10s |
Seconds |
Date terms also accept a "granularity" condition, this can help improve query performance. This is applied using a ||/[interval] for absolute and /[interval] for intervals. When this is used SOC.OS won't check further than a required accuracy when performing queries.
When performing a date query across months, day accuracy might be appropriate: created:["2021-01-01||/d" to "2021-03-01||/d"] .
To retrieve all clusters within the last seven days, hour accuracy might be appropriate: created>"now-7d/h" .
¶ Operators
Field expressions, and grouped term queries, can be chained together using operators. The supported operators are and (default if none is specified), or, and not, or not and not.
and is the default, making not the equivilent to and not. For example, the queries title:cluster alertType:"Brute Force" not score>20 is the functionally the same as title:cluster and alertType:"Brute Force" and not score>20".
¶ Null Values
An empty or 'null' value is stored as __NULL__ to prevent conflicting with any likely data. To return all clusters that have no priority set, the query would be priority:__NULL__.
¶ The Order by Statement
The results can be sorted by adding an order by statement to the end of the query. The order by is in the format order by [field] [asc/desc] for example, order by score desc or order by created asc.
Only keyword, numerical and date fields can be used to order the results.
¶ Sample Queries
The following is a list of sample queries and subsequent explanations:
- Return all clusters updated the last 7 days, scored higher than 200, that have not been prioritised yet:
score>200 and priority:__NULL__ and lastModified>"now-7d/h" - Return all clusters that have entities with an Abuseipdb Abuse Confidence Score of greater than 0:
entities.attributes:{ kind.keyword:"Abuseipdb Abuse Confidence Score" and value>0 } - Return any clusters that are either tagged with our
tag:credit-serveror have a high priority alert present:
tags.keyword:"server:credit-server" or alertScores>200 - Return any clusters that have a high score, but exclude a source system we're not interested in:
score>200 and not sourceSystems:"Darktrace"
¶ Available Fields
These are the available fields that can be searched in SOC.OS:
| Clusters | ||
|---|---|---|
| Field | Type | Description |
title |
text |
The title of the cluster |
title.keyword |
keyword |
|
score |
numeric |
The cluster score |
created |
date |
The date the cluster was created |
clusterStatus.status |
keyword |
The cluster status |
clusterStatus.created |
date |
The date the cluster status was set |
clusterStatus.comment |
text |
Any comment associated with the curret cluster status |
clusterStatus.reason |
text |
Any reason associated with the current cluster status |
clusterStatus.reason.keyword |
keyword |
|
threatTypes.threatId |
keyword |
The IDs of any MITRE threat types in this cluster |
threatTypes.acronym |
keyword |
The acronym of any MITRE threat types in this cluster |
threatTypes.name |
text |
The name of any MITRE threat types in this cluster |
threatTypes.name.keyword |
keyword |
|
internalEntityCount |
numeric |
The number of internal entities within the cluster (approximately) |
externalEntityCount |
numeric |
The number of external entities within the cluster (approximately) |
entityCount |
numeric |
The number of entities within the cluster (approximately) |
alertCount |
numeric |
The number of alerts within the cluster (approximately) |
actionedAlertCount |
numeric |
The number of actioned alerts within the cluster (approximately) |
unactionedAlertCount |
numeric |
The number of unactioned alerts within the cluster (approximately) |
alertTypeCount |
numeric |
The number of alert types within the cluster |
sourceSystems |
text |
The source systems contained with the cluster |
sourceSystems.keyword |
keyword |
|
sourceSystemCount |
numeric |
The number of source systems within the cluster |
assignee.userId |
keyword |
The IDs of the user assigned to the cluster |
assignee.name |
text |
The name of the user assigned to the cluster |
assignee.name.keyword |
keyword |
|
priority |
keyword |
The priority of the cluster (P1, P2, P3 or P4) |
reactivatedDate |
date |
The date the cluster was reactivated by SOC.OS from an archived state. |
lastModified |
date |
The date the cluster was last modified. |
mergeParent |
keyword |
The cluster ID of the parent if this cluster is merged. |
mergeChildren |
keyword |
The cluster IDs of any child clusters that were merged into this cluster. |
deactivated |
keyword |
The deactivated state of this cluster |
tags |
text |
Any tags, in the form of 'kind:value' that are associated with this cluster |
tags.keyword |
keyword |
|
tagIds |
keyword |
The raw hashed tag IDs associated with this cluster |
entityIds |
keyword |
The raw hashed entity IDs associated with this cluster |
clusterEntities.entityKind |
keyword |
A flattened field of all the entity kinds within this cluster. |
clusterEntities.entityValue |
text |
A flattened field of all the entity values within this cluster. |
clusterEntities.entityValue.keyword |
keyword |
|
entityAttributeIds |
keyword |
A flattened field of all the raw hashed entity attribute ids within entities in this cluster |
alertIds |
keyword |
A flattened field of all the alert IDs within this cluster |
alertTimestamps |
date |
A flattened field of all the alert timestamps within this cluster. |
alertTypes |
text |
A flattened field of all the alert types within this cluster |
alertTypes.keyword |
keyword |
|
actionCategories |
keyword |
A flattened field of all the alert action categories within this cluster |
actions |
text |
A flattened field of all the alert actions within this cluster |
actions.keyword |
keyword |
|
alertScores |
numeric |
A flattened field of all the alert scores within this cluster |
baseScores |
numeric |
A flattened field of all the alert base scores within this cluster |
alerts |
nested |
A nested field for searching the alerts within a cluster, see below for the alert fields |
entities |
nested |
A nested field for searching the entities within a cluster, see below for the entity fields |
| Alerts | ||
| Field | Type | Description |
timestamp |
date |
The alert timestamp |
alertType |
text |
The alert type |
alertType.keyword |
keyword |
|
actionCategory |
keyword |
The action category of the alert |
action |
text |
The action of the alert |
action.keyword |
keyword |
|
alertScore |
numeric |
The score of the alert |
baseScore |
numeric |
The base score of the alert |
sourceSystem |
text |
The source system the alert is from |
sourceSystem.keyword |
keyword |
|
threatType.threatId |
keyword |
The ID of the MITRE threat type for this alert |
threatType.acronym |
keyword |
The acronym of the MITRE threat type for this alert |
threatType.name |
text |
The name of the MITRE threat type for this alert |
threatType.name.keyword |
keyword |
|
entities |
keyword |
A flattened field of all the entities this alert linked to in the form kind:value |
externalEntities |
keyword |
A flattened field of all the external entities this alert linked to in the form kind:value |
internalEntities |
keyword |
A flattened field of all the internal entities this alert linked to in the form kind:value |
attributes |
nested |
A nested field for searching the attributes within an alert, see below for the attribute fields |
| Entities | ||
| Field | Type | Description |
kind |
keyword |
The entity kind such as "ipV4" or "hostname" |
value |
text |
The entity value, such as ip address or hostname |
value.keyword |
keyword |
|
value.date |
date |
|
value.numeric |
numeric |
|
category |
keyword |
The entity category (internal or external) |
attributes |
nested |
A nested field for searching the attributes within an alert, see below for the attribute fields |
| Attributes | ||
| Field | Type | Description |
kind |
text |
The attribute kind |
kind.keyword |
keyword |
|
value |
text |
The attribute value |
value.keyword |
keyword |
|
value.date |
date |
|
value.numeric |
numeric |
|
For more information on Advanced Search, check out our August 2021 Release Notes.