ElasticSearch
- ES为什么快?
What is Elasticsearch
You know, for search (and analysis)
Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack. Logstash and Beats facilitate collecting, aggregating, and enriching your data and storing it in Elasticsearch. Kibana enables you to interactively explore, visualize, and share insights into your data and manage and monitor the stack. Elasticsearch is where the indexing, search, and analysis magic happens.
Elasticsearch provides near real-time search and analytics for all types of data. Whether you have structured or unstructured text, numerical data, or geospatial data, Elasticsearch can efficiently store and index it in a way that supports fast searches. You can go far beyond simple data retrieval and aggregate information to discover trends and patterns in your data. And as your data and query volume grows, the distributed nature of Elasticsearch enables your deployment to grow seamlessly right along with it.
While not every problem is a search problem, Elasticsearch offers speed and flexibility to handle data in a wide variety of use cases:
- Add a search box to an app or website
- Store and analyze logs, metrics, and security event data
- Use machine learning to automatically model the behavior of your data in real time
- Automate business workflows using Elasticsearch as a storage engine
- Manage, integrate, and analyze spatial information using Elasticsearch as a geographic information system (GIS)
- Store and process genetic data using Elasticsearch as a bioinformatics research tool
We’re continually amazed by the novel ways people use search. But whether your use case is similar to one of these, or you’re using Elasticsearch to tackle a new problem, the way you work with your data, documents, and indices in Elasticsearch is the same.
Quick Start
https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html
Mapping
Aggregate
Stores pre-aggregated numeric values for metric aggregations. An aggregate_metric_double
field is an object containing one or more of the following metric sub-fields: min
, max
, sum
, and value_count
.
When you run certain metric aggregations on an aggregate_metric_double
field, the aggregation uses the related sub-field’s values. For example, a min
aggregation on an aggregate_metric_double
field returns the minimum value of all min
sub-fields.
An
aggregate_metric_double
field stores a single numeric doc value for each metric sub-field. Array values are not supported.min
,max
, andsum
values aredouble
numbers.value_count
is a positivelong
number.
Alias
An alias
mapping defines an alternate name for a field in the index. The alias can be used in place of the target field in search requests, and selected other APIs like field capabilities.
PUT trips
{
"mappings": {
"properties": {
"distance": {
"type": "long"
},
"route_length_miles": {
"type": "alias",
"path": "distance"
},
"transit_mode": {
"type": "keyword"
}
}
}
}
GET _search
{
"query": {
"range" : {
"route_length_miles" : {
"gte" : 39
}
}
}
}
The path to the target field. Note that this must be the full path, including any parent objects (e.g. object1.object2.field
).
Almost all components of the search request accept field aliases. In particular, aliases can be used in queries, aggregations, and sort fields, as well as when requesting docvalue_fields
, stored_fields
, suggestions, and highlights. Scripts also support aliases when accessing field values. Please see the section on unsupported APIs for exceptions.
https://www.elastic.co/guide/en/elasticsearch/reference/current/field-alias.html
Arrays
In Elasticsearch, there is no dedicated array
data type. Any field can contain zero or more values by default, however, all values in the array must be of the same data type. For instance:
- an array of strings: [
"one"
,"two"
] - an array of integers: [
1
,2
] - an array of arrays: [
1
, [2
,3
]] which is the equivalent of [1
,2
,3
] - an array of objects: [
{ "name": "Mary", "age": 12 }
,{ "name": "John", "age": 10 }
]
Arrays of objects
Arrays of objects do not work as you would expect: you cannot query each object independently of the other objects in the array. If you need to be able to do this then you should use the [nested
](dfile:///Users/trylife/Library/Application Support/Dash/DocSets/ElasticSearch/ElasticSearch.docset/Contents/Resources/Documents/www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) data type instead of the [object
](dfile:///Users/trylife/Library/Application Support/Dash/DocSets/ElasticSearch/ElasticSearch.docset/Contents/Resources/Documents/www.elastic.co/guide/en/elasticsearch/reference/current/object.html) data type.
This is explained in more detail in [Nested](dfile:///Users/trylife/Library/Application Support/Dash/DocSets/ElasticSearch/ElasticSearch.docset/Contents/Resources/Documents/www.elastic.co/guide/en/elasticsearch/reference/current/nested.html).
When adding a field dynamically, the first value in the array determines the field type
. All subsequent values must be of the same data type or it must at least be possible to [coerce](dfile:///Users/trylife/Library/Application Support/Dash/DocSets/ElasticSearch/ElasticSearch.docset/Contents/Resources/Documents/www.elastic.co/guide/en/elasticsearch/reference/current/coerce.html) subsequent values to the same data type.
Arrays with a mixture of data types are not supported: [ 10
, "some string"
]
An array may contain null
values, which are either replaced by the configured [null_value
](dfile:///Users/trylife/Library/Application Support/Dash/DocSets/ElasticSearch/ElasticSearch.docset/Contents/Resources/Documents/www.elastic.co/guide/en/elasticsearch/reference/current/null-value.html) or skipped entirely. An empty array []
is treated as a missing field — a field with no values.
Nothing needs to be pre-configured in order to use arrays in documents, they are supported out of the box:
PUT my-index-000001/_doc/1
{
"message": "some arrays in this document...",
"tags": [ "elasticsearch", "wow" ],
"lists": [
{
"name": "prog_list",
"description": "programming list"
},
{
"name": "cool_list",
"description": "cool stuff list"
}
]
}
PUT my-index-000001/_doc/2
{
"message": "no arrays in this document...",
"tags": "elasticsearch",
"lists": {
"name": "prog_list",
"description": "programming list"
}
}
GET my-index-000001/_search
{
"query": {
"match": {
"tags": "elasticsearch"
}
}
}
The tags field is dynamically added as a string field. |
|
---|---|
The lists field is dynamically added as an object field. |
|
The second document contains no arrays, but can be indexed into the same fields. | |
The query looks for elasticsearch in the tags field, and matches both documents. |
Multi-value fields and the inverted index
The fact that all field types support multi-value fields out of the box is a consequence of the origins of Lucene. Lucene was designed to be a full text search engine. In order to be able to search for individual words within a big block of text, Lucene tokenizes the text into individual terms, and adds each term to the inverted index separately.
This means that even a simple text field must be able to support multiple values by default. When other data types were added, such as numbers and dates, they used the same data structure as strings, and so got multi-values for free.
Binary
Boolean
Boolean fields accept JSON true
and false
values, but can also accept strings which are interpreted as either true or false:
False values | false , "false" , "" (empty string) |
---|---|
True values | true , "true" |
For example:
PUT my-index-000001
{
"mappings": {
"properties": {
"is_published": {
"type": "boolean"
}
}
}
}
POST my-index-000001/_doc/1?refresh
{
"is_published": "true"
}
GET my-index-000001/_search
{
"query": {
"term": {
"is_published": true
}
}
}
Indexing a document with "true" , which is interpreted as true . |
|
---|---|
Searching for documents with a JSON true . |
Aggregations like the terms
aggregation use 1
and 0
for the key
, and the strings "true"
and "false"
for the key_as_string
. Boolean fields when used in scripts, return true
and false
:
POST my-index-000001/_doc/1?refresh
{
"is_published": true
}
POST my-index-000001/_doc/2?refresh
{
"is_published": false
}
GET my-index-000001/_search
{
"aggs": {
"publish_state": {
"terms": {
"field": "is_published"
}
}
},
"sort": [ "is_published" ],
"fields": [
{"field": "weight"}
],
"runtime_mappings": {
"weight": {
"type": "long",
"script": "emit(doc['is_published'].value ? 10 : 0)"
}
}
}
Parameters for boolean
fields
The following parameters are accepted by boolean
fields:
boost |
Mapping field-level query time boosting. Accepts a floating point number, defaults to 1.0 . |
---|---|
doc_values |
Should the field be stored on disk in a column-stride fashion, so that it can later be used for sorting, aggregations, or scripting? Accepts true (default) or false . |
index |
Should the field be searchable? Accepts true (default) and false . |
null_value |
Accepts any of the true or false values listed above. The value is substituted for any explicit null values. Defaults to null , which means the field is treated as missing. Note that this cannot be set if the script parameter is used. |
on_script_error |
Defines what to do if the script defined by the script parameter throws an error at indexing time. Accepts fail (default), which will cause the entire document to be rejected, and continue , which will register the field in the document’s _ignored metadata field and continue indexing. This parameter can only be set if the script field is also set. |
script |
If this parameter is set, then the field will index values generated by this script, rather than reading the values directly from the source. If a value is set for this field on the input document, then the document will be rejected with an error. Scripts are in the same format as their runtime equivalent. |
store |
Whether the field value should be stored and retrievable separately from the _source field. Accepts true or false (default). |
meta |
Metadata about the field. |
Date
JSON doesn’t have a date data type, so dates in Elasticsearch can either be:
- strings containing formatted dates, e.g.
"2015-01-01"
or"2015/01/01 12:10:30"
. - a number representing milliseconds-since-the-epoch.
- a number representing seconds-since-the-epoch (configuration).
Values for milliseconds-since-the-epoch must be non-negative. Use a formatted date to represent dates before 1970.
Internally, dates are converted to UTC (if the time-zone is specified) and stored as a long number representing milliseconds-since-the-epoch.
Queries on dates are internally converted to range queries on this long representation, and the result of aggregations and stored fields is converted back to a string depending on the date format that is associated with the field.
Dates will always be rendered as strings, even if they were initially supplied as a long in the JSON document.
Date formats can be customised, but if no format
is specified then it uses the default:
"strict_date_optional_time||epoch_millis"
This means that it will accept dates with optional timestamps, which conform to the formats supported by strict_date_optional_time
or milliseconds-since-the-epoch.
For instance:
PUT my-index-000001
{
"mappings": {
"properties": {
"date": {
"type": "date"
}
}
}
}
PUT my-index-000001/_doc/1
{ "date": "2015-01-01" }
PUT my-index-000001/_doc/2
{ "date": "2015-01-01T12:10:30Z" }
PUT my-index-000001/_doc/3
{ "date": 1420070400001 }
GET my-index-000001/_search
{
"sort": { "date": "asc"}
}
The date field uses the default format . |
|
---|---|
This document uses a plain date. | |
This document includes a time. | |
This document uses milliseconds-since-the-epoch. | |
Note that the sort values that are returned are all in milliseconds-since-the-epoch. |
Dates will accept numbers with a decimal point like
{"date": 1618249875.123456}
but there are some cases (#70085) where we’ll lose precision on those dates so should avoid them.
Multiple date formats
Multiple formats can be specified by separating them with ||
as a separator. Each format will be tried in turn until a matching format is found. The first format will be used to convert the milliseconds-since-the-epoch value back into a string.
PUT my-index-000001
{
"mappings": {
"properties": {
"date": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
}
}
}
}
Parameters for date
fields
The following parameters are accepted by date
fields:
boost |
Mapping field-level query time boosting. Accepts a floating point number, defaults to 1.0 . |
---|---|
doc_values |
Should the field be stored on disk in a column-stride fashion, so that it can later be used for sorting, aggregations, or scripting? Accepts true (default) or false . |
format |
The date format(s) that can be parsed. Defaults to strict_date_optional_time||epoch_millis . |
locale |
The locale to use when parsing dates since months do not have the same names and/or abbreviations in all languages. The default is the ROOT locale, |
ignore_malformed |
If true , malformed numbers are ignored. If false (default), malformed numbers throw an exception and reject the whole document. Note that this cannot be set if the script parameter is used. |
index |
Should the field be searchable? Accepts true (default) and false . |
null_value |
Accepts a date value in one of the configured format ’s as the field which is substituted for any explicit null values. Defaults to null , which means the field is treated as missing. Note that this cannot be set of the script parameter is used. |
on_script_error |
Defines what to do if the script defined by the script parameter throws an error at indexing time. Accepts fail (default), which will cause the entire document to be rejected, and continue , which will register the field in the document’s _ignored metadata field and continue indexing. This parameter can only be set if the script field is also set. |
script |
If this parameter is set, then the field will index values generated by this script, rather than reading the values directly from the source. If a value is set for this field on the input document, then the document will be rejected with an error. Scripts are in the same format as their runtime equivalent, and should emit long-valued timestamps. |
store |
Whether the field value should be stored and retrievable separately from the _source field. Accepts true or false (default). |
meta |
Metadata about the field. |
Epoch seconds
If you need to send dates as seconds-since-the-epoch then make sure the format
lists epoch_second
:
PUT my-index-000001
{
"mappings": {
"properties": {
"date": {
"type": "date",
"format": "strict_date_optional_time||epoch_second"
}
}
}
}
PUT my-index-000001/_doc/example?refresh
{ "date": 1618321898 }
POST my-index-000001/_search
{
"fields": [ {"field": "date"}],
"_source": false
}
Which will reply with a date like:
{
"hits": {
"hits": [
{
"_id": "example",
"_index": "my-index-000001",
"_type": "_doc",
"_score": 1.0,
"fields": {
"date": ["2021-04-13T13:51:38.000Z"]
}
}
]
}
}
{
"hits": {
"hits": [
{
"_id": "example",
"_index": "my-index-000001",
"_type": "_doc",
"_score": 1.0,
"fields": {
"date": ["2021-04-13T13:51:38.000Z"]
}
}
]
}
}
Date nanoseconds
This data type is an addition to the date
data type. However there is an important distinction between the two. The existing date
data type stores dates in millisecond resolution. The date_nanos
data type stores dates in nanosecond resolution, which limits its range of dates from roughly 1970 to 2262, as dates are still stored as a long representing nanoseconds since the epoch.
Dense vector
Flattened
By default, each subfield in an object is mapped and indexed separately. If the names or types of the subfields are not known in advance, then they are mapped dynamically.
The flattened
type provides an alternative approach, where the entire object is mapped as a single field. Given an object, the flattened
mapping will parse out its leaf values and index them into one field as keywords. The object’s contents can then be searched through simple queries and aggregations.
This data type can be useful for indexing objects with a large or unknown number of unique keys. Only one field mapping is created for the whole JSON object, which can help prevent a mappings explosion from having too many distinct field mappings.
On the other hand, flattened object fields present a trade-off in terms of search functionality. Only basic queries are allowed, with no support for numeric range queries or highlighting. Further information on the limitations can be found in the Supported operations section.
The flattened
mapping type should not be used for indexing all document content, as it treats all values as keywords and does not provide full search functionality. The default approach, where each subfield has its own entry in the mappings, works well in the majority of cases.
An flattened object field can be created as follows:
PUT bug_reports
{
"mappings": {
"properties": {
"title": {
"type": "text"
},
"labels": {
"type": "flattened"
}
}
}
}
POST bug_reports/_doc/1
{
"title": "Results are not sorted correctly.",
"labels": {
"priority": "urgent",
"release": ["v1.2.5", "v1.3.0"],
"timestamp": {
"created": 1541458026,
"closed": 1541457010
}
}
}
During indexing, tokens are created for each leaf value in the JSON object. The values are indexed as string keywords, without analysis or special handling for numbers or dates.
Querying the top-level flattened
field searches all leaf values in the object:
POST bug_reports/_search
{
"query": {
"term": {"labels": "urgent"}
}
}
To query on a specific key in the flattened object, object dot notation is used:
POST bug_reports/_search
{
"query": {
"term": {"labels.release": "v1.3.0"}
}
}
Supported operations
Because of the similarities in the way values are indexed, flattened
fields share much of the same mapping and search functionality as keyword
fields.
Currently, flattened object fields can be used with the following query types:
term
,terms
, andterms_set
prefix
range
match
andmulti_match
query_string
andsimple_query_string
exists
When querying, it is not possible to refer to field keys using wildcards, as in { "term": {"labels.time*": 1541457010}}
. Note that all queries, including range
, treat the values as string keywords. Highlighting is not supported on flattened
fields.
It is possible to sort on an flattened object field, as well as perform simple keyword-style aggregations such as terms
. As with queries, there is no special support for numerics — all values in the JSON object are treated as keywords. When sorting, this implies that values are compared lexicographically.
Flattened object fields currently cannot be stored. It is not possible to specify the store
parameter in the mapping.
Retrieving flattened fields
Field values and concrete subfields can be retrieved using the fields parameter. content. Since the flattened
field maps an entire object with potentially many subfields as a single field, the response contains the unaltered structure from _source
.
Single subfields, however, can be fetched by specifying them explicitly in the request. This only works for concrete paths, but not using wildcards:
PUT my-index-000001
{
"mappings": {
"properties": {
"flattened_field": {
"type": "flattened"
}
}
}
}
PUT my-index-000001/_doc/1?refresh=true
{
"flattened_field" : {
"subfield" : "value"
}
}
POST my-index-000001/_search
{
"fields": ["flattened_field.subfield"],
"_source": false
}
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 1.0,
"hits": [{
"_index": "my-index-000001",
"_type" : "_doc",
"_id": "1",
"_score": 1.0,
"fields": {
"flattened_field.subfield" : [ "value" ]
}
}]
}
}
You can also use a Painless script to retrieve values from sub-fields of flattened fields. Instead of including doc['<field_name>'].value
in your Painless script, use doc['<field_name>.<sub-field_name>'].value
. For example, if you have a flattened field called label
with a release
sub-field, your Painless script would be doc['labels.release'].value
.
For example, let’s say your mapping contains two fields, one of which is of the flattened
type:
PUT my-index-000001
{
"mappings": {
"properties": {
"title": {
"type": "text"
},
"labels": {
"type": "flattened"
}
}
}
}
Index a few documents containing your mapped fields. The labels
field has three sub-fields:
"script": {
"source": """
if (doc['labels.release'].value.equals('v1.3.0'))
{emit(doc['labels.release'].value)}
else{emit('Version mismatch')}
"""
Parameters
boost |
Mapping field-level query time boosting. Accepts a floating point number, defaults to 1.0 . |
---|---|
depth_limit |
The maximum allowed depth of the flattened object field, in terms of nested inner objects. If a flattened object field exceeds this limit, then an error will be thrown. Defaults to 20 . Note that depth_limit can be updated dynamically through the update mapping API. |
doc_values |
Should the field be stored on disk in a column-stride fashion, so that it can later be used for sorting, aggregations, or scripting? Accepts true (default) or false . |
eager_global_ordinals |
Should global ordinals be loaded eagerly on refresh? Accepts true or false (default). Enabling this is a good idea on fields that are frequently used for terms aggregations. |
ignore_above |
Leaf values longer than this limit will not be indexed. By default, there is no limit and all values will be indexed. Note that this limit applies to the leaf values within the flattened object field, and not the length of the entire field. |
index |
Determines if the field should be searchable. Accepts true (default) or false . |
index_options |
What information should be stored in the index for scoring purposes. Defaults to docs but can also be set to freqs to take term frequency into account when computing scores. |
null_value |
A string value which is substituted for any explicit null values within the flattened object field. Defaults to null , which means null sields are treated as if it were missing. |
similarity |
Which scoring algorithm or similarity should be used. Defaults to BM25 . |
split_queries_on_whitespace |
Whether full text queries should split the input on whitespace when building a query for this field. Accepts true or false (default). |
Geo-point
Geo-shape
Histogram
IP
Join
Keyword
The keyword family includes the following field types:
keyword
, which is used for structured content such as IDs, email addresses, hostnames, status codes, zip codes, or tags.constant_keyword
for keyword fields that always contain the same value.wildcard
for unstructured machine-generated content. Thewildcard
type is optimized for fields with large values or high cardinality.
Keyword fields are often used in sorting, aggregations, and term-level queries, such as term
.
Avoid using keyword fields for full-text search. Use the text
field type instead.
Keyword field type
Below is an example of a mapping for a basic keyword
field:
PUT my-index-000001
{
"mappings": {
"properties": {
"tags": {
"type": "keyword"
}
}
}
}
Mapping numeric identifiers
Not all numeric data should be mapped as a numeric field data type. Elasticsearch optimizes numeric fields, such as integer
or long
, for range
queries. However, keyword
fields are better for term
and other term-level queries.
Identifiers, such as an ISBN or a product ID, are rarely used in range
queries. However, they are often retrieved using term-level queries.
Consider mapping a numeric identifier as a keyword
if:
- You don’t plan to search for the identifier data using
range
queries. - Fast retrieval is important.
term
query searches onkeyword
fields are often faster thanterm
searches on numeric fields.
If you’re unsure which to use, you can use a multi-field to map the data as both a keyword
and a numeric data type.
Parameters
The following parameters are accepted by keyword
fields:
boost |
Mapping field-level query time boosting. Accepts a floating point number, defaults to 1.0 . |
---|---|
doc_values |
Should the field be stored on disk in a column-stride fashion, so that it can later be used for sorting, aggregations, or scripting? Accepts true (default) or false . |
eager_global_ordinals |
Should global ordinals be loaded eagerly on refresh? Accepts true or false (default). Enabling this is a good idea on fields that are frequently used for terms aggregations. |
fields |
Multi-fields allow the same string value to be indexed in multiple ways for different purposes, such as one field for search and a multi-field for sorting and aggregations. |
ignore_above |
Do not index any string longer than this value. Defaults to 2147483647 so that all values would be accepted. Please however note that default dynamic mapping rules create a sub keyword field that overrides this default by setting ignore_above: 256 . |
index |
Should the field be searchable? Accepts true (default) or false . |
index_options |
What information should be stored in the index, for scoring purposes. Defaults to docs but can also be set to freqs to take term frequency into account when computing scores. |
norms |
Whether field-length should be taken into account when scoring queries. Accepts true or false (default). |
null_value |
Accepts a string value which is substituted for any explicit null values. Defaults to null , which means the field is treated as missing. Note that this cannot be set if the script value is used. |
on_script_error |
Defines what to do if the script defined by the script parameter throws an error at indexing time. Accepts fail (default), which will cause the entire document to be rejected, and continue , which will register the field in the document’s _ignored metadata field and continue indexing. This parameter can only be set if the script field is also set. |
script |
If this parameter is set, then the field will index values generated by this script, rather than reading the values directly from the source. If a value is set for this field on the input document, then the document will be rejected with an error. Scripts are in the same format as their runtime equivalent. Values emitted by the script are normalized as usual, and will be ignored if they are longer that the value set on ignore_above . |
store |
Whether the field value should be stored and retrievable separately from the _source field. Accepts true or false (default). |
similarity |
Which scoring algorithm or similarity should be used. Defaults to BM25 . |
normalizer |
How to pre-process the keyword prior to indexing. Defaults to null , meaning the keyword is kept as-is. |
split_queries_on_whitespace |
Whether full text queries should split the input on whitespace when building a query for this field. Accepts true or false (default). |
meta |
Metadata about the field. |
Constant keyword field x-pack
Constant keyword is a specialization of the keyword
field for the case that all documents in the index have the same value.
PUT logs-debug
{
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"message": {
"type": "text"
},
"level": {
"type": "constant_keyword",
"value": "debug"
}
}
}
}
constant_keyword
supports the same queries and aggregations as keyword
fields do, but takes advantage of the fact that all documents have the same value per index to execute queries more efficiently.
It is both allowed to submit documents that don’t have a value for the field or that have a value equal to the value configured in mappings. The two below indexing requests are equivalent:
POST logs-debug/_doc
{
"date": "2019-12-12",
"message": "Starting up Elasticsearch",
"level": "debug"
}
POST logs-debug/_doc
{
"date": "2019-12-12",
"message": "Starting up Elasticsearch"
}
However providing a value that is different from the one configured in the mapping is disallowed.
In case no value
is provided in the mappings, the field will automatically configure itself based on the value contained in the first indexed document. While this behavior can be convenient, note that it means that a single poisonous document can cause all other documents to be rejected if it had a wrong value.
Before a value has been provided (either through the mappings or from a document), queries on the field will not match any documents. This includes exists
queries.
The value
of the field cannot be changed after it has been set.
Parameters for constant keyword fields
The following mapping parameters are accepted:
meta |
Metadata about the field. |
---|---|
value |
The value to associate with all documents in the index. If this parameter is not provided, it is set based on the first document that gets indexed. |
Wildcard field type
The wildcard
field type is a specialized keyword field for unstructured machine-generated content you plan to search using grep-like wildcard
and regexp
queries. The wildcard
type is optimized for fields with large values or high cardinality.
Mapping unstructured content
You can map a field containing unstructured content to either a text
or keyword family field. The best field type depends on the nature of the content and how you plan to search the field.
Use the text
field type if:
- The content is human-readable, such as an email body or product description.
- You plan to search the field for individual words or phrases, such as
the brown fox jumped
, using full text queries. Elasticsearch analyzestext
fields to return the most relevant results for these queries.
Use a keyword family field type if:
- The content is machine-generated, such as a log message or HTTP request information.
- You plan to search the field for exact full values, such as
org.foo.bar
, or partial character sequences, such asorg.foo.*
, using term-level queries.
Choosing a keyword family field type
If you choose a keyword family field type, you can map the field as a keyword
or wildcard
field depending on the cardinality and size of the field’s values. Use the wildcard
type if you plan to regularly search the field using a wildcard
or regexp
query and meet one of the following criteria:
- The field contains more than a million unique values.
AND
You plan to regularly search the field using a pattern with leading wildcards, such as
*foo
or*baz
. - The field contains values larger than 32KB. AND You plan to regularly search the field using any wildcard pattern.
Otherwise, use the keyword
field type for faster searches, faster indexing, and lower storage costs. For an in-depth comparison and decision flowchart, see our related blog post.
Switching from a text
field to a keyword field
If you previously used a text
field to index unstructured machine-generated content, you can reindex to update the mapping to a keyword
or wildcard
field. We also recommend you update your application or workflow to replace any word-based full text queries on the field to equivalent term-level queries.
Internally the wildcard
field indexes the whole field value using ngrams and stores the full string. The index is used as a rough filter to cut down the number of values that are then checked by retrieving and checking the full values. This field is especially well suited to run grep-like queries on log lines. Storage costs are typically lower than those of keyword
fields but search speeds for exact matches on full terms are slower. If the field values share many prefixes, such as URLs for the same website, storage costs for a wildcard
field may be higher than an equivalent keyword
field.
You index and search a wildcard field as follows
PUT my-index-000001
{
"mappings": {
"properties": {
"my_wildcard": {
"type": "wildcard"
}
}
}
}
PUT my-index-000001/_doc/1
{
"my_wildcard" : "This string can be quite lengthy"
}
GET my-index-000001/_search
{
"query": {
"wildcard": {
"my_wildcard": {
"value": "*quite*lengthy"
}
}
}
}
Parameters for wildcard fieldsThe following parameters are accepted by wildcard
fields:
null_value |
Accepts a string value which is substituted for any explicit null values. Defaults to null , which means the field is treated as missing. |
---|---|
ignore_above |
Do not index any string longer than this value. Defaults to 2147483647 so that all values would be accepted. |
Limitations
wildcard
fields are untokenized like keyword fields, so do not support queries that rely on word positions such as phrase queries.- When running
wildcard
queries anyrewrite
parameter is ignored. The scoring is always a constant score.
Nested
The nested
type is a specialised version of the object
data type that allows arrays of objects to be indexed in a way that they can be queried independently of each other.
When ingesting key-value pairs with a large, arbitrary set of keys, you might consider modeling each key-value pair as its own nested document with key
and value
fields. Instead, consider using the flattened data type, which maps an entire object as a single field and allows for simple searches over its contents. **Nested documents and queries are typically expensive, so using the flattened
data type for this use case is a better option.
How arrays of objects are flattened
Elasticsearch has no concept of inner objects. Therefore, it flattens object hierarchies into a simple list of field names and values. For instance, consider the following document:
**
PUT my-index-000001/_doc/1
{
"group" : "fans",
"user" : [
{
"first" : "John",
"last" : "Smith"
},
{
"first" : "Alice",
"last" : "White"
}
]
}
The user
field is dynamically added as a field of type object
.
The previous document would be transformed internally into a document that looks more like this:
{
"group" : "fans",
"user.first" : [ "alice", "john" ],
"user.last" : [ "smith", "white" ]
}
The user.first
and user.last
fields are flattened into multi-value fields, and the association between alice
and white
is lost. This document would incorrectly match a query for alice AND smith
:
GET my-index-000001/_search
{
"query": {
"bool": {
"must": [
{ "match": { "user.first": "Alice" }},
{ "match": { "user.last": "Smith" }}
]
}
}
}
Using nested
fields for arrays of objects
If you need to index arrays of objects and to maintain the independence of each object in the array, use the nested
data type instead of the object
data type.
Internally, nested objects index each object in the array as a separate hidden document, meaning that each nested object can be queried independently of the others with the nested
query:
Numeric
The following numeric types are supported:
long |
A signed 64-bit integer with a minimum value of -263 and a maximum value of 263-1 . |
---|---|
integer |
A signed 32-bit integer with a minimum value of -231 and a maximum value of 231-1 . |
short |
A signed 16-bit integer with a minimum value of -32,768 and a maximum value of 32,767 . |
byte |
A signed 8-bit integer with a minimum value of -128 and a maximum value of 127 . |
double |
A double-precision 64-bit IEEE 754 floating point number, restricted to finite values. |
float |
A single-precision 32-bit IEEE 754 floating point number, restricted to finite values. |
half_float |
A half-precision 16-bit IEEE 754 floating point number, restricted to finite values. |
scaled_float |
A floating point number that is backed by a long , scaled by a fixed double scaling factor. |
unsigned_long |
An unsigned 64-bit integer with a minimum value of 0 and a maximum value of 264-1 . |
Below is an example of configuring a mapping with numeric fields:
PUT my-index-000001
{
"mappings": {
"properties": {
"number_of_bytes": {
"type": "integer"
},
"time_in_seconds": {
"type": "float"
},
"price": {
"type": "scaled_float",
"scaling_factor": 100
}
}
}
}
The double
, float
and half_float
types consider that -0.0
and +0.0
are different values. As a consequence, doing a term
query on -0.0
will not match +0.0
and vice-versa. Same is true for range queries: if the upper bound is -0.0
then +0.0
will not match, and if the lower bound is +0.0
then -0.0
will not match.
Which type should I use?edit
As far as integer types (byte
, short
, integer
and long
) are concerned, you should pick the smallest type which is enough for your use-case. This will help indexing and searching be more efficient. Note however that storage is optimized based on the actual values that are stored, so picking one type over another one will have no impact on storage requirements.
For floating-point types, it is often more efficient to store floating-point data into an integer using a scaling factor, which is what the scaled_float
type does under the hood. For instance, a price
field could be stored in a scaled_float
with a scaling_factor
of 100
. All APIs would work as if the field was stored as a double, but under the hood Elasticsearch would be working with the number of cents, price*100
, which is an integer. This is mostly helpful to save disk space since integers are way easier to compress than floating points. scaled_float
is also fine to use in order to trade accuracy for disk space. For instance imagine that you are tracking cpu utilization as a number between 0
and 1
. It usually does not matter much whether cpu utilization is 12.7%
or 13%
, so you could use a scaled_float
with a scaling_factor
of 100
in order to round cpu utilization to the closest percent in order to save space.
If scaled_float
is not a good fit, then you should pick the smallest type that is enough for the use-case among the floating-point types: double
, float
and half_float
. Here is a table that compares these types in order to help make a decision.
Type | Minimum value | Maximum value | Significant bits / digits |
---|---|---|---|
double |
2-1074 |
(2-2-52)·21023 |
53 / 15.95 |
float |
2-149 |
(2-2-23)·2127 |
24 / 7.22 |
half_float |
2-24 |
65504 |
11 / 3.31 |
Mapping numeric identifiers
Not all numeric data should be mapped as a numeric field data type. Elasticsearch optimizes numeric fields, such as integer
or long
, for range
queries. However, keyword
fields are better for term
and other term-level queries.
Identifiers, such as an ISBN or a product ID, are rarely used in range
queries. However, they are often retrieved using term-level queries.
Consider mapping a numeric identifier as a keyword
if:
- You don’t plan to search for the identifier data using
range
queries. - Fast retrieval is important.
term
query searches onkeyword
fields are often faster thanterm
searches on numeric fields.
If you’re unsure which to use, you can use a multi-field to map the data as both a keyword
and a numeric data type.
Parameters for numeric fieldsedit
The following parameters are accepted by numeric types:
coerce |
Try to convert strings to numbers and truncate fractions for integers. Accepts true (default) and false . Not applicable for unsigned_long . Note that this cannot be set if the script parameter is used. |
---|---|
boost |
Mapping field-level query time boosting. Accepts a floating point number, defaults to 1.0 . |
doc_values |
Should the field be stored on disk in a column-stride fashion, so that it can later be used for sorting, aggregations, or scripting? Accepts true (default) or false . |
ignore_malformed |
If true , malformed numbers are ignored. If false (default), malformed numbers throw an exception and reject the whole document. Note that this cannot be set if the script parameter is used. |
index |
Should the field be searchable? Accepts true (default) and false . |
null_value |
Accepts a numeric value of the same type as the field which is substituted for any explicit null values. Defaults to null , which means the field is treated as missing. Note that this cannot be set if the script parameter is used. |
on_script_error |
Defines what to do if the script defined by the script parameter throws an error at indexing time. Accepts fail (default), which will cause the entire document to be rejected, and continue , which will register the field in the document’s _ignored metadata field and continue indexing. This parameter can only be set if the script field is also set. |
script |
If this parameter is set, then the field will index values generated by this script, rather than reading the values directly from the source. If a value is set for this field on the input document, then the document will be rejected with an error. Scripts are in the same format as their runtime equivalent. Scripts can only be configured on long and double field types. |
store |
Whether the field value should be stored and retrievable separately from the _source field. Accepts true or false (default). |
meta |
Metadata about the field. |
Parameters for scaled_float
scaled_float
accepts an additional parameter:
scaling_factor |
The scaling factor to use when encoding values. Values will be multiplied by this factor at index time and rounded to the closest long value. For instance, a scaled_float with a scaling_factor of 10 would internally store 2.34 as 23 and all search-time operations (queries, aggregations, sorting) will behave as if the document had a value of 2.3 . High values of scaling_factor improve accuracy but also increase space requirements. This parameter is required. |
---|---|
Object
JSON documents are hierarchical in nature: the document may contain inner objects which, in turn, may contain inner objects themselves:
PUT my-index-000001/_doc/1
{
"region": "US",
"manager": {
"age": 30,
"name": {
"first": "John",
"last": "Smith"
}
}
}
Internally, this document is indexed as a simple, flat list of key-value pairs, something like this:
{
"region": "US",
"manager.age": 30,
"manager.name.first": "John",
"manager.name.last": "Smith"
}
An explicit mapping for the above document could look like this:
PUT my-index-000001
{
"mappings": {
"properties": {
"region": {
"type": "keyword"
},
"manager": {
"properties": {
"age": { "type": "integer" },
"name": {
"properties": {
"first": { "type": "text" },
"last": { "type": "text" }
}
}
}
}
}
}
}
-
Properties in the top-level mappings definition.
-
The manager field is an inner object field.
-
The manager.name field is an inner object field within the manager field.
You are not required to set the field type
to object
explicitly, as this is the default value.
Parameters for object
fieldsedit
The following parameters are accepted by object
fields:
dynamic |
Whether or not new properties should be added dynamically to an existing object. Accepts true (default), false and strict . |
---|---|
enabled |
Whether the JSON value given for the object field should be parsed and indexed (true , default) or completely ignored (false ). |
properties |
The fields within the object, which can be of any data type, including object . New properties may be added to an existing object. |
If you need to index arrays of objects instead of single objects, read Nested first.