Define Mappings in Elasticsearch

2 hours
  • 2 Learning Objectives

About this Hands-on Lab

Dynamic templates in Elasticsearch make it very easy to index data without having to create explicit mappings for every field. However, sometimes you might prefer to create explicit mappings, or even disable dynamic mapping altogether, in order to have a tighter control over your index structure and datatype requirements. In this learning activity, you are given the opportunity to create explicit field mappings for an index containing log data. Specifically, you will exercise how to:

* Create analyzed string fields with a specific analyzer
* Create non-analyzed string fields with character limits
* Create `geo_point` field mappings
* Create numerical field mappings
* Create date field mappings
* Create IP field mappings
* Create nest field mappings (objects)
* Reindex data from one index into another with different mappings

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Create the index with explicit field mappings.

Use the Kibana console tool to execute the following:

PUT logs_new
{
  "mappings": {
    "properties": {
      "@message": {
        "type": "text"
      },
      "@tags": {
        "type": "keyword",
        "ignore_above": 128
      },
      "@timestamp": {
        "type": "date"
      },
      "@version": {
        "type": "keyword",
        "ignore_above": 256
      },
      "agent": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "bytes": {
        "type": "long"
      },
      "clientip": {
        "type": "ip"
      },
      "extension": {
        "type": "keyword",
        "ignore_above": 256
      },
      "geo": {
        "properties": {
          "coordinates": {
            "type": "geo_point"
          },
          "dest": {
            "type": "keyword",
            "ignore_above": 128
          },
          "src": {
            "type": "keyword",
            "ignore_above": 128
          },
          "srcdest": {
            "type": "keyword",
            "ignore_above": 128
          }
        }
      },
      "headings": {
        "type": "keyword",
        "ignore_above": 256
      },
      "host": {
        "type": "keyword",
        "ignore_above": 256
      },
      "ip": {
        "type": "ip"
      },
      "links": {
        "type": "keyword",
        "ignore_above": 256
      },
      "machine": {
        "properties": {
          "os": {
            "type": "keyword",
            "ignore_above": 256
          },
          "ram": {
            "type": "long"
          }
        }
      },
      "memory": {
        "type": "long"
      },
      "phpmemory": {
        "type": "long"
      },
      "referer": {
        "type": "keyword",
        "ignore_above": 256
      },
      "relatedContent": {
        "properties": {
          "article:modified_time": {
            "type": "date"
          },
          "article:published_time": {
            "type": "date"
          },
          "article:section": {
            "type": "keyword",
            "ignore_above": 128
          },
          "article:tag": {
            "type": "keyword",
            "ignore_above": 128
          },
          "og:description": {
            "type": "text"
          },
          "og:image": {
            "type": "keyword",
            "ignore_above": 256
          },
          "og:image:height": {
            "type": "long"
          },
          "og:image:width": {
            "type": "long"
          },
          "og:site_name": {
            "type": "keyword",
            "ignore_above": 256
          },
          "og:title": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "og:type": {
            "type": "keyword",
            "ignore_above": 128
          },
          "og:url": {
            "type": "text",
            "analyzer": "simple",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "twitter:card": {
            "type": "keyword",
            "ignore_above": 128
          },
          "twitter:description": {
            "type": "text"
          },
          "twitter:image": {
            "type": "keyword",
            "ignore_above": 256
          },
          "twitter:site": {
            "type": "keyword",
            "ignore_above": 128
          },
          "twitter:title": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 128
              }
            }
          },
          "url": {
            "type": "text",
            "analyzer": "simple",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "request": {
        "type": "text",
        "analyzer": "simple",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "response": {
        "type": "keyword",
        "ignore_above": 128
      },
      "spaces": {
        "type": "text",
        "analyzer": "whitespace"
      },
      "url": {
        "type": "text",
        "analyzer": "simple",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "utc_time": {
        "type": "date"
      },
      "xss": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 512
          }
        }
      }
    }
  },
  "settings": {
    "number_of_shards": 4,
    "number_of_replicas": 3
  }
}
Reindex the documents from the logs index into the logs_new index.

Use the Kibana console tool to execute the following:

POST _reindex
{
  "source": {
    "index": "logs"
  },
  "dest": {
    "index": "logs_new"
  }
}

Additional Resources

You work as an Elasticsearch administrator for a 6-node cluster and you are tasked with fixing some field mappings for an index containing log data. The current log data was ingested without any preparation so most of the fields created are not the desired type. The existing logs index will need to be reindexed into a new index called logs_new. You are being tasked to create the logs_new index with the following mapping requirements:

--------------------------------------+-----------+-----------+-----------
Field                                 | Datatype  | Character | Analyzer  
                                      |           | Limit     |           
--------------------------------------+-----------+-----------+-----------
@message                              | text      |           | standard  
--------------------------------------+-----------+-----------+-----------
@tags                                 | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
@timestamp                            | date      |           |           
--------------------------------------+-----------+-----------+-----------
@version                              | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
agent                                 | text      |           | standard  
--------------------------------------+-----------+-----------+-----------
agent.keyword                         | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
bytes                                 | long      |           |           
--------------------------------------+-----------+-----------+-----------
clientip                              | ip        |           |           
--------------------------------------+-----------+-----------+-----------
extension                             | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
geo.coordinates                       | geo_point |           |           
--------------------------------------+-----------+-----------+-----------
geo.dest                              | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
geo.src                               | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
geo.srcdest                           | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
headings                              | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
host                                  | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
ip                                    | ip        |           |           
--------------------------------------+-----------+-----------+-----------
links                                 | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
machine.os                            | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
machine.ram                           | long      |           |           
--------------------------------------+-----------+-----------+-----------
memory                                | long      |           |           
--------------------------------------+-----------+-----------+-----------
phpmemory                             | long      |           |           
--------------------------------------+-----------+-----------+-----------
referer                               | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.article:modified_time  | date      |           |           
--------------------------------------+-----------+-----------+-----------
relatedContent.article:published_time | date      |           |           
--------------------------------------+-----------+-----------+-----------
relatedContent.article:section        | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.article:tag            | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.og:description         | text      |           | standard  
--------------------------------------+-----------+-----------+-----------
relatedContent.og:image               | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.og:image:height        | long      |           |           
--------------------------------------+-----------+-----------+-----------
relatedContent.og:image:width         | long      |           |           
--------------------------------------+-----------+-----------+-----------
relatedContent.og:site_name           | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.og:title               | text      |           | standard  
--------------------------------------+-----------+-----------+-----------
relatedContent.og:title.keyword       | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.og:type                | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.og:url                 | text      |           | simple    
--------------------------------------+-----------+-----------+-----------
relatedContent.og:url.keyword         | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.twitter:card           | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.twitter:description    | text      |           | standard  
--------------------------------------+-----------+-----------+-----------
relatedContent.twitter:image          | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.twitter:site           | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.twitter:title          | text      |           | standard  
--------------------------------------+-----------+-----------+-----------
relatedContent.twitter:title.keyword  | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
relatedContent.url                    | text      |           | simple    
--------------------------------------+-----------+-----------+-----------
relatedContent.url.keyword            | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
request                               | text      |           | simple    
--------------------------------------+-----------+-----------+-----------
request.keyword                       | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
response                              | keyword   | 128       |           
--------------------------------------+-----------+-----------+-----------
spaces                                | text      |           | whitespace
--------------------------------------+-----------+-----------+-----------
url                                   | text      |           | simple    
--------------------------------------+-----------+-----------+-----------
url.keyword                           | keyword   | 256       |           
--------------------------------------+-----------+-----------+-----------
utc_time                              | date      |           |           
--------------------------------------+-----------+-----------+-----------
xss                                   | text      |           | standard  
--------------------------------------+-----------+-----------+-----------
xss.keyword                           | keyword   | 512       |           
--------------------------------------+-----------+-----------+-----------

The logs_new index should retain the same index settings as the logs index for the number of primary and replica shards. Once the logs_new index has been created, you can reindex the documents into the new index.

To use Kibana, navigate to the public IP address of the coordinator-1 node in your web browser and login with:

  • Username: elastic
  • Password: la_elastic_409

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Get Started
Who’s going to be learning?

How many seats do you need?

  • $499 USD per seat per year
  • Billed Annually
  • Renews in 12 months

Ready to accelerate learning?

For over 25 licenses, a member of our sales team will walk you through a custom tailored solution for your business.


$2,495.00

Checkout
Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!