apiVersion parsing failed

[hyogyunbang]

• Version: Logstash 2.3.4
• Operating System: CentOS
• Config File (if you have sensitive info, please remove it):
  input {
  s3 {
  access_key_id => "XXX"
  secret_access_key => "XXX"
  region => "XXX"
  bucket => "XXX"
  prefix => "XXX"
  tags => [ "XXX" ]
  type => "cloudtrail"
  codec => "cloudtrail"
  }
  }
• Sample Data:
• Steps to Reproduce:

The following is a error log of logstash.
:response=>{"create"=>{"_index"=>"XXX", "_type"=>"cloudtrail", "_id"=>"XXXXX", "status"=>400, "error"=>"MapperParsingException[failed to parse [apiVersion]]; nested: MapperParsingException[failed to parse date field [2016_01_28], tried both date format [dateOptionalTime], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: "2016_01_28" is malformed at "_01_28"]; "}}, :level=>:warn}

[hyogyunbang]

I looked into the original logs of the cloudtrail. Sometimes apiVersion format of cloudtrail is "YYYY_MM_DD".

[rdesgrange-lf]

I have this problem too, do you know any work around ?

[catalinvr]

You can create an index_template and set the "apiVersion" as string instead of date.

[michalkm]

Hey,

The above solution doesn't work, it is still being detected as "date" (ES date_detection is enabled by default, I suspect that disabling it will affect also @timstamp field.

Any idea ?

[wvidana]

I guess it can be done on logstash using a grok filter to change _ to -

[vinogradovs]

Hi,
such template helped to avoid the issue:

curl -XPUT localhost:9200/_template/cloudtrail-template -d '
{
  "template" : "cloudtrail-*",
  "mappings" : {
    "apiVersion_type" : {
	   "properties": {
      "apiVersion" : { "type" : "string", "index" : "not_analyzed" }
    }
  }}
}
'

[ravenbyron]

if anyone comes across this same issue with ES7 this is what the mapping needs to look

  "mappings": {
    "numeric_detection": false,
    "properties": {
      "apiVersion": {
        "type": "text"
      },
      "requestParameters.limit":{
          "type": "long"
      }
    }
  },