Capitalization on search queries

[birdsarah]

http://annotateit.org/api/search?suid=2dg930tfdSR3oMtKzT4LiF

returns no results, when it should, but if you lowercase the suid, then it will return expected results

http://annotateit.org/api/search?suid=2dg930tfdsr3omtkzt4lif

[birdsarah]

although on searching on a user field, behavior is as expected:
http://annotateit.org/api/search?user=timdavies - no results
http://annotateit.org/api/search?user=TimDavies - expected results

[nickstenning]

So I chatted to @birdsarah about this yesterday. The problem here is that by default new fields in elasticsearch are tokenized/analyzed by the default analyzer, which normalises them to lowercase:

$ curl -s http://localhost:9200/_analyze -d 'CaseSensitiveOrNot' | json
{
  "tokens": [
    {
      "token": "casesensitiveornot",
      "start_offset": 0,
      "end_offset": 18,
      "type": "<ALPHANUM>",
      "position": 1
    }
  ]
}

There are a couple of ways we could change this:

1. normalize all incoming query param values to lowercase before passing them to elasticsearch
2. set the default analyzer to the keyword analyzer.
3. create a root object template that defaults all new string fields to not_analyzed

[tilgovi]

Default of not_analyzed seems like a good idea if we're not culling unknown fields, simply for the disk space impact.

[tilgovi]

And performance, etc.

[tilgovi]

It's strange, though, that the search endpoint doesn't follow the same default, isn't it?

[nickstenning]

It's strange, though, that the search endpoint doesn't follow the same default, isn't it?

Not sure what you mean here...

[tilgovi]

Not sure what you mean here...

I mean it's strange that the default analyzer for indexing would lowercase the tokens but the default search analyzer wouldn't. I thought ES also analyzed the query, although maybe it's the case that it doesn't when we specify the terms query directly instead of a query string.

[nickstenning]

maybe it's the case that it doesn't when we specify the terms query directly instead of a query string.

I think that's the killer here, yes.

[nickstenning]

Fixed by #88.