mongodb - When putting data into elasticsearch, how do you handle fields that sometimes have different structures? -

i'm getting several mapping exceptions when trying insert data mongodb elastic. after investigative work, seems error comes fact have field in db , array of strings, while other times array of objects.

meaning, documents in mongo have this:

{"my_field" : ["one", "two"] 

while others

{"my_field": [{"key":"value", "key2":"value"}, {"key":"value", "key2":"value"}, ...] 

i'm having difficult time in pinning down how situation handled in elastic.

you need massage data before indexed conform elasticsearch's rules. 1 approach my_field nested document - 1 document might have

{"my_field": {"string_value": ["one", "two"]}} 

and another

{"my_field": {"doc_value": {"key":"value", "key2":"value"}}} 

this assumes values key , key2 have same type , there small number of possible keys in document. if document contains arbitrary data might better off indexing

{"my_field": [{"key": "key1", "string_value": "value"},             {"key": "key2", "int_value": "123"}]} 

as how massage, 1 option before send data elasticsearch. downside the _source attribute contained transformed data.

another approach send data elasticsearch as-is, have transform in mapping elasticsearch run transform data before indexing.