I am currently migrating our existing ELK application from EC2 to Amazon Elasticsearch Service. Firstly, I decided to keep my existing Kibana server and only switch it to point to the new ES domain. So I had changed my existing kibana configuration, as advised in this section: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-kibana
Then, I started to have some issues, which most of them were addressed in this article: Kibana won't connect to Elasticsearch on Amazon's Elasticsearch Service Except for step 4. Kibana was still requiring authentication via the header. Since I am not using CLI or a AWS SDK to authenticate via access keys, but using HTTP/S calls instead (i.e. http://search-[es-domain]:80 inside the kibana config file), it is looking like I would need to sign my HTTP requests anyway (as suggested in step 4 on the previous link) however, I am looking for other options to overcome this issue, in order to have my own Kibana server, cluster, with the proper level of access control from the outside internet, whilst still protecting access to the Elasticsearch for only certain IAM users/roles.
I then decided to try another option: to implement my own proxy (in this case use my existing Nginx server to make it point to the new ES domain. This approach is suggested in slide 56 on the Reinvent bdt209-launch-amazon-elasticsearch-for-realtime-data-analytics slideshare. That way, I could make the web server accessible to the world (via ports 443/80), with of course basic web authentication, whilst protecting the ES Domain with very restrictive access policy only allowing IP-based (as well as IAM roles) access to the Elasticsearch cluster.
However I still got into the same issue. I'm getting this message as a reponse:
"message": "'a2liYW5hYWRtaW46YWRtaW5AMTIz' not a valid key=value pair (missing equal-sign) in Authorization header: 'Basic a2liYW5hYWRtaW46YWRtaW5AMTIz'."
Which means I would need to sign the request. Can I please have some suggestions on how to overcome this issue? Do I really need to sign the web requests programatically? Or are there other options without affecting security and access control (meaning allowing restricted public access to kibana, with very limited access (role and IP-based) to ES cluster) ?
Thanks very much.