10

I am currently migrating our existing ELK application from EC2 to Amazon Elasticsearch Service. Firstly, I decided to keep my existing Kibana server and only switch it to point to the new ES domain. So I had changed my existing kibana configuration, as advised in this section: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-kibana

Then, I started to have some issues, which most of them were addressed in this article: Kibana won't connect to Elasticsearch on Amazon's Elasticsearch Service Except for step 4. Kibana was still requiring authentication via the header. Since I am not using CLI or a AWS SDK to authenticate via access keys, but using HTTP/S calls instead (i.e. http://search-[es-domain]:80 inside the kibana config file), it is looking like I would need to sign my HTTP requests anyway (as suggested in step 4 on the previous link) however, I am looking for other options to overcome this issue, in order to have my own Kibana server, cluster, with the proper level of access control from the outside internet, whilst still protecting access to the Elasticsearch for only certain IAM users/roles.

I then decided to try another option: to implement my own proxy (in this case use my existing Nginx server to make it point to the new ES domain. This approach is suggested in slide 56 on the Reinvent bdt209-launch-amazon-elasticsearch-for-realtime-data-analytics slideshare. That way, I could make the web server accessible to the world (via ports 443/80), with of course basic web authentication, whilst protecting the ES Domain with very restrictive access policy only allowing IP-based (as well as IAM roles) access to the Elasticsearch cluster.

However I still got into the same issue. I'm getting this message as a reponse:

 "message": "'a2liYW5hYWRtaW46YWRtaW5AMTIz' not a valid key=value pair (missing equal-sign) in Authorization header: 'Basic a2liYW5hYWRtaW46YWRtaW5AMTIz'."

Which means I would need to sign the request. Can I please have some suggestions on how to overcome this issue? Do I really need to sign the web requests programatically? Or are there other options without affecting security and access control (meaning allowing restricted public access to kibana, with very limited access (role and IP-based) to ES cluster) ?

Thanks very much.

2 Answers 2

9

The root of the issue was Nginx having added the Authorization header (enabled via Basic Authentication module) that includes an encrypted string with the login details to my custom Kibana instance.

If this header is passed to the ES domain endpoint, then ES domain will try to use that header to authenticate against the receiving server. Since it is not preset for my custom authorization credentials, it will simply reject with the Authorization error.

So I had to disable Header Forwarding from the NGINX proxy to the ElasticSearch endpoint replacing my location stanza (in my NGINX config) with this one:

location / {
     proxy_pass http://localhost:5601/;
     proxy_pass_request_headers off;
     proxy_redirect http://localhost:5601/ /;
     proxy_set_header Host $host;
     proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
     auth_basic "Restricted";
     auth_basic_user_file /etc/nginx/nginx.auth;
}
3
  • thanks. now i get "Request must contain an kbn-xsrf header" error when kibana sends a request to es.
    – halil
    Mar 3, 2017 at 6:59
  • @AlbertVacaCintora you can check my answer below. sorry for lateness.
    – halil
    Apr 13, 2017 at 8:56
  • I found that you need to whitelist the header "kbn-version" from the proxy. Current versions of Kibana use "kbn-version" instead of "kbn-xsrf", even though they didn't update the error message. That was confusing me, but in the end I managed to make it work :) Apr 15, 2017 at 17:29
8

I know there is an approved answer here but it did not completely worked for me and i want to present my solution, maybe someone can use it. My environment is simple, usual aws linux on a t2 instance, apt-get installed nginx and untar kibana folder.

my nginx.conf file ;

server {
        listen 80;
        server_name localhost;
        location / {
                proxy_set_header X-Real-IP  $remote_addr;
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_set_header Host $host;
                proxy_set_header User-Agent $http_user_agent;
                auth_basic_user_file /etc/nginx/.htpasswd;
                auth_basic "Auth Required";
                proxy_pass http://search-xxxxxx.eu-west-1.es.amazonaws.com/;
                proxy_redirect http://search-xxxxxx.eu-west-1.es.amazonaws.com/ /;

                proxy_set_header Authorization "";
                proxy_hide_header Authorization;

        }
}

I believe last two lines are important and what is missing in other answer. When you send any kind of Authentication header to Aws ES, it tries to find aws api auth method like signature as value, so it does not work with auth header.

I hope this help someone too, i spent 4 hours for that as a nginx newbie.

2
  • 2
    Adding the two lines at the end fixes this problem correctly. I believe the csrf token is passed when we only blank out the Authorization header. The accepted answer doesn't pass the csrf token and thus end up with more problems.
    – Dat Chu
    Aug 23, 2017 at 18:42
  • @halil I'm getting this "User is not authorized to perform this action" with your config above. am I missing something? Sep 12, 2019 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.