14

We have hosted some apps on Amazon EC2 and are using an Elastic Load Balancer (ELB) to manage several instances of one app. Also, we have set up ELB alarms to get notified about Unhealthy Hosts, i.e. when an instance has gone down.

So far, I could not figure out where to check which instance exactly has gone down when the alarm goes off, except for the ELB status page in the AWS console. However, if the instance comes back to In Service state again, this won't help me either. The e-mail notification sent out by the ELB does not contain this information; and I couldn't find it anywhere in the alarms history in the console either.

Is there a way to tell which instance an ELB alarm has been triggered for, even if the instance has come back into OK state in the meantime?

Cheers, Alex

5 Answers 5

7

Sadly Amazon does not provide a health check log, so its impossible to find out which instance failed the health check afterwards, assuming that the server is no longer unhealthy. You can only use Per-Az metrics to know in which AZ is the instance.

But, you could know which instance is down if you query AWS api during the issue. So, I have thought of a possible workaround:

  • Set up a new SNS topic, and add an HTTP action to a custom URL that triggers a job that enumerates the instances and send you that info by mail.
  • Then setup a CloudWatch alarm for UnHealthyHostCount > 0 and setup the action to the SNS topic.

The difficult part is that your URL should handle the SNS subscription & confirmation described here.

The command to know which instance is currently OutOfService is:

elb-describe-instance-health *LoadBalancerName* --region *YourRegion*
1
  • aws elb describe-instance-health --load-balancer-name XXX Mar 2, 2018 at 4:33
2

You could probably use the AWS SDK gem or other AWS library that can get status. Use it to create a cron task that regularly gets the status of each instance and records it somewhere. Either that will give you what you need or the disappearance of the status for one instance will tell you which one went bad.

2

We are using the following Lambda function to make up for the lack of Health Check logging:

'use strict';

var AWS = require('aws-sdk');
var elb = new AWS.ELB();

exports.handler = (event, context, callback) => {

    var params = {
        LoadBalancerName: "<elb_name_here>"
    };
    elb.describeInstanceHealth(params, function(err, data) {
        if (err) console.log(err, err.stack); // an error occurred
        else     console.log(data);           // successful response
    });
};

It does not produce the prettiest logs in CloudWatch, but the data is there. It allows us to see if there is a particular instance which tends to drop more often, etc. It is set up much like Gerardo Grignoli's answer above. I added a CloudWatch alarm to send an SNS message to the Lambda function when the alarm was triggered. It doesn't do anything with the message itself - the message is merely the triggering mechanism for the Lambda function to run and log the instance status.

1
  • Exactly what I was hoping to see as an answer; however, it could perhaps be cleaner to publish the details about the incident to something like a DynamoDB table or RDS, where you can then retrieve the metrics about an instance without having to fish through CloudWatch logs. (E.g., perhaps a DynamoDB with: "EventId" (some UUID; primary key), "InstanceId", "IncidentTime" (timestamp; sort-key), "Expires" (time-to-live for the record).) IMO, that would make it exceptionally easy to see things like, "this instance tends to drop more frequently than the rest."
    – Spencer D
    Nov 17, 2018 at 1:14
-3

No. The ELB metrics in CloudWatch do not provide you with that level of details and IMHO from the design perspective they should not. If a host is unhealthy the monitoring on the specific host should report the details for that not the ELB. If a node goes out of service in ELB, it should not be a problem for ELB. Although, in load balancer it makes sense to figure out an alarming state where 3 out of 6 of your machines go into Not In Service state. Take a look at CloudWatch metrics

2
  • Self-monitoring instances and/or setting up external monitoring would solve the problem - agreed. However, I still think that this information might be included in ELB alarms, as the ELB obviously knows which instances it is monitoring and which ones are unhealthy. Setting up external monitoring that basically does the same as the ELB when it comes to health checking just seems like redundant extra work. Feb 19, 2014 at 11:21
  • 2
    I disagree. I would love to have access to the data in question (precisely which instance(s) went unhealthy) for post mortem analyses.
    – Sankalp
    Feb 1, 2016 at 13:10
-3

Go to load balancer and find load balancer associated with you ELB. Then look at instances that OutofService

1
  • Sorry, but this answer is not helpful. The question was how to determine which instance was unhealthy after all instances have gone back to In Service state. Dec 8, 2015 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.