7

Please help get past this. I have to enter 3 logical ids to rollback my update, but the regex in cloudformation doesn't allow for that because the regex doesn't appear to allow commas. But the continue ROLLBACK instructions from AWS clearly state to use commas:

"To skip resources, type a list of comma-separated logical resource IDs. Include only the resources that are blocking the rollback."

But the commas are not allowed by the cloudformation regex???? What am I missing here? I can't go forward or back. My cloudformation stack is stuck.

My error on trying to roll back:

Failed to rollback: 1 validation error detected: Value '[rRoute10,rRoute192,rRoute172]' at 'resourcesToSkip' failed to satisfy constraint: Member must satisfy constraint: [Member must satisfy regular expression pattern: [a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*.[a-zA-Z0-9]+]

My errors in cloudformation:

14:06:04 UTC-0400 UPDATE_FAILED AWS::EC2::Route rRoute192 The gateway ID 'vgw-0e7d969e316a7b5d5' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidGatewayID.NotFound; Request ID: 6d4cdb5f-31c5-4cf3-8777-3e3eb361d594)

14:06:04 UTC-0400 UPDATE_FAILED AWS::EC2::Route rRoute10 The gateway ID 'vgw-0e7d969e316a7b5d5' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidGatewayID.NotFound; Request ID: 315ecc3a-d70c-46b7-b1cd-05f4c6765edd)

14:06:04 UTC-0400 UPDATE_FAILED AWS::EC2::Route rRoute172 The gateway ID 'vgw-0e7d969e316a7b5d5' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidGatewayID.NotFound; Request ID: 2a6e20f0-4d41-4647-85f0-ffbfc0326680)

For background, someone deleted our VPG and created a different gateway. They also manually updated the routes. Now, I am trying to get the stack in sync on the route tables. However, the stack failed to update and now it fails to roll back.

Screenshot: enter image description here

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
    RDS

    SAM Template for creating an RDS in a secure Fault-Tolerant fashion...

Parameters:
  pDBName:
    Default: MyDatabase
    Description: The database name
    Type: String
    MinLength: '1'
    MaxLength: '64'
    AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
    ConstraintDescription: must begin with a letter and contain only alphanumeric characters.
  pDBUser:
    NoEcho: 'true'
    Description: The database admin account username
    Type: String
    MinLength: '1'
    MaxLength: '16'
    AllowedPattern: '[a-zA-Z][a-zA-Z0-9]*'
    ConstraintDescription: must begin with a letter and contain only alphanumeric
      characters.
  pDBPassword:
    NoEcho: 'true'
    Description: The database admin account password
    Type: String
    MinLength: '1'
    MaxLength: '41'
    AllowedPattern: '[a-zA-Z0-9]+'
    ConstraintDescription: must contain only alphanumeric characters.
  pDBAllocatedStorage:
    Default: '20'
    Description: The size of the database (Gb)
    Type: Number
    MinValue: '20'
    MaxValue: '16384'
    ConstraintDescription: must be between 5 and 1024Gb.
  pDBInstanceClass:
    Description: The database instance type
    Type: String
    Default: db.t2.micro
    AllowedValues: [db.t1.micro, db.m1.small, db.m1.medium, db.m1.large, db.m1.xlarge,
      db.m2.xlarge, db.m2.2xlarge, db.m2.4xlarge, db.m3.medium, db.m3.large, db.m3.xlarge,
      db.m3.2xlarge, db.m4.large, db.m4.xlarge, db.m4.2xlarge, db.m4.4xlarge, db.m4.10xlarge,
      db.r3.large, db.r3.xlarge, db.r3.2xlarge, db.r3.4xlarge, db.r3.8xlarge, db.m2.xlarge,
      db.m2.2xlarge, db.m2.4xlarge, db.cr1.8xlarge, db.t2.micro, db.t2.small, db.t2.medium,
      db.t2.large]
    ConstraintDescription: must select a valid database instance type.

    # TODO - Future DBEngine allowed values: [aurora-mysql, aurora-postgresql, mariadb, oracle-ee, oracle-se2, sqlserver-ee, sqlserver-se, sqlserver-ex, sqlserver-web]
  pDBEngine:
    Description: The database type to create
    Type: String
    Default: oracle-se2
    AllowedValues: [aurora-mysql, oracle-ee, oracle-se2]
    ConstraintDescription: must select a valid database engine
  pDBEngineVersion:
    Description: The version of database to create
    Type: String
  pMultiAZ:
    Description: Multi-AZ master database
    Type: String
    Default: 'false'
    AllowedValues: ['true', 'false']
    ConstraintDescription: must be true or false.
  pDatabaseDeletionProtection:
    Description: Do we even allow a database to be deleted?
    Type: String
    Default: 'false'
    AllowedValues: ['true', 'false']
    ConstraintDescription: must be true or false.
  pDatabaseLicenseModel:
    Description: What licensing do we use?
    Type: String
    Default: "license-included"
    AllowedValues: ["license-included", "bring-your-own-license", "general-public-license"]
    ConstraintDescription: must be license-included or bring-your-own-license or general-public-license
  pDBParameterGroupName:
    Description: Use an existing parameter group in Amazon RDS - must match the database.
    Type: String
    Default: "default.aurora-mysql5.7"
  pStorageType:   # https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_CreateDBInstance.html
     Description: The type of storage to be used by the database.
     Type: String
     Default: 'standard'
     AllowedValues: ['standard', 'gp2', 'io1']
  pRDSVPC:
    Description: The vpc to use?
    Type: AWS::EC2::VPC::Id
  pCreateNewSubnets:
    Description: Set to true if we want new subnets for our databases
    Type: String
    Default: "true"
    AllowedValues: ['true', 'false']
    ConstraintDescription: must be true or false.    
  pExistingSubnetId1:
    Description: Use an id of an existing private subnet
    Type: String
    Default: "subnet-0455e1cfa66facd17"
  pExistingSubnetId2:
    Description: Use an id of a second existing private subnet
    Type: String
    Default: "subnet-0536176832ba42dfd"
  pSubnetCIDR1:
    Description: The cidrblock to use?
    Type: String
  pSubnetCIDR2:
    Description: The cidrblock to use?
    Type: String
  pGatewayID:
    Description: The virtual private gateway id for the vpc, to manage our databases.
    Type: String
    Default: "vgw-0e7d969e316a7b5d5"
  pOperatorEMail:
    Description: EMail address to notify if there are any operational issues
    Type: String
    AllowedPattern: >-
      ([a-zA-Z0-9_\-\.]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([a-zA-Z0-9\-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?)
    ConstraintDescription: must be a valid email address.

# This next parameter is a hold-over from older pipelines.  It must exist, or there will be a cloudformation error.
  BuildBucket:
    Description: Unused in this template
    Type: String
    Default: ""


#  Warning, I haven't actually tested all these databases or these mappings - be ready to debug :-)
Mappings: 
  DBPortMap: 
    aurora-mysql: 
      port: 3306
    aurora-postgresql: 
      port: 5432
    mariadb: 
      port: 3306
    mysql: 
      port: 3306
    oracle-ee: 
      port: 1521
    oracle-se2: 
      port: 1521
    postgres: 
      port: 5432
    sqlserver-ee: 
      port: 1433
    sqlserver-se: 
      port: 1433
    sqlserver-ex: 
      port: 1433
    sqlserver-web: 
      port: 1433

Conditions: 
  CreateMultiAZ: !Equals [ !Ref pMultiAZ, "true" ]
  CreateNewSubnets: !Equals [ !Ref pCreateNewSubnets, "true" ]
  CreateAurora: !Or
    - !Equals [ !Ref pDBEngine, "aurora-mysql" ]
    - !Equals [ !Ref pDBEngine, "aurora-postgresql" ]
  CreateNonAurora: !Not [Condition: CreateAurora ]
  CreateAuroraMultiAZ: !And
    - Condition: CreateAurora
    - Condition: CreateMultiAZ

Resources:
  # TODO - SECURE THE PASSWARD PARAMETERS IN AWS PARAMETER STORE
  #   See: https://github.com/aws-samples/aws-aurora-cloudformation-samples/blob/master/cftemplates/Aurora-Postgres-DB-Cluster.yml 
  # TODO - DBClusterParameterGroups and DBParameterGroups are not auto-created.  


# Subnets
#
# Create one subnet if pCreateNewSubnets is true
# Create two new subnets if both pCreateNewSubnets and pMultiAZ are true...
# Otherwise, the user better have specified existing subnet(s) in pExistingSubnetId1 and pExistingSubnetId2
#
# Note that this template assumes we use Availability Zones 0 and 1.  Perhaps I should make those parameters?
#
  rDBSubnet1:
    Type: AWS::EC2::Subnet
    Condition: CreateNewSubnets
    Properties:
      VpcId: !Ref 'pRDSVPC'
      CidrBlock: !Ref 'pSubnetCIDR1'
      AvailabilityZone: !Select 
        - 0
        - !GetAZs 
          Ref: 'AWS::Region'
      Tags:
        - Key: "Name"
          Value: !Join [ "-", [!Ref "AWS::StackName", !Ref "AWS::Region", "RDS-AZA-SUBNET"] ]

  rDBSubnet2:
    Type: AWS::EC2::Subnet
    Condition: CreateNewSubnets
    Properties:
      VpcId: !Ref 'pRDSVPC'
      CidrBlock: !Ref 'pSubnetCIDR2'
      AvailabilityZone: !Select 
        - 1
        - !GetAZs 
          Ref: 'AWS::Region'
      Tags:
        - Key: "Name"
          Value: !Join [ "-", [!Ref "AWS::StackName", !Ref "AWS::Region", "RDS-AZB-SUBNET"] ]

# Route Table
#
# If we create new subnets, we need to use a route table.  I thought about reusing the existing route table for the 
# primary private subnet, but this is a different class of traffic.  I feel it is best practice to create new route tables.
#
  rCustomRouteTable:
    Type: AWS::EC2::RouteTable
    Condition: CreateNewSubnets
    Properties:
      VpcId: !Ref 'pRDSVPC'
      Tags:
        - Key: "Name"
          Value: !Join [ "-", [!Ref "AWS::StackName", !Ref "AWS::Region", "RTB-RDS"] ]

  rRoute10:
    Type: AWS::EC2::Route
    Condition: CreateNewSubnets
    Properties:
      RouteTableId: !Ref rCustomRouteTable
      DestinationCidrBlock: 10.0.0.0/8
      GatewayId: !Ref pGatewayID

  rRoute172:
    Type: AWS::EC2::Route
    Condition: CreateNewSubnets
    Properties:
      RouteTableId: !Ref rCustomRouteTable
      DestinationCidrBlock: 172.16.0.0/12
      GatewayId: !Ref pGatewayID

  rRoute192:
    Type: AWS::EC2::Route
    Condition: CreateNewSubnets
    Properties:
      RouteTableId: !Ref rCustomRouteTable
      DestinationCidrBlock: 192.168.0.0/16
      GatewayId: !Ref pGatewayID

 # attach route tables to our previously created subnets.
  rSubnetRouteTableAssociation1:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Condition: CreateNewSubnets
    Properties:
      SubnetId: !Ref rDBSubnet1
      RouteTableId: !Ref rCustomRouteTable

  rSubnetRouteTableAssociation2:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Condition: CreateNewSubnets
    Properties:
      SubnetId: !Ref rDBSubnet2
      RouteTableId: !Ref rCustomRouteTable

# Security group.
#
# This is to make sure our databases only allow traffic on the correct port.
#
# Technically, we could be more specific than 0.0.0.0/0

  rDBEC2SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Open database for access
      VpcId: !Ref pRDSVPC
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: !FindInMap
        - DBPortMap
        - !Ref 'pDBEngine'
        - port
        ToPort: !FindInMap
        - DBPortMap
        - !Ref 'pDBEngine'
        - port
        CidrIp: 0.0.0.0/0
        Description: !Ref 'pDBEngine'
      Tags:
        - Key: "Name"
          Value: !Join [ "-", [!Ref "AWS::StackName", !Ref "AWS::Region", "RDS-SEC"] ]

# Subnet Groups
# 
# Here is the deal.  We don't attach subnets directly to databases.  Rather, we group subnets and attach the group to the 
# databases.  So, here is the complex "if" logic to determine which subnets to put into our database subnet group.
  rDBSubnetGroup: 
    Type: "AWS::RDS::DBSubnetGroup"
    Properties: 
      DBSubnetGroupDescription: "description"
      SubnetIds: 
        - !If [CreateNewSubnets, !Ref rDBSubnet1, !Ref pExistingSubnetId1 ]
        - !If [CreateNewSubnets, !Ref rDBSubnet2, !Ref pExistingSubnetId2 ]

# Database Builds...
#
# Aurora builds... first
  rDatabaseCluster:
    Type: AWS::RDS::DBCluster
    Condition: CreateAurora
    Properties:
      DatabaseName: !Ref 'pDBName'
      Engine: !Ref 'pDBEngine'
      EngineVersion: !Ref pDBEngineVersion
      MasterUsername: !Ref 'pDBUser'
      MasterUserPassword: !Ref 'pDBPassword'
      DBSubnetGroupName: !Ref rDBSubnetGroup
      DBClusterParameterGroupName: !Ref pDBParameterGroupName
      DeletionProtection: !Ref 'pDatabaseDeletionProtection'
      VpcSecurityGroupIds: [!GetAtt [rDBEC2SecurityGroup, GroupId]]

  rAuroraDB1:
    Type: AWS::RDS::DBInstance
    Condition: CreateAurora
    Properties:
      CopyTagsToSnapshot: true
      DBInstanceClass: !Ref 'pDBInstanceClass'
      DBClusterIdentifier: !Ref rDatabaseCluster
      DBSubnetGroupName: !Ref rDBSubnetGroup
      Engine: !Ref 'pDBEngine'
      EngineVersion: !Ref pDBEngineVersion
      DBParameterGroupName: !Ref pDBParameterGroupName
      PubliclyAccessible: false
      LicenseModel: !Ref 'pDatabaseLicenseModel'

  rAuroraDB2:
    Type: AWS::RDS::DBInstance
    Condition: CreateAuroraMultiAZ
    Properties:
      CopyTagsToSnapshot: true
      DBInstanceClass: !Ref 'pDBInstanceClass'
      DBClusterIdentifier: !Ref rDatabaseCluster
      DBSubnetGroupName: !Ref rDBSubnetGroup
      Engine: !Ref 'pDBEngine'
      EngineVersion: !Ref pDBEngineVersion
      DBParameterGroupName: !Ref pDBParameterGroupName
      PubliclyAccessible: false
      LicenseModel: !Ref 'pDatabaseLicenseModel'

# Now for the Non-Aurora builds...
  rMasterDB:
    Type: AWS::RDS::DBInstance
    Condition: CreateNonAurora
    Properties:
      AllocatedStorage: !Ref 'pDBAllocatedStorage'
      CopyTagsToSnapshot: true
      DBInstanceClass: !Ref 'pDBInstanceClass'
      DBName: !Ref 'pDBName'
      DBSubnetGroupName: !Ref rDBSubnetGroup
      Engine: !Ref 'pDBEngine'
      EngineVersion: !Ref pDBEngineVersion
      DBParameterGroupName: !Ref pDBParameterGroupName
      MasterUsername: !Ref 'pDBUser'
      MasterUserPassword: !Ref 'pDBPassword'
      MultiAZ: !Ref 'pMultiAZ'
      PubliclyAccessible: false
      StorageType: !Ref 'pStorageType'
      VPCSecurityGroups: [!GetAtt [rDBEC2SecurityGroup, GroupId]]
      LicenseModel: !Ref 'pDatabaseLicenseModel'
      DeletionProtection: !Ref 'pDatabaseDeletionProtection'

# Cloudwatch alarms
#
# Nothing too special here.  I just make sure the DB is operational.
  rAlarmTopic:
    Type: 'AWS::SNS::Topic'
    Properties:
      Subscription:
        - Endpoint: !Ref pOperatorEMail
          Protocol: email

  rCPUAlarmHighMasterDB:
    Type: 'AWS::CloudWatch::Alarm'
    Condition: CreateNonAurora
    Properties:
      EvaluationPeriods: 10
      Statistic: Average
      Threshold: 50
      AlarmDescription: >-
        Alarm if CPU too high or metric disappears indicating the RDS database
        instance is having issues
      Period: 60
      Namespace: AWS/RDS
      MetricName: CPUUtilization
      Dimensions:
        - Name: DBInstanceIdentifier
          Value: !Ref rMasterDB
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - !Ref rAlarmTopic
      InsufficientDataActions:
        - !Ref rAlarmTopic

  rCPUAlarmHighAuroraDB1:
    Type: 'AWS::CloudWatch::Alarm'
    Condition: CreateAurora
    Properties:
      EvaluationPeriods: 10
      Statistic: Average
      Threshold: 50
      AlarmDescription: >-
        Alarm if CPU too high or metric disappears indicating the RDS database
        instance is having issues
      Period: 60
      Namespace: AWS/RDS
      MetricName: CPUUtilization
      Dimensions:
        - Name: DBInstanceIdentifier
          Value: !Ref rAuroraDB1
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - !Ref rAlarmTopic
      InsufficientDataActions:
        - !Ref rAlarmTopic

  rCPUAlarmHighAuroraDB2:
    Type: 'AWS::CloudWatch::Alarm'
    Condition: CreateAuroraMultiAZ
    Properties:
      EvaluationPeriods: 10
      Statistic: Average
      Threshold: 50
      AlarmDescription: >-
        Alarm if CPU too high or metric disappears indicating the RDS database
        instance is having issues
      Period: 60
      Namespace: AWS/RDS
      MetricName: CPUUtilization
      Dimensions:
        - Name: DBInstanceIdentifier
          Value: !Ref rAuroraDB2
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - !Ref rAlarmTopic
      InsufficientDataActions:
        - !Ref rAlarmTopic
# 
# Consider adding JDBC string...
#
Outputs:
  Name:
    Description: Aurora Stack Name
    Value: !Ref AWS::StackName
    Export:
      Name: !Sub ${AWS::StackName}-Name
  RDSEndPointAddress:
    Description: Database Endpoint Address
    Value: !GetAtt [rMasterDB, Endpoint.Address]
    Export:
      Name: !Sub ${AWS::StackName}-RDSEndPointAddress
    Condition: CreateNonAurora
  RDSEndPointPort:
    Description: Database Endpoint port
    Value: !GetAtt [rMasterDB, Endpoint.Port]
    Export:
      Name: !Sub ${AWS::StackName}-RDSEndPointPort
    Condition: CreateNonAurora
  AuroraClusterId:
    Description: Aurora Cluster ID
    Value: !Ref rDatabaseCluster
    Export:
      Name: !Sub ${AWS::StackName}-AuroraClusterID
    Condition: CreateAurora
  AuroraEndPointAddress:
    Description: Database Endpoint Address
    Value: !GetAtt [rDatabaseCluster, Endpoint.Address]
    Export:
      Name: !Sub ${AWS::StackName}-AuroraDatabaseURL
    Condition: CreateAurora
  AuroraEndPointPort:
    Description: Database Endpoint port
    Value: !GetAtt [rDatabaseCluster, Endpoint.Port]
    Export:
      Name: !Sub ${AWS::StackName}-AuroraDatabasePort
    Condition: CreateAurora
  EndPointDBName:
    Description: Database Name
    Value: !Ref 'pDBName'
    Export:
      Name: !Sub ${AWS::StackName}-DBName
  JDBCConnectionString:
    Description: JDBC connection string for a mysql database
    Value: !Join ['', ['jdbc:mysql://', !GetAtt [rDatabaseCluster, Endpoint.Address], ':', !GetAtt [
          rDatabaseCluster, Endpoint.Port], /, !Ref 'pDBName']]
    Export:
      Name: !Sub ${AWS::StackName}-MySQLJDBCString
    Condition: CreateAurora
12
  • Can you please add a screenshot of the all the failed events listed under the Events tab? Commented Mar 28, 2019 at 21:59
  • Absolutely. Note that the order of events was as follows: 1) deployed a stack with routes that included VPG. 2) someone manually changed the VPG and manually altered the route tables. 3) updated stack to have correct route tables. 4) stack update failed (Reason: "Route did not stabilize in expected time") 5) stack went into "UPDATE_ROLLBACK_FAILED" 6) now, we can't get out of that status. Commented Mar 28, 2019 at 22:23
  • Are you using nested stacks, can you post your CloudFormation yaml here (obfuscate any critical info)? Commented Mar 28, 2019 at 22:33
  • No nested stacks... Commented Mar 28, 2019 at 22:45
  • 1
    Nope, it is not a bug I think. CLI just shows the programmatic way of doing it so that would be more accurate anyway. I think the console just takes the input without quotes and [ ]. I used the option a while ago so don't remember but will reproduce this and test it from the console as well. Commented Mar 29, 2019 at 9:54

2 Answers 2

10

Step 1: Bring stack back to "UPDATE_ROLLBACK_COMPLETE" state by following below steps

  • Goto AWS CloudFormation and select the stack you want to update, choose Actions, and then choose Continue Update Rollback
  • Then include the failing resource identifier under Advanced -> Resources to Skip

Step 2 : Fixed the issue which cause the deployment fails and deploy stack again

0
5

So after a little bit of digging, this seems to be the best way to do it.
AWS CLI documentation is a bit more clear on this and expects syntax for logical IDs in this format as per the link above: "string" "string" ... with continue-update-rollback operation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.