2

I am trying to use Dr Brian Gladman's famous AES encryption routines, in this case the Galois Counter Mode version, and I'm getting a segfault and can't find the cause.

I've written a little demo program that shows what I'm trying to do and in fact it appears to work, but at the end it segfaults.

I am a little rusty in C, so the error may be obvious to wiser heads.

Here's my program, which I hope is as short and readable as possible:

/* This program tries to be a demo for Dr Brian Gladman's AES-GCM encryption code.
   The plan is to use AES-GCM to encrypt a short message, and provide an authentication key.
   And then to decrypt the ciphertext, verifying the authentication tag on the way.*/

#include<stdio.h>
#include<assert.h>

#include "aes-modes/gcm.h"

/*note that uint_8t is Dr Gladman's type, not to be confused with uint8_t from C99*/

void print_uint_8t_array(uint_8t *a, int len, char* name)
{
  int i;
  printf("%-10s:", name);

  for(i=0; i< len; i++){
    if(a[i]==0x00) printf("-");
    else if(a[i]>=0x20 && a[i]<=0x7e) printf("%c", a[i]); 
    else printf("~");
  }
  printf("\n");
}

#define p8(name) (print_uint_8t_array((name), (sizeof(name)), (#name)))

/*Dr Gladman's functions return 0,1 or -1, to be interpreted so:*/
void interpret_retval(AES_RETURN retval){
  switch(retval){
  case      0: 
    printf("RETURN_GOOD\n");
    break;
  case 1:
    printf("RETURN_WARN\n");
    break;
  case -1:
    printf("RETURN_ERROR\n");
    break;
  default: printf("Unknown return value");
    assert(0);
    break;
  }
}

int main(void){
  printf("GCM-AES encryption/decryption/authentication example\n");

  aes_init(); 
  /*I think that this is only needed if you've compiled the libraries to make up
    their tables at runtime. If the tables are compiled in, then it's
    unnecessary but probably doesn't do any harm.*/

  uint_8t key[32]="01234567890123456789012345678901"; 
  uint_8t iv[32]="01234567890123456789012345678901";

  uint_8t header[]="Here is a header, which is not to be encrypted, but which is nevertheless to be proved to have been placed in the message by someone who knows the key.";
  uint_8t message[]="Here is some text which is to be protected from the prying eyes of those who do not know the key, whilst at the same time it is to carry along with it an unencrypted header, and a with them both a tag proving beyond reasonable doubt that the two were packaged together and run through the aes-gcm algorithm by someone who knew the key."; 
  uint_8t tag[32]="01234567890123456789012345678901";

  void printtexts(void)
  {
    printf("-----------------------------------\n");
    p8(key);
    p8(iv);
    printf("-----------------------------------\n");
    p8(header);
    p8(message);
    p8(tag);
    printf("-----------------------------------\n");
  }

  printtexts();

  {
    gcm_ctx ecx[1];
    printf("encrypting....\n");
    gcm_init_and_key(key, sizeof(key), ecx);
    interpret_retval(gcm_encrypt_message(iv, sizeof(iv), header, sizeof(header), message, sizeof(message), tag, sizeof(tag), ecx));
    gcm_end(ecx);
    printf("done\n");
  }

  printtexts();

  {
    gcm_ctx dcx[1];

    printf("decrypting....\n");
    gcm_init_and_key(key, sizeof(key), dcx);
    interpret_retval(gcm_decrypt_message(iv, sizeof(iv), header, sizeof(header), message, sizeof(message), tag, sizeof(tag), dcx));
    gcm_end(dcx);
    printf("done\n");
  }

  printtexts();

  return 0;

}

And here's the result. The encryption/decryption and tagging all appear to work, but then there's a bad return code and a segfault.

jla@jaspden-desktop$ ./test3
GCM-AES encryption/decryption/authentication example
-----------------------------------
key       :01234567890123456789012345678901
iv        :01234567890123456789012345678901
-----------------------------------
header    :Here is a header, which is not to be encrypted, but which is nevertheless to be proved to have been placed in the message by someone who knows the key.-
message   :Here is some text which is to be protected from the prying eyes of those who do not know the key, whilst at the same time it is to carry along with it an unencrypted header, and a with them both a tag proving beyond reasonable doubt that the two were packaged together and run through the aes-gcm algorithm by someone who knew the key.-
tag       :01234567890123456789012345678901
-----------------------------------
encrypting....
RETURN_GOOD
done
-----------------------------------
key       :01234567890123456789012345678901
iv        :01234567890123456789012345678901
-----------------------------------
header    :Here is a header, which is not to be encrypted, but which is nevertheless to be proved to have been placed in the message by someone who knows the key.-
message   :~(}Hr~|o~~:~D~~~~3~~T~C~C(l~~~'~~~UC~D~~~~&O~ ~L~-~~~E&~~~~!~~~~~K~~~~M<|l%<ho~"~[A~0~~O~3T~%8K~~~L~~~~~~~~~7~~~~~~~~e~~~~;~392~8<~~ ~~v,E~~~~~~~~~~~W~yd~~C$H~~~*r~~~_~~~~O~u~h\9s~~`%~~~~~&~~~~~~.~~Q~~~Y~Ix~A~~~a5~~~~|~~9J~~~~~~~~ ~~ZOiX~~~~~~f~~`e~~~Ju~Z~~~X~g~OX~C~~~~~~~~~(~~gyT~~x~L~4>~Z~~ge~~~08~;~@~s~3~~WA~~~~Z-~~~~/~~~~~~Tj~bL~~
tag       :|~~,~2~~Ec~~~A~~~~~_j{~~~~`~~~u~
-----------------------------------
decrypting....
RETURN_ERROR
done
-----------------------------------
key       :01234567890123456789012345678901
iv        :01234567890123456789012345678901
-----------------------------------
header    :Here is a header, which is not to be encrypted, but which is nevertheless to be proved to have been placed in the message by someone who knows the key.-
message   :Here is some text which is to be protected from the prying eyes of those who do not know the key, whilst at the same time it is to carry along with it an unencrypted header, and a with them both a tag proving beyond reasonable doubt that the two were packaged together and run through the aes-gcm algorithm by someone who knew the key.-
tag       :|~~,~2~~Ec~~~A~~~~~_j{~~~~`~~~u~
-----------------------------------
Segmentation fault (core dumped)
2
  • 2
    If I'm reading the output right, it looks like the segfault is happening right at the end, after the last call to printtexts(). Perhaps something is corrupting the stack? Note that you're getting an error during decryption -- figure out what's causing that, and maybe your segfault will also be solved. Oct 6, 2011 at 19:00
  • Thanks, that's excellent advice, although in fact I ended up finding where the stack was getting smashed, and that led to the 'key too large for local variable in library' source of the problem. Googling then led to the fact that the AES-GSM tag is limited to 16 bytes. Oct 6, 2011 at 19:57

1 Answer 1

4

Self-answering, in case anyone else is interested in using these routines.

Apparently the maximum tag size for AES-GSM is 16 bytes. Change the tag-length to 16 or less (I have it as 32 here), and everything works fine.

This limit appears to be hard-coded into gcm_decrypt_message as BLOCK_SIZE. Changing that to 32 also seems to work, but the Lord alone knows what that does to the cryptographic properties of the algorithm.

On a debugging note, gcc has an option -fstack-protector-all which can catch libraries in the act of smashing the return stack, which was what was happening here.

As far as I can tell, the segfault is caused when main tries to return to a completely fictional address which has been written onto the stack by the library.

1
  • 2
    The GCM tag can be maximally one full block size of the block cipher. As AES has 16-byte blocks, more than that is not possible. (I didn't look at the implementation to see what effect a larger size would have, but certainly you wouldn't have the standard AES-GCM.) Oct 7, 2011 at 0:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.