9

I am trying to write csv data from S3 to DynamoDB using AWS Lambda function. I am currently receiving the following error 'BatchWriteItem operation: The provided key element does not match the schema'.

Is there a quick fix to this problem?

import boto3

s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')

def csv_reader(event, context):

    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']

    obj = s3.get_object(Bucket=bucket, Key=key)

    rows = obj['Body'].read().split('\n')

    table = dynamodb.Table('customer_id')

    with table.batch_writer() as batch:
        for row in rows:
            batch.put_item(Item={

                'Customer-ID':row.split(',')[0],
                'Name':row.split(',')[1]
            })

Error looks like this:

An error occurred (ValidationException) when calling the BatchWriteItem operation: The provided key element does not match the schema: ClientError Traceback (most recent call last):

File "/var/task/lambda_function.py", line 22, in csv_reader 'Name':row.split(',')[1]

File "/var/runtime/boto3/dynamodb/table.py", line 156, in exit self._flush()

File "/var/runtime/boto3/dynamodb/table.py", line 137, in _flush RequestItems={self._table_name: items_to_send})

File "/var/runtime/botocore/client.py", line 314, in _api_call return self._make_api_call(operation_name, kwargs)

File "/var/runtime/botocore/client.py", line 612, in _make_api_call raise error_class(parsed_response, operation_name)

ClientError: An error occurred (ValidationException) when calling the BatchWriteItem operation: The provided key element does not match the schema

I would expect the data to write from the CSV file to the DynamoDB. Completely new to AWS & Python so any help would be appreciated.

2
  • If it says The provided key element does not match the schema, you should provide the schema, and maybe some example input you're using
    – ChatterOne
    May 13, 2019 at 7:49
  • 2
    I received this error because I was passing data in a dynamodb format but this method expects pure values, i.e., it wants items in the form {'pk': 'abcd', 'sk': 'efgh'} as opposed to {'pk': {'S': 'abcd'}, 'sk': {'S': 'efgh'}} Oct 25, 2022 at 16:22

2 Answers 2

8

When putting an item you must provide all the mandatory fields, otherwise it will fail with the mentioned exception.

Double check your Table schema and check the mandatory values: at least a Hash Key is mandatory. If you have configured a Hash Key with a Range key, then you must provide both values. Without further information from your table it is difficult for us to guess, but I will try to summarise the problem in the next couple of lines. If it still does not suffice, please update your question with the Table Schema.

So let's say your HashKey is named id, your code should look like this:

batch.put_item(Item={
            'id': 'some-id'
            'Customer-ID':row.split(',')[0],
            'Name':row.split(',')[1]
        })

If a range key is also set and it's named, let's say, test, then your code should look like this:

batch.put_item(Item={
            'id': 'some-id',
            'test': 'some-value',
            'Customer-ID':row.split(',')[0],
            'Name':row.split(',')[1]
        })

If you're not sure how Hash and Range key work together, I suggest you read the docs

0

You're probably getting this error because your CSV contains numbers. What you have to remember is that when you do row.split(','), it splits the comma-separated values, and they are strings at this point. Thus, your primary key which you've probably declared as an int is getting a string value.

Do:

'Customer-ID':int(row.split(',')[0])

That should solve your problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.