I know the whole design should be based on natural aggregates (documents), however, I'm thinking to implement a separate table for localisations (lang, key, text) and then use keys in other tables. However, I was unable to find any example on doing this.
7 Answers
You are correct, DynamoDB is not designed as a relational database and does not support join operations. You can think about DynamoDB as just being a set of key-value pairs.
You can have the same keys across multiple tables (e.g. document_IDs), but DynamoDB doesn't automatically sync them or have any foreign-key features. The document_IDs in one table, while named the same, are technically a different set than the ones in a different table. It's up to your application software to make sure that those keys are synced.
DynamoDB is a different way of thinking about databases and you might want to consider using a managed relational database such as Amazon Aurora: https://aws.amazon.com/rds/aurora/
One thing to note, Amazon EMR does allow DynamoDB tables to be joined, but I'm not sure that's what you're looking for: http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/EMRforDynamoDB.html
-
1Thanks! Having a join would be an exception rather than a design rule/requirement. Do you have any thoughts on performance/billing/.. penalties when querying documents by one and then joining on application side? I still think DynamoDB will be a better fit in my case, however I don't know will there be any significant disadvantage for exceptional cases like this. Commented Apr 21, 2016 at 6:36
-
2@Centurion, if you know your document_id (or similar) beforehand, then you can get just that associated record from each table. Getting a single record from each table is not expensive and joining them after seems very reasonable. The expensive stuff happens when you need to query or scan: DynamoDB charges for each record retrieved in the scan / query, even if you don't actually return them to the application. Whenever you're querying or scanning, that presents an opportunity to examine that operation in more detail to try and eliminate the query / scan. Commented Apr 21, 2016 at 14:40
With DynamoDB, rather than join I think the best solution is to store the data in the shape you later intend to read it.
If you find yourself requiring complex read queries you might have fallen into the trap of expecting DynamoDB to behave like an RDBMS, which it is not. Transform and shape the data you write, keep the read simple.
Disk is far cheaper than compute these days - don't be afraid to denormalise.
-
1
Update: This answer is well within the defined community guidelines and not a non-answer speaking only about a commercial solution.
One solution I have seen come up multiple times in this space is to sync from DynamoDB into a separate database that is more well suited for the types of operations you're looking for.
I wrote a blog about this topic comparing various approaches I've seen people take to this very problem, but I'll summarize some of the key takeaways here so you don't have to read all of it.
DynamoDB secondary indexes
What's good?
- Fast and no other systems needed!
- Good for a very specific analytic feature you're building (like a leaderboard)
Considerations
- Limited # of secondary indexes, limited fidelity of queries
- Expensive if you're depending on scans
- Security and performance concerns using production database directly for analytics
DynamoDB + Glue + S3 + Athena
What's good?
- All components are “serverless” and require no provisioning of infrastructure
- Easy to automate ETL pipeline
Considerations
- High end-to-end data latency of several hours, which means stale data
- Query latency varies between tens of seconds to minutes
- Schema enforcement can lose information with mixed types
- ETL process can require maintenance from time to time if structure of data in source changes
DynamoDB + Hive/Spark
What's good?
- Queries over latest data in DynamoDB
- Requires no ETL/pre-processing other than specifying a schema
Considerations
- Schema enforcement can lose information when fields have mixed types
- EMR cluster requires some administration and infrastructure management
- Queries over the latest data involves scans and are expensive
- Query latency varies between tens of seconds to minutes directly on Hive/Spark
- Security and performance implications of running analytical queries on an operational database
DynamoDB + AWS Lambda + Elasticsearch
What's good?
- Full-text search support
- Support for several types of analytical queries
- Can work over the latest data in DynamoDB
Considerations
- Requires management and monitoring of infrastructure for ingesting, indexing, replication, and sharding
- Requires separate system to ensure data integrity and consistency between DynamoDB and Elasticsearch
- Scaling is manual and requires provisioning additional infrastructure and operations
- No support for joins between different indexes
DynamoDB + Rockset
What's good?
- Completely serverless. No operations or provisioning of infrastructure or database required
- Live sync between DynamoDB and the Rockset collection, so that they are never more than a few seconds apart
- Monitoring to ensure consistency between DynamoDB and Rockset
- Automatic indexes built over the data enabling low-latency queries
- SQL query serving that can scale to high QPS
- Joins with data from other sources such as Amazon Kinesis, Apache Kafka, Amazon S3, etc.
- Integrations with tools like Tableau, Redash, Superset, and SQL API over REST and using client libraries.
- Features including full-text search, ingest transformations, retention, encryption, and fine-grained access control
Considerations
- Not a great fit for storing rarely queried data (like machine logs)
- Not a transactional datastore
(Full Disclosure: I work on the product team @ Rockset) Check out the blog for more details on the individual approaches.
-
2Fantastic response... this is an area I have been wondering about and will also check out Rockset! Commented Jun 13, 2019 at 19:30
You must query the first table, then iterate through each item with a get request on the next table.
The other answers are unsatisfactory as 1) don't answer the question and, more importantly, 2) how can you design your tables in advance to knowing their future application? The technical debt is just too high to reasonably cover unbounded future possibilities.
My answer horribly inefficient but this is the only current solution to the posed question.
I eagerly await a better answer.
-
3
-
You really should be working backwards from the customer and designing your tables based on their future application rather than having to build some ugly acrobatics in code to fit your data that doesn't consider how the clients will ask for it.– Anna TCommented Jul 13, 2022 at 9:47
I know that my response is slightly late, by a couple of years. However, I was able to dig up some additional information, regarding Amazon DynamoDB & Joins, which might benefit you (or perhaps another individual, who may stumble upon this discussion, while researching this information, in the future).
To get to the point, I was able to locate some documentation on the Amazon DynamoDB Website, which states that the Apache HiveQL Query Language can be utilized, to perform Joins on Amazon DynamoDB Tables, Columns & Data, etc.
Querying Data in DynamoDB (w/ HiveQL): https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EMRforDynamoDB.Querying.html
Working w/ Amazon DynamoDB & Apache Hive: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EMRforDynamoDB.Tutorial.html
Processing Amazon DynamoDB Data with Apache Hive on Amazon EMR: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EMRforDynamoDB.html
I hope this information helps someone out, if not the original poster.
-
Careful! Even Hive can't do magic, it's just a convenience layer on top, with some caching built-in. If you want to aggregate data across multiple DynamoDB entities, you will incur read cost for each of these entities. You might even end up with expensive scan operations. See docs.aws.amazon.com/amazondynamodb/latest/developerguide/… Commented May 1, 2019 at 20:45
-
Thank you for the Tip as well as the Link to the Developer Guide. I greatly appreciate your feedback.– MattiCommented Aug 5, 2021 at 3:42
When I have needed to do this I have made use of pandas in python to do the joins across tables in memory.
Its not ideal as like already said, dynamo DB is not a relational database, but there are times when you need to do something like maintain mapping between ID's in two tables and if this happens to you, using a library like pandas along with the SDK can help you out.
I have an application I am using dynamo DB on that I now wish I just opted to use postgres for.
Recently I have the same requirement to use join and aggregate function like avg and sum with dynamoDb, to solve this I used the Cdata JDBC driver and it worked perfectly. It support join as well aggregate functions. Although, I am also searching for the solution to avoid using cdata because of license cost of Cdata.