2

Facts

  • Database: PostgreSQL (latest)
  • Programming language: Java

Problem statement (simplified)

We have 2 tables - overview and details. There could be millions of rows in "overview" and each row of "overview" can have millions of rows associated with it in "details". The foreign key details.overview_id refers to overview.id. Most queries are of the general form

SELECT * FROM details WHERE overview_id = xxx AND details.id > yyy AND details.id < zzz;

If we have a single table for details, the queries will be too slow (although the queries on details are almost always on primary keys).

More on the nature of DB activities: INSERT and UPDATE on overview happens infrequently. INSERT on details happen at a rapid pace, while UPDATE on the same table almost never happens and bulk DELETE happens sometimes.

What we already have

In the past we used raw SQL to partition the table "details" against each row in "overview". (In practice, we did not actually partition, instead we created new tables based on a template. These tables did not have any column called overview_id (saving storage space), instead we had a separate table that did the mapping between overview.id and the table-name of the specific partition table.) So, as you can understand, the partitions had to be generated on the fly as new rows were inserted in overview and partitions were dropped as rows were deleted from overview. All of this was managed inside the application. The application-database interaction has been blazing fast, but the application code is fairly complex, implying it is hard to maintain. Also, with raw SQL lying around everywhere, it is hard to scale the DB horizontally - we have to reinvent what most JPA providers have already done.

Current goal

Currently we are exploring options for a mechanism by which this partitioning can happen behind the scene - possibly by a JPA provider (I understand that this is not part of the JPA spec), so that we can focus on the application while the underlying framework/layer takes care of the scalability issues.

I looked at openJPA Slice and EclipseLink. Both of them provide partition (shard) management across hosts. We certainly need that. But we also need partition management within a single host. However, if there is a better or more elegant solution to this or if there is a totally different angle to look at this, I will be really glad to know about that.

I will appreciate any insight you can provide.

Thanks.
Prajesh

2
  • what have you done with regards to indexing details on the parent overview foreign key? If you're always accessing a specific subset, an index should help. Jul 23, 2012 at 17:05
  • @wrschneider99 I think I understand what you are suggesting. Data for any overview_id can be requested at any time, including single requests for multiple overview_id's. So, I guess, not much joy there. Jul 23, 2012 at 17:32

2 Answers 2

2

Have you looked into using Postgres's table partitioning?

http://www.postgresql.org/docs/9.1/static/ddl-partitioning.html

1
0

Thank you all for your comments/answers till date. We decided to stick to what we already have (see the section named "what we already have"), with minor modifications.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.