0

I have a model to which I have to record a PositiveSmallIntegerField to the object, that is updated daily with the relevant score.

class Student(models.Model):
    name = models.CharField(max_length=20)
    grade = models.ForeignKey(Grade)
    rank = ??

The number of objects with this model will never exceed 100 and the scores/ranks must be retained for a period of 180 days. The database is Postgresql 9.2.

The rank is calculated daily on the score from another app, which I want to store in the database related to the student model, where I'm stuck with the model design, I have no Idea, what should be done for the ranks? Is there a repeating field in Django?

Any clues or experiences will be much appreciated

thanks.

Update:(Adding an example)

The database must look something like this,

+---------+-------+----------+----------+----------+----------+----------+----------+
| Student | Grade | 08-01-15 | 08-02-15 | 08-03-15 | 08-04-15 | 08-05-15 | 08-06-15 |
+---------+-------+----------+----------+----------+----------+----------+----------+
| Alex    |     5 |        2 |        1 |        1 |        2 |        3 |        2 |
| John    |     5 |        3 |        2 |        3 |        4 |        2 |        4 |
| Susan   |     5 |        1 |        4 |        2 |        1 |        1 |        1 |
| Zara    |     5 |        4 |        3 |        4 |        3 |        4 |        3 |
+---------+-------+----------+----------+----------+----------+----------+----------+

The rank of the student must be stored for the days like shown here, for the day 1, the ranks must be stored in a column/anything similar, and the number of days must go on for the consecutive 180 days, the ranks for each day must be added to the consecutive days.

I'm not stuck with the save method, but about the field where to save the calculated ranks.

8
  • Maybe this can fit your need.
    – Gocht
    Sep 6, 2015 at 20:45
  • @Gocht the source looks like a counter which logs PATH_INFO and REMOTE_ADDR, a visitor counter; while I'm trying to store an Integer for all the model objects daily as per the scores from examinations, attendance and more.
    – user5170375
    Sep 6, 2015 at 20:49
  • What's the problem with a simple update? Studen.objects.get(pk=pk).update(rank=F('rank')+score). rank's value starts in 0, I guess.
    – Gocht
    Sep 6, 2015 at 20:52
  • 1
    @Gocht does this overwrite the value? I want the rank to be accessible for a period of 180 days!
    – user5170375
    Sep 6, 2015 at 20:57
  • 1
    So my friend, you need store every rank for 180 days, you need a new table with the ranks related to a Student object and a created field to control 180 days. Were you looking for a different solution?
    – Gocht
    Sep 6, 2015 at 21:06

4 Answers 4

7
+50

I would suggest something similar to what e4c5 suggested, but I would also:

  • Generate an index on the date of the ranks so that obtaining all the ranks on any single day can be optimized.

  • Mark the date and student as unique_together. This prevents the possibility of recording two ranks for the same student on the same date.

The models would look like this:

from django.db import models

class Grade(models.Model):
    pass  # Whatever you need here...

class Student(models.Model):
    name = models.CharField(max_length=20)
    grade = models.ForeignKey(Grade)

class Rank(models.Model):

    class Meta(object):
        unique_together = (("date", "student"), )

    date = models.DateField(db_index=True)
    student = models.ForeignKey(Student)
    value = models.IntegerField()

In a full-fledged application I'd also expect to have some uniqueness constraints on Grade and Student but the problem presented in the question does not provide enough details about these models.

You could then run a task every day with cron or whatever task manager you want to use (Celery is also an option), to run a command like the following that would update the ranks according to some computation and purge the old records. The following code is an illustration of how it can be done. The real code should be designed to be generally idempotent (the following code is not because the rank computation is random) so that if the server is rebooted in the middle of an update, the command can just be rerun. Here's the code:

import random
import datetime
from optparse import make_option
from django.utils.timezone import utc

from django.core.management.base import BaseCommand
from school.models import Rank, Student

def utcnow():
    return datetime.datetime.utcnow().replace(tzinfo=utc)

class Command(BaseCommand):
    help = "Compute ranks and cull the old ones"
    option_list = BaseCommand.option_list + (
        make_option('--fake-now',
                    default=None,
                    help='Fake the now value to X days ago.'),
    )

    def handle(self, *args, **options):
        now = utcnow()
        fake_now = options["fake_now"]
        if fake_now is not None:
            now -= datetime.timedelta(days=int(fake_now))
            print "Setting now to: ", now

        for student in Student.objects.all():
            # This simulates a rank computation for the purpose of
            # illustration.
            rank_value = random.randint(1, 1000)
            try:
                rank = Rank.objects.get(student=student, date=now)
            except Rank.DoesNotExist:
                rank = Rank(
                    student=student, date=now)
            rank.value = rank_value
            rank.save()

        # Delete all ranks older than 180 days.
        Rank.objects.filter(
            date__lt=now - datetime.timedelta(days=180)).delete()

Why not pickles?

Multiple reasons:

  1. It is a premature optimization, and overall probably not an optimization at all. Some operations may be faster, but other operations will be slower. If the ranks are pickled into a field on Student then, loading a specific student in memory means loading all the rank information into memory together with that student. This can be mitigated by using .values() or .values_list() but then you are no longer getting Student instances from the database. Why have Student instances in the first place and not just access the raw database?

  2. If I change the fields in Rank, Django's migration facilities easily allow performing the needed changes when I deploy the new version of my application. If the rank information is pickled into a field, I have to manage any structure change by writing custom code.

  3. The database software cannot access values in a pickle and so you have to write custom code to access them. With the model above, if you want to list students by rank today (and the ranks for today have already been computed), then you can do:

    for r in Rank.objects.filter(date=utcnow()).order_by("value")\
        .prefetch_related():
        print r.student.name
    

    If you use pickles, you have to scan all Students and unpickle the ranks to extract the one for the day you want, and then use a Python data structure to order the students by rank. Once this is done, you then have to iterate over this structure to get the names in order.

1
  • If you want a little nonrel you can do so on the bleeding edge of Django 1.9 with postgres. I have a database I'm converting from mongo to postgres and it doesn't quite fit in the relational model. For convenience I'll be making an extra_data JSONB in all my models from now on (some place to stick incidentals till I figure out the best way to structure the tables). Sep 25, 2015 at 17:25
2

There is a much better way. Use redis Sorted sets

With sorted sets you can add, remove, or update elements in a very fast way (in a time proportional to the logarithm of the number of elements). Since elements are taken in order and not ordered afterwards, you can also get ranges by score or by rank (position) in a very fast way. Accessing the middle of a sorted set is also very fast, so you can use Sorted Sets as a smart list of non repeating elements where you can quickly access everything you need: elements in order, fast existence test, fast access to elements in the middle! In short with sorted sets you can do a lot of tasks with great performance that are really hard to model in other kind of databases

How to make use of this feature? Make a sorted set for each of the 180 days that you are interested in. You can either use the string representation of the date or just count them up like day1, day2 etc. Then when you calculate the rank for each user. Add it into the redis set (code borrowed from e4c5's answer).

def save(self, *args, **kwargs):
    r = redis.Redis()
    super(Exam,self).save(*args,**kwargs)
    student = Student.objects.get(id = self.student_id)
    rank = some_calculation
    student.save()

    r.zadd('dayx', self.name, self.rank)

Then you can retrieve rankings for any given day by r.zrange('dayx'). And really other than the imports and whatever logic to calculate the rankings. This is all there is to it.

1

Updating my answer after you updated the question with an example.

You should not do this with one table, you need two. ONe should be the student model which would look like this.

class Student(models.Model):
    name = models.CharField(max_length=20)
    grade = models.ForeignKey(Grade)

The other would be the rank model which might look like this.

class Rank(models.Model):
    data = models.DateField()
    student = models.ForeignKey(Student)

The following information is for the original question but parts of it will still be relevant I think.

1) Override save method in the Student model.

def save(self, *args, **kwargs):
    super(Exam,self).save(*args,**kwargs)
    student = Student.objects.get(id = self.student_id)
    student.rank = some_calculation
    student.save()

2) Use the post_save signal on the Exam object.

Similar to above

3) Use a trigger.

Since you are using postgresql, you can use the much more elegant solution of creating an AFTER INSERT trigger

8
  • oops the question in misinterpreted, have added an example in the question. please have a look. thanks
    – user5170375
    Sep 7, 2015 at 7:53
  • Oops, sorry to tell you but I think you are completely on the wrong track with your database design. Updating my answer
    – e4c5
    Sep 7, 2015 at 8:45
  • I guess this method increase the number of relationships and associated objects linearly, I think dumping a json with date and rank to a models.TextField is much efficient, hence storing much data than 180 days!.
    – user5170375
    Sep 7, 2015 at 18:01
  • 1
    There is no reason to shy away from increasing the number of relations in a database. Storing a JSON will be very inefficient (unless you are using the JSONB data type in postgres) if you want to retrieve data only for a particular day and comparisions will kill you.
    – e4c5
    Sep 8, 2015 at 0:01
  • 1
    You are forgetting that databases have indexes.
    – e4c5
    Sep 8, 2015 at 11:06
0

If you really want to have just an object field instead of joining two objects you can use something like

https://github.com/shrubberysoft/django-picklefield

serializing into a single text field a dictionary where key/val is date/rank

( this library is quite old, I don't think it will work out of the box with a modern django project )

Anyway, unless you are really forced by some external constraint to use this ugly solution the better (right) way is to just join two meaningful objects like @e4c5 suggested.

I once used such a field in a project where auth_user was a db view shared between multiple projects. That way I asked for just one migrate on the central db to add the "pickled" field and I was then able to add every user option i wished without tampering the original model further.

Anyway:

  1. You can't query for stuff like "average rank on 12 september" with Django ORM.
  2. It's not faster, you are not taking into account the time to depickle the text string into an object, quite sure is slower, probably is way slower.

In my case I had just a couple of options per user without need to query on them, more than that I think you're asking for future troubles mantaining your code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.