8

I've spent hours searching for examples of how to use the bsddb module and the only ones that I've found are these (from here):

data = mydb.get(key)
if data:
    doSomething(data)
#####################
rec = cursor.first()
while rec:
    print rec
    rec = cursor.next()
#####################
rec = mydb.set()
while rec:
    key, val = rec
    doSomething(key, val)
    rec = mydb.next()

Does anyone know where I could find more (practical) examples of how to use this package?

Or would anyone mind sharing code that they've written themselves that used it?

Edit:

The reason I chose the Berkeley DB was because of its scalability. I'm working on a latent semantic analysis of about 2.2 Million web pages. My simple testing of 14 web pages generates around 500,000 records. So doing the math out... there will be about 78.6 Billion records in my table.

If anyone knows of another efficient, scalable database model that I can use python to access, please let me know about it! (lt_kije has brought it to my attention that bsddb is deprecated in Python 2.6 and will be gone in 3.*)

1

5 Answers 5

8

These days, most folks use the anydbm meta-module to interface with db-like databases. But the API is essentially dict-like; see PyMOTW for some examples. Note that bsddb is deprecated in 2.6.1 and will be gone in 3.x. Switching to anydbm will make the upgrade easier; switching to sqlite (which is now in stdlib) will give you a much more flexible store.

9
  • but how scalable is SQLLite? One of the reasons I chose to use the Berkeley DB was because "Berkeley DB scales up extremely well. It can manage multi-terabyte tables with single records as large as four gigabytes."
    – tgray
    Commented Apr 1, 2009 at 20:48
  • I think sqlite can handle databases up to 2TB, though I haven't pushed it nearly that far myself. Your quote seems to come from Oracle's db documentation. I don't believe that that has much to do with the implementations supported by Python. What exactly are you trying to do?
    – lt_kije
    Commented Apr 1, 2009 at 21:14
  • Ah -- your new comment helps. ;) At that scale, I think you're best off using an RDBMS (PostgreSQL, MySQL, etc). SQLite will be a good starting place, since it provides a DBAPI interface that will be compatible with the major RDBMS connectors in Python.
    – lt_kije
    Commented Apr 1, 2009 at 21:17
  • 8
    bsddb is deprecated only because it was too difficult for the python team to maintain, it is still going to be developed as an external module. SQLLite is a SQL Database and as such has more overhead than bsddb
    – Ed L
    Commented Jul 31, 2009 at 16:34
  • 5
    In my experience working with 33GB of key-blobvalue data, I found Berkely DB was several times faster to build the store and iterate over the store than SQLite.
    – Grover
    Commented Jan 12, 2012 at 16:02
5

Look at: Lib3/bsddb/test after downloading the source from http://pypi.python.org/pypi/bsddb3/

The current distribution contains the following tests that are very helpful to start working with bsddb3:

test_all.py
test_associate.py
test_basics.py
test_compare.py
test_compat.py
test_cursor_pget_bug.py
test_dbenv.py
test_dbobj.py
test_db.py
test_dbshelve.py
test_dbtables.py
test_distributed_transactions.py
test_early_close.py
test_fileid.py
test_get_none.py
test_join.py
test_lock.py
test_misc.py
test_pickle.py
test_queue.py
test_recno.py
test_replication.py
test_sequence.py
test_thread.py
1
  • The tests are sometimes the only documentation in some projects I've seen.
    – gak
    Commented Oct 9, 2012 at 23:50
5

I'm assuming this thread is still active, so here we go. This is rough code and there's no error checking, but it may be useful as a starting point.

I wanted to use PHP's built-in DBA functions and then read the database using a Python (2.x) script. Here's the PHP script that creates the database:

<?php 
$id=dba_open('visitor.db', 'c', 'db4');
dba_optimize($id);
dba_close($id);
?>

Now, here's the PHP code to insert an entry: I use JSON to hold the "real" data:

<?php 
/* 
    record a visit in a BSD DB
*/
$id=dba_open('visitor.db', 'w', 'db4');
if (!$id) {
    /* dba_open failed */
    exit;
}
$key  = $_SERVER['REQUEST_TIME_FLOAT']; 
$rip  = $_SERVER['REMOTE_ADDR'];
$now  = date('d-m-Y h:i:s a', time()); 
$data = json_encode( array('remote_ip' => $rip, 'timestamp' => $now) );
$userdata=array($key => $data);
foreach ($userdata as $key=>$value) {
dba_insert($key, $value, $id);
}
dba_optimize($id);
dba_close($id);
?>

Now, here's the code that you and I are actually interested in, and it uses Python's bsddb3 module.

#!/usr/bin/env python
from bsddb3 import db
import json

fruitDB = db.DB()
fruitDB.open('visitor.db',None,db.DB_BTREE,db.DB_DIRTY_READ)
cursor = fruitDB.cursor()
rec = cursor.first()

while rec:
    print rec
    visitordata = rec[1]
    print '\t' + visitordata
    jvdata = json.loads(visitordata)
    print jvdata
    rec = cursor.next()
    print '\n\n'
print '----';

fruitDB.close()
1
  • The None shouldn't be the database name db4?
    – pbies
    Commented Aug 2, 2022 at 20:59
4

Searching for "import bsddb", I get:

...but personally I'd heavily recommend you use sqlite instead of bsddb, people are using the former a lot more for a reason.

5
  • Thanks for telling me how you found them too. I'd forgotten that trick.
    – tgray
    Commented Apr 1, 2009 at 20:40
  • Unfortunately I don't think sqlite will scale well enough for my application (updated question). If you know that sqlite will work (with some certainty), please let me know!
    – tgray
    Commented Apr 1, 2009 at 21:11
  • I'm not sure sqlite will scale that well, but I'm also not sure bsddb will scale well either. If you are creating the data and then accessing it a lot, cdb might be your best bet. Commented Apr 6, 2009 at 18:41
  • I'm using Windows, so I don't think cdb is an option. At least, the docs say it is for UNIX.
    – tgray
    Commented Aug 3, 2009 at 15:29
  • the links are dead Commented Mar 25, 2018 at 0:28
1

The Gramps genealogy program uses bsddb for its database

Not the answer you're looking for? Browse other questions tagged or ask your own question.