Questions tagged [datajoint]

For general questions or debugging development errors as a result of using DataJoint, a framework for scientific workflow management. For assistance with debugging, please include: Client OS, DataJoint Client (Python|MATLAB|LabBook|Pharus), DataJoint Version, and Database Version (MySQL 5.7|MariaDB 10.5|etc).

Filter by
Sorted by
Tagged with
0 votes
1 answer
28 views

How to host a DataJoint external store on the same server as the DataJoint MySQL database

I'm trying to set up an external store to work with my pipeline. I am self hosting my database on a small server which my lab connect to. This is currently working well and we can all insert and query ...
rbedford's user avatar
0 votes
1 answer
33 views

How can I connect my DataJoint LabBook instance to my DataJoint MySQL server?

I'm trying to host DataJoint LabBook on our lab's server so our team can interact with our database. I'm afraid I don't know much about web development (I'm a datascientist) so am not finding it easy. ...
rbedford's user avatar
0 votes
1 answer
25 views

External file cleanup routines in datajoint

I have a lot of blob data stored externally (S3) and recalculate results regularly. I am running datajoint python 0.13.8. I have read the documentation under https://docs.datajoint.org/python/admin/5-...
Horst's user avatar
  • 167
0 votes
2 answers
54 views

Datajoint locking the table? Populating on multiple machines

I have a table that runs some heavier computation (process length ~ 5 minutes per key). I want to reserve jobs and run it on multiple machines. I noticed that computers get locked out from the table ...
Horst's user avatar
  • 167
0 votes
1 answer
34 views

Cannot drop AttributeAdapter tables -- "Declaration context is not set"

I have created a dj.AttributeAdapter subclass for a custom datatype (attribute_type = filepath@external). However it does not seem to be working right -- when I attempt to drop the table for instance ...
zagaroo's user avatar
  • 21
2 votes
0 answers
70 views

How do I migrate my database schema in a backwards-incompatible way with DataJoint?

I work with a neuroscience lab that uses DataJoint and a shared lab database to manage our data and data pipelines during data preprocessing and analysis. We have tables for different data types, both ...
Ryan Ly's user avatar
  • 51
0 votes
1 answer
45 views

Datajoint Lost Connection Error (Server connection lost due to an interface error)

I am running datapoint in Pycharm on a Mac. When I am connecting to the Database via the Pycharm Terminal, I can execute the code. Also, using the Pycharm Python Console, I am able to connect to the ...
Verena's user avatar
  • 1
0 votes
1 answer
161 views

Fetching data from RDS on AWS returns empty array

I have a mysql db set up on RDS on AWS (I ported the db from my personal computer), and have set up a datajoint connection to the db on Sagemaker in a Jupyter notebook. Say I have a table mouse, which ...
eric's user avatar
  • 7,301
2 votes
0 answers
48 views

external storage table already exists error datajoint matlab

Whenever I want to create a table with multiple external storage fields I get the following error: Table '~external_behavior_time_spatial_blobs' already exists This is a code snippet of the table I ...
Alvaro Luna's user avatar
0 votes
0 answers
62 views

Foreign key reference could not be resolved immediately after defining

The following code @schema class A(dj.Manual): definition = """ a: varchar(1) """ a = A() a.insert1(["a"], skip_duplicates=True) @schema ...
zagaroo's user avatar
  • 21
-1 votes
1 answer
69 views

Auto incrementing composite key

I have a relatively simple need -- I have a table with a composite primary key that includes an integer index to be autoincremented. However, I cannot figure out how to do this. I have read the ...
zagaroo's user avatar
  • 21
1 vote
1 answer
32 views

Moving files within a datajoint filestore

We have a file store set up for analysis files, and we've realized that we need to create a subdirectory structure within in. The dj.config settings are below, and the schema definition is @schema ...
Loren Frank's user avatar
0 votes
1 answer
74 views

Storing wide-form dataframes in datajoint table [closed]

Say I have some analysis that spits out a wide-form pandas dataframe with a multiindex on the index and columns. Depending on the analysis parameters, the number of columns may change. What is the ...
zagaroo's user avatar
  • 21
1 vote
0 answers
40 views

What’s the best way to profile python code when using datajoint?

We have tried using line_profiler in juypter but it just tells us that the auto populate function is using all of the cpu time. Is there a simple way for us to get a more detailed profiling report ...
Loren Frank's user avatar
0 votes
1 answer
27 views

Generate a table subset with a list

I have a set of subjects in a table, and I want delete a subset (e.g., subject1 and subject2). Is there a way to fetch a subset using a list? Or do I need to have an attribute I can use to select the ...
Chris Broz's user avatar
0 votes
2 answers
66 views

Alternative way to list all part tables that have existing entries/tuples

I have one single master Subject table which have several part tables. One convenient thing about Table.delete() is that is displays a prompt with all existing entries that were created under that ...
Sahil's user avatar
  • 7
1 vote
0 answers
47 views

DataJointError: Class ________ is not properly declared (schema decorator not applied?) when trying to diagram

DataJoint version 0.13.4 Python 3.9, running in PyCharm 2022.1.1 Pro Edition MacOS 12.3.1 I've been working with several of the datajoint-elements packages just trying to get an overall schema created ...
Spider Pig 212's user avatar
0 votes
2 answers
75 views

Is there a way to stop datajoint from asking for username and password when I import it

In Datajoint, you are prompted for username and password. I am creating a readthedocs website with Sphinx and I need the system to not prompt for username and password. How can I turn it off?
Phil's user avatar
  • 2,153
0 votes
2 answers
201 views

Nested MATLAB struct is difficult to parse when fetched in DataJoint Python

From DataJoint Python and DataJoint MATLAB, I have inserted the same values into a longblob attribute. From DataJoint Python it was inserted as a dictionary and from DataJoint MATLAB it was inserted ...
kabilar's user avatar
0 votes
1 answer
106 views

validating parameter dictionaries before insertion in datajoint schema

We'd like to enforce parameter checking before people can insert into a schema as shown below, but the code below doesn't work. Is there a way to implement pre-insertion parameter checking? @schema ...
Loren Frank's user avatar
1 vote
1 answer
34 views

`delete()` vs. `fetch()` with underspecified relative restrictions

My database has a number of sessions. @schema class Session(dj.Manual): definition = """ -> Subject session_datetime: datetime """ I could fetch or ...
Chris Broz's user avatar
0 votes
1 answer
30 views

What is the purpose of FreeTable? It does not have the same behavior as user tables. It is not covered in the documentation

Environment: DataJoint version (0.12.9) on Ubuntu with MySQL 5.7 The concept of FreeTable is nice. Provide a dj.conn() object and full_table_name and a table you can manipulate is created, independent ...
spapadop's user avatar
0 votes
1 answer
52 views

MySQL OperationalError when deleting from DataJoint table after restricting with its part table: "You can't specify target table : 'part_table'"

Using DataJoint Python (0.12.9) on Ubuntu with MySQL 5.7 . Am running into an error with what I believed to be a fairly standard operation. Have a table named Table with a part Table.Part. Restricting ...
spapadop's user avatar
0 votes
1 answer
98 views

Specifying table dependency when many entries in upstream table for one entry in downstream table

I would like to create a datajoint computed table that depends on multiple entries (about 20) from an upstream table. I know it is possible to create a table that depends on multiple entries from an ...
jguides's user avatar
  • 27
0 votes
1 answer
64 views

Querying a varchar entry for a partial string

I am wondering if there is a way to query a table for a partial match to a string. For instance, I have a table that lists drug treatments, but sometimes there are multiple drugs in a single treatment....
anna1ntegrated's user avatar
1 vote
1 answer
96 views

Does datajoint "join" operator require tables to have the same value at shared secondary attributes?

I am a researcher in Loren Frank's lab at UCSF. When calling populate on a Computed datajoint table that depends on two upstream tables which have entries that have the same value for shared primary ...
jguides's user avatar
  • 27
1 vote
0 answers
283 views

InvalidXMLError: InvalidXMLError: message: "BucketLocationConstraintResult" XML is not parsable

I am setting up a local database for datajoint following instructions here: https://github.com/datajoint/mysql-docker. The mySQL database works but now I wanted to use MinIO for external storage. I ...
user72328's user avatar
0 votes
1 answer
65 views

Datajoint dependencies with custom key_source

I use non-dependent tables to restrict using a custom key source. As an example, my pipeline stages are all parameterized and use a mixin key_source to specify which parameters are used for each ...
HoosierDaddy's user avatar
0 votes
1 answer
40 views

Setting datajoint key to a different variable in make fxn

Wondering why and when it's needed to set the key to a different variable in a make/maketuples function in an auto-populated table in datajoint, as in the documentation here. In this example, the part ...
jbussell's user avatar
0 votes
1 answer
93 views

Drop a Datajoint schema in Matlab? List schemas on the database server?

Looking for how to remove an entire Datajoint schema from the SQL database server using the Matlab 3.4 implementation of datajoint (the equivalent of schema.drop(True) in Python) as well as how to ...
jbussell's user avatar
0 votes
1 answer
53 views

Cannot repopulate ElectrodeGroup datajoint table

I'm a researcher in Loren Frank's lab at UCSF using datajoint and files in the nwb format. I made some changes to our code for defining entries in our ElectrodeGroup table, and was hoping to test ...
jguides's user avatar
  • 27
2 votes
1 answer
80 views

Is it possible to implement <populate> such that it utilizes <insert> instead of <insert1>?

Is it possible to implement populate such that it utilizes insert instead of insert1 ? For large data sets, I am getting a server time out error as using insert1 takes hours to populate a table. Here ...
Juux's user avatar
  • 21
2 votes
2 answers
66 views

DataJoint: IntegrityError while dropping table

I am in the process of designing our database and have a table called Session in the common_exp schema with the following definition: @schema class Session(dj.Manual): definition = ""&...
Hendrik Heiser's user avatar
1 vote
1 answer
101 views

Fastest way to backup / delete from table

I am trying to backup some data from a table with > 50M rows. The table looks like this # entry_id : int unsignedauto_increment # unique entry id --- time=CURRENT_TIMESTAMP : ...
Horst's user avatar
  • 167
1 vote
1 answer
40 views

Datajoint: suggestions for how to handle stages needing manual review in a neuroscience data pipeline?

Wondering if there are good examples or suggestions for how to handle steps that require manual review in a database-based scientific data pipeline (datajoint). For example, we'd like to handle the ...
jbussell's user avatar
0 votes
1 answer
50 views

Datajoint scientific pipeline: how to efficiently store data and design pipeline when attributes can change and data come from arrays

We need to load the output from our neuroscience animal behavior training sessions into our datajoint database pipeline. We run experiments with a platform called Bpod that implements a finite state ...
jbussell's user avatar
0 votes
1 answer
180 views

Datajoint autopopulate make function error: DataJointError: A relation must have primary dependencies for auto-populate to work

I am getting the following error: "DataJointError: A relation must have primary dependencies for auto-populate to work" with what I think is a very basic start defining first a manual table ...
jbussell's user avatar
1 vote
1 answer
370 views

Is there a preferred way to have a downstream table that can take data from one of two upstream tables in datajoint?

We are trying to build flexible schema where, for example, we might have single units that come from one of two places, either a SpikeSorting pipeline that begins from raw data, or imported from a ...
Loren Frank's user avatar
1 vote
1 answer
64 views

Renaming/replacing primary key to query across schemas

I currently have two separate schemas for processed calcium imaging data, in which I've used two different preprocessing methods. The big goal of what I am trying to do is use sources from 1 schema as ...
A__'s user avatar
  • 11
0 votes
2 answers
28 views

NotImplementedError when calling .alter() (0.13.1)

I am on datajoint python 0.13.1. When executing .alter() on a table in my schema I am getting the following error message: --------------------------------------------------------------------------- ...
Horst's user avatar
  • 167
1 vote
1 answer
55 views

DataJointError: Found a part table without its master table

I am using datajoint python '0.12.8'. When trying to delete from an upstream table, I get the following error message: DataJointError: Found a part table "someschema"."table_part-table&...
Horst's user avatar
  • 167
3 votes
1 answer
82 views

Datajoint LabBook - how to change ports

I am running Datajoint LabBook through the provided docker container (https://datajoint.github.io/datajoint-labbook/user.html#installation) and wondered whether there is a way to move it away from the ...
Horst's user avatar
  • 167
1 vote
0 answers
77 views

Datajoint LabBook - can't get past login screen

I set up the datajoint lab book (docker, https://datajoint.github.io/datajoint-labbook/user.html#installation) and it seems to run. However, I can’t seem to get past the login screen (credentials / db ...
Horst's user avatar
  • 167
1 vote
1 answer
141 views

Is it possible to alter a table to include reference to an upstream table in datajoint for python?

We want to alter a table to include a non-primary key reference to a new table. The old definition is: @schema class SpikeSortingParameters(dj.Manual): definition = """ # Table ...
Loren Frank's user avatar
2 votes
1 answer
125 views

Dependency from a part table, but not its master table, to an upstream table suddenly fails

I have a table Downstream with a part table DownstreamPart. DownstreamPart, but not Downstream, has a dependency to an upstream table Upstream (the only other additional dependency is to Downstream). ...
lara's user avatar
  • 21
0 votes
1 answer
41 views

datajoint-matlab implementation of filepath@data datatype

For datajoint-matlab, #327 seems to indicate that File external storage (#143, PR #197) should be implemented in the current version. I can create a table with the datatype filepath@data, after ...
Gary Yellen's user avatar
0 votes
1 answer
257 views

Is there a way to run a transaction multiple time at once?

First, I would like to apologize in advance. I just started learning this a few months ago, so I need stuff broken down completely. I have a project using python and datajoint (makes sql code shorter ...
DONALD DAY's user avatar
2 votes
1 answer
221 views

Use a docker image to build a different docker image using a dockerfile

I am trying to build a docker image of a flask app using a docker file. The flask app uses a docker image of specific sql version datajoint/mysql (using docker-compose). But I get the following error: ...
ishaan arora's user avatar
1 vote
1 answer
211 views

Datajoint: Lock wait timeout exceeded errors

While trying to delete from a table I sometimes get Lock wait timeout exceeded errors. This seems to happen when many .populate commands (with reserve_jobs=True) are running in parallel. Removing ...
Horst's user avatar
  • 167
3 votes
1 answer
306 views

What is the correct ORDER BY syntax in DataJoint for Python?

In DataJoint for Python, what is the correct syntax of the ORDER BY argument used in the fetch command? The current documentation (as of 2018-08-17) only specifies the following syntax for DataJoint ...
Vathes's user avatar
  • 61