9

I'm starting to develop a Datomic-backed Clojure app, and I'm wondering what's the best way to declare the schema, in order to address the following concerns:

  1. Having a concise, readable representation for the schema
  2. Ensuring the schema is installed and up-to-date prior to running a new version of my app.

Intuitively, my approach would be the following:

  1. Declaring some helper functions to make schema declarations less verbose than with the raw maps
  2. Automatically installing the schema as part of the initialization of the app (I'm not yet knowledgeable enough to know if that always works).

Is this the best way to go? How do people usually do it?

6 Answers 6

4

I Use Conformity for this see Conformity repository. There is also a very useful blogpost from Yeller Here which will guide you how to use Conformity.

1
  • thanks! That seems to solve concern #2. Do you have any advice about #1 ? Jul 14, 2015 at 20:55
4
  1. Raw maps are verbose, but have some great advantages over using some high level api:

    • Schema is defined in transaction form, what you specify is transactable (assuming the word exists)
    • Your schema is not tied to a particular library or spec version, it will always work.
    • Your schema is serializable (edn) without calling a spec API.
    • So you can store and deploy your schema more easily in a distributed environment since it's in data-form and not in code-form.

For those reasons I use raw maps.

  1. Automatically installing schema.

This I don't do either.

Usually when you make a change to your schema many things may be happening:

  • Add new attribute
  • Change existing attribute type
  • Create full-text for an attribute
  • Create new attribute from other values
  • Others

Which may need for you to change your existing data in some non obvious and not generic way, in a process which may take some time.

I do use some automatization for applying a list of schemas and schema changes, but always in a controlled "deployment" stage when more things regarding data updating may occur.

Assuming you have users.schema.edn and roles.schema.edn files:

(require '[datomic-manage.core :as manager])
(manager/create uri)
(manager/migrate uri [:users.schema
                      :roles.schema])
7
  • 1
    It seems to me that some helpers for defining a schema are harmless, I would see it only as a more practical notation than map literals in the case your language is Clojure. As long as it's still data I believe it's okay. Jul 15, 2015 at 14:00
  • 1
    Yes the problem of using a helper is your schema stops being data and starts being an "api call", the distinction is important if you consider that data to be already in transact form.
    – guilespi
    Jul 15, 2015 at 14:02
  • I don't really agree with this assertion. To me a function that outputs data is just as much data-oriented and transactable as Clojure data structure literals. The only difference I can think of is that it's less language-agnostic. Jul 15, 2015 at 14:06
  • 1
    Not really, if you want to deploy your schema in a distributed environment for instance you need to execute your function before having serializable data. Your approach assumes your schema will always be in code-form, having your schema in data literal form allows for alternative storage, serialization and communication.
    – guilespi
    Jul 15, 2015 at 14:08
  • I'm thinking this may be a good use case for transaction functions. We could use them to reduce noise without compromising portability. Jul 15, 2015 at 14:38
2

For #1, datomic-schema might be of help. I haven't used it, but the example looks promising.

1
  • Now I want to validate both answers. I had it coming, since my question complected 2 concerns :) Jul 14, 2015 at 21:30
1

My preference (and I'm biased, as the author of the library) lies with datomic-schema - It focusses on only doing the transformation to normal datomic schema - from there, you transact the schema as you would normally.

I am looking to use the same data to calculate schema migration between the live datomic instance and the definitions - so that the enums, types and cardinality gets changed to conform to your definition.

The important part (for me) of datomic-schema is that the exit path is very clean - If you find it doesn't support something (that I can't implement for whatever reason) down the line, you can dump your schema as plain edn, save it off and remove the dependency.

Conformity will be useful beyond that if you want to do some kind of data migration, or more specific migrations (cleaning up the data, or renaming to something else first).

1

Proposal: using transaction functions to make declaring schema attributes less verbose in EDN, this preserving the benefits of declaring your schema in EDN as demonstrated by @Guillermo Winkler's answer.

Example:

;; defining helper function
[{:db/id #db/id[:db.part/user]
  :db/doc "Helper function for defining entity fields schema attributes in a concise way."
  :db/ident :utils/field
  :db/fn #db/fn {:lang :clojure
                 :require [datomic.api :as d]
                 :params [_ ident type doc opts]
                 :code [(cond-> {:db/cardinality :db.cardinality/one
                                 :db/fulltext true
                                 :db/index true
                                 :db.install/_attribute :db.part/db

                                 :db/id (d/tempid :db.part/db)
                                 :db/ident ident
                                 :db/valueType (condp get type
                                                 #{:db.type/string :string} :db.type/string
                                                 #{:db.type/boolean :boolean} :db.type/boolean
                                                 #{:db.type/long :long} :db.type/long
                                                 #{:db.type/bigint :bigint} :db.type/bigint
                                                 #{:db.type/float :float} :db.type/float
                                                 #{:db.type/double :double} :db.type/double
                                                 #{:db.type/bigdec :bigdec} :db.type/bigdec
                                                 #{:db.type/ref :ref} :db.type/ref
                                                 #{:db.type/instant :instant} :db.type/instant
                                                 #{:db.type/uuid :uuid} :db.type/uuid
                                                 #{:db.type/uri :uri} :db.type/uri
                                                 #{:db.type/bytes :bytes} :db.type/bytes
                                                 type)}
                                doc (assoc :db/doc doc)
                                opts (merge opts))]}}]

;; ... then (in a later transaction) using it to define application model attributes
[[:utils/field :person/name :string "A person's name" {:db/index true}]
 [:utils/field :person/age :long "A person's name" nil]]
2
  • This is really an "out of the box" approach! Any pros, cons?
    – onetom
    Dec 28, 2015 at 1:26
  • onetom I haven't tried it, so far I'm happy with using a DSL in my application code. Jul 30, 2016 at 21:57
1

I would suggest using Tupelo Datomic to get started. I wrote this library to simplify Datomic schema creation and ease understanding, much like you allude in your question.

As an example, suppose we’re trying to keep track of information for the world’s premiere spy agency. Let’s create a few attributes that will apply to our heroes & villains (see the executable code in the unit test).

  (:require [tupelo.datomic   :as td]
            [tupelo.schema    :as ts])

  ; Create some new attributes. Required args are the attribute name (an optionally namespaced
  ; keyword) and the attribute type (full listing at http://docs.datomic.com/schema.html). We wrap
  ; the new attribute definitions in a transaction and immediately commit them into the DB.
  (td/transact *conn* ;   required              required              zero-or-more
                      ;  <attr name>         <attr value type>       <optional specs ...>
    (td/new-attribute   :person/name         :db.type/string         :db.unique/value)      ; each name      is unique
    (td/new-attribute   :person/secret-id    :db.type/long           :db.unique/value)      ; each secret-id is unique
    (td/new-attribute   :weapon/type         :db.type/ref            :db.cardinality/many)  ; one may have many weapons
    (td/new-attribute   :location            :db.type/string)     ; all default values
    (td/new-attribute   :favorite-weapon     :db.type/keyword ))  ; all default values

For the :weapon/type attribute, we want to use an enumerated type since there are only a limited number of choices available to our antagonists:

  ; Create some "enum" values. These are degenerate entities that serve the same purpose as an
  ; enumerated value in Java (these entities will never have any attributes). Again, we
  ; wrap our new enum values in a transaction and commit them into the DB.
  (td/transact *conn*
    (td/new-enum :weapon/gun)
    (td/new-enum :weapon/knife)
    (td/new-enum :weapon/guile)
    (td/new-enum :weapon/wit))

Let’s create a few antagonists and load them into the DB. Note that we are just using plain Clojure values and literals here, and we don’t have to worry about any Datomic specific conversions.

  ; Create some antagonists and load them into the db.  We can specify some of the attribute-value
  ; pairs at the time of creation, and add others later. Note that whenever we are adding multiple
  ; values for an attribute in a single step (e.g. :weapon/type), we must wrap all of the values
  ; in a set. Note that the set implies there can never be duplicate weapons for any one person.
  ; As before, we immediately commit the new entities into the DB.
  (td/transact *conn*
    (td/new-entity { :person/name "James Bond" :location "London"     :weapon/type #{ :weapon/gun :weapon/wit   } } )
    (td/new-entity { :person/name "M"          :location "London"     :weapon/type #{ :weapon/gun :weapon/guile } } )
    (td/new-entity { :person/name "Dr No"      :location "Caribbean"  :weapon/type    :weapon/gun                 } ))

Enjoy! Alan

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.