Rob Behnke
August 21st, 2023
They say 'No man is an island,' meaning, of course, that no human is isolated. This principle also applies to enterprise architecture components, such as a System of Record (abbreviated as SOR) or a transaction platform, which is the role that a Daml/Canton ledger typically fulfills. Even if you use it as a standalone application, you will want to communicate with it through a web front end and/or a mobile application.
So a Daml/Canton ledger is always integrated with some client system(s) running on behalf of one or more ledger parties. In this blog post, we give an overview about the most important options to implement such integrations. (We wrote earlier about the front-door and back-door security features of the Daml smart contract platform and the Canton blockchain, and why and how workflow composition is supported by them, so we won’t repeat these aspects here.)
On top of the usual system integration challenges, some additional ones arise from the special purpose and design of the Daml platform:
Daml is an expressive, business-friendly programming language. It can closely mimic high-level business and legal concepts through algebraic data types (ADT for short). This means that one single data type can express alternative variants under one concept with all variants having their own sets of details. An example: we can use an ADT for expressing the opening hours of a business for every weekday separately, even adding some on-call contact options for the weekends. ADTs in Daml are inherited from Haskell, but are not natively supported in many popular client languages. So translating ADTs (and other Daml data types) to client languages is a special challenge.
A Daml ledger is an event source system, meaning that what the ledger stores is actually an event graph rather than distinct system states. The full event graph is important for integrity and auditability, but client systems are typically interested in the current snapshot of the system state, because this is what limits interactions. The current, relevant system state (also called active contract set, or ACS for short) needs to be computed from the event graph, which takes more time than just retrieving data from a database. Speeding up the retrieval of the ACS is a special challenge.
A Daml ledger implements the legal architecture of a business model. The actors of the business model are represented as ledger parties. All client applications operate on behalf of one or more ledger parties, creating new contracts or exercising choices on existing contracts. So every API call needs to contain a valid access token proving the link between the calling entity and one or more ledger parties, containing custom claims required by the platform. Issuing Daml-compatible access tokens is a special challenge.
A Daml model is rigid on purpose. This rigidity serves legal stability. This is what guarantees that all consequences of signing a Daml contract are predictable at the time of signing it. This also means that if for whatever reason (business and/or regulatory) you want to change the structure of contracts, you have to deploy a new Daml model on the ledger and upgrade existing contracts, acquiring the consent of their signatories. The change also poses a special challenge for the client applications.
Directly calling external APIs from a Daml ledger is not allowed. This limitation, like the previous one, serves legal stability and is intentional. Unpredictable results of such calls would jeopardize the predictability of the Daml model. The workaround here is that a client application listens to signals coming from the ledger, makes the API call, retrieves the result, handles errors, and feeds back the answer to the Daml ledger via exercising a choice on behalf of an authorized ledger party. There are no success guarantees for the API call on the Daml side, but the usual guarantees can be provided on the client side.
In the following sections of this blog post, we will also touch upon the Daml-specific challenges of client communication, and some of the solutions for handling these challenges.
At the bottom level, the Daml platform exposes a gRPC API. The protobuf message and service definitions are available as part of the Daml source code. Grpc is a highly language-agnostic protocol, meaning there are some officially supported languages and other languages for which there exist libraries to use it. For prototyping client interactions, gRPCurl can also be used.
Daml also exposes an HTTP JSON API, built on top of the gRPC API with a somewhat limited functionality. The JSON API exposes a limited set of endpoints, most notably for querying or creating contracts and exercising choices on existing contracts. The complexity of calls stemming from the actual Daml model is transferred to the JSON payload. JSON is language-agnostic, so the JSON API can be used with practically any client language. JSON is not type safe, so it’s the client’s responsibility to create the appropriate representation of the Daml data types, following these rules.
The freedom of language choice, of course, comes with a price tag. Writing a gRPC integration or JSON generator from scratch is not an easy task. To alleviate this burden, there are some client bindings, some of which also provide generated code containing the equivalents of the Daml data types in the client language:
The Java bindings for using the full functionality of the gRPC API. It contains a ledger client interface implementation and a data access layer, based on generated Java classes. Click here for an example project to see how it works.
The JavaScript/TypeScript based ledger library to communicate with the somewhat limited JSON API of the Daml ledger, and a React library to build React based single page applications on top of it. The ledger library can be used with Node.js applications and other UI frameworks as well. The JavaScript/TypeScript code generator generates the JavaScript equivalents of the Daml types and TypeScript type annotations for type checking (no class constructors, unlike the Java binding). Click here for an example project with a Daml ledger and a React SPA.
The Python bindings (formerly called Dazl). Unlike the former ones, the Python bindings don’t come with codegen. Users of the Python bindings either need to write functions to generate the Python dictionary equivalents of the Daml types, which are similar to the JSON required by the JSON API, or write Python classes for the Daml types manually. The Python bindings communicate with the Daml gRPC API, but only exposes roughly the same functionality as the JSON API.
The above libraries are provided by Digital Asset, and are actively maintained and supported. As a curiosity, there is an unofficial Rust binding for Daml, also providing codegen. It’s not actively maintained any more, the last Daml-LF version supported is 1.14. The current Daml-LF version is 1.15, which contains support for Daml interfaces. But as long as you don’t need interfaces, it’s fine to use. Alternatively, if you need Rust and Daml interfaces, you can upgrade it yourself.
One challenge with using the generated code is that it depends on the actual Daml model. Whenever the Daml model changes, the code needs to be generated again, and the client applications need to be updated.
This can be burdensome in the early phase of the project, when the Daml model is not yet stable, so there is no use putting much effort into building client applications and UI, but still you have to demonstrate how the Daml part works to external stakeholders. The Daml platform contains the Navigator UI, which looks pretty, and can display archived contracts as well as active contracts, but it’s not very convenient for a large number of contracts or contracts with a lot of fields. In these cases, some dataframe libraries, like Python Pandas or its Java/JavaScript/Rust equivalents can come to the rescue to filter and display the ledger content.
Another challenge is that codegen, naturally, only generates serializable data types. So there is no way to generate the functions contained in the Daml model for the client applications.
For testing and demonstration, Daml Script can be used. Daml Script is an ordinary client application, which happens to be written in Daml and can import data types and functions from the Daml model. One serious limitation of Daml Script is that it currently doesn’t have IO capability, so it cannot read data from files to use for ledger communication. If you have to test your ledger with a lot of data, this limitation can be overcome by generating Daml Script code using any language which can manipulate text, using the source data. Daml Script can be run with authorization, but it’s not usual to use it in production, except for maybe ledger setup.
A third challenge, as mentioned before, is that many client languages don’t support algebraic data types (ADTs). (A word of warning: ADTs are called “variants” in the Daml docs, probably reflecting the usual gRPC wording. So the word “variant” means the whole ADT data structure as well as one of its options.)
Among the languages with Daml codegen, the closest to Daml ADTs are Rust enums, which are richer than enums in other languages. The JSON representation of an ADT (variant data type) contains a “tag” and a “value” field for the constructor and the argument. The TypeScript type annotation is similar, and separates options with the usual pipe symbol. The generated Java code represents the ADT (variant data type) with an abstract class, and the options with classes implementing the parent abstract class.
With languages for which we don’t have codegen out of the box, the challenge of course is to write codegen, when the Daml model is large and complex enough to justify the effort. For this purpose the DAML LF API Type Signature library can be used, with the caveat that this is a public, but internal Scala library with no documentation.
As mentioned before, the contents of a Daml ledger is an event graph, from which the current snapshot of system state (usually referred to as active contract set or ACS) needs to be computed. This takes more time than fetching data from a database.
Ledger users are free to apply any mechanism to speed up this query. The feature the Daml platform offers is the Query Store functionality of the JSON API, for which a supported SQL database (PostgreSQL or Oracle, the latter only for the enterprise edition) can be configured as data storage. Another approach can be to store the ledger event graph in a graph database which is optimized for fast graph search. This can work because the ACS consists of such “create” nodes in the event graph for which no consuming exercise was recorded on the ledger.
As mentioned before, all client applications operate on behalf of a ledger party. The platform doesn’t perform authentication, just checks if the access tokens supplied with the API request are valid and contain the party id of a ledger party which is authorized to perform the required change.
Daml-compatible access tokens need to contain custom claims, which is not supported by all OAuth service providers. One such OAuth service provider which does support Daml-compatible access tokens is Auth0. The Daml documentation describes in detail how to configure Auth0 for this.
An alternative to directly issuing party access tokens is to use the user management feature of Daml. Users can be configured on participant nodes locally to act on behalf of one or more ledger parties. Rights provided for users can be granted or revoked dynamically. User access tokens are simpler than party access tokens.
As mentioned before, smart contract instances on a Daml ledger are tied to a specific Daml model, identified by a ledger id, which is a hash of the Daml source code. Consequently, if you need to upgrade your Daml model during production, you need to migrate the existing contracts on the ledger from the old model to the new one. The following challenges need to be tackled for such an upgrade:
Probably the most serious challenge is the CRM (Customer Relationship Management) part: acquiring the authorization of the signatories of the original contracts. This needs to be handled on the business side.
On the Daml side, you have to create a third, transitory Daml model which will serve as a bridge between the old and the new model, containing the functions to convert the old data types to the new ones, and templates to get the authorization from the signatories. This is an example for doing the migration. For large, complex Daml models writing the functions for type conversion is challenging, and requires either a lot of typing, or using codegen.
On the client side, the challenge arises when the transition is not complete, so you have to handle two sets of contracts belonging to two different Daml models in one client application. If in the Daml model you don’t want to rename the templates and choice input data types, you have to handle the difference in the client app. For Java, this can be handled by introspection.
Calling an external API directly from a Daml ledger is not possible.
Daml Hub, Digital Asset’s cloud platform for running Daml ledgers offers the option of integrations, which are applications connecting a Daml ledger with external APIs.
Another, more flexible option is to signal the need for an API call by exercising a nonconsuming choice on the ledger, for which a client application listens. It’s the client application’s responsibility to pick up the message, call the appropriate API and feed back the response to the ledger by exercising the appropriate choice.
To learn more, please check out our blog series on Daml and Canton.