Message

Message is the envelope by which coordinated data exchange can happen between parties in the network. Data is passed by reference in these messages, and a chain of hashes covering the data and the details of the message, provides a verification against tampering.

A message is made up of three sections:

The header - a set of metadata that determines how the message is ordered, who should receive it, and how they should process it
The data - an array of data attachments
Status information - fields that are calculated independently by each node, and hence update as the message makes it way through the system

Hash¶

Sections (1) and (2) are fixed once the message is sent, and a hash is generated that provides tamper protection.

The hash is a function of the header, and all of the data payloads. Calculated as follows:

The hash of each Data element is calculated individually
A JSON array of [{"id":"{{DATA_UUID}}","hash":"{{DATA_HASH}}"}] is hashed, and that hash is stored in header.datahash
The header is serialized as JSON with the deterministic order (listed below) and hashed
JSON data is serialized without whitespace to hash it.
The hashing algorithm is SHA-256

Each node independently calculates the hash, and the hash is included in the manifest of the Batch by the node that sends the message. Because the hash of that batch manifest is included in the blockchain transaction, a message transferred to a node that does not match the original message hash is rejected.

Tag¶

The header.tag tells the processors of the message how it should be processed, and what data they should expect it to contain.

If you think of your decentralized application like a state machine, then you need to have a set of well defined transitions that can be performed between states. Each of these transitions that requires off-chain transfer of private data (optionally coordinated with an on-chain transaction) should be expressed as a type of message, with a particular tag.

Every copy of the application that runs in the participants of the network should look at this tag to determine what logic to execute against it.

Note: For consistency in ordering, the sender should also wait to process the state machine transitions associated with the message they send until it is ordered by the blockchain. They should not consider themselves special because they sent the message, and process it immediately - otherwise they could end up processing it in a different order to other parties in the network that are also processing the message.

Topics¶

The header.topics strings allow you to set the the ordering context for each message you send, and you are strongly encouraged to set it explicitly on every message you send (falling back to the default topic is not recommended).

A key difference between blockchain backed decentralized applications and other event-driven applications, is that there is a single source of truth for the order in which things happen.

In a multi-party system with off-chain transfer of data as well as on-chain transfer of data, the two sets of data need to be coordinated together. The off-chain transfer might happen at different times, and is subject to the reliability of the parties & network links involved in that off-chain communication.

A "stop the world" approach to handling a single piece of missing data is not practical for a high volume production business network.

The ordering context is a function of:

Whether the message is broadcast or private
If it is private, the privacy group associated with the message
The topic of the message

When an on-chain transaction is detected by FireFly, it can determine the above ordering - noting that privacy is preserved for private messages by masking this ordering context message-by-message with a nonce and the group ID, so that only the participants in that group can decode the ordering context.

If a piece of off-chain data is unavailable, then the FireFly node will block only streams of data that are associated with that ordering context.

For your application, you should choose the most granular identifier you can for your topic to minimize the scope of any blockage if one item of off-chain data fails to be delivered or is delayed. Some good examples are:

A business transaction identifier - to ensure all data related to particular business transaction are processed in order
A globally agreed customer identifier - to ensure all data related to a particular business entity are processed in order

Using multiple topics¶

There are some advanced scenarios where you need to merge streams of ordered data, so that two previously separately ordered streams of communication (different state machines) are joined together to process a critical decision/transition in a deterministic order.

A synchronization point between two otherwise independent streams of communication.

To do this, simply specify two topics in the message you sent, and the message will be independently ordered against both of those topics.

You will also receive two events for the confirmation of that message, one for each topic.

Some examples:

Agreeing to join two previously separate business transactions with ids 000001 and 000002, by discarding business transaction 000001 as a duplicate
Specify topics: ["000001","000002"] on the special merge message, and then from that point onwards you would only need to specify topics: ["000002"].
Agreeing to join two previously separate entities with id1 and id2, into a merged entity with id3.
Specify topics: ["id1","id2","id3"] on the special merge message, and then from that point onwards you would only need to specify topics: ["id3"].

Transaction type¶

By default messages are pinned to the blockchain, within a Batch.

For private messages, you can choose to disable this pinning by setting header.txtype: "unpinned".

Broadcast messages must be pinned to the blockchain.

In-line data¶

When sending a message you can specify the array of Data attachments in-line, as part of the same JSON payload.

For example, a minimal broadcast message could be:

{
    "data": [
        {"value": "hello world"}
    ]
}

When you send this message with /api/v1/namespaces/{ns}/messages/broadcast:

The header will be initialized with the default values, including txtype: "batch_pin"
The data[0] entry will be stored as a Data resource
The message will be assembled into a batch and broadcast

Example¶

{
    "header": {
        "id": "4ea27cce-a103-4187-b318-f7b20fd87bf3",
        "cid": "00d20cba-76ed-431d-b9ff-f04b4cbee55c",
        "type": "private",
        "txtype": "batch_pin",
        "author": "did:firefly:org/acme",
        "key": "0xD53B0294B6a596D404809b1d51D1b4B3d1aD4945",
        "created": "2022-05-16T01:23:10Z",
        "namespace": "ns1",
        "group": "781caa6738a604344ae86ee336ada1b48a404a85e7041cf75b864e50e3b14a22",
        "topics": [
            "topic1"
        ],
        "tag": "blue_message",
        "datahash": "c07be180b147049baced0b6219d9ce7a84ab48f2ca7ca7ae949abb3fe6491b54"
    },
    "localNamespace": "ns1",
    "state": "confirmed",
    "confirmed": "2022-05-16T01:23:16Z",
    "data": [
        {
            "id": "fdf9f118-eb81-4086-a63d-b06715b3bb4e",
            "hash": "34cf848d896c83cdf433ea7bd9490c71800b316a96aac3c3a78a42a4c455d67d"
        }
    ]
}

Field Descriptions¶

Field Name	Description	Type
`header`	The message header contains all fields that are used to build the message hash	`MessageHeader`
`localNamespace`	The local namespace of the message	`string`
`hash`	The hash of the message. Derived from the header, which includes the data hash	`Bytes32`
`batch`	The UUID of the batch in which the message was pinned/transferred	`UUID`
`txid`	The ID of the transaction used to order/deliver this message	`UUID`
`state`	The current state of the message	`FFEnum`: `"staged"` `"ready"` `"sent"` `"pending"` `"confirmed"` `"rejected"` `"cancelled"`
`confirmed`	The timestamp of when the message was confirmed/rejected	`FFTime`
`rejectReason`	If a message was rejected, provides details on the rejection reason	`string`
`data`	The list of data elements attached to the message	`DataRef[]`
`pins`	For private messages, a unique pin hash:nonce is assigned for each topic	`string[]`
`idempotencyKey`	An optional unique identifier for a message. Cannot be duplicated within a namespace, thus allowing idempotent submission of messages to the API. Local only - not transferred when the message is sent to other members of the network	`IdempotencyKey`

MessageHeader¶

Field Name	Description	Type
`id`	The UUID of the message. Unique to each message	`UUID`
`cid`	The correlation ID of the message. Set this when a message is a response to another message	`UUID`
`type`	The type of the message	`FFEnum`: `"definition"` `"broadcast"` `"private"` `"groupinit"` `"transfer_broadcast"` `"transfer_private"` `"approval_broadcast"` `"approval_private"`
`txtype`	The type of transaction used to order/deliver this message	`FFEnum`: `"none"` `"unpinned"` `"batch_pin"` `"network_action"` `"token_pool"` `"token_transfer"` `"contract_deploy"` `"contract_invoke"` `"contract_invoke_pin"` `"token_approval"` `"data_publish"`
`author`	The DID of identity of the submitter	`string`
`key`	The on-chain signing key used to sign the transaction	`string`
`created`	The creation time of the message	`FFTime`
`namespace`	The namespace of the message within the multiparty network	`string`
`topics`	A message topic associates this message with an ordered stream of data. A custom topic should be assigned - using the default topic is discouraged	`string[]`
`tag`	The message tag indicates the purpose of the message to the applications that process it	`string`
`datahash`	A single hash representing all data in the message. Derived from the array of data ids+hashes attached to this message	`Bytes32`
`txparent`	The parent transaction that originally triggered this message	`TransactionRef`

TransactionRef¶

Field Name	Description	Type
`type`	The type of the FireFly transaction	`FFEnum`:
`id`	The UUID of the FireFly transaction	`UUID`

DataRef¶

Field Name	Description	Type
`id`	The UUID of the referenced data resource	`UUID`
`hash`	The hash of the referenced data	`Bytes32`