Traditional applications produce massive volumes of continually changing data. Databases are used by programs to organize and manage information, making it simpler to store, retrieve, and update as required. Decentralized apps, like their conventional counterparts, generate massive amounts of constantly changing data. Until recently, however, the lack of a feasible decentralized solution for storing structured and dynamic data hampered the development of next-generation Web3 apps, especially those aimed at consumers. Fortunately, an increasing number of projects have developed to answer the demand for decentralized databases, providing feature parity with centralized systems while also bringing Web3-native capabilities like data composability and ownership.

Web3's Application Stack

As it stands, the majority of Web3 programs feature at least one centralized component, most notably a centralized database. Databases, although being a key component of the Web2 application stack, have been largely disregarded in Web3, owing to the notion that data can be kept in decentralized file storage networks or general-purpose smart contract platforms. These possibilities, however, have not proven viable for developers, causing many to turn to centralized solutions.

Decentralized databases should be integrated into the Web3 application stack in order to develop completely decentralized apps. The application stack is separated into four parts: the smart contract layer, file storage, database, and general infrastructure layer.

The smart contract layer acts as an application's backend server, while the general infrastructure layer comprises, but is not limited to, an oracle, RPC, access control, identification, off-chain computing, and indexing network.

Although less obvious, file storage and database networks are also critical in the creation of Web3 applications. They offer the architecture required for storing both organized and unstructured data, which is required for all types of applications. Due to the nature of this study, both components are discussed in further depth below.

Databases

Databases are an effective method to handle and regulate enormous volumes of dynamic data. Data schemas, which specify the structure and connections between data types, may also be enforced.

Databases are typically used to store structured data, which is information that can be arranged in a specified manner, such as tables. This kind of data storage makes it easy to update and alter data on a regular basis.

Structured data also has various additional advantages, including:

Queryability: Structured data enables quicker and more precise querying, which is critical for finding and displaying relevant information to clients.
Scalability: As applications increase in size, structured data helps to preserve performance and manageability by ensuring that data is stored, retrieved, and updated effectively.
Consistency: Structured data ensures consistency across the program and facilitates the process of integrating and sharing data between apps.
Data analysis: Because the data is organized and categorized, it is simpler to examine and extract insights.

Storage

When it comes to unstructured data, decentralized file storage networks (DFSNs) like Filecoin, Arweave, and Sia are particularly built for storing material that does not adhere to a set standard. As a result, DFSNs excel in storing text, picture, audio, and video data. However, since their primary goal is to guarantee data immutability, they are not suitable for storing dynamic data.

arweave permaweb

Source: Techcrunch

When storing data on Filecoin, for example, users must first choose a storage provider and negotiate an agreement with them. If the data has to be modified, a new file must be uploaded, and a new transaction must be completed. When the data has to be recovered, the user must locate the storage provider and request a retrieval, which costs money and may take up to 5 hours.

As a result, DFSNs are not suited for data that needs regular changes, and the lack of built-in querying capabilities makes fast data retrieval problematic.

What are Developers Using Right Now?

Smart contracts and centralized databases are the two major ways used by developers to store structured, changeable data. Because each alternative includes drawbacks, none of them entirely meets the requirements of Web3 app developers. These developers need:

Affordable, Scalable Infrastructure: During the 2022 FIFA World Cup, Twitter experienced 24,000 tweets per second. Supporting that many transactions in a Web3 alternative must be affordable.
Mutability: Data should be readily updated and altered.
Composability: Data should be available and useable by other programs for composition.
Queryability: Data retrieval must be quick and exact.
Data Ownership: Individuals should be able to own and manage their data.
Developer Experience: The infrastructure should be simple to use and integrate into developers' applications.

Smart Contracts

Using smart contracts to store data on a general-purpose L1/L2 may seem to be a decent idea at first. Data availability is high, data is completely composable, and data permanence is assured for the duration of the underlying chain. However, owing to scalability expenses, this strategy is not practicable. Because each new write or update activity would need an on-chain transaction, the accompanying gas prices would soon mount.

Furthermore, data saved directly in a smart contract is difficult to query. Data queries would be sluggish and inefficient without an indexing network.

Centralized database

Currently, most developers choose centralized databases supplied by huge cloud providers since they are cost-effective, changeable, and readily queryable. These centralized databases, however, have considerable downsides. Individual data ownership, interoperability, and composability are all lacking.

Furthermore, centralized systems hinder other developers from freely accessing, creating, and reading data contained in these databases. To put it another way, there is no Web3-native solution for providing access and encouraging cooperation.

Benefits of Decentralized Databases

One of the primary benefits of decentralized databases is their capacity to provide data composability. This enables developers to reuse and modify apps rather than starting fresh.

Another notable benefit is the inclusion of Web3-native permissions. Developers may use capabilities like crypto addresses, decentralized IDs (DID), or Lens profiles to enable cooperation and specify read and write rights for certain databases. This degree of granular control guarantees that data access is confined to approved persons or organizations, boosting Web3 application security and privacy.

Furthermore, Web3-native access controls enable users to own and govern their own data, reducing the danger of abuse by centralized institutions. Users would have total control over their data, enabling them to quickly move it to other platforms in order to prevent platform lock-in. Users will be able to monetize their data as well.

Decentralized databases also provide the following advantages:

Data that is verifiable and tamperproof: Cryptographic proofs (zero-knowledge or Merkle proofs) may be used to ensure that database write modification were done successfully and that the data was not tampered with or damaged. Furthermore, data may be confirmed to have come from the proper owner.
Increased Reliability and Availability: Because data is duplicated across several nodes, the danger of data loss and application downtime due to centralized provider failures or outages is eliminated.
Because data is dispersed among several nodes, no organization may erase or limit access to an individual's data.

Overview of a Decentralized Database

Background Information on Databases

Databases are classified into two types: relational and non-relational. Relational databases store and arrange data in tables with rows and columns that are linked together by shared properties. This creates a logical structure that can be queried and changed using SQL.

Non-relational databases, often known as NoSQL databases, vary from typical table-based databases in that they use other data schemas, such as key-value, document-oriented, and graph-based data. These options provide more flexibility in terms of what data may be saved and how it is stored and maintained.

Access Management

Databases rely heavily on access control. It assures that only authorized persons or systems have access to specified data reading and writing. Traditional databases handle access control by user authentication, in which users log in with unique credentials, and their access is limited depending on their allocated rights.

Decentralized databases, on the other hand, provide built-in programmable access control logic that uses ZK-proofs, public/private key pairs, whitelisting addresses (public key only), or decentralized IDs. Granular control over data access is enabled by these systems, allowing for fine-tuned permissions and security standards.

While some databases contain built-in encryption, others use the Lit protocol, a distributed key management method, when data must be kept secret.

The landscape of Decentralized Databases

ComposeDB

ComposeDB is a graph database based on Ceramic, a framework for developing apps with composable data. To understand ComposeDB, you must first understand Ceramic and how it works.

Anyone may use Ceramic to explore, produce, exchange, and build on top of existing certified data. Each piece of information is represented by Ceramic as a Stream, which is simply an immutable event log that records every modification made over time. Each Stream defines a StreamType, which is a framework that enables developers to deploy different kinds of data to the network while defining their own criteria for how the data is processed and stored. StreamTypes are in charge of enforcing the Stream's rules and logic, establishing everything from the data structure and content form to access control/authentication and the consensus mechanism.

StreamTypes enables developers to establish their own access control rules to permit new data to be uploaded to a Stream, with DIDs being the most frequent and powerful technique.

ComposeDB extends Ceramic to give developers a composable graph database for efficiently building apps. To establish the graph database schema for an application, ComposeDB provides Composites, which are composable building pieces made up of groupings of data streams. Furthermore, ComposeDB improves data reusability on Ceramic and provides a Composites marketplace.

Ceramic's horizontal scalability is one of its primary characteristics. In contrast to typical blockchains, which have a global execution environment, each Ceramic node functions as its own virtual execution environment. This method enables nodes to choose which streams to check and store, enabling Ceramic to easily manage the size of the world's data.

Tableland

tableland database model

Source: Tableland

Tableland is a SQL database that divides a traditional relational database into two major components:

The access control mechanism is handled by an on-chain smart contract that uses EVM.
An off-chain permissionless network of relational database nodes.

Each table (database) is issued as an NFT (ERC-721 token) on an EVM-compatible base layer (now supported are Ethereum, Arbitrum, Optimism, and Polygon). The table's access rights are controlled by the on-chain owner. The owner may restrict who can write to the table, but read queries can be performed by anybody since they happen off-chain through the Tableland gateway. Off-chain table construction and changes also occur on the Tableland network, as shown by the on-chain smart contract.

Tableland's hybrid method has the benefit of protecting data on-chain while still enabling native composability through off-chain queries. However, since it is on-chain, the cost of writing to the database is depending on the selected base chain. As a result, use cases with lower write frequency and less emphasis on performance are better suited for Tableland.

Space and Time

Source: Space and Time

Space and Time is a one-of-a-kind SQL database that serves as a hybrid transactional/analytical processing (HTAP) data warehouse (as opposed to conventional databases that solely do transaction processing). The data warehouse is a network of database and computes clusters that enable developers to effectively handle and analyze data contained in their databases using ZK-proofs. The network also indexes key blockchains, allowing developers to effortlessly merge on-chain and off-chain data in SQL format.

Space and Time is essentially a serverless backend for Web3 applications. Space & Time may appeal to developers looking to speed up the construction of decentralized apps owing to the availability of off-chain computation, indexing services, and databases.

Polybase

Polybase is a ZK-rollup key-value database with an emphasis on privacy. It provides SDK-based capability equivalent to Firebase or Supabase, but with integrated Web3 permissions. To ensure correct data storage, query processing, and access authorization enforcement, ZK-proofs are used. Polybase's design allows for the inclusion of several data storage and indexing providers, giving developers the opportunity to determine the tradeoffs for their specific use case.

Databases based on Arweave

Arweave acts as a global, permanent hard drive, allowing users to store data, retrieve it quickly, query it in a variety of ways, and employ smart contracts. Its particular design makes it ideal for building databases on top of it.

Arweave takes a novel way to compute. Computations on conventional blockchains are performed on-chain by full nodes, which subsequently construct and maintain the network's global state. Arweave, on the other hand, only employs its blockweave for data storage and performs calculations off-chain on the user's end. As a consequence, the cost of Arweave is completely dictated by the quantity of data kept on-chain.

The Storage-based Consensus Paradigm (SCP), which separates computing from the chain, tackles the scalability issue with blockchains. The SCP works on Arweave because of two factors: the network's determinism and its cost-effective approach to permanent data storage. Off-chain computations provide the same state as on-chain computations since data inputs are kept on-chain.

The effective deployment of SCP has paved the way for the creation of a plethora of databases on Arweave.

Arweave Databases

Arweave databases, in general, work as follows:

A transaction with an initial state is saved together with logic that determines the database's data structure and access rights.
If the data written to the database is legitimate, it is checked against the original transaction and logged as an updated operation.
A SmartWeave client uses the contract source code to run the history of inputs sequentially to determine the status of the database.

The process is dependent on two key pieces of infrastructure: Warp and Bundlr. Warp is an Arweave-based smart contracts technology that implements the SmartWeave protocol. Bundlr is placed on top of Arweave as a layer. It combines transactions to decrease expenses and increase scalability while providing rapid data access.

Arweave supports four separate databases:

WeaveDB: A smart contract-based document database that leverages whitelisting addresses for access control logic.
HollowDB is a smart contract-based key-value database that combines whitelisting addresses and ZK-proofs to secure data verifiability. ZK-proofs are also used to confirm the veracity of data.
Kwil: An SQL database that runs its own peer-to-peer network but utilizes Arweave as its storage layer. Its access control logic is based on public/private key pairs, and it has its own consensus process for data verification.
Glacier: a NoSQL database designed as a ZK-rollup that uses Arweave as its data availability layer. It employs public/private key pairs in its access control logic and ZK-proofs in its data verification.

Use Cases for a Decentralized Database

Decentralized databases not only improve existing Web3 apps, but also allow the creation of new applications and experiences in a variety of industries.

Decentralized Social – By keeping the social network on a decentralized database, users will have more control over their data, will be able to move across platforms, and will have more potential for content monetization.

Gaming — An key part of blockchain-based games is managing and preserving player data, in-game assets, user settings, and other game-related information. Decentralized databases can guarantee that this data is easily traded and added to other applications and games.

Dynamic NFT information may be kept in a decentralized database to allow the issuance of on-chain credentials as NFTs that can change depending on particular user activities or successes. It may also be used to monitor the movement of physical products in a supply chain, where the metadata mutates to offer real-time information on the goods' condition and position.

Machine Learning & AI — Using a decentralized database with ZK-proofs inside a federated learning environment would allow data to be shared with just the nodes engaged in the training process and ensure that only authorized individuals may edit the dataset.

Decentralized Science (DeSci) — Decentralized databases allow DeSci DAOs to store and regulate data access while guaranteeing that data producers are properly recognized and paid.

Decentralized Physical Infrastructure (DePIN) — DePIN networks create real-time data that must be saved, such as hardware device location, performance, consumption, and uptime statistics. Using a decentralized database to store this information would enable other apps to integrate and utilize it smoothly.

Arguments Against Decentralized Databases

Lens protocol, the premier Web3 social graph, demonstrates that decentralized databases are not a one-size-fits-all solution. Lens was initially designed as a smart contract on Polygon, with profiles, social graphs, and social activities (posting, following, etc.) performed on-chain. The actual post content, on the other hand, was kept on IPFS and Arweave. The Lens team chose to construct their own indexer that fed the on-chain data into their centralized SQL database, making it accessible through their API.

Momoka, a scalability solution that executes Polygon transactions off-chain, was recently revealed by Lens. While Polygon will continue to analyze profiles and social graphs, Momoka will manage social activities. Transactions are generated on Momoka, but instead of broadcasting them to the network, they are sent to Arweave, which acts as the data availability layer. On Momoka, anybody may run a trustless verifier node, which also indexes the data on Arweave.

Lens is able to employ an established and tested centralized database while yet keeping some degree of decentralization since anybody may index the data on Polygon and Momoka. While users must rely on the team to provide correct query responses, the ability for others to validate Lens' activities helps to mitigate any dangers.

The choice to incorporate Arweave into Momoka highlights the scalability challenges associated with putting data directly on-chain. If additional protocols use Arweave for storing transactions and indexing data in a centralized database, they may be able to strike a balance of data composability and ownership without completely decentralizing the database. While this model may be effective for certain applications, such as Lens, others, such as DeSci, may benefit more from a truly decentralized database.

Conclusion

Each decentralized database project examined in this article used a unique strategy with varied tradeoffs aimed at certain market groups. As a result, developers will choose a decentralized database depending on the specific needs of their application. They must examine aspects such as the data format, frequency of data updates, and the appropriate security and access restrictions.

Scalability is the primary risk associated with projects developed as smart contracts or rollups. In the future, use cases may need thousands of data write or changes per second, causing these databases to struggle to provide the necessary throughput. Due to the network's storage-based consensus model, Arweave's smart contract-based databases seem to avoid these scaling concerns.

Decentralized databases may not be required for all applications. Depending on the individual demands of an application, it may be sufficient to store data directly on-chain and index this data into a centralized database, given the application does not need significant rates of data writes or changes. The main benefit of a decentralized database, however, is its potential to ease the app development process by providing developers with a decentralized solution for all of their data storage, retrieval, and updating requirements.

Disclaimer:
This article is intended for and only to be used for reference purposes only. No such information provided constitutes advice or a recommendation that any investment or trading strategy is suitable for any specific name.

Decentralized Database: The last piece for web3 stack