If you’ve been keeping an eye on the developments in the DePIN sector of the Web3 realm, you might have heard that our partners, the Streamr Network, have successfully released Streamr 1.0.
The core updates in this release focus on full decentralisation, bringing several notable innovations:
- Coordination of nodes without trackers, using a Distributed Hash Table instead.
- Improved message compression during transportation by opting for binary formatting.
- Measuring and leveraging the shortest distance between nodes required for message transportation from software A to software Z.
- Proof of relay node inspection layer involves verifying peer-node message delivery to enable automated flagging and misbehaviour detection.
- New tokenomics to support additional incentives for a secure and speedy network.
Additionally, Sponsorships are an incentivised mechanism where intermediary nodes help propagate messages from publishers to subscribers, optimising message delivery for security and speed.
We know all of this because we’re developers who read and leverage Streamr technology to enable the Log Store Network — a decentralised, replicated, and fault-tolerant message store for data transported over Streamr and other real-time data systems.
The key takeaway with this upgrade is that it is essentially a hard fork of the original network, now referred to as Streamr Brubeck.
You can determine which network you’re accessing based on the version of the packages you are using to interact with the Streamr Network.
For our developers:
streamr-clientversion 8.5.5 is the final state of the Brubeck Network and is marked for deprecation.@streamr/sdkversion 100.* represents the state of the live 1.0 Network.
The Log Store is Compatible with Streamr 1.0
The Log Store Network is designed to extend the Streamr Network with caching, storage, and message delivery capabilities. With the Streamr 1.0 upgrade, the Log Store Network has also undergone necessary updates.
We are pleased to announce that this upgrade has been successfully completed, and the Log Store Network is now fully compatible with the latest Streamr 1.0.
However, this was no simple task.
Due to the deep integration and reliance on various development environment technologies, Usher Labs needed to implement compatibility with the new message transport formats, new cryptographic primitives, and new historic data lookup paradigms. Additionally, we had to reconsider how data verification takes place on various destination systems that use the Log Store as a component to enable data protocols—a set of conditions enforced on data as it’s received and processed by a blockchain.
This consideration is also critical for various projects in the DePIN sector aiming to leverage real-time data to verify its origin, frequency, and source. Verification plays a significant role in data protocols as it authenticates external data to facilitate digital asset management and reward distributions to Technology Resource Entities (TREs), such as physical resources in IoT and compute networks. Authenticating this data maximises user trust by minimising reliance on the data technology that facilitates digital asset management. You can learn more about this issue in a research article: Secure Web3 with Trusted Data.
The result of this upgrade not only ensures compatibility but also allows our team to collaborate on minor fixes and set a new roadmap for the Log Store. This roadmap aims to improve sovereignty, reducing bugs and network issues when utilising both Streamr and Log Store. An example of sovereignty is how IPFS can be used independently of Filecoin, yet together they form a decentralised storage solution for large (> 32GB) data blobs. Despite this, it is clear that Streamr has successfully delivered technology and a decentralised network for peer-to-peer data transport, which the Log Store integrates to receive, store, and enable access with high availability.
Our goal for the Log Store moving forward is to become the IPFS for real-time data, with persistent identifiers for rapidly growing datasets.
The Bottlenecks of Real-Time Data
If you’ve managed real-time data in the past, you’re probably familiar with the challenges of packet loss and data consistency. Streamr is no exception to these issues, which is why the Sponsorships improvement was introduced.
Packet loss can occur due to faulty internet connections, downtime, or other issues affecting either the publisher or the subscriber. This can result in discrepancies in packets between nodes due to various factors, including network instability. While there is no complete solution to this phenomenon, technology systems typically mitigate the risk through increased data replication and horizontal scaling. These measures significantly reduce the likelihood of packet loss but do not eliminate it entirely. Apache Kafka, a widely adopted centralised streaming solution, exemplifies this approach. Kafka ensures reliable data replication by allowing multiple nodes to replicate data, but only one node needs to acknowledge the receipt of a message to ensure its durability. This approach guarantees strong data replication consistency and high availability through its distributed architecture and scaling methodologies.
Packet consistency is a design methodology in streaming networks that balances the speed of data delivery with consistency in data replication and availability. Streamr opts for an eventually consistent data delivery approach. This allows packets to be shared and propagated through the network with fewer checks and balances, which can result in scenarios where data appears in one subscriber node earlier than another. Latency issues exacerbate this problem, especially when one node is in Helsinki, Finland, and another is in Sydney, Australia. The propagation mechanism in Streamr Sponsorships incentivises intermediary nodes to ensure secure data delivery to destination subscribers by reducing the latency between each node’s packet communication. This improves data replication and availability, preventing issues like connection timeouts or latency from jeopardising the security and speed of data delivery.
The Log Store Network doubles as another mechanism through which a network of nodes can cache data in a highly replicated manner. This ensures that destination systems can always look up and query data at a later date on an ad-hoc basis.
The Underrated Aspect of the Streamr Protocol
While the Streamr Network is widely recognised as a solution for real-time data transport in the Web3 realm, an often undervalued aspect is its embedded protocol.
For publishers to communicate with subscribers, each packet includes metadata that forms the cryptographic security underpinning the network’s peer-to-peer capabilities. Every message must be signed, ensuring that anyone receiving the data can trace its origin.
The research article mentioned earlier details how cryptography is a mechanism to enable new knowledge. In Streamr’s case, it enables high-frequency data traceability. This traceability is crucial not only for DePINs attributing rewards to TREs but also for decentralised tracking technologies, providing shared and trust-minimised observability over an entire decentralised network.
The Solana Blockchain already employs solutions such as Solana Status and Solana Metrics, showcasing an advanced approach in some respects. However, trust in these metrics is assumed by the operator of these centralised technologies. While this may suffice in certain aspects, the evolving Web3 realm with app-chains necessitates incentives for OS-level telemetry to enhance the security of app-chains. These metrics can indicate the health of an operator’s node, pre-emptive security risks in specific node operators, and allow developers to assess the computational resource requirements of their network.
Coupling this with the Log Store enables anyone to launch a local portal for analysing an entire app-chain based on highly trusted and traceable observation data.
The Log Store is Now Fully Streamable and Inconsistently Distributed
What sets the Log Store apart from all other storage solutions, especially those handling real-time data, is its radical and unique approach:
- Nodes do not need to share the same data.
- Queries can be any size.
Due to the traceable nature of the real-time data collected by the Log Store and the inherent bottlenecks of real-time data, Log Store Nodes supporting the storage of messages in transit do not need to have the exact same data replicated. In essence, trying to achieve exact replication would eventually become physically impossible. Instead, the Log Store allows nodes to observe incoming data and store it independently, without regard for how other nodes manage the data. At the point of observation, via a query to the Log Store, nodes coordinate through a unique gossip protocol, where they eventually propagate the active state of the network relevant to the given query.
For example, Node A may have different data from Nodes C and B. If a query is made to Node B, Node B will aggregate the total data across all nodes to serve the query.
What’s even more impressive, and now enabled through this upgrade, is that queries to the Log Store have no size limit. If you query for data since the dawn of time for a particular data stream, the Log Store’s internal gossip protocol will deliver a response in the form of a data stream. Furthermore, there is no data duplication, and the data reflects all the information stored by all nodes in the Log Store. This represents a massive improvement in how decentralisation and real-time data integration function together.
Consider a video streaming network. An entire video can be inconsistently distributed among all its viewers. A new viewer watching the video can experience hyper-low latency by sampling data from all prior viewers. This is what we mean by IPFS for real-time data.
Who’s Using Log Store?
Usher Labs has been hard at work integrating the Log Store into various prominent technologies to facilitate adoption by forward-thinking companies and projects.
Our biggest highlight is an integration with Kwil. Here, DePIN projects can achieve end-to-end decentralised data management, where TREs can push real-time data over Streamr and have it collected by Log Store Nodes that function as sidecars to Kwil Nodes. The result is real-time data streams integrated into a decentralised Postgres platform, allowing a DePIN operator to be fully transparent about the networks and data. This transparency enables their communities to engage, analyse, and cross-reference the network’s data.
Powerpod is an early adopter of this technology, enabling their team to focus entirely on offering the best device and hardware technology, as well as customer-centric support and experience.
The Log Store is also integrated with Streamr’s native storage interface. As detailed in a previous post, the Log Store acts as Streamr Storage ++, offering high replication and fault tolerance for anything published to the Streamr Network via their existing SDKs and interfaces.
Finally, projects such as Opsec, Truflation, and ChainSight on the Internet Computer are preparing for integration with the Log Store. These use cases span decentralised data availability for real-time applications, network observability for security and trust analysis, and enabling open participation in data feeds and the creation of data protocols relevant to Oracle operations.
Getting Started with the Log Store
Learn more about the Log Store Network by visiting the Log Store Documentation.
To review how the Log Store Network works in practice using Streamr SDKs, view our tutorial.
To operate a Log Store Node for data storage in a completely standalone manner, check out the Node operators guide.
To learn more and stay up to date with Usher Labs, the team behind the Log Store, connect with us via Twitter and Discord.
