Guest User Guest User

Protocol Design in Action: Insights from Irene Giacomelli

Step into the world of cryptography and protocol design with Irene Giacomelli, Protocol Researcher at FilOz, as she shares how she bridges theory and real-world application to tackle the unique challenges of Web3. 

Background 

We love the title protocol researcher. Can you tell us a bit about your background as well as what a protocol researcher does?

Irene Giacomelli: I came from “theory land” in the sense that I started my job career in academia. I have a master degree in mathematics and PhD in informatics and I spent a few years in academia publishing papers on cryptography before joining the industry. I started with Protocol Labs in 2019. The fun fact is that when I joined it, I didn't know much about blockchain or Web3. I just knew the basic definitions in the space.

I joined to work on the cryptographic aspect of Filecoin – which at that moment in time was still in the design phase. They needed a protocol and cryptographic researcher to understand and overcome the challenge of constructing a blockchain that used useful space as the underlying resource for consensus.

There was no such thing at the time and even today there are few blockchains that use space as a provable resource for consensus. There were people already working on these topics before I joined and I learned a lot from them. I focused on bridging the theory with the real world. Meaning that results from academic papers often need to be adapted to work with the constraints from a real-word protocol like Filecoin. That is a crucial part of being a protocol researcher.

I kept working on Filecoin core protocol even after we launched the network, mainly shipping Filecoin Improvement Proposals (FIPs) and keeping an eye on the security of the network. Now that core protocol is more time-tested, the focus for protocol designers like me has shifted from the initial core work to adding new features and capabilities. We transitioned to where we’re saying, “Let's look at this as a product, as a storage network.” to “What features are needed to increase adoption?” and “What can we build – or even better – what can we unlock so that others can build these types of features for the network?”

The mission of FilOz is to support Filecoin and allow everyone in the ecosystem to be able to use it and improve it. This fits very well with the line of work I was doing. And so in April, I was happy when they asked me to join the team and I was happy to say yes.

DESIGNING Web3 PROTOCOLS

Let's talk about protocols. What makes working with Web3 protocol so challenging?

Irene Giacomelli: There are a few challenges when you try to design a protocol in Web3. One challenge comes from the fact that web3 protocols need to work in the decentralized world. So here you have a bunch of parties that can collude or create Sybil attacks and there is no trusted central authority to fall back on. Which makes for a design challenge because in classic cryptography it's natural to assume that two different entities – two parties in the protocol – won't collude because they have different interests. In decentralized protocols, this is not the case. You cannot assume this and so you need to evaluate all possible attack vectors. Also, more parties means that the design needs to be built to be able to scale, which is not always trivial.

Another difference is how to use crypto incentives – how to use economic incentives the right way.. In classical cryptography, you usually want to prove that something is impossible to happen or that the probability for it to happen is very low. In protocols with economic incentives, it's not always white on black, there are multiple scenarios that can occur and you rank all these using profit functions. This is a challenge but also a resource – incentives can solve problems that classical methods cannot.

And so you are using incentives to solve problems. So can you give examples?

Irene Giacomelli: Filecoin itself is an example of this, right? In general, we have cryptographic proofs that act as a certificate of the service and that can trigger payments. The great idea of Filecoin is linking that to block mining. So now, storage providers can publish proofs that grant them the right to be elected to create a block and earn the related reward.

You might think of doing the same with retrievals using a proof of delivery , but it's actually impossible to do so in classical cryptography. We actually can prove that such a thing like“proof of delivery” is impossible to exist. This is a known limitation in cryptography that is linked to the well-studied fair-exchange problem. However, with the right incentives, we can design a protocol that overcomes this limitation and provides a workable solution.

I worked on this at CryptoNet Lab (Protocol Labs), and the result was retriev.org. This is a retrieval insurance protocol that uses crypto incentives to ensure guaranteed delivery in decentralized storage networks like Filecoin. It’s something that I hope to work more on after a few of these more near-term projects are completed.

What motivates you to do protocol research and design?

Irene Giacomelli: For me in particular it’s to see something that I worked on in theory as a protocol design go live. Let's say I design or co-design a protocol with a set of drawings and instructions, detailing what it should do and how it should work. That’s nice, but what I really care about is that this can go into the dev pipeline and be implemented and be used – so that what I designed really makes an impact.

RECENT WORK

What problems are you working on now, especially when it comes to opening up new opportunities for the network?

Irene Giacomelli: We are working on exactly what you said – opening up new opportunities. In this case we are working on shipping a new proof system. What we have today on Filecoin is a proof system based on two pieces. First we have the PoRep (Proof of Replication) that allows a storage provider to prove that they have your data encoded in an incompressible form. Then we have repeated PoSt (Proof of Spacetime), which allows storage providers to prove that the encoded data is kept stored for a specific amount of time. The incompressibility property is expensive to achieve but it's needed for consensus. Indeed, in Filecoin, consensus power is proportional to the “spacetime” (i.e. space through time) resource that is committed to the network.

We are adding a new proof system that while it doesn't offer incompressibility guarantees and cannot be used for consensus, it is much more efficient to generate and it can still be used to prove data possession to both the network and clients. This allows the Filecoin network to offer different types of storage services and business models.

For example, with the new proof system, storage providers can store data in an unencoded format for fast retrievals, which is not possible (without additional copies) with the current PoRep + PoSt system. This opens up new use cases for the network and new markets for storage providers.

What's this new proof called?

Irene Giacomelli: It’s called PDP – Proof of Data Possession and it is not necessarily a new concept. This has been known in the literature and used in classic client-server storage systems. What we are doing at FilOz is – and this goes back to what we're discussing about challenges for web3 protocols – is taking care of solving the challenges and making this protocol scalable to be shipped via smart contracts on a blockchain and used by applications and other protocols as a native proof.

SOLVING REAL PROBLEMS

What other things excites you about the space? Are there other proofs? Are there other protocol elements that really excite you?

Irene Giacomelli: Retrievability for sure. It's something that I would like to see improved for Filecoin. Another interesting open problem is designing a decentralized protocol for reporting performance metrics without a trusted authority. This could apply to retrievals as well as other services. Typically, a centralized entity tests providers and reports on their performance but decentralization removes this option. In Web3, there is no single entity that is testing.

The members of a network can perform tests but there are few problems with this. For example, how do you handle cases where nodes report different values for the same metric? How can we ensure the network doesn’t collude with providers? So a colleague and I worked on this for a little time with an academic researcher’s help on this. But it's still an open problem and I think it is a really exciting problem for both academic research and practical Web3 solutions.

What should the community know about you and the other protocol researchers and core devs?

Irene Giacomelli: That what we care about is solving the real problems. We always try to talk with storage providers and possible clients and people who are interested in using Filecoin to better understand the problems. Our goal is to keep expanding Filecoin so that there are many meaningful and practical storage applications and supporting protocols.

They should ask us about a feature or complain to us if one is missing. Let us know the pain points they are having because we like to solve these issues. Because in principle, as a protocol designer, I could just design a beautiful protocol that solves a random problem, right?

There are beautiful protocols everywhere and I can get excited about these because they are actually beautiful theory work, but if they're not used and if they don't solve a real problem of Filecoin, we miss the real occasion to perform.

As a group, we all like to work on proofs and protocols that go in the direction of solving real problems and getting real use.

 

Contact Irene or follow her work at:
Website: https://www.filoz.org
X: https://x.com/Irene_2911 
LinkedIn: https://www.linkedin.com/in/igiacomelli
Filecoin Slack: @irene
GitHub: https://github.com/irenegia

Read More
Guest User Guest User

The DSA Announces Leap Forward in ZK Proof Computation

Select ZK Proof Advancements Now Integrated in Filecoin


MARCH 19, 2024 - The Decentralized Storage Alliance (DSA) recently released dramatic cryptographic processing improvements that stand to benefit zero-knowledge proof (zk proofs) processing within the Filecoin network as well as throughout the Web3 infrastructure stack. These advances optimized several key computational algorithms, as well as improved CPU and GPU processing flows to reduce cost and improve performance by up to 80%. The advances reduce what used to take many servers and combine it into a single server thereby gaining additional processing efficiencies. Filecoin development teams have integrated select improvements directly into Filecoin, making them accessible to the over 3,000 Storage Providers in the Filecoin network.

The Decentralized Storage Alliance (DSA) brings together technology industry leaders to help enterprises make the transition to decentralized storage technologies through reference architectures, standards, education, advocacy, and best practices. The work was part of an initiative by the DSA to improve performance and energy usage within the Filecoin network and was led by Supranational, a US-based firm that designs and develops hardware accelerated cryptography for verifiable and confidential computing. The team improved the process to onboard data onto the Filecoin network with emphasis on phases of the process that are heavily zk-SNARK dependent, which is a specific type of zk proof.

ZK proof processing is particularly time and resource intensive. And while the Filecoin network already supports GPU acceleration to assist with zk-SNARK processing, these recent advances further optimize the estimated over 100,000 GPUs in use – representing around $100M of deployed capital – operating within the Filecoin network.

Filecoin Network is the Highest User of ZK-SNARKS in Web3

ZK-SNARKs allows one party to prove to another that a certain statement is true, without revealing any information about the statement itself. Zk proofs in general, and zk-SNARKs in particular, have become fundamental building blocks for storing trusted and verifiable data within blocks and data sectors contained within blockchains.

ZK-SNARKS are heavily used in the sealing process of the Filecoin network, which is the process by which data gets onboarded or loaded on the network in a cryptographically secure, but verifiable manner.

Every day, over 1.7 million zk-SNARK proofs are created and verified on the Filecoin network – making the Filecoin network the highest known user of zk-SNARKs in the world and which is larger than Ethereum, zCash, Mina and many other networks combined.

This evidence is derived from gas metrics from FilFox, a Filecoin Network block explorer, which shows ~170,000 ProveCommitSectors operations and ~90,000 SubmitWindowedPoSt operations. The first operation processes 10 zk-SNARKs each and the second processes an additional zk-SNARK – the total of which then adds up to 1.7+M zk-SNARKs.

Optimization of ZK-SNARK Computational Primitives

A key to achieving the 80% reduction in cost and latency in generating Filecoin’s SNARKs was the use of improved algorithms and software implementations. Optimizations to the GPU algorithms were enabled for the specific computations that are being employed in the Filecoin SNARKs. These include multi-scalar multiplications (MSM), number theoretic transformations (NTT), and Poseidon hashing, three core computations found in many zk proof systems.

  • Multi-Scalar Multiplications (MSM) – MSM is a key computational primitive in zk proofs. The algorithm is used to calculate the sum of multiple scalar multiplications, and it is often used to commit to a polynomial in SNARK systems. MSM operations were accelerated through the use of optimized assembly that improves the efficiency of large integer arithmetic.

  • Number Theoretic Transformations (NTT) – NTT is a mathematical approach that facilitates efficient polynomial multiplication. This implementation makes use of Fast Fourier Transform-style algorithms giving the algorithm a computational complexity of O(nlogn). NTT operations were accelerated by developing a more efficient implementation of the core NTT algorithm.

  • Poseidon Hashing – Poseidon is a ‘SNARK-friendly’ cryptographic hash function. It is often used in SNARK systems as it requires up to 8x fewer constraints per message bit than previous ‘SNARK-friendly’ hashes such as the Pedersen hash. Poseidon hashing was accelerated through an improved kernel implementation that leverages more efficient data transfers. 

This work advanced the state-of-the-art performance for Groth16 proof generation. Groth16 is a widely used proof system throughout the blockchain ecosystem and as such this project represents a breakthrough in the performance of ZK cryptography more broadly for blockchains and beyond. The work is open sourced and can be used to accelerate other zk proof-based systems.

Sealing/Onboarding Data on the Filecoin Network

Two phases of the sealing or data onboarding process on the Filecoin network rely heavily on zk-SNARKs in their operations. One phase computes a number of zk-SNARK proofs, and the other verifies them as part of the continual proof phase.

Whereas the term “block” is used for blocks of transaction in transaction-based blockchain networks, the “sector” is used for data stored on the Filecoin network. Before a sector can be onboarded onto the network, a Storage Provider within the network must seal the sector or in other words, encode the data in the sector to prepare it for the proving process.

Sealing a sector is done via a Proof-of-Replication process which is a computation-intensive operation that results in a unique encoding of the sector. Once data is sealed, Storage Providers (1) generate a proof-of-replication, (2) run a SNARK on the proof to compress it, and (3) submit the result of the compression to the blockchain as a certification of the storage commitment.

Project Origins and Team

Supranational, the developer behind these performance improvements, is a long time contributor to the Filecoin ecosystem. Supranational develops blst, the core cryptography library used in Filecoin’s consensus protocol, and also assisted with initial optimizations to Filecoin’s zk proof system before the network’s launch. 

Recently, the team has been working on sppark, a library designed to accelerate a variety of zk proof systems. The sppark library is leveraged in the recent improvements to Filecoin’s proving system, and is also used by other Filecoin ecosystem participants including Lurk Lab’s implementation of the Nova proof system.

“The software optimization work for this project was extensive,” said Kelly Olson, one of the leaders of the team, “While there is always room for additional improvements, the current software is quite efficient and the cost of producing these proofs is low enough that it is no longer a concern for storage providers.”

Release and Availability of the ZK-SNARK Enhancements

All Storage Providers can now take advantage of select improvements that have been integrated directly into Filecoin.

“These enhancements will reduce costs for the entire network, resulting in further cost savings for end users of decentralized storage technologies" said Daniel Leon, Founding Advisor of the DSA.

Storage providers are able to purchase common single-server hardware solutions from vendors with these enhancements included. These servers are plug-and-play ready meaning that they can be powered and connected to the network and perform sealing operations with limit to no customizations.

Future Possibilities  

Building off of these enhancements, Supranational, in collaboration with Protocol Labs and the DSA, have defined a ‘SNARK-as-a-Service’ API which will allow these proofs to be outsourced to providers who specialize in proof computations, thereby further simplifying operations and enabling specialization in the Filecoin ecosystem. 

This type of zk-SNARK compute specialization is one of the firsts of its kind in the Web3 / blockchain space but is likely to be replicated across networks that are heavily zk proof dependent. Different algorithmic enhancements and circuit-specific implementations might have to be employed but the model for doing so using GPUs is one that holds great promise for reducing costs and compute times across the entire Web3 ecosystem

**

About Supranational

Supranational designs and develops hardware accelerated cryptography for verifiable and confidential computing. The team has decades of experience in algorithmic optimization across CPU, GPU, FPGA, and ASIC platforms. Accelerated cryptography developed by Supranational is currently in production in blockchain networks such as Ethereum, Filecoin, Optimism, Polygon, Aptos, Sui, Chia, and more. For more information visit https://www.supranational.net/ or e-mail hello@supranational.net.

About Decentralized Storage Alliance (DSA)

​​The Decentralized Storage Alliance (DSA) is a professional organization launched in October 2022 by a community of companies who imagine a better path to storing data through decentralized storage solutions. Its charter is to drive industry growth, bridge the gap between Web2 and Web3 storage technologies, and highlight advances in technology an

Read More
Valeria Valeria

Decentralized Storage Reduces Costs by 40%

Sealing Optimizations Release (GA) open sourced and ready for testing.

“The new open source software optimizations have achieved remarkable performance improvements and cost reductions for onboarding data to the Filecoin network. This first DSA release will remove data onboarding bottlenecks and lower the overall cost of storage for the decentralized storage industry by up to 40%.”

Daniel León, Alliance Lead, Protocol Labs

The Decentralized Storage Alliance (DSA) brings together technology industry leaders to help enterprises make the transition to decentralized storage technologies through education, advocacy, and best practices. The DSA, in collaboration with Protocol Labs and Supranational, unveil the first release of an advanced software and reference configuration that will fundamentally optimize data onboarding for the Filecoin network. The new software significantly reduces costs for network participants, making it easier to both become a Filecoin Storage Provider and onboard data to the network. These network optimizations push forward decentralized storage technologies, advancing the industry and enabling further adoption by enterprises.

The Filecoin Network relies on computations performed on the data being onboarded in order to ensure network security and enable cryptographic proving, verifiability, and data immutability. However, the hardware required to execute this process, known as "sealing," often accounts for more than 50% of the total hardware expenses. This poses a challenge for new Storage Providers aiming to join the network and for existing ones seeking to enhance their data onboarding capacity. To address this issue, we are excited to announce the release of new software that optimizes data onboarding for the Filecoin network, resulting in sealing server cost reductions of up to 90%. By leveraging this software, network participants can benefit from decreased compute costs while experiencing increased compute power. Please see below for a breakdown of cost and performance metrics calculations. Although the benchmarks are for CC,  we expect to see similar results for non-CC as well.

We are pleased to introduce the new software optimizations along with optimized hardware reference configurations. The impact on Storage Providers will vary based on individual hardware and setups; nevertheless, the entire ecosystem can anticipate improved performance and reduced costs with 75-85% increased throughput to PC2 and C2 that will be integrated directly into Lotus. The new optimized software with optimized hardware reduces sealing server costs by up to 90% for Storage Providers, leading to an up to 40% overall cost reduction to decentralized storage. As an example of cost savings, it used to cost over $200,000 to purchase sealing servers to seal 20 TiBs per day. It will now cost under $25,000 to seal 20 TiBs per day. Furthermore, Storage Providers will benefit from a noticeable reduction in operational costs as a single server can now efficiently handle all sealing tasks, achieving a remarkable 6x+ higher throughput.

“Improving the performance of the cryptographic operations in Filecoin’s proof-of-replication and zero knowledge proofs is key to enabling new use cases and scaling the network. By dramatically lowering the costs associated with this cryptography, decentralized storage is able to better compete with centralized offerings.”

Kelly Olson, Co-Founder, Supranational

This initial GA early release is open sourced under a permissive license, allowing it to be accessed by all Storage Providers in the Filecoin network today. Network participants can now explore and start testing the new software. Alongside the release of the software, the DSA is releasing optimized hardware reference configurations that will enable Storage Providers and service providers interested in the full suite of optimizations to start planning on establishing optimized hardware configurations for this new optimized software.

While this GA early release provides basic preliminary access to the new software, additional following releases will make it easier for the ecosystem to realize the benefits of the optimizations. Open source orchestration software and DSA reference architectures will make it easier to use the new software off-the-shelf, hardware upgrade instructions will provide guidance on how to update current hardware configurations, and PC2 and C2 optimizations will be integrated directly into Lotus in Q3.

To explore and test this early release, visit the DSA website: https://dsalliance.io/

DSA Release: SupraSeal
About DSA
About Supranational
About Protocol Labs

Read More