[2020 Theme Proposal] Solid foundation for future growth

[momack2]

Note, this is part of the 2020 Theme Proposals Process - feel free to create additional/alternate proposals, or discuss this one in the comments!

Theme description

In 2020, IPFS should invest in a solid foundation for future growth that creates a highly reliable, extensible, and performant base to iterate and improve upon. This means doubling down on testing, reliability, and performance improvements; creating new programs to support longer-term community growth; and making IPFS a much easier project to contribute to - through good specs, simplifying refactors, and better contributor support.

If achieved, this should make all current users of IPFS very happy with the performance and reliability of their current IPFS usage, while also unlocking new user groups that won’t consider IPFS until it is highly stable, performant, and ready to scale (like most web2 enterprises). This would also involve investing in our team and community - to ensure we’re ready and able to grow with healthy habits, well-supported maintainers, and easy on-ramps for new contributors. Finally, it should make us proud of our code health and development velocity - including investing in automation, tooling, and refactors that will make each IPFS developer more productive and impactful.

Core needs & gaps

Current developers of IPFS-powered apps need a solid, performant network to power their dapps and ensure they can deliver highly reliable decentralized services to their end users. A big pain point in the past has been DHT performance, but this also encompasses further investment in a solid release process, high-fidelity testing that ensures developers can rely on non-buggy releases, and good benchmarks that measure and improve performance for core use. Gaps in these areas have made it difficult for current users to develop large production-ready services on IPFS, and potentially dissuaded large-scale applications from relying on the IPFS network for their needs. Without focused investment in network reliability/performance and team support - significant investment to continue scaling IPFS will lead to fire-drills, missed expectations, and both user and team frustration.

Why focus this year

IPFS saw huge growth in 2019, and needs to continue landing performance improvements and stability fixes to be able to support the next set of demands from new dapps and ecosystems in alpha/beta right now. Continuing to invest in our foundations (testing, performance, reliability) will help ensure IPFS can scale gracefully to be the default file system for dapps built on Ethereum 2.0 and Filecoin. Similarly, the more/earlier we invest in contributor support and development velocity, the faster we can move and the more we can accelerate our value creation into future years.

Milestones / rough roadmap

This theme involves 4 simultaneous strands of work over the course of the year - with a heavier focus on 1 & 3 in H1, and increased investment in 2 & 4 in H2.

1. Scale the team

• onboard additional senior maintainers for the core implementations of IPFS and important functions like community, testing, infra, and more
• create a smooth contributor on ramp starting with visibility and outreach through how-tos, demos, and guides; to area hackathons and meetup programs; through the contributor onboarding flow and tooling; through joining working groups and feeling part of the community via {channels}; to ramping up contributions with support, metrics/tracking, and the things that keep people coming back!

2. Support contributor velocity

• heavy investment in team training ensures that all contributors are able to participate actively in review, triage, debugging, test writing, etc
• create tools and automation that leverage our efforts and help us work smarter

3. Solidify investment in testing / benchmarks

• all active core development work is validated and benchmarked by multiple layers of network testing
• all contributors can easily add tests and benchmarks for their use case or the module they’re refactoring
• testing is well automated and insights are visual and trigger alerts/actions when relevant

4. IPFS network and infra are solid and performant

• IPFS gateways and other network infra are ready for 10x growth (with runbooks, provisioning, chaos testing to prepare for the unexpected)
• testing validates that network performance “just works” and we eradicate inefficiencies and unpredictability through focused iteration cycles

Desired / expected impact

• We have a great foundation for future growth
• There are more senior core maintainers, and they are well supported and sustainable in their efforts
• We see many more contributors joining the IPFS project
• Current IPFS users are very happy with network performance / reliability
• New users with greater demands on network scalability, performance, and reliability are in active development/evaluation of the IPFS network for their needs
• The project is able to ship more things faster - because we’ve invested in busting velocity blockers
• Contributors feel empowered and proud of the quality and impact of their work
• We can more clearly measure, improve, and report on our performance due to improved tooling / benchmarking
• IPFS velocity, scale, and impact continue to accelerate in 2021 and beyond due to this foundational investment
• IPFS feels "ready for 1.0" in terms of reliability and stability

[jessicaschilling]

@momack2 - This is a fantastic summary of why a number of efforts combine into a single aim, but it seems documentation should be involved, too, since it’s especially critical to onboarding, scaling and continued adoption. Suggestion for primary summary statement, to be expanded upon later down as well:

This means doubling down on testing, reliability, and performance improvements; ensuring that both the on-ramp to and continuous use of IPFS are as frictionless as possible due to robust, well-presented documentation; creating new programs to support longer-term community growth; and making IPFS a much easier project to contribute to - through good specs, simplifying refactors, and better contributor support.

[momack2]

Agreed - documentation and developer usability (debugging tools, getting started guides, examples, how tos, diagrams, etc) definitely continues to be a significant part of contributor on-ramp & velocity, and end user reliability (since you can't rely on something if you don't know what it does / what to expect from it)

[Stebalien]

I wonder if there's a way to work "papercuts" into this. We have a lot of issues like:

• I try to install IPFS but then I can't run it because it's not in my path.
• I get an error installing IPFS because ipfs-update.
• I run ipfs cat ... and get an error about my repo not being initialized.
• I try to access the WebUI and it hangs because, e.g., I'm offline and haven't downloaded it yet.
• "Why can't anyone download files from me?" (e.g., warn about NATs when starting the daemon).
• Slightly broken progress bars.

On the other hand, these aren't usually issues for developer adoption, just end-user adoption, and that's not the focus.

[mikeal]

On the other hand, these aren't usually issues for developer adoption, just end-user adoption, and that's not the focus.

I don’t necessarily agree with this. You can expect developers to work through/around these more than end-users but this is also a big factor in developer adoption. The more “friction” a tool has the more proficient a developer will have to be in order to use it and work through that friction. Which means that the less friction the wider the pool for potential developer adoption.

That said, the most effective solution to solving “papercuts” I’ve seen is to broaden the contributor base. These are, by definition, not difficult to fix bugs, so they make great “first contribution” issues and PRs. But you can’t effectively grow the contributor base around issues like this if the review and maintainer process isn’t optimized for handling a larger volume of contributions.

[nuke-web3]

IPFS saw huge growth in 2019, and needs to continue landing performance improvements and stability fixes to be able to support the next set of demands from new dapps and ecosystems in alpha/beta right now.

AFAIK it is unclear if increased usage of IPFS will scale to enhance the usability, speed, and reliability of the network overall, or harm it. If the latter, it is critical to identify how to correct such that increased usage increases overall performance metrics. The default behavior of the clients, nodes, and gateways should optimize the network at the (reasonable and non-cumbersome) expense of the host machine. Ideally this tends toward removing the need for gateways in general.

[rklaehn]

I just want to chime in on this as well. I really like the gist of this proposal.

I think the most important thing right now is to make the core of IPFS solid and as fast as physically possible.

Ipfs is currently associated with sky-high ambition and a lot of good theoretical ideas. It is currently not associated with reliability or performance. Not just in my experience. Just read the comments for recent articles on hacker news about ipfs.

Let's change that. Let's make IPFS the thing in the stack that is so reliable and fast that people forget that it is there.

edit: I found the sessions at IPFS camp about the DHT very interesting. But as an application developer I should not have to know how kademlia works, just like when I open a TCP connection to australia I don't need to know about the details of the Border Gateway Protocol. If I open a TCP connection to wherever, all I have to know is that this amazing world spanning machine called the internet will do whatever is necessary to enable communication. And that is how it should be with IPFS.

[vasa-develop]

IPFS saw huge growth in 2019, and needs to continue landing performance improvements and stability fixes to be able to support the next set of demands from new dapps and ecosystems in alpha/beta right now.

AFAIK it is unclear if increased usage of IPFS will scale to enhance the usability, speed, and reliability of the network overall, or harm it. If the latter, it is critical to identify how to correct such that increased usage increases overall performance metrics. The default behavior of the clients, nodes, and gateways should optimize the network at the (reasonable and non-cumbersome) expense of the host machine. Ideally this tends toward removing the need for gateways in general.

Well said. I feel the same thing. The way the infrastructure services are growing in the community(I mean a few reliable gateways, giving it a centralized touch), can support IPFS only in the short term.

We need a p2p approach to this, just like filecoin is doing(building an incentivization system). As the number of users(where users also serve data) grows, the platform becomes more resilient. As this is a big differentiator b/w centralized and p2p models, we should see how we can transition from gateways to much p2p approach, in a smooth fashion.

[npfoss]

I'd like to second this theme as the objective for 2020, in particular because it would make literally every other theme proposal easier (or even impossible without it). I think the words "stable" and "usability" appear in almost every other proposal.

I think the main reason to choose this objective over any other is that the IPFS team cannot build all of the cool things we know it enables. There are just too many. Picking one as a focus would be a disservice to equally valuable alternatives, and you can't predict what's going to take off anyways. The only way all of those things get built is if other people want to build them, and the best way to make that happen is to provide a nice developer experience and good performance guarantees.

Developers are people too, and like anyone else they're only going to put up with a bad UX if they're getting a huge improvement over the alternative. Likewise, their users are only going to put up with the poor performance of their IPFS-backed app if they're getting a huge improvement. Making the experience better lowers the activation energy to switch to IPFS at all levels, not just developers.

---

Other descriptive names for this theme could be "Usability and Reliability" or "IPFS 1.0.0". (What's the threshold for version 1.0.0 anyways? When it's "stable enough", "functional enough", and "future-proofed enough"? That seems like a good target to hit ASAP, especially as more people are joining the network and community.)

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.