Congratulations team. I was involved in the project for quite some time years ago. Indeed I ported the original “Read the full index.xml into Java memory using a giant DB class” with the first “Stream entries from XML into the database, and make use of ContentProviders” because they seemed like the “Android way to do things” at the time. I also worked on the migration from XML to JSON metadata. At the time this was done, we needed to updated the metadata format to support internationalisation of metadata and the inclusion of images.
To see Torsten and others working on replacing my crufty-passed-their-use-by-date ContentProvider code with a modern Kotlin + Room implementation is heart warming. If any of you are reading this, please accept my deepest sympathies for having to pull that code out and rework it - that is not something I would have enjoyed doing! It is even better that it all lives in libraries that other clients can adopt if they choose so they don’t need to reinvent the wheel.
For those interested, yes, I am also responsible for writing the bulk of the code for the “new” (now several years old) UI in the offical client which often gets maligned on HN (and this thread is no exception). At the time I did my best to fight off edge cases and quirks of the Android system. F-Droid needs to ask Android which apps are installed and what their signatures are, then cross verify that with its own database to tell you whether updates are available. It also needs to know whether new apps have been installed outside of F-Droid since last time you opened it, etc. It also needs to pass downloaded .apk files to the system and request for them to be installed, then wait for confirmation from the system. The API’s Android provide for this kind of work, but they always seemed flakey, unwieldily, and slow to respond. The whole experience is full of race conditions, and each bistro seems to handle it ever so slightly differently.
I am still proud of how much we managed to achieve, and I’m also very pleased with the fact we were able to do so much great work around internationalised metadata, screenshots, encouraging donations to app developers, etc. But I’d be lying if I said I wasn’t still a little disappointed that I couldn’t iron out all of the kinks.
Life circumstances meant that I drifted away from the project and no longer contribute (other than via Liberapay donations to the wonderful fdroiddata team). However I would love to revisit the project in the future to see if I can address some of those edge cases to ease the user experience for all of us. You may have noticed that for such an amazing project, used by many people (not just the client, but the entire infrastructure), there has been comparatively few contributors. Despite that, I am still extremely fond of the value they bring to the Free and Open Source ecosystem.
Thank you for everything that you've done! The Android ecosystem is uniquely hostile, and it's hard for an outsider to understand the public and private battles that you've inevitably faced over the years.
Hey @pserwylo, great to hear from you, and I especially appreciate your message here with the background. And I'd like to highlight your point that there is much less contributor time than people imagine. If anything, the F-Droid client UX design process back in 2016 has proven to be an immensely efficient exercise it putting together a UX that still works decently. Plus we managed to predict that bottom nav would rise in popularity and those sliding sidebars, which were recommended back in 2016, would fade.
You're of course welcome to contribute again! The hard part is that app stores are large and complicated apps, when done fully, so that makes it hard to contribute to. We do mark issues with "first-timer" https://gitlab.com/fdroid/fdroidclient/-/issues/?label_name%... and "help-wanted" https://gitlab.com/fdroid/fdroidclient/-/issues/?label_name%... if anyone is looking for a place to jump in. I think we can also see this in all the various other clients like G-Droid, M-Droid, Foxy, Droid-ify, NeoStore, etc. Many rapidly stop being maintained, and others leave out key functionality like localization and automatic mirror selection because it is a lot of work to implement. The new libraries should make it a lot easier for forks to implement these features.
What is (or was) the benefit of using ContentProviders compared to accessing the database directly? They seem useful for exchanging data between apps, but I can't figure out what their role is when they're internal to the app (not exported).
Is it just that they're required for using CursorLoader, so that you can easily load things in the background without having to create a custom Loader implementation? And they're basically just a fairly inconvenient database abstraction?
When faced with the previous system [0] - where full table scans were the norm and filtering/sorting was done in Java afterwards - we certainly could have gone to proper database queries. I think it is mostly the first thing - that we wanted to move into the background instead of full table scans on the UI thread, and indeed CursorLoaders were an agreed upon way to do that with the (then, relatively new) RecyclerView.
But yes, you are correct that they are indeed an inconvenient database abstraction.
The only thing I will say is that despite the crummy abstraction, we did put the effort in to have good test coverage of each ContentProvider. It took a while to get the infrastructure up and running so that we even could test them, but once the plumbing was added, it became simple to add new tests to ensure they worked as expected.
This is important when you need to take into account things like "Get me all the apps, but filter on category, and then also limit those Apps to ones for which they have at least one Apk which is installable on my hardware, meets my AntiFeatures requirements, and then also pull back data about whether there is a version that can be upgraded to or not based on currently installed apps. I fell in love with the SQLite explain output. I found it really good at explaining what was going on with these mildly complex joins - much easier than the MySQL explain output I was familiar with.
To see Torsten and others working on replacing my crufty-passed-their-use-by-date ContentProvider code with a modern Kotlin + Room implementation is heart warming. If any of you are reading this, please accept my deepest sympathies for having to pull that code out and rework it - that is not something I would have enjoyed doing! It is even better that it all lives in libraries that other clients can adopt if they choose so they don’t need to reinvent the wheel.
For those interested, yes, I am also responsible for writing the bulk of the code for the “new” (now several years old) UI in the offical client which often gets maligned on HN (and this thread is no exception). At the time I did my best to fight off edge cases and quirks of the Android system. F-Droid needs to ask Android which apps are installed and what their signatures are, then cross verify that with its own database to tell you whether updates are available. It also needs to know whether new apps have been installed outside of F-Droid since last time you opened it, etc. It also needs to pass downloaded .apk files to the system and request for them to be installed, then wait for confirmation from the system. The API’s Android provide for this kind of work, but they always seemed flakey, unwieldily, and slow to respond. The whole experience is full of race conditions, and each bistro seems to handle it ever so slightly differently.
I am still proud of how much we managed to achieve, and I’m also very pleased with the fact we were able to do so much great work around internationalised metadata, screenshots, encouraging donations to app developers, etc. But I’d be lying if I said I wasn’t still a little disappointed that I couldn’t iron out all of the kinks.
Life circumstances meant that I drifted away from the project and no longer contribute (other than via Liberapay donations to the wonderful fdroiddata team). However I would love to revisit the project in the future to see if I can address some of those edge cases to ease the user experience for all of us. You may have noticed that for such an amazing project, used by many people (not just the client, but the entire infrastructure), there has been comparatively few contributors. Despite that, I am still extremely fond of the value they bring to the Free and Open Source ecosystem.