A user recently notified us of an issue causing a video to not be decrypted. We have identified a potential issue and are rolling out a fix, while continuing to investigate. We'll update this document as we have more details.
Details
Since the initial report, we have been combing through the code that deals with uploads to find where the issue might lie.
In the web or desktop code, we haven't found any issue so far. Still, we've added some more assertions just as an extra safety check.
We found a scenario with partial chunked reads, however based on the evidence it seems unlikely that this was the cause in practice. In any case, we've reworked that code, and have also added extra logging.
In the mobile code, we found a potential issue in the auto-resumable multipart background upload functionality that was recently enabled. Based on our current information, this seems to be the most likely candidate. We have added a fix, and we're rolling it as I write this.
The issue in not reproducible, however based on the circumstantial evidence - large video recently uploaded from mobile in a manner which would go through this recently enabled code path - we do think the last case is the reason. So the next section explains case 3 in more detail.
Safeguards against concurrent multipart upload resumptions
Multipart uploads is a feature which allows the Ente app to improve the reliability and resilience of large (> 20 MB) file uploads by breaking them into chunks and uploading them individually. This both allows the app to retry a particular part which fails due to network issue (without retrying the entire video). This is also essential for self-hosters who use Cloudflare's free plan, since Cloudflare limits their network requests to 100 MB each.
Multipart uploads have been in the web and desktop app for a couple of years. Last year, we added the same functionality to the mobile app.
However, since this was affecting a critical part of the code, we didn't release it immediately after testing, but instead further dogfooded it for a couple of months on our own personal devices to make sure it stood up to real world usage.
When we didn't find any issue, we started gradually (20%, then 50%, then 100%) rolling it out to non-internal users a couple of months ago.
In parallel, we've been improving our background upload tasks, with a major improvement going in there in the latest release last month that makes the background uploads run in more scenarios.
The potential issue we've identified happens in the combination of these two features:
- An upload starts both in the foreground and the background simultaneously.
- And neither of them succeeds.
- The upload is later resumed.
In theory, SQL does not guarantee the order of retrieval of rows, so during step 3 it is possible that the remainder of the upload (post resumption) gets encrypted with a different key than the prior chunks.
In practice, we've not been able to reproduce this, which is why we are not certain this is the issue, but as I mentioned previously, the circumstantial evidence seems to point here. So we're immediately rolling out the fix for this.
Postscript
Apart from technical and process changes to reduce such things from happening in the future, we are also considering social changes like a beta testers program to get even more real world usage for important features before we start rolling them out.
If you have uploaded large videos from our mobile app to Ente, please upgrade your app to v1.2.8 or higher and verify if you were impacted by this issue by exporting your data using our desktop app or CLI. If you were impacted, please reach out to support@ente.io.
v1.2.8 is already out on the App Store, Play Store, and on Ente's GitHub releases, and it should be out on F-Droid soon.
Code reviews, tests, QA are all important, and we'll continue iterating on them. But this incident has again shown the vital importance of being responsive to reports from users. We are grateful to all our community members who take time to report issues to us.