Performance Clarification

[benjaminy]

Is my understanding correct that push and pull operations using this remote require [up/down]loading the entire (compressed) repo? That seems like a bummer in situations where the repo is large and the deltas are small. Does anyone know how feasible it would be to make a remote that works more incrementally?

[mih]

Please note that this project is no longer active. The presently most advanced implementation is here https://docs.datalad.org/projects/next/en/latest/generated/datalad_next.gitremotes.datalad_annex.html

A more incremental behavior is feasible, but the Q is what a use case for it would be?

In practice, many storage systems have significant request latencies, but little streaming throughput limits. For reasonably large repos we found that a plain reupload is faster and simpler. When more complex patterns are needed, it is often better to use a git-based remote directly, because it also addresses other issues that come with this simplification.

[benjaminy]

Thank you for the quick reply. My long-term goal is to build a framework for (asynchronous) collaboration applications (shared calendar, address book, etc) that has the smallest possible dependence on any 3rd party service/server. My working architecture sketch involves each group member storing a (encrypted) copy of the team's data on a commodity cloud service. If I can make this work with a Git repo, that seems like it will provide a lot of useful concurrent versioning functionality.