Proto commits in mutagen-io/mutagen

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2024-01-30

all: remove license handling code Signed-off-by: Jacob Howard <jacob@mutagen.io>

all: remove license handling code Signed-off-by: Jacob Howard <jacob@mutagen.io>

The documentation is generated from this commit.

2023-07-25

sync: rename IgnoreSyntax type and value This commit renames IgnoreSyntax to Syntax. It also renames the "git" syntax to "mutagen" because (a) it's more appropriate and understandable and (b) we stray a little too far from gitignore syntax to reasonably claim compatibility. Signed-off-by: Jacob Howard <jacob@mutagen.io>

sync: implement support for Docker-style ignore syntax and semantics This commit implements the new Docker ignore syntax defined in the previous commit. This is accomplished via a new "phantom directory" entry type that is reified (before reconciliation) by a combined walk over ancestor, alpha, and beta entries. Signed-off-by: Jacob Howard <jacob@mutagen.io>

sync: define ignore syntax configuration types and wire to CLI Signed-off-by: Jacob Howard <jacob@mutagen.io>

2023-02-16

all: use vendor-prefixed build tags to avoid collisions Signed-off-by: Jacob Howard <jacob@mutagen.io>

all: implement Mutagen Pro licensing mechanism for SSPL builds Most SSPL features (starting with Zstandard and XXH128 support) are going to require a paid license to use. This license will be the same as that required to use the Mutagen extension for Docker Desktop. This change is fundamental to supporting continued Mutagen development. This commit implements the client-side portion of the licensing mechanism. This only applies to SSPL builds; no licensing code is included in MIT-only builds. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2023-02-09

sync: relocate and refactor hashing algorithm code to match compression This commit relocates the hashing algorithm code to a more appropriate location, similar to the way that we structured the compression code. Signed-off-by: Jacob Howard <jacob@mutagen.io>

sync: make compression configurable and add Zstandard support This commit makes synchronization session compression configurable when working with remote endpoints. It adds support for Zstandard compression and the ability to disable compression. The default compression algorithm is still DEFLATE. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2023-01-11

sync: add support for SHA-256 and XXH128 content hashing This commit adds support for SHA-256 and XXH128 content hashing when computing synchronization content digests. The content hashing algorithm used by a synchronization session is also now configurable (via the command line or YAML configuration) and displayed in session listings. XXH128 provides significant performance enhancements over SHA-1 on systems without SHA-1 hardware acceleration. SHA-256, while not a performance enhancer, was easy enough to add and will allow users to have even stronger collision avoidance in content digests, which may be desirable in certain cases (though almost certainly not necessary). SHA-1 will remain the default. XXH128 will likely be a paid extension for Mutagen, so we've added its adapter code under an SSPL license, and it is not currently included in core builds (which will remain purely MIT-licensed for now). Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-08-22

sync: add fully manual permissions propagation mode While Mutagen's "portable" permissions propagation mode remains the best default for development, there do exist valid use cases for a fully manual mode. This commit adds a new PermissionsMode enumeration (along with the associated machinery) and a new "manual" permissions mode that allows for fully explicit permission mode setting. This enumeration also opens the door to future modes such as "posix-raw" and possibly even "windows-raw". The existing behavior has been designated "portable" for the time being. The only other sensible name would be "executability", but that's too tedious and error-prone to type, even if it is the default. This commit also fixes a minor bug in transitioner.createSymbolicLink where the directory mode was being used as the permissions base for symbolic links. The file mode should have been used instead. The effect is minor since permission bits on symbolic links aren't used for access control anyway (except for readlink on macOS, where read permissions are required to read the symbolic link's target path (though even there the target itself is still subject to access controls with its own permission bits)). Also, given that directory permissions are typically the same as file permissions with executability bits AND'd in, it's unlikely that any observable differences will manifest in this particular use case. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-06-28

sync: renamed directory, file, and symbolic link count fields The names of these fields weren't consistent with the naming convention for other counting fields. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-06-23

sync: restrict synchronizable content printing to scanned endpoints Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-06-17

forward: add data volume statistics to state and monitoring This commit adds statistics on transmitted and received data volume to forwarding session states. It also updates the forward monitor command to present this information. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-06-16

sync: adopt more idiomatic Protocol Buffers usage in rsync This commit modifies the rsync state logic to make better use of built-in Protocol Buffers utility functions. It also fixes some typos and ambiguity in comments. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-06-15

sync: add sub-file rsync progress Signed-off-by: Jacob Howard <jacob@mutagen.io>

forward: restructure session state to match sync This commit restructures the session state representation to match the refactoring done for synchronization. Signed-off-by: Jacob Howard <jacob@mutagen.io>

sync: restructure session state and output endpoint statistics This commit restructures the internal and external session state representations, encapsulating endpoint-specific state into a separate structure. This makes it easier to propagate the endpoint content statistics that are now available in Snapshot, and also cleans up a lot of the previous code repetition for endpoint-specific state fields. This commit also modifies synchronization session listings to include these statistics. Signed-off-by: Jacob Howard <jacob@mutagen.io>

sync: record scan statistics for entry counts and total file size This commit extends the Snapshot type to include pre-computed statistics for synchronizable entry counts and the total synchronizable file size. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-05-25

cli: use non-gRPC data structures for template contexts The gRPC types used by the internal API are not suitable for use by other tools. At the very least, Protocol Buffers enumeration types require non-numeric representations, but more significant structural issues also exist that result in less-than-idiomatic JSON. This isn't unexpected, especially given that Protocol Buffers types are typically flat and have a non-idiomatic JSON representation, but it does mean that we need to define public-facing types for templating. Given that we eventually want to switch to (or at least support) a JSON-based REST API for the daemon, it makes sense to model the template types after the eventual REST API models. This commit adds a new api package that defines these model types, which will be useful for both API clients (eventually) as well as tools that want data structures into which they can decode JSON output. We probably won't make any API or format stability guarantees just yet, but embedding applications can sipmly pin versions for now (like they already do with Mutagen itself). Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-05-19

sync: rename successful cycle count field Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-03-31

sync: add shorthand optimization for remote staging paths In the vast majority of staging operations, especially large initial staging operations, the endpoint is going to require all of the requested paths. In this scenario, there's not point in sending all of them back over the wire, so this commit introduces a shorthand into the remote synchronization endpoint protocol to sidestep this overhead. In theory, we could also look at using path indices rather than paths themselves, but that would affect the signature of the Stage method itself (or require additional computation) and at that point it's probably better to look at full pipelining. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-03-30

sync: clarify and fix comments This commit fixes two outdated comments and some comment punctuation. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2022-03-17

sync: adopt new watching and scanning APIs in synchronization This commit adopts the new watching and scanning APIs in the synchronization controller and endpoints. This refactoring is a follow up to #319, which is what originally motivated these additional changes. Most of the ideas of #319 are still present, though certain techniques (like transition change detection) have been refined and/or simplified. The most important change is that scan acceleration is no longer always disabled in Transition for recursive watching. Signed-off-by: Jacob Howard <jacob@mutagen.io>

sync: add Snapshot message type to encapsulate scan results The results of Scan have been a little too complicated for a while now, with filesystem contents being transmitted separately from behavioral information. To remedy this issue, this commit adds a new Protocol Buffers message type called Snapshot, which fully encapsulates the content and behavioral results of filesystem scans (cache and ignore cache information are still returned separately, since they needn't be transmitted from endpoints to the controller). Unicode decomposition behavior is also now returned for all scan roots that exist on disk, including file roots, and ErrScanCancelled has been exported. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2021-11-06

Added internal staging mode. This commit adds a new "internal" staging mode where files are synchronized directly into a subdirectory of the staging root. This actually works better than expected because the synchronization loop will retry on missing files, meaning that even if the synchronization root doesn't initially exist, it will still be created by transition, and thus staging will succeed on the next synchronization cycle. However, this isn't super efficient, because an entire staging operation will have to fail, so we may want to figure out a mechanism to push failure notifications earlier, but that's fine because this staging mechanism is not something we'll recommend in general due to its strict requirements. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2021-05-10

Switched daemon service to RESTful API. This commit is the first in a series that will convert the daemon API to a RESTful design using JSON. It adds an HTTP server to the daemon TCP port and API endpoints for the daemon service. It also removes the old daemon gRPC service, which required disabling some functionality until this work is complete, though that functionality can currently be accessed via manual HTTP requests. Because the daemon gRPC service was previously responsible for housekeeping, some refactoring/restructuring of the housekeeping infrastructure was also required, though the result is better than what existed previously. Signed-off-by: Jacob Howard <jacob@mutagen.io>

2021-04-06

Replaced usage of "symlink" with "symbolic link". Historically, Mutagen has tended to use the term "symlink", but it has also opted for more verbose terminology for non-user-facing elements of the code, so it makes more sense to use the term "symbolic link". Signed-off-by: Jacob Howard <jacob@havoc.io>

2021-04-01

Added unsynchronizable content tracking and refactored core tests. This commit teaches Mutagen how to handle and track unsynchronizable content. Mutagen was already capable of handling untracked content (due to unsupported types or ignores), but now it's able to track where untrack content exists and evaluate potential conflicts earlier in a synchronization cycle. Mutagen can also now handle, track, and synchronize around all problematic content, whereas certain problem conditions would have previously halted a synchronization cycle. Enabling this required massive refactoring due to the new entry kinds and updated reconciliation algorithm. As a result, tests for the core synchronization package had to be entirely rewritten, though they should now have significantly better coverage and be significantly more extensible. Signed-off-by: Jacob Howard <jacob@havoc.io>

2021-03-03

Clarified mutability for most Protocol Buffers message types. This commit adds documentation for most Protocol Buffers message types, clarifying which should be considered immutable and which need to be copied. It doesn't cover core synchronization types because those will be updated separately in another commit, though they should all be treated as immutable. Signed-off-by: Jacob Howard <jacob@havoc.io>

2021-02-26

Fixed comment formatting in Protocol Buffers file. This commit makes the comment formatting in url.proto clearer and avoids having the note about field 7 folded into the comments for field 8 in generated code. Signed-off-by: Jacob Howard <jacob@havoc.io>

Removed tunnels. This commit removes tunnel support from Mutagen. Signed-off-by: Jacob Howard <jacob@havoc.io>

2020-12-18

Fixed comment formatting in Protocol Buffers file. This commit makes the comment formatting in url.proto clearer and avoids having the note about field 7 folded into the comments for field 8 in generated code. Signed-off-by: Jacob Howard <jacob@havoc.io>

Removed tunnels. This commit removes tunnel support from Mutagen. Signed-off-by: Jacob Howard <jacob@havoc.io>

2020-07-30

Fixed typo and regenerated Protocol Buffers code. This commit fixes a small typo and regenerates Protocol Buffers code. There are no behavioral changes. Signed-off-by: Jacob Howard <jacob@havoc.io>

2020-06-08

Added conflict and problem truncation in list. We've had several cases where some underlying issue (e.g. permissions, different base paths, etc.) generates significant numbers of conflicts and/or problems (sometimes in the thousands or even tens of thousands). This is problematic because gRPC has a message size limit (which we've already increased to 25 MB) that can prevent list operations from working and thus prevent any insight into the underlying issue. It's also problematic because it could (in theory) be a performance issue for the daemon if there were frequent list operations or a large number of concurrent list operations for such a session. To remedy this, this commit introduces conflict and problem truncation. This occurs at the service level and uses fixed maximum lenghts. These lengths need to be sufficient to diagnose underlying issues, but not so large that they cause sessions to exceed the gRPC message size limit (not to mention making list output unreadable). Signed-off-by: Jacob Howard <jacob@havoc.io>

2020-06-04

Decoupled prompting from services. This commit is the first half of an effort to decouple prompting from the forwarding, synchronization, and tunneling services. It extends the prompting service to support a hosting method that implements stream-based prompting, allowing other methods to avoid streaming requests and responses. This commit only converts service code - command implementations have not yet been converted. Signed-off-by: Jacob Howard <jacob@havoc.io>

Added parameters field to URLs. This commit adds a parameters field to URLs that can be used to store internal parameters for a transport. The exact nature of these parameters is undefined and transport-dependent. They'll be needed to support Docker URLs that take into account command line parameters. They may also be useful later for passing command line options to OpenSSH and other transport commands. Signed-off-by: Jacob Howard <jacob@havoc.io>

2020-02-11

Fixed typo in tunnel Protocol Buffers definitions. This commit fixes a small typo in Protocol Buffers comments. Signed-off-by: Jacob Howard <jacob@havoc.io>

2020-02-10

Removed heartbeats from tunnels. This commit removes heartbeats from tunnels. Their setup was a bit iffy anyway since we couldn't guarantee data channel establishment order. But they were also rendered basically useless by head-of-line blocking when saturating the tunnel with data. Instead, we'll rely directly on errors from the transport layer. Signed-off-by: Jacob Howard <jacob@havoc.io>

Renamed prompt packages to prompting. This commit renames the prompt and prompt service packages to use the name prompting, which is more consistent with our other noun service names. Signed-off-by: Jacob Howard <jacob@havoc.io>

Fixed Protocol Buffers message formatting. This commit makes a minor tweak to the spacing of Protocol Buffers message brackets. Signed-off-by: Jacob Howard <jacob@havoc.io>

2020-02-09

Implemented transition preemption and fixed earlier code. This commit implements transition preemption. It also cleans up (and in some case fixes) the preemption protocol for scan and transition. Signed-off-by: Jacob Howard <jacob@havoc.io>

2020-02-03

Made scan operations preemptable. This commit adds full preemption to scan operations, both locally and over the network. A bit of structural cleanup in the remote endpoint client and server also took place to better separate transmission and execution errors. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-12-11

Implemented "reset" command for synchronization sessions. This commit adds a "reset" command for synchronization sessions that allows users to clear the ancestor data for a synchronization session. This is useful in cases where storage on one endpoint has been reset (e.g. when a container volume that was recreated) and the user wants to once again rely upon the initial both-created-same behavior without having to recreate the session (or project). In the wake of our new safety checks, this will probably be a useful convenience. Signed-off-by: Jacob Howard <jacob@havoc.io>

Added additional safety check focused on container volumes. This commit adds an additional safety check designed to detect cases where (in a bidirectional synchronization mode) one endpoint has deleted all synchronization root contents. The threshold for detection is two or more content entries. The idea is to detect cases where a non-persistent (or recreated) container volume was used to store synchronization contents. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-12-05

Implemented "reset" command for synchronization sessions. This commit adds a "reset" command for synchronization sessions that allows users to clear the ancestor data for a synchronization session. This is useful in cases where storage on one endpoint has been reset (e.g. when a container volume that was recreated) and the user wants to once again rely upon the initial both-created-same behavior without having to recreate the session (or project). In the wake of our new safety checks, this will probably be a useful convenience. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-12-04

Added additional safety check focused on container volumes. This commit adds an additional safety check designed to detect cases where (in a bidirectional synchronization mode) one endpoint has deleted all synchronization root contents. The threshold for detection is two or more content entries. The idea is to detect cases where a non-persistent (or recreated) container volume was used to store synchronization contents. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-11-30

Renamed "tunnel host parameters" to "tunnel host credentials". This commit just performs some internal type and variable renaming to make our tunnel host credential nomenclature more consistent with documentation and intuition. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-11-29

Disabled user specification in tunnel URLs. User specification was never supported in tunnels, so there was no point in having the code in place to handle it. Having tunnel agents run as the hosting user is the better security model in any case. If we do decide to add this back, we'll need to create a new tunnel version anyway. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-11-01

Added tunneling heartbeat mechanism. This commit adds a heartbeat data channel to all tunneling connections in order to better detect connection breakage. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-10-30

Added support for mutagen.io tunnels. This commit implements support for mutagen.io tunnels. Some polish and almost all tests are still missing, but the end-to-end connectivity works. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-10-27

Added support for tunnel URLs. This commit adds support for tunnel URLs. It currently supports user specification, though we may disabled this when we eventually ship tunnel support. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-07-18

Removed fuzzy matching in favor of explicit name/identifier matching Now that sessions have user-defined (and user-friendly) names by which they can be addressed, the fuzzy matching can likely be removed. This behavior was ill-defined and name matching is far more explicit. We've also removed the match on identifier prefixes, which probably isn't going to be missed since most people just copied/pasted session identifiers.

Added names to sessions This commit adds a name parameter to synchronization and forwarding sessions. This name parameter will replace the fuzzy matching currently used by session managers. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-07-16

Added the ability to create pre-paused sessions This commit adds a -p/--paused flag to the forward create and sync create commands to allow for the creation of pre-paused sessions. Fixes #115. Signed-off-by: Jacob Howard <jacob@havoc.io>

2019-07-15

Updated module path to new location. Mutagen now lives at github.com/mutagen-io/mutagen, so this commit updates the module paths (and all import paths) to reflect that new location. Our custom dependencies have also been relocated to the mutagen-io organization, so their license information links and import paths have also been updated.

2019-07-14

Minor refactoring, mostly for style and consistency. This commit makes a few minor changes to the forwarding and synchronization packages to improve their consistency.

Added Unix domain socket setting to forwarding. This commid adds Unix domain socket configuration options to forwarding. It also modifies the list and monitor commands to display these configuration options in the same way as for synchronization sessions.

2019-07-13

Added support for forwarding sessions. This commit adds support for creating and managing network forwarding sessions. Forwarding sessions are separate from synchronization sessions and have slightly different semantics, but they are built on much of the same underlying technology. Forwarding can be performed between arbitrary pairs of TCP (including IPv4-only and IPv6-only TCP variants) and Unix domain sockets.

2019-07-12

Removed ephemeral synchronization session support. This commit removes support for ephemeral synchronization sessions since we're going to implement them another way. The implementation being removed was not particularly clean, especially in terms of the changes to the session controller.

Relocated sync and rsync packages into synchronization. This commit relocates the sync and rsync packages into the synchronization package, renaming sync to "core".

Renamed session package to synchronization. This commit renames the session package to synchronization (in preparation for the additional session types that we'll be adding). The sync and rsync packages will likely be relocated under this package, with the sync package being renamed to avoid confusion.

2019-07-11

Extracted label selection infrastructure into separate package. The label selection infrastructure will be used for other object types as well (e.g. forwarding sessions), so this commit extracts it into its own package and better encapsulates its Kubernetes-based underpinnings.

Added support for forwarding URLs. This commit modifies the URL package and other code to parse forwarding URLs. Tests have been extended to cover the new parsing paths, but the URL tests need a bit of refactoring before more extensive tests are added.

Changed URL member names and add documentation. This commit makes the member names for URL a little less verbose and a little more general. It also adds documentation comments for URL and Protocol.

Added the ability to create ephemeral synchronization sessions. This commit modifies the session package to enable to creation of ephemeral sessions. Ephemeral sessions avoid writing any files to disk (with the exception of endpoint staging directories, which are forced into neighboring mode) and thus terminate when the corresponding Controller is shut down. This commit also starts threading logging facilities into the session package.

2019-07-06

Renamed remote endpoint files. The file names in the remote endpoint package were a bit repetitive, so this commit renames them to be a bit simpler.

2019-05-21

Added check for missing stager files in Transition. This commit modifies Transition to watch for missing files when calling Provider.Provide. This is almost certainly an indication of concurrent modifications during staging. This requires cooperation from the Provider implementation and is documented for the Provider.Provide method. Fortunately this is easy to implement for stager. If missing files are detected, controller will now force another synchronization cycle immediately, with the idea being that the concurrent modifications are ephemeral and another round of staging should solve the problem.

2019-05-20

Added Endpoint.Scan parameter to force full scans, enabled for flush. This commit adds a "full" parameter to Endpoint.Scan that forces a full (warm) scan, bypassing any acceleration that might be available. It also modifies controller to use this flag when processing a flush request.

Renamed/isolated mode definitions, code, and tests into their own files. This commit gives all mode definitions, code, and tests their own files, making their organization a little bit clearer and more explicit.

Separated session.Version code. This commit separates the session.Version code into separate files from the session.Session code. The former is starting to become quite large as we continue to add additional configuration options, so it makes sense to split it into separate files.

Added scan mode to control scan acceleration features. This commit adds a "scan mode" concept to Mutagen that allows users to control whether or not scan acceleration is enabled.

Redistributed staging package into session and session/endpoint/local. The staging package being separate from session and endpoint was primarily a crutch to work around the session package's dependency on the configuration package (we needed a third package where the staging mode could live), but since that dependency no longer exists, it doesn't make sense for staging to stand on its own as a separate package. Instead, we've redistributed StagingMode infrastructure into the session package (and renamed it to StageMode for consistency with other modes) and moved the stager infrastructure into the session/endpoint/local package.

Relocated WatchMode to session package. The filesystem watching package has been pared down to a more minimalist set of primitives, and as such the watch mode abstraction no longer really belongs there. Instead, it makes more sense in the session package, where it is actually understood.

2019-05-14

Fixed a number of code comments related to probe modes. This commit fixes a number of code comments related to probe modes.

Added "neighboring" staging mode and consolidated housekeeping. This commit adds a "staging mode" concept to Mutagen and establishes two staging modes: "mutagen" and "neighboring". The former is the standard staging within the Mutagen data directory, while the latter creates staging directories that neighbor their corresponding synchronization roots. This is designed to solve the chronic performance problems that users experience when the Mutagen data directory resides on a different filesystem than the synchronization root (and atomic rename operations have to be emulated). This commit also consolidates housekeeping functions into a single package, though this unfortunately brings in a large number of secondary changes.

2019-05-11

Restructured and expanded handling of TOML-based configuration files. This commit restructures how TOML-based configuration files are processed. These files are now processed by the create command and merged before being sent to the daemon. This refactoring allows two new features: the ability to ignore the global configuration file and the ability to specify a custom configuration file. The merging infrastructure behind this is such that we could allow an arbitrary number of configuration files to be specified on the command line, but we'll start with a single file for the time being. Allowing multiple files to be specified can be done in a backwards-compatible manner.

2019-05-10

Added daemon version match check. Since we don't currently guarantee daemon API compatibility across Mutagen versions, this commit adds a version check to ensure that the Mutagen daemon version matches the client version. In order to fully realize this goal, we had to add tag information to the daemon version API.

2019-05-09

Relocated filesystem behavior probing and watching code. This commit relocates the behavior probing and watching code in the filesystem package to dedicated subdirectories. This is part of an effort to chop up the filesystem package. Unfortunately, a lot of the filesystem package's internals rely on low-level primitives that need to live in the same package, but the behavior probing and watching code operate at a higher level and are ideal candidates for extraction. A lot of code has been updated to account for these relocations, but there shouldn't be much in the way of behavioral changes.

Revert "Removed probe modes." This reverts commit 567752ad74bee44ef84fde088d959213fad7ec61. It turns out that fast paths are not going to be entirely sufficient to avoid probing with temporary files, and there will be cases that need manual intervention to avoid these temporary files.

2019-05-04

Removed probe modes. This commit removes the "probe mode" concept from Mutagen. Instead, we'll continue to rely on automatic probing behavior, now optimized with fast paths for majority use cases and workarounds for problematic filesystems. Worth noting in this removal is that printEndpoint hadn't been updated to include probe modes, so that should be fixed if reverting this commit.

2019-05-03

Added support for session labels. This commit adds support for session labels and label selectors. This makes Mutagen significantly easier to use from scripting tools. To keep familiarity and consistency, we've adopted the Kubernetes syntax for labels and selectors, and fortunately we're able to re-use the Kubernetes implementation.

2019-04-25

Split protocols/local package and relocated code. This commit extracts the endpoint implementation from pkg/protocols/local into a separate package that lives under the session package (so the local protocol implementation is just a stub). The remote endpoint implementation has also been relocated to live under the session package.

Synchronized *Mode nomenclature. This commit updates IgnoreVCSMode and SymlinkMode to use a nomenclature that's consistent with more recently added Mode enumerations. The motivation behind this switch is primarily consistency, though the original motivation behind this repetitive naming scheme is that the Go Protocol Buffers generator will eventually generate more idiomatic Go enumerations, at which point we'll want enumeration type names in corresponding enumeration value names.

2019-04-24

Added the ability to specify filesystem probing behavior. This commit adds a "probe mode" concept to synchronization sessions that allows users to specify how endpoints probe filesystem behavior (specifically Unicode decomposition and executability preservation behavior). In addition to the default "probe" mode which uses probe files, users can now specify an "assume" mode which tells Mutagen to assume behavior based on the platform without creating probe files.

2019-04-19

Updated CacheEntry.Mode documentation. This field no longer stores os.FileMode values on POSIX systems, instead opting for the st_mode member of struct stat. Now that we've changed the definition of this field's definition once and realized that there are no consequences to doing so (other than a full rescan), it also doesn't seem like the long-winded justification for this field's definition was needed.

2019-02-21

Modified flush command the block by default. This command modifies the flush command to block by default, making its behavior consistent with the other session modification commands (i.e. pause, resume, and terminate). The -w/--wait flag has been replaced with a --skip-wait flag that enables the non-blocking behavior.

2019-02-13

Added -w/--wait flag to flush command. This commit adds a -w/--wait flag to the flush command which will cause the command to block until the session for which flushing has been requested completes a successful synchronization cycle. This also cleans up and improves upon the logic in 46b4d67.

2019-02-12

Refactored ownership specifications and changed Windows Chmod. This commit refactors ownership specifications to use the more general term "owner", which is more idiomatic and supports the notion that groups may be owners on Windows. This commit also removes the third-party Chmod implementation for Windows and instead reverts to the os-based implementation. The third-party implementation had issues with well-known SIDs, and even once fixed, the behavior wasn't very intuitive. We may expand mode bit behavior on Windows at some point in the future, but I don't think it makes sense to ship at this point in time. We can still document how mode bits behave on Windows - i.e. just like the os.Chmod function.

2019-02-11

Major refactor of configuration mechanism. This commit is a major refactor of endpoint-specific configuration handling. Endpoints now have their own configuration objects that are merged with the core configuration at runtime.

Implemented "no-watch" mode and flush command. This commit adds a "no-watch" watching mode that makes filesystem watching a no-op. Synchronization still occurs, but only when forced, otherwise it just sits idle. To that end, a "flush" command has also been added that forces a synchronization cycle to occur for one or more sessions. This can be used in conjunction with normal watching as well.

2019-02-10

Added anti-DoS parameters for endpoints. This commit adds two parameters for endpoints: maximum number of entries and maximum staging file size. Together, these two parameters should protect endpoints from DoS issues with malfunctioning or malicious remote endpoints and/or controllers.

Implemented synchronization root sandboxing. This commit uses the new race-free filesystem traversal infrastructure to sandbox access to sychronization roots. Paths passed from a remote can no longer escape the synchronization root via incorrect formatting or a symbolic link. This change affected both the endpoint implementation and rsync implementation. In order to facilitate this, an Opener type was added to the filesystem package, which provides race-free filesystem traversal that is "fast" if paths are accessed in DFS traversal order, which fortunately they are in our case.

2019-02-09

Changed permission specification system and updated transition code. This commit changes the permission specification mechanism (removing the permission exposure level concept) and updates the transition infrastructure to take advantage of the new race-free filesystem infrastructure. Unfortunately this had to be done in the same commit because the permission behavior in the transition code was too difficult to shim. Tests are passing, though the code still needs an additional read-through.

2019-02-08

Converted Scan infrastructure to use new filesystem infrastructure. This commit converts the Scan infrastructure to use the new filesystem infrastructure. There are no observable API changes, except that FileID has been added to CacheEntry to perform more exacting comparison checks, and mode now has a more platform-native definition. A few tests are currently failing due to the fact that the Transition infrastructure has not been transitioned to the new filesystem infrastructure. In particular, differing mode bits are causing problems. This should fix itself automatically once the code migration is complete. A few changes and bug fixes were necessary in the filesystem package. Performance seems more or less on par with the existing implementation. For larger directory hierarchies, the openat infrastructure without symbolic link resolution seems to have a marginally positive performance effect.

2019-01-28

Renamed "permissions" files to "permission". This is simply a rename commit, with a few associated fixups for Protocol Buffers definition files. The goal of the commit is simply to maintain naming consistency.

Added permission exposure levels. This commit extends the current (implicit) "portable" permission propagation mode with a set of "permission exposure levels". This was chosen (instead of explicit POSIX-inspired octal permission codes) in order to maintain a portable abstraction that'll work across different platform and to avoid the complexity associated with explicitly specifying octal mode values that would be tricky to parse, difficult to validate, and (most importantly) run up against algorithmic difficulties when dealing with executability propagation. These levels can currently be specified on a general basis (in which case they'll apply to both endpoints), and endpoint-specific basis, or a combination thereof.

2019-01-25

Renamed synchronization modes. This commit renames synchronization modes with the following mappings: symmetric -> two-way-safe source-wins -> two-way-resolved mirror-safe -> one-way-safe mirror-exact -> one-way-replica These names are more consistent and intuitive. No other behavior has changed.

Replaced conflict resolution implementation and added unidirectionality. This commit replaces the existing conflict resolution implementation and adds support for unidirectionality. Both concepts have been unified into a single "synchronization mode" concept, which uses a spectrum of symmetry breaking between endpoints to provide a more minimal and intuitive implementation of these features. The mode names are still likely to change, and additional reconciliation tests may be added.