brig - decentralized & secure synchronization¶

Step 0: Installing Go¶

This is only required if you don’t already have Go installed. Please consult your package manager for that.

If you did not do that, you gonna need to install Go. Refere here for possible ways of doing so. Remember to set the GOPATH environment variable to a place where you’d like to have your .go sources being placed. For example you can put this in your .bashrc:

# Place the go sources in a "go" directory inside your home directory:
export GOPATH=~/go
# This is needed for the go toolchain:
export GOBIN="$GOPATH/bin"
# Make sure that our shell finds the go binaries:
export PATH="$GOPATH/bin:$PATH"

By choosing to have the GOPATH in your home directory you’re not required to have sudo permissions later on. You also need to have git installed for the next step.

Step 1: Compile & Install brig¶

This step requires setting GOPATH, as discussed in the previous section.

$ go get -d -v -u github.com/sahib/brig  # Download the sources.
$ cd $GOPATH/src/github.com/sahib/brig   # Go to the source directory.
$ go run mage.go                         # Build the software.

All dependencies of brig are downloaded for you during the first step. Execution might take a few minutes though because all of brig is being compiled during the go run mage.go step.

If you cannot or want to install git for some reason, you can manually download a zip from GitHub and place its contents into $GOPATH/src/github.com/sahib/brig. In this case, you can skip the go get step.

Step 2: Test if the installation is working¶

If everything worked, there will be a brig binary in $GOBIN.

$ brig help

If above command prints out documentation on how to use the program’s commandline switches then the installation worked. Happy file shipping!

Precursor: The help system¶

Before we dive in, we go over a few things that will make your life easier along the way. brig has some built-in helpers to serve as support for your memory. If you’re not interested in that you can skip right to the next section.

$ brig help stage
NAME:
   brig stage - Add a local file to the storage

USAGE:
   brig stage [command options] (<local-path> [<path>]|--stdin <path>)

CATEGORY:
   WORKING TREE COMMANDS

DESCRIPTION:
   Read a local file (given by »local-path«) and try to read
   it. This is the conceptual equivalent of »git add«. [...]

EXAMPLES:

   $ brig stage file.png                         # gets added as /file.png
   $ brig stage file.png /photos/me.png          # gets added as /photos/me.png
   $ cat file.png | brig stage --stdin /file.png # gets added as /file.png

OPTIONS:
   --stdin, -i  Read data from stdin

source $GOPATH/src/github.com/sahib/brig/autocomplete/bash_autocomplete

source $GOPATH/src/github.com/sahib/brig/autocomplete/zsh_autocomplete

$ brig bug

Passwords¶

brig needs a password to securely encrypt your repository. In the example above we specified a password helper (-w 'echo my-password'). That’s simply a program that will output the correct password to stdout. Obviously, using echo for this job is not perfect (although it works fine in test environments). Instead we recommend to use a password manager like pass and to initialize your repo like that:

# Generate a password and store it in "pass":
$ pass generate brig/ali -n 20
$ brig init ali@woods.org/desktop -w "pass brig/ali"

The advantage here is that your password is protected with a master password that will be handled by pass. If you already set it up like above you can simply change the password helper command:

$ brig cfg set repo.password_command "pass brig/ali"

There are two alternatives to using a password manager (which we do not recommend):

1. Do not use the -w / --password-helper flag. You will be asked to enter a new password. The more secure the password is you entered, the greener the prompt gets [1]. The clear disadvantage here is that you need to re-enter the password every time you restart the daemon.
2. Do not use a password. You can do this by passing -x to the init command. This is obviously not recommended.

Choosing and finding names¶

You might wonder what the name you pass to init is actually for. As previously noted, there is no real restriction for choosing a name, so all of the following are indeed valid names:

• ali
• ali@woods.org
• ali@woods.org/desktop
• ali/desktop

It’s however recommended to choose a name that is formatted like a XMPP/Jabber-ID. Those IDs can look like plain emails, but can optionally have a »resource« part as suffix (separated by a »/« like desktop). Choosing such a name has two advantages:

• Other peers can find you by only specifying parts of your name. Imagine all of the Smith family members use brig, then they’d possibly those names:

  • dad@smith.org/desktop
  • mom@smith.org/tablet
  • son@smith.org/laptop

  When dad now sets up brig on his server, he can use brig net locate -m domain 'smith.org' to get all fingerprints of all family members. Note however that brig net locate is not secure. Its purpose is solely discovery, but is not able to verify that the fingerprints really correspond to the persons they claim to be. This due to the distributed nature of brig where there is no central or federated authority that coordinate user name registrations. So it is perfectly possible that one name can be taken by several repositories - only the fingerprint is unique.

• Later development of brig might interpret the user name and domain as email and might use your email account for verification purposes.

Having a resource part is optional, but can help if you have several instances of brig on your machines. i.e. one user name could be dad@smith.org/desktop and the other dad@smith.org/server.

Logging¶

Unless you pass the -s (--log-to-stdout flag) as above, all logs are being piped to the system log. You can follow the log like this:

# Follow the actual daemon log:
$ journalctl -ft brig

This assumes you’re using a systemd-based distribution. If not, refer to the documentation of your syslog daemon.

Using several repositories in parallel¶

It can be useful to run more than one instance of the brig daemon in parallel. Either for testing purposes or as actual production configuration. In order for the brig client to know what daemon to talk to, you have to be specific about the repository (--repo) path. Here is an example:

# Be explicit
$ brig --repo /tmp/ali init ali -x --ipfs-path ~/.ipfs
$ brig --repo /tmp/bob init bob -x --ipfs-path ~/.ipfs2

# Since you specified --repo we know what daemon to talk to.
# You can also set BRIG_PATH for the same effect:
$ BRIG_PATH=/tmp/ali brig ls
<file list of ali>

# Add some alias to your .bashrc to save you some typing:
$ alias brig-ali="brig --repo /tmp/ali"
$ alias brig-bob="brig --repo /tmp/bob"

# Now you can use them normally,
# e.g. by adding them as remotes each:
$ brig-ali remote add bob $(brig-bob whoami -f)
$ brig-bob remote add ali $(brig-ali whoami -f)

Remote access¶

Working with remote data does often not work extremely well with the file abstraction that does not play well with timeouts. This often causes applications to hang for indefinite times, since they are not most of the time not build for data that might not be delivered immediately. For this very common case we have the --offline flag. It will error out immediately on files that are not in our local cache:

$ brig mount /tmp/mount --offline
# Or with fstab:
$ brig fstab add some-mount /tmp/mount --offline

If you have a remote file you want to read, you can do this to make it cached locally:

$ brig cat /remote-file > /dev/null

After brig cat run, you should be able to view the file normally in the mount.

Making mounts permanent¶

All mounts that are created via brig mount will be gone after a daemon restart. If you a typical set of mounts, you can persist them with the brig fstab facility:

$ brig fstab add tmp_rw_mount /tmp/rw-mount
$ brig fstab add tmp_ro_mount /tmp/ro-mount -r
$ brig fstab
NAME          PATH           READ_ONLY  ROOT  ACTIVE
tmp_ro_mount  /tmp/ro-mount  yes        /
tmp_rw_mount  /tmp/rw-mount  no         /
$ brig fstab apply
$ brig fstab
NAME          PATH           READ_ONLY  ROOT  ACTIVE
tmp_ro_mount  /tmp/ro-mount  yes        /     ✔
tmp_rw_mount  /tmp/rw-mount  no         /     ✔
$ brig fstab apply -u
NAME          PATH           READ_ONLY  ROOT  ACTIVE
tmp_ro_mount  /tmp/ro-mount  yes        /
tmp_rw_mount  /tmp/rw-mount  no         /

Et Voilà, all mounts will be created and mounted once you enter brig fstab apply or restart the daemon. The opposite can be achieved by executing brig fstab apply --unmount.

CAVEATS: The FUSE filesystem is not yet perfect and somewhat experimental. Keep those points in mind:

• Performance: Writing to FUSE is currently somewhat memory and CPU intensive. Generally, reading should be fast enough for most basic use cases, but also is not enough for high performance needs. If you need to edit a file many times, it is recommended to copy the file somewhere to your local storage (e.g. brig cat the_file > /tmp/the_file), edit it there and save it back for syncing purpose. Future releases will work on optimizing the performance.
• Timeouts: Although it tries not to look like one, we’re operating on a networking filesystem. Every file you access might come from a different computer. If no other machine can serve this file we might block for a long time, causing application hangs and general slowness. This is a problem that still needs a proper solution and leaves much to be desired in the current implementation.

Data retrieval¶

If the data is not on your local machine, where is it then? Thanks to IPFS it can be transferred from any other peer that caches this particular content. Content is usually cached when the peer either really stores this file or if this peer recently used this content. In the latter case it will still be available in its cache. This property is particularly useful when having a small device for viewing data (e.g. a smartphone, granted brig would run there) and a big machine that acts as storage server (e.g. a desktop).

How are the files secure then if they essentially could be everywhere? Every file is encrypted by brig before giving it to IPFS. The encryption key is part of the metadata and is only available to the peers that you chose to synchronize with. Think of each brig repository only as a cache for the whole network it is in.

Partial synchronisation¶

Sometimes you only want to share certain things with certain people. You probably want to share all your /photos directory with your significant other, but not with your fellow students. On the other hand you maybe want to share the /lectures folder with them. In brig you can define what folder you want to share with what remote. If you do not limit this, all folders will be open to a remote by default. Also note, that if a remote already got some content of a folder you did not want to share, he will still be able to access it. If you’re unsure, you should better be restrictive than too permissive.

To add a folder for a specific remote, you can use the folders subcommand of brig remote:

# Starting with next sync, bob will only see the /videos folder:
$ brig remote folder add bob /videos
$ brig remote folder ls bob
/videos

If you’re tired of typing all of this, be reminded that there are very short aliases for most subcommands:

$ brig rmt f a bob /videos

In some cases you might not trust your peers with some folders or don’t want to have modifications in that specific folder. For this case, brig supports adding a folder as --read-only. Other remotes still will have access to the folder, but whenever we sync with them the changes they made are ignored. You can add a read-only folder by adding the --read-only switch to the command above:

$ brig rmt f a bob /videos --read-only

$ brig remote folder set bob /videos -c embrace --read-only

Conflicts¶

Whenever two repositories have a file at the same path, brig needs to do some conflict resolving. If those files are equal or if they share common history and did not diverge there is nothing to fear. But what if both sides have different versions of a file without common history? In this case brig offers you to handle conflict by one of the three strategies:

• ignore: Ignore the change from the remote side.
• embrace: Ignore our state and take over the remote’s change.
• marker: Create a conflict file with the same name but a .conflict ending. Leave it to the user to resolve the conflict. This is the default.

You can configure this behavior by using brig cfg:

$ brig cfg set fs.sync.conflict_strategy marker

In some cases this might not be enough though. Sometimes you might want to say »I trust this remote, always accept their changes«. You can do this by setting the conflict strategy per remote. If no specific conflict strategy is set, fs.sync.conflict_strategy is used. You can set the strategy by using a subcommand of the brig remote family:

# Always take the versions of bob on conflicts:
$ brig remote conflict-strategy embrace bob

Still not enough? You can also set the conflict strategy per folder. This will trump the per-remote folder strategy:

# Use the default in all folders but use "embrace" in this one:
$ brig remote folder add bob /collab -c embrace

Automatic Updating¶

If you do not want to hit brig sync every time somebody in the network changed something, you can enable the automatic updating for any remote you like. Let’s suppose we are ali and want to receive updates on every change of bob, we should simply add the following:

$ brig remote auto-update enable bob

# (You can also abbreviate most of that:)
# brig rmt au e bob

Alternatively, we could have used the -a switch when adding bob as remote:

$ brig remote add bob -a

In any case, an initial sync is performed with this remote and a sync on every change that bob published. Keep in mind that bob will not receive your updates by default, he needs to decide to use auto updating for himself. You can watch the times when your repository was updated automatically by looking at brig log:

$ brig log
     -       Sun Dec 16 18:24:27 CET 2018 • (curr)
W1kGKKviWCBY Sun Dec 16 18:24:27 CET 2018 sync due to notification from »bob« (head)
...

Pushing changes¶

As you saw above, doing a brig sync won’t do a bidirectional synchronisation. It will only fetch metadata from the remote and modify our local state with it. In some cases you might want to push data to a remote - especially when it is on one of your machines and you use for example as archival repository. By default pushing to a remote is rejected. You can enable it on a per-remote basis with this command out of the brig remote family of commands:

# Allow bob and charlie to auto push to us.
$ brig remote auto-push enable bob charlie

Now either bob or charlie can do this from their machines:

# bob's machine:
$ brig push ali

This will simply ask ali to do a sync with bob.

Key concepts¶

I’d like you to keep the following mantra in your head when thinking about versioning (repeating before you go to sleep may or may not help):

Metadata and actual data are separated. This means that a repository may contain metadata about many files, including older versions of them. However, it is not guaranteed that a repository caches all actual data for each file or version. This is solely controlled by pinning described in the Pinning section. If you check out earlier versions of a file, you’re always able to see the metadata of it, but being able to view the actual data depends on having a peer that is being able to deliver the data in your network (which might be yourself). So in short: brig only versions metadata and links to the respective data for each version.

This is a somewhat novel approach to versioning, so feel free to re-read the last paragraph, since we’ve found that it does not quite fit what most people are used to. Together with pinning this offers a high degree of freedom on how you can decide what repositories store what data. The price is that this fine-tuned control can get a little annoying. Future versions of brig will try to solve that.

For some more background, you can invoke brig info to see what metadata is being saved per file version:

$ brig show README.md
Path          /README.md
User          ali
Type          file
Size          832 bytes
Inode         4
Pinned        yes
Explicit      no
ModTime       2018-10-14T22:46:00+02:00
Tree Hash     W1gX8NMQ9m8SBnjHRGtamRAjJewbnSgi6C1P7YEunfgTA3
Content Hash  W1pzHcGbVpXaePa1XpehW4HGPatDUJs8zZzSRbpNCGbN2u
Backend Hash  QmPvNjR1h56EFK1Sfb7vr7tFJ57A4JDJS9zwn7PeNbHCsK

Most of it should be no big surprise. It might be a small surprise that three hashes are stored per file. The Backend Hash is really the link to the actual data. If you’d type ipfs cat QmPvNjR1h56EFK1Sfb7vr7tFJ57A4JDJS9zwn7PeNbHCsK you will get the encrypted version of your file dumped to your terminal. The Content Hash is being calculated before the encryption and is the same for two files with the same content. The Tree Hash is a hash that uniquely identifies this specific node for internal purposes. The Inode is a number that stays unique over the lifetime of a file (including moves and removes). It is used mostly in the FUSE filesystem.

Commits¶

Now that we know that only metadata is versioned, we have to ask »what is the smallest unit of modification that can be saved?«. This smallest unit is a commit. A commit can be seen as a snapshot of the whole repository.

The command brig log shows you a list of commits that were made already:

      -      Sun Oct 14 22:46:00 CEST 2018 • (curr)
W1kAySD3aKLt Sun Oct 14 22:46:00 CEST 2018 user: Added ali-file (head)
W1ocyBsS28SD Sun Oct 14 22:46:00 CEST 2018 user: Added initial README.md
W1D9KsLNnAv4 Sun Oct 14 22:46:00 CEST 2018 initial commit (init)

Each commit is identified by a hash (e.g. W1kAySD3aKLt) and records the time when it was created. Apart from that, there is a message that describes the commit in some way. In contrast to git, commits are rarely done by the user themselve. More often they are done by brig when synchronizing.

All commits form a long chain (no branches, just a linear chain) with the very first empty commit called init and the still unfinished commit called curr. Directly below curr there is the last finished commit called head.

Sometimes you might want to do a snapshot or »savepoint« yourself. In this case you can do a commit yourself:

$ brig touch A_NEW_FILE
$ brig commit -m 'better leave some breadcrumbs'
$ brig log | head -n 2
      -      Mon Oct 15 00:27:37 CEST 2018 • (curr)
W1hZoY7TrxyK Sun Oct 14 22:46:00 CEST 2018 user: better leave some bread crumbs (head)

This snapshot can be useful later if you decide to revert to a certain version. The hash of the commit is of course hard to remember, so if you need it very often, you can give it a tag yourself. Tags are similar to the names, curr, head and init but won’t be changed by brig and won’t move therefore:

# instead of "W1hZoY7TrxyK" you also could use "head" here:
$ brig tag W1hZoY7TrxyK breadcrumbs
$ brig log | grep breadcrumbs
$ W1hZoY7TrxyK Sun Oct 14 22:46:00 CEST 2018 user: better leave some bread crumbs (breadcrumbs, head)

File history¶

Each file and directory in brig maintains its own history. Each entry of this history relates to exactly one distinct commit. In the life of a file or directory there are four things that can happen to it:

• added: The file was added in this commit.
• moved: The file was moved in this commit.
• removed: The file was removed in this commit.
• modified: The file’s content (i.e. hash changed) was altered in this commit.

You can check an individual file or directorie’s history by using the brig history command:

# or "hst" for short:
$ brig hst README.md
CHANGE  FROM  TO              WHEN
added   INIT  W1ocyBsS28SD    Oct 14 22:46:00
$ brig mv README.md README_LATER.md
$ brig hst README_LATER.md
CHANGE  FROM  TO            HOW                           WHEN
moved   HEAD  CURR          /README.md → /README_LATER.md Oct 15 00:27:37
added   INIT  W1ocyBsS28SD                                Oct 14 22:46:0

As you can see, you will be shown one line per history entry. Each entry denotes which commit the change was in. Some commits were nothing was changed will be jumped over except if you pass --empty.

Viewing differences¶

If you’re interested what changed in a range of your own commits, you can use the brig diff command as shown previously. The -s (--self) switch says that we want to compare only two of our own commits (as opposed to comparing with the commits of a remote).

# Let's compare the commit hashes from above:
$ brig diff -s W1hZoY7TrxyK W1kAySD3aKLt
•
└── + A_NEW_FILE

Often, those hashes are quite hard to remember and annoying to look up. That’s why you can the special syntax <tag or hash>^ to denote that you want to go »one commit up«:

brig diff -s head head^
•
└── + A_NEW_FILE
# You can also use this several times:
brig diff -s head^^^ head^^^^^
•
└── + README.md

If you just want to see what you changed since head, you can simply type brig diff. This is the same as brig diff -s curr head:

$ brig diff
•
└── README.md → README_LATER.md
$ brig diff -s curr head
•
└── README.md → README_LATER.md

Reverting to previous state¶

Until now we were only looking at the version history and didn’t modify it. The most versatile command to do that is brig reset. It is able to revert changes previously made:

# Reset to the "init" commit (the very first and empty commit)
$ brig reset init
$ brig ls  # nothing, it's empty.

The key here is that you did not loose any history:

$ brig log | head -2
       -     Mon Oct 15 00:51:12 CEST 2018 • (curr)
W1hZoY7TrxyK Sun Oct 14 22:46:00 CEST 2018 user: better leave some bread crumbs (breadcrumbs)

As you can see, we still have the previous commits. brig revert did one thing more than restoring the state of init and put that result in curr. This also means that you can’t really modify history. But you can revert it. Let’s revert your complete wipe-out:

# Reset to the state we had in »breadcrumbs«
$ brig reset breadcrumbs

brig reset cannot only restore old commits, but individual files and directories:

$ brig reset head^^ README.md

Garbage collection¶

Strongly related to pinning is garbage collection. Whenever you need to clean up some space, you can just type brig gc to remove all unpinned files from the cache.

By default, the garbage collector is also run once every hour. You can change this interval by setting brig config set repo.autogc.interval to 30m for example. You can also disable this automatic garbage collection by issuing brig config set repo.autogc.enabled false.

Repinning¶

Repinning allows you to control how many versions of each file you want to store and/or how much space you want to store at most. The repinning feature is controlled by the following configuration variables:

• fs.repin.quota: Maximum amount of data to store in a repository.
• fs.repin.min_depth: Keep this many versions definitely pinned. Trumps quota.
• fs.repin.max_depth: Unpin versions beyond this depth definitely. Trumps quota.
• fs.repin.enabled: Wether we should allow the repinning to run at all.
• fs.repin.interval: How much time to wait between calling repinning automatically.

Normally repinning will run for you every 15 minutes. You can also trigger it manually:

$ brig pin repin

By default, brig will keep 1 version definitely (fs.repin.min_depth) and delete all versions starting with the 10th (fs.repin.max_depth). The default quota (fs.repin.quota) is 5GB. If repin detects files that need to be unpinned, then it will first unpin all files that are beyond the max depth setting. If this is not sufficient to stay under the quota, it will delete old versions, layer by layer starting with the biggest version first.

Gateway Screenshots¶

The gateway UI consists of several tabs, which are briefly shown below to give you a short impression of it.

Login screen¶

Allows you to login. You can also come back here to change the user. It is also possible to login anonymously, as you will see below.

File Browser¶

The main view. Lists the directory tree and file attributes. Allows for modification, uploading and everything what you’d expect.

Changelog View¶

A list of commits. You are able to jump back to a specific commit.

Trashbin¶

A list of deleted files. If you deleted something you will be able to get it back here.

Remote List¶

If your user is privileged enough, you can see and edit the list of remotes and adjust settings in it.

Remote Add Dialog¶

A sample dialog. The UI uses many of them.

Introduction¶

Many users will not run brig themselves, so you won’t be able to brig sync with them. Chances are that you still want to send or present them your files without too much hassle. brig features a Gateway to HTTP(S), which comes particularly handy if you happen to run a public server and/or want to provide a GUI to your users. It also includes an easy to use UI that is enabled by default.

Before you do anything, you need to a »user« to your gateway. This user is different than remotes and describes what credentials can be used to access the gateway. You can add add a new user like this:

$ brig gateway user add admin my-password
# or shorter:
# brig gw u a admin my-password
$ brig gateway user list
NAME  FOLDERS
admin /

The gateway is disabled by default. If you want to start it, use this command:

$ brig gateway start

Without further configuration, this will create a HTTP (not HTTPS!) server on port 6001, which can be used already. If you access it under http://localhost:6001 you will see a login mask where you can log yourself in with the credentials you entered earlier.

If you’d like to use another port than 6001, you can do so by setting the respective config key:

$ brig cfg set gateway.port 7777

$ brig gateway status

The gateway can be stopped anytime with the following command:

$ brig gateway stop

There is also a small helper that will print you a nice hyperlink to a certain file called brig gateway url:

$ brig gateway url README.md
http://localhost:6001/get/README.md

Folder management¶

You probably do not want to offer your files to everyone that have a link. Therefore you can restrict access to a few folders (/public for example) for individual users. By default a user is allowed to see everything. If you want a user that can only access the /public folder simply add him as follows:

$ brig gw user add my-new-user /public

Now only the files in /public (and including /public itself) are accessible from the gateway.

User right management¶

We already discussed the adding of a user above. There is a little more to that though. You can add users with different rights. In total there are 5 different rights currently:

• fs.view: View and list all files.
• fs.edit: Edit and create new files.
• fs.download: Download file content.
• remotes.view: View the remotes tab.
• remotes.edit: Edit the remotes tab.

When you add users you can give a new user a comma separated list of rights via the -r switch:

$ brig gw user add my-new-user -r 'remotes.view,remotes.edit'

For your convenience there are a bunch of presets which will do the work for you in 99% of the cases:

• --role-admin, -a: Add this user as admin (short for »-r ‘fs.view,fs.edit,fs.download,remotes.view,remotes.edit’«)
• --role-editor, -b: Add this user as collaborator (short for »-r ‘fs.view,fs.edit,fs.download,remotes.view’«)
• --role-collaborator, -c: Add this user as collaborator (short for »-r ‘fs.view,fs.edit,fs.download’«)
• --role-viewer, -d: Add this user as viewer (short for »-r ‘fs.view,fs.download’«)
• --role-link-only, -e: Add this user as linker (short for »-r ‘fs.download’«)

Running the gateway with HTTPS¶

The gateway has built-in support for LetsEncrypt. If the gateway is reachable under a DNS name, it is straightforward to get a TLS certificate for it. In total there are three methods:

Method one: Automatic: This works by telling the gateway the domain name. Since the retrieval process for getting a certificate involves binding on port 80, you need to prepare the brig binary to allow that without running as root:

# You need to restart the brig daemon for that.
# Every next brig command will restart it.
$ brig daemon quit
$ sudo setcap CAP_NET_BIND_SERVICE=+ep $(which brig)

Afterwards you can set the domain in the config. If the gateway is already running, it will restart immediately.

$ brig cfg set gateway.cert.domain your.domain.org

You can check after a few seconds if it worked by checking if the certfile and keyfile was set:

$ brig cfg get gateway.cert.certfile
/home/user/.cache/brig/your.domain.org_cert.pem
$ brig cfg get gateway.cert.keyfile
/home/user/.cache/brig/your.domain.org_key.pem
$ curl -i https://your.domain.org:6001
HTTP/2 200
vary: Accept-Encoding
content-type: text/plain; charset=utf-8
content-length: 38
date: Wed, 05 Dec 2018 11:53:57 GMT

<html>
...
</html>

This method has the advantage that the certificate can be updated automatically before it expires.

Method two: Half-Automated:

If the above did not work for whatever reasons, you can try to get a certificate manually. There is a built-in helper called brig gateway cert that can help you doing that:

$ brig gateway cert your.domain.org
You are not root. We need root rights to bind to port 80.
I will re-execute this command for you as:
$ sudo brig gateway cert nwzmlh4iouqikobq.myfritz.net --cache-dir /home/sahib/.cache/brig

A certificate was downloaded successfully.
Successfully set the gateway config to use the certificate.
Note that you have to re-run this command every 90 days currently.

If successful, this command will set the certfile and keyfile config values for you. You can test if the change worked by doing the same procedure as in method one. Sadly, you have to re-execute once the certificate expires.

Method three: Manual:

If you already own a certificate you can make the gateway use it by setting the path to the public certificate and the private key file:

$ brig cfg set gateway.cert.certfile /path/to/cert.pem
$ brig cfg set gateway.cert.keyfile /path/to/key.pem

If you do not own a certificate yet, but want to setup an automated way to download one for usages outside of brig, you should look into certbot.

Redirecting HTTP traffic¶

This section only applies to you if you choose method one from above and want to run the gateway on port 80 (http) and port 443 (https). This has the advantage that a user does not need to specify the port in a gateway URL have which looks a little bit less »scary«. With this setup all traffic on port 80 will be redirected directly to port 443.

$ brig cfg set gateway.port 443
$ brig cfg set gateway.cert.redirect.enabled true
$ brig cfg set gateway.cert.redirect.http_port 80

Allowing anonymous access¶

If you want to run a public gateway (for example for a group of friends), then you might want to enable anonymous access. In this mode you will be logged in right away to the gateway without facing the login screen. You still have the option to go to the login screen and become another user.

You can enable the anonymous mode like this:

$ brig cfg set gateway.auth.anon_allowed true

Additionally you have to create an anon user. This allows you to define what rights the anonymous users have and what folders they may access:

# Give the anonymous users only access to /public and don't let them modify anything:
$ brig gw u add anon anon --role-viewer /public

If you want to change the name of the anon user to something else (for whatever reason) you can do so by setting the auth.anon_user variable. You also have to re-add the user above with the new name.

$ brig cfg set gateway.auth.anon_user some_other_anon_name_that_is_not_used

Profiles¶

1. Init¶

Before you can do anything with brig you need to create a repository. During this step, also your online identity will be created. So make sure to use a sane username (sahib@wald.de) and resource (laptop).

As an alternative to entering your password manually, you can use an existing password manager:

2. Adding files¶

Before synchronizing them, you need to stage them. The files will be stored encrypted (and possibly compressed) in blobs on your hard disks.

3. Coreutils¶

brig provides implementations of most file related core utils like mv, cp, rm, mkdir or cat. Handling of files should thus feel familiar for users that know the command line.

4. Mounting¶

For daily use and for use with other tools you might prefer a folder that contains the file you gave to brig. This can be done via the built-in FUSE layer.

If you wish to always have the mount when brig is running, you should look into Making mounts permanent.

5. Commits¶

In it’s heart, brig is very similar to git and also supports versioning via commits. In contrast to git however, there are no branches and you can’t go back in history – you can only bring the history back up front.

6. History¶

Each file (and directory) maintains a history of the operations that were done to this file.

7. Discovery & Remotes¶

In order to sync with your buddies, you need to add their fingerprint as remotes. How do you get their fingerprint? In the best case by using a separate side channel like telephone, encrypted email or elsewhise. But brig can assist finding remotes via the brig net locate command.

8. Sync & Diff¶

Once both parties have setup each other as remotes, we can easily view and sync with their data.

9. Pinning¶

By default brig will only keep the most recent files. All other files will be marked to deletions after a certain timeframe. This is done via Pins. If a file is pinned, it won’t get deleted. If you don’t need a file in local storage, you can also unpin it. On the next access brig will try to load it again from a peer that provides it (if possible).

General questions¶

Technical questions¶

1. How is the encryption working?¶

A stream is chunked into equal sized blocks that are encrypted in GCM mode using AES-256. Additionally ChaCha20 (with Poly1305) is currently supported but it might be removed soon. The overall file format is somewhat similar to NaCL secretboxes, but it is more tailored to supporting efficient seeking.

The current default is ChaCha20, although machines with the aes-ni instruction set might yield significant higher throughput. The source of the encryption layer can be found here. Here’s a basic overview over the format:

The key of each file is currently being derived from the content hash of the file (See also Convergent Encryption). If the content changes later, the key does not change since the key is only generated once during the first staging of the file.

Please refer to the implementation for all implementation details for now. No security audits of the implementation have been done yet, therefore I’d appreciate every pair of eyes. Especially while everything is still in flux and won’t harm any users.

2. Is there compression implemented?¶

Yes. The compression is being done before encryption and is only enabled if the file looks compression-worthy. The »worthiness« is determined by looking at its header to guess a mime-type. Depending on the mime-type either snappy or lz4 is selected or no compression is added at all.

The source of the compression layer can be found here. Here’s a basic overview over the format:

5. How fast is the I/O when using brig?¶

Here are some rather outdated graphs where you can get a rough feeling how fast it can be. There are a few rules of thumb with mostly obvious content:

• It it goes over the network, it’s the network speed plus a smaller constant overhead.
• If it comes over FUSE, it is quite a bit slower than over brig cat.
• If you do not use compression, writing and reading will be faster.

The graphs below only measure in-memory performance compared to a dd like speed (see the »baseline« line).

Your mileage may vary and you better do your own benchmarks for now.

Todo

Explain/Update those graphs.

Encrypted and compression built-in¶

• All data is encrypted during storage and transport using AES-256 in GCM mode.
• Optional compression algorithm is selected based on the file type.
• Keys are stored as part of the metadata during synchronisation.

Easy Version control¶

• Simplified git-like version control only limited by your storage space.
• Synchronization algorithm that can handle moved files and empty directories and files.
• Auto-updating facility that will sync on any change.
• Configurable conflict handling.

Separation between data and metadata¶

• Your data does not need to be stored on the device you are currently using.
• Pin the data you want to use to your local storage. Every repository acts as cache of all the files you have access to.
• Keep a range of versions cached locally and delete older versions if they exceed a quota.

Truly Distributed¶

• No central server at all. All infrastructure is based on IPFS.
• Still, central architectures can be build with brig.
• Simple user identification and discovery.
• You do not store data you don’t want to.

FUSE filesystem¶

• FUSE filesystem that makes all files seem like a normal directory.
• Allows your normal tools to work seamlessly with brig.
• Mounts can be persisted to stay where they are.

Gateway and Web UI¶

• Gateway to share normal HTTP/S links with other users.
• Simple UI provided to execute the typical tasks without the need of a command line.
• User and right management included.

100% Open-Source¶

• Completely free software under the terms of the AGPL.
• Development driven by the community.
• Written in Go and Elm.

Current state¶

The first real release (0.3.0 »Galloping Galapagos«) was released on the 7th December 2018. It includes all basic features and is working somewhat. The original goals were met:

• Stable command line interface.
• Git-like version control
• User discovery
• User authentication
• Fuse filesystem

For day-to-day use there are quite some other features that make brig easier to use and capable of forming a Dropbox-like backend out of several nodes.

There will be no stability guarantees before version 1.0.0.

Future¶

Those features should be considered after releasing the first prototype. A certain amount of first user input should be collected to see if the direction we’re going is valid.

• Gateway: Provide a built-in (and optional) http server, that can »bridge« between the internal ipfs network and people that use a regular browser. Instances that run on a public server can then provide hyperlinks of files to non-brig users. Done as of version 0.3.0.
• Config profiles: Make it easy to configure brig in a way to serve either as thin client or as archival node. Archival nodes can be used in cases where a brig network spans over computers that lie in a different timezone. The archival node would accumulate all changes and repositories would see it as some sort of “blessed repository” which holds the latest and greatest state.
• Automatic syncing: Automatically publish changes after a short amount of time. If an instance modified some file other nodes are notified and can decide to pull the change. Done as of version 0.4.0.
• Intelligent pinning strategies: By default only the most recent layer of files are being kept. This is very basic and can’t be configured currently. Some users might only want to have only the last few used files pinned, archive instances might want to pin almost everything up to a certain depth. Done as of version 0.4.0 (see repinning)
• Improve read/write performance: Big files are currently hold in memory completely by the fuse layer (when doing a flush). This is suboptimal and needs more intelligent handling and out-of-memory caching of writes. Also, the network performance is often very low and ridden by network errors and timeouts. This can be tackled since IPFS v0.4.19 supports an –offline switch to error out early if a file is not available locally.
• More automated authentication scheme: E-Mail-like usernames could be used to verify a user without exchanging fingerprints. This could be done by e.g. sending an activation code to the email of an user (assuming the brig name is the same as his email), which the brig daemon on his side could read and send back.
• Format and read fingerprint from QR-Code: Fingerprints are hard to read and not so easy to transfer and verify. QR-Code could be a solution here, since we could easily snap a picture with a phone camera or print it on a business card.

Far Future¶

Those features are also important, but require some more in-depth research or more work and are not the highest priority currently.

• Port to other platforms: Especially Windows and eventually Android. This relies on external help, since I’m neither capable of porting it, nor really a fan of both operating systems.
• Implement alternative to FUSE: FUSE currently only works on Linux and is therefore not usable outside of that. Windows has something similar (called Dokan). Alternatively we could also go on by implementing a WebDAV server, which can also be mounted.
• Implement the encryption in IPFS: Having the encryption/compression layer in brig effectively disables the usage of deduplication. This is unfortunate and could be mitigated by either implementing deduplication ourselves or moving to a block based encryption scheme.

• Ensure N-Copies: It should be possible to define a minimum amount of copies a file has to have on different peers. This could be maybe incorporated into the pinning concept. If a user wants to remove a file, brig should warn him if he would violate the min-copies rule. This idea is shamelessly stolen from git-annex.

What to improve¶

We try to open a ticket for anything that can be worked right now

The following general improvements are of course also greatly appreciated:

• Bug reports & fixes.
• Documentation improvements.
• Writing more and better tests.
• Porting to other platforms.

Workflow¶

Please adhere to the general GitHub workflow_, i.e. fork the repository, make your changes and open a pull request that can be discussed.

Here’s a small checklist before publishing your pull request:

• Did you go fmt all code?
• Does your code style fit with the rest of the code base?
• Did you run go run mage.go dev:lint?
• Did you write tests if necessary?
• Did you consider if changes to the docs are necessary?
• Did you check if you need something to CHANGELOG.md?

Thank you for your contribution.