Back in 2018 a reader responded to a post about a headset, I had written, saying how the fix I had described used to work but now no longer did. Which led to an interesting, collabarative debugging expedition, trying to understand why my headset was still working but his wasn’t. That was a completely unexpected but amazing side effect of having shared a few tips on this blog.
The reason I’m bringing this up is that another reader has reached out and very kindly shared his discoveries. Again, this blows me away, seeing as this blog is not really a forum and I have no fake internet points to award. Which makes it feel all the more special and community-like and stuff. Sniff, I’m getting all choked up. Anyway, thank you for sharing, Knut. This post is really just a write-up of Knut’s suggestions on my first Syncopoli post with some of my own comments and experiences thrown in.
Last year I set up an Rsync daemon on my homeserver to share some music files with my phone. Using Syncopoli on Android – which is no longer just the most recently updated rsync client on fdroid but the only – I managed to connect to the daemon over the rsync protocol and download the files to the phone. As the rsync protocol is unencrypted, I felt the need to stress many times over that a) it was obviously not private, b) I didn’t think it was unsafe but who could really say when nobody really uses rsync that way and c) that it should only be a temporary measure until I could move it to SSH, the way rsync is almost always used nowadays.
Here’s what I’m setting out to do: I want to be able to synchronise a remote server directory to a local phone directory using rsync over ssh, using Syncopoli on the client side and OpenSSH on the server side. . I will be using an ssh key on the phone to authenticate with the server. The benefits of this is that I can remove any local or public facing rsync daemon I have had running and that my traffic to and from the server now will be secure and private.
There are downsides, though, make no mistake. First, due to limitations of dropbear (and possibly Syncopoli) the private SSH key will not be password protected. Second, the private key will reside on the phone’s storage available to all applications with permission to access “Files and media”.
In other words, anybody who can lay their hands on the phone will be able to gain access to the account. Needless to say, if you adopt this approach, the account in question should be as circumscribed as possible, and ideally the host in question a virtual machine that you are able and willing to implode and kill with the flick of a switch the second you think someone has accessed your phone.
Why not just use SSH with passwords? Two simple reasons: First, I don’t know how Syncopoli stores passwords but I suspect a hands-on-phone attacker would be able to extract a password as easily as copying the private key (and it does not currently have the ability to ask the user for the password when the sync job is run). Second, and more importantly, that would remove the best protection that you can use on your SSH server next to banning root login through SSH, i.e. requiring keys, not passwords.
Making a key pair
As with any assymetric encryption solution, we need to generate a pair of keys: One private and one public. The server gets the public key, the phone holds the private one. Problem number one: Syncopoli cannot use the openssh private keys that most linux distros use, it can only use dropbear keys. Dropbear is an alternative to OpenSSH. Fortunately, there is a way to do this without switching my server from OpenSSH over to dropbear:
Generate an OpenSSH key pair and convert the private key to dropbear’s format. Syncopoli can still talk to my OpenSSH server and the dropbear private key still fits the OpenSSH public key
Normally I would generate the key pair on the client because then I would only have to move the not-secret public key off of the machine. However, I don’t have the necessary tools (
dropbearconvert) on the client in this instance, what with Android not being a proper linux distro. So here I’m going to have to generate the key pair on the server and move the private key off of it without spilling the secret.
Now, Knut tried using Ed25519 keys but could not make it work. Only RSA keys seemed to work. I have not found anything in the dropbear conversion tool or in Syncopoli, explicitly stating this, but I have taken his word for it. It certainly mirrors my experience using PuTTY and similar non-OpenSSH clients where support for anything but RSA were exceptionally slow to arrive.
Note: According to Wikipedia Dropbear has supported at least ECDSA since 2013 though and I did at least find one example that suggests that Syncopoli can work with ECDSA. Also, release notes for dropbear 2020.79 (June 2020) mention support for Ed25519, though none of my current distros have such a recent version. Of course, there are a plethora of reasons why these types might work on a pure dropbear setup but not in this scenario. In short, I think it warrants more experimentation but for now I’m sticking with RSA. If anybody has any experience with using either elliptic curve certificate type in this setup, please share in the comments.
Right, on to the command:
ssh-keygen -t rsa -b 4096 -m PEM
Fairly simple: Type (
-t) is RSA, size in bits (
-b) is 4096 (what the Arch wiki goes with, all the while noting that adding length to RSA keys don’t really make them much harder to crack) and
-m PEM specifies that the output should use “the previously-used PEM format private keys” which is required for the later conversion to work. The
ssh-keygen man page goes on to note that the newer OpenSSH specific non-PEM format has some advantages and also allows inserting comments but we have to use PEM. From this we can infer that it’s no use trying to add a comment to the key pair using the
-c flag because you can’t have comments in the PEM format.
Do not enter a password with which to encrypt the certificate. Just hit Enter twice. The reason is this line in the dropbearconvert man page:
Encrypted private keys are not supported, use ssh-keygen(1) to decrypt them first.
There is no point in encrypting and then (permanently) decrypting the key.
This gives me the following in my ~/.ssh folder
-rw------- 1 owner group 3.2K Oct 25 22:44 .ssh/id_rsa -rw------- 1 owner group 750 Oct 25 22:44 .ssh/id_rsa.pub
Putting the public key into service is as simple as adding the contents to
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
Note the “double arrow” for concatenate rather than overwrite in case I already have other keys in that users’
Finally I check that RSA is among the the accepted type of hostkey by looking through the lines beginning with “HostKey” or “#Hostkey” in my sshd config file:
$ grep ^#*HostKey /etc/ssh/sshd_config HostKey /etc/ssh/ssh_host_ed25519_key HostKey /etc/ssh/ssh_host_ecdsa_key HostKey /etc/ssh/ssh_host_rsa_key
Depending on distro it might need to be uncommented and the sshd process restarted.
^#*HostKey is regex for “lines beginning with either HostKey or #Hostkey”.
Converting private key
On debian and derivatives the tool
dropbearconvert is part of the dropbear-bin package.
dropbear-bin is a dependency of the dropbear package but that one will include a bunch of other stuff that isn’t necessary or relevant seeing as I won’t be running the dorpbear server. So might as well just pick the one I need.
Once installed all I need to do is run
dropbearconvert which for some reason is hidden away in the
/usr/lib rather than the
# Usage: /usr/lib/dropbear/dropbearconvert <inputtype> <outputtype> <inputfile> <outputfile> /usr/lib/dropbear/dropbearconvert openssh dropbear ./id_rsa ./id_rsa_db
I cannot say much about the openssh vs. dropbear types but when cat’ting the certificates it becomes clear that the OpenSSH PEM certificate is base64 encoded (i.e. the data can be represented by text characters), so you can print it and inspect it as an orderly block of characters whereas the dropbear certificate seems to be just raw binary data.
As mentioned before it is obviously crucial that when I move the converted key off of the server onto the phone it is done so by secure and private means.
I had Syncopoli configured for an rsync protocol setup so I am going to describe what settings I changed and why. This is valid for Syncopoli version 0.6.
First in “Settings” I change a number of things. Note that Syncopoli forces you to pick one global protocol (SSH or Rsync) as well as one global host and stick with them. So you have to completely abandon any Rsync protocol based jobs now. Yes, there is an open issue for that as well.
Protocol: Change from
Port: Change from whatever port I’m running the Rsync daemon on (default: 873) to the port the SSH daemon is listening on (default: 22).
User: Should be changed so as to correspond to the username for whom the certificate was made on the server .
Private Key File: Should be set to the location of the private key file that I just moved from the server to the phone. Some Android file managers (Amaze, for one) have the capability to copy file locations to the clipboard as it is otherwise a pain to enter the location manually.
All other settings can be carried over from a working rsync protocol setup. Note that I do not set
SSH Password as I’m using a certificate instead of password authentication (and password authentication is not allowed on my server).
After updating the settings I add a new Syncopoli profile by returning to the “main screen” (the list of profiles) and hit the “+” button in the lower right corner. Or press and hold the old Rsync profile and choose “Copy Profile”. Apart from naming the new profile somthing distinct that should identify it as SSH based , there is really only one thing to change:
In a remote-to-local Rsync protocol profile
Origins refers to the name of a “share” in the Rsync daemon config file. In an SSH protocol profile it’s simply the absolute path of the remote directory with which I wish to sync. So I just enter the the remote absolute path and check that the user in question has the necessary read permission to that directory. If I were instead setting up a local-to-remote profile my user would obviously also need write permissions.
As for rsync options there should be no difference – all the usual ones apply, like
--recursive, etc. I did have one small issue which was that after switching Syncopoli would sometimes re-download the same file multiple times even though it had not changed on the server side.
--size-only fixed that quite handily.
All that’s left now is to check whether I am indeed connecting to the machine I think I am. Now, technically I should just be able to go into Syncopoli’s settings, press “Trust host fingerprint” and verify whether or not the fingerprint matches that of my server. Three things, however, complicate things a bit:
First, which public key is actually used to identify the host? Logically, sshd cannot know which user I want to log in as beforehand. So it cannot present one specific user’s public key fingerprint. Instead it uses the host keys found in
/etc/ssh. A quick look there and a nother question asks itself: Which of the keys there is used? RSA, ECDSA, Ed25519, or something else entirely?
Second, Syncopoli presents an MD5 fingerprint/hash of the host key, whereas by default OpenSSH uses SHA256.
Finally, Syncopoli’s “Trust host fingerprint” dialog only tells me the fingerprint and nothing else.
Here’s what I did that worked: I ran the job after revoking all trust (“Clear trusted host” in Settings). That is obviously going to fail but it will fail in an informative manner. Host such and such, Syncopoli tells me, is not in the trusted hosts file. Well, duh, I just erased that, so obviously. But it then goes on to tell me what type of certificate it has been presented with and it’s fingerprint. The first part is the crucial difference beween this and the “Trust host fingerprint” UI. Here I’m told that the certificate it is informing me about is not actually an RSA one but an ECDSA one (“ecdsa-sha2-nistp256 fingerprint md5 89:xx:xx:…”)
This led to run the “tell me the fingerprint” command on the ECDSA certificate in
/etc/ssh on my server and finally got a match.
There is an ECDSA certificate there, I suspect, because by default Ed25519, ECDSA and RSA are all enabled by default on my distro. Why ECDSA and not ED25519 or RSA, the ones I actually use? I have no idea. Alphabetically first? Anyway, here’s the command:
ssh-keygen -l -E md5 -f /etc/ssh/ssh_host_ecdsa_key.pub
-l is for show me the fingerprint instead of making a new file, -E md5 tells keygen to use MD5 rather than SHA256 that has supplanted it (because that’s what Syncopoli will show me) and -f points to the file. Now that I know which fingerprint to match, I finally go to “Trust host fingerprint” and verify that the fingerprint Syncopoli is showing me, matches the one I just saw ssh-keygen spit out.
Next time I run the sync job, I should get a line through.
I will try to revisit this in a third and, hopefully, final part where I will try to run some tests to see if some combination of dropbear, OpenSSH and Syncopoli allows for using ECDSA and Ed25519. No promises, though. Although, it would be nice to finally put RSA keys out to pasture.
The [Syncopoli] launcher icon is designed by Armin Moradi and distributed under Creative Commons Attribution-ShareAlike 4.0 International Public Licensehttps://gitlab.com/fengshaun/syncopoli/blob/master/LICENSE.icons
Hi Mads, kudos for this excellent write-up (pt I + II). I started struggeling with Syncopoli way back but there were simply too many issues to deal with, I never got it working. The project has matured since then, but it’s still in a AS IS-state in my opinion. I sincerely hope we’ll see further development or active forks.
My use case is syncing files (i.e. pictures) from my phone to a local NAS running OpenMediaVault located in my home apartment (no worries running this as unencrypted rsync). I found it convenient configuring an rsync module “server side” and just referencing it by name in a Syncopoli profile. After endless attempts with “Sync failed”, turning on max verbosity in rsyncd.conf etc I switched to rsync over SSH, following your instructions and the sync-job finally finished.
Now, after jumping down quite a few rabbit holes, I’d like to point out a few things that might be missing in your write-up:
* Rsync over SSH does not depend on rsyncd (rsync daemon). Just make sure rsync (client) is installed both locally and on the remote system.
* Modules configured in rsyncd.conf cannot be used when doing rsync over SSH. Instead you’ll have to enter a valid remote path (absolute or relative) as destination in Syncopoli.
* Troubleshoot auth and permissions issues by browsing through auth.log (or wherever sshd is spitting out logs). The ssh daemon can be very picky about permissions so make sure to restrict ownerships of authorized_keys, home-dirs etc.
* adb can also supply you with some valuable clues by browsing through debug messages from Syncopoli with logcat. Turn on development tools and USB debugging on your phone, connect through USB and run adb shell (see further instructions in Syncopoli README.md)
* rsync error code 12 is probably due to connection timeouts (I got this after a sync failed with a partial set of files). Try setting parameters ServerAliveInterval and ServerAliveCount in your ~/.ssh/config and parameters ClientAliveInterval and ClientAliveCount in /etc/ssh_config
Good tips and clarifications, thank you, Fredrik 👍
For the security minded: the concern about having a private ssh-key accessible by other Android apps can be addressed by restricting ssh to issue a predefined command-line when the ssh session is initiated with the aforementioned key.
The predefined command is supplied along with the public key in authorized_keys. This will effectively override any rsync command-line sent by Syncopoli (or other apps using the same private key), which renders it a seemingly useless solution.
There is however a utility called sshdo which enables multiple command to be executed in the context of a single key (the environment variable $SSH_ORIGINAL_COMMAND will be matched by predefined commands stored in /etc/sshdoers).
Disclaimer: I haven’t tried this myself (yet).
A couple of things… When you see “ecdsa-sha2-nistp256 fingerprint md5 89:xx:xx:…” you’re looking at a fingerprint hash of the ecdsa key. This does not mean that any any keys are stored in the, now insecure, md5 format. Don’t set the encrypting format for your private key to md5! Leave it the default rsa — it works just fine. It’s also worth pointing out that 0.6 is still somewhat buggy; behaviour is not always as expected when adding a trusted host. I think I needed to back out of the app for it to save properly. ‘Last updated’ doesn’t update whilst the app is in the foreground, either.
you have a little typo in your manual: If you check the ssh fingerprint of your host (server) and the output should be in md5 (to compare with Syncopoli) and, as described, you created an RSA key, your check up should be also check up the RSA file, not the ECDSA.
ssh-keygen -l -E md5 -f /etc/ssh/ssh_host_ecdsa_key.pub
But it should be:
ssh-keygen -l -E md5 -f /etc/ssh/ssh_host_rsa_key.pub
Makes sense 🙂 Thanks, Knut 👍
EDIT: No, wait, now that I’m actually going through the proces again, I’m not sure, I agree. As I wrote, the server is verified by the ECDSA host key, not by my user’s public key. I don’t know why it’s the ECDSA host key rather than one of the other host keys but that was the one that worked.