Using Rsync on Android: Getting Syncopoli talking to the rsync daemon

Transferring files from your linux host to your phone using good ol’ rsync

Syncthing has largely taken the FOSS throne of continuous file synchronization on both servers and desktops. When it comes to my phone, however, I can still see a use for discrete, on-demand file synchronization – at least if changes only appear on one end, be it on the phone or the server. And for that purpose, there is really nothing else to turn to than good ol’ rsync.

Syncopoli seems to be the most up-to-date rsync client on Android, seeing as it’s code has at least been updated within the past year. Even though I believe I know how the basics of rsync on the command line, Syncopoli is not the most intuitive of mobile apps and presents some special problems. In the following I will detail both setting up the rsync daemon on the server and getting a Syncopoli setup that matches the server.

This initial setup will use an unencrypted rsync protocol because there were enough things that could go wrong and few enough debugging tools without adding encryption and keys into the mix. Hopefully, I will get on to switching to an SSH-based setup later but seeing as I’m only using it to sync publicly available mp3 files, I’m not too concerned. It does however call for the following…

Disclaimer: This is not a privacy focused setup. The files I’m syncing are not considered private in any way. I’m happy for the world to see that I’m downloading a new episode of Newshour and even – should the worst come to the worst – to share said episode with the world. It is, however, concerned with security. Nobody should be able to upload anything to my server. Nobody should be able to take over my server using rsyncd as a wedge.

October 2021: I wrote a follow-up where I configured Syncopoli to communicate with an OpenSSH server, rather than the rsync daemon. It builds on the understanding from the setup detailed in this post so even if you want to switch to SSH it might still be worth at least reading this first.

Rsync daemon

First, I had grown accustomed to rsync just working as a client without any setup. This is due to the fact that the machines I had been sending files to always have an SSH daemon running which was what was responding to my client. In order for Syncopoli to use the rsync protocol, I had to set up an rsync daemon listening instead.

Fortunately, the rsync daemon is fairly easy to configure, especially if you’ve ever set up network filesharing, like NFS of Samba, before. The configuration file defaults to /etc/rsyncd.conf and follows an ini-like format. First come the global settings, then a section for each file share (or module in rsync parlance). Sections are indicated by the “[Module Name]” header. The bare minimum looks like this

pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsync.log

[MODULE_NAME]
path = /path/to/folder
comment = My Files
read only = true
timeout = 300

The daemon can most easily be started by running the systemd service:

systemctl start rsync.service

and tested from a client linux machine

rsync -r rsync://host.ip.on.lan/MODULE_NAME /local/folder

First and foremost: Testing the daemon is in my view a lot easier from a linux command line than from Syncopoli because of the feedback, reusing commands, faster transfer speeds etc. Only once I’m sure that the server works as I intend, do I start mucking about with Syncopoli.

Secondly, MODULE_NAME in the command obviously refer to the MODULE_NAME in the configuration above, and not the path in the module. Rsync-over-SSH uses absolute paths, rsync-over-rsync uses module names (although you can specify paths under the module using relative path notation).

In Syncopoli (v. 0.5.3) I set my global settings as follows:

  • Protocol: Rsync
  • Server address: The server’s address on my local network
  • Port: 873 (the default for the rsync protocol)
  • User: Anything (I haven’t yet restricted the users allowed to sync)
  • rsync options: the defaults (-vHrlt), i.e.
    • -v for verbose
    • -H for preserve hardlinks (not that I use them)
    • -r for recursive, i.e. include subfolders
    • -l for copying symlinks as symlinks
    • -t for preserve modification times
    • and at some later point probably --delete so that files that disappear from the server also disappear from the phone

Syncopoli

The I add a Syncopoli profile by clicking the big plus-sign on the main page/profile list. A profile in Syncopoli can match an Rsync module one-for-one – or it can target some subfolder or sync with special options. If I just want a copy of the entire module, I do something like

  • Remote to local (rather than Local to remote)
  • Profile name: Test (this does not refer to anything on the server side, it’s just for Syncopoli’s list so it can be whatever I want)
  • Origins: MODULE_NAME
  • Destination: /storage/emulated/0/RsyncFolder (in order to minimise the things that can go wrong I pick a folder on the phone’s storage rather than an external SD card due to permissions)
  • Additional rsync options: (left empty)

If I want a subfolder rather than the entire module, I would add the relative path (from the absolute path in the rsyncd module specification) to the Origins setting.

Starting the sync job is done by pressing the profile’s play button. Viewing the daemon’s response and progress of the sync job is done by just clicking on the profile name.

Debugging

When Syncopoli works, it just works. When it doesn’t, well, I may know about or just as likely, I won’t. Here’s a few tips to help figure out what’s wrong.

Verify connection

The Syncopoli settings have an option to “Verify connection”. It seems I’m not the only one to have been confused by this as the following note has been added to the apps README file:

Starting v0.5, you must press “Verify Connection” in settings and verify that your host’s fingerprint matches before you can sync.

https://gitlab.com/fengshaun/syncopoli/blob/master/README.md

The reference to fingerprints make me think that this about matching a private SSH key file to the host. The somewhat misleading error message “Failed to verify host” for a long time made me think that Syncopoli coul not see my rsync daemon. That is not the case. It just means that the non-existant SSH private key doesn’t match the SSH server running on the host IP.

Empty logs

After any operation check the output in Syncopoli and on the server side (the value of the log file setting, usually /var/log/rsync.log. If there are no (new) lines in the server log, there was no connection made. This will usually correspond to empty output on the client, just the date or lines like “Log file not found”. A solution of sorts: Kill Syncopoli, i.e. go to Android’s Settings | Apps | Syncopoli and choose Force Stop. Or just reboot to be on the safe side. In fact, Force Stop should be used after any changes are made server-side or client-side. Once I have gone over every line in my config and found no good reason for the error, it has inevitably been a matter of killing the app and starting over.

Name lookups

The server side logs are filled with lines such as these:

2020/01/12 18:38:26 [27080] name lookup failed for 192.168.1.40: Name or service not known
2020/01/12 18:38:26 [27080] connect from UNKNOWN (192.168.1.40)

This looks pretty bad but is not – necessarily – a show stopper. I believe a name lookup failure did at one point cause the daemon to refuse the connection but the only way to replicate the issue seems to be to use the module specific setting hosts allow to a host name that doesn’t match the client machine. E.g. using any mDNS-like name such as hostname.local results in lines like the following:

2020/01/12 19:29:29 [27842] name lookup failed for 192.168.1.21: Name or service not known
2020/01/12 19:29:29 [27842] connect from UNKNOWN (192.168.1.21)
2020/01/12 19:29:29 [27842] rsync denied on module MODULE_NAME from UNKNOWN (192.168.1.21)

To skip the reverse DNS lookups (PTR) check – only really useful is one is sync’ing named server to named server – I add the following to my global settings:

reverse lookup = no

The result of this is that the PTR attempt is skipped and the connecting machine is referred to as UNDETERMINED rather than UNKNOWN. hosts allow should default to allow any and all machine, regardless of name or IP address, so I don’t know how I would have got denials.

Success

A successful connection (and transfer if any changes were discovered can easily be gleaned from the server logs by the presence of the line

2020/01/12 18:09:00 [27586] building file list

Authorization

Syncopoli supports authentication using the rsync protocol. To tell the truth I’m not sure if there’s much point, seeing as the transfer is not encrypted and so the password should be visible to anybody who is able to sniff the traffic, i.e. a network administrator on your local network and uhm a lot of people on the internets?

For what it’s worth the setup on the daemon side is adding a line to rsyncd.conf:

auth users = my_user
secrets file = /etc/rsyncd.secrets

my_user does not have to correspond to a system user. The user name is looked up in a table in the secrets file, which should have 600 permissions (rw-) and owned by root. The format of the file is as simple as can be

my_user:cleartext_password

Needless to say, the daemon needs to be restarted after any such additions. For Syncopoli this is a global setting, found in Settings (similar to how you can only have one server). Set User to my_user and Rsync Password to cleartext_password. Leave settings, Force Stop and start the app again. It may be worth checking the new setup from the linux command line first, like so:

rsync -r rsync://my_user@192.168.1.30/MODULE_NAME/ /local/folder/

To root or not to root

Rsyncd…

… must run with root privileges if you wish to use chroot, to bind to a port numbered under 1024 (as is the default 873), or to set file ownership.

rsyncd.conf(5)

Which is better for security: A chrooted process run by root or a non-chrooted process run by a non-root user with no other privileges apart from access to the relevant files? I’m guessing it depends on which is more likely to happen: Breaking out of chroot jail or a privileges escalation? All while taking control of the rsync daemon?

My decision to go with a non-privileged user rests on two simple foundations: 1) It’s a less complex setup that I understand (and therefore more secure than something I don’t really know what is) and 2) as a result it’s a lot easier to set up.

The rsync daemon does have some settings (“uid = 1234”) to allow for running file operations as a specific user, rather than root. I figured that would mean that the daemon drops privileges, say after reading the secrets file. Not so. The daemon stays running as root which means that this is more about access (to the files) than security. In hindsight I realize that this is obvious, given that the setting is module specific and not global. So instead I opted to run the entire thing as a dedicated user.

Here’s what I did to make it work. First, I created a custom systemd service file, using systemctl’s inbuilt feature:

 systemctl edit --full rsync.service

This will start me up with a copy of the /lib/systemd/system service file in /etc/system/system. --full means that I’m not writing an addendum to the main file, I’m writing a replacement (using the system one as a starting point). On the minus side, I’m missing out on improvements from updates. On the plus side, I’m a bit safer against surprises from said improvements. My only change is to add two lines to the [Service] section:

User=MyRsyncUser
Group=MyRsyncUserGroup

With this I can run the rsync daemon entirely as a non-privileged user. Which will definitely fail badly unless I alter the config file too, so here goes.

My rsync user needs to have read and write access to the pid, lock and log files. To avoid having to recreate them with each reboot or alter permissions on system directories, I created a var/run and var/log directory in the rsync user’s home directory and set the config file to use those instead.

pid file = /home/MyRsyncUser/var/run/rsyncd.pid
lock file = /home/MyRsyncUser/var/run/rsync.lock
log file = /home/MyRsyncUser/var/log/rsync.log

Next I define the port to use, as port 873 is now inaccessible:

port = 8873

Finally, I need to give the user read access to the /etc/rsyncd.secrets file. It should obviously still be owned by root but I set it’s group to be the rsync users unique group and give read access on the file to that group, i.e.

chmod 640 /etc/rsyncd.secrets

Depending on distribution an rsync daemon may default to try running chroot’ed. It will not check whether this makes sense (i.e. am I even root?) and a non-root user trying to chroot fails badly. So it’s best to make explicit that chroot’ing should not be attempted by adding the following line to each module in /etc/rsyncd.conf:

use chroot = false

Halfway conclusion

I know it’s not the ideal setup but seeing as Syncopoli is not that geared towards helping the user set up a sync job or inform them of what goes wrong when it goes wrong, I figured I wanted to take it one thing at a time.

Hopefully this is useful to others who want a simple, no-nonsense approach to file syncing on an increasingly nonsense platform. And equally hopefully, I will soon switch over to, and report on, using rsync and Syncopoli over SSH.

The [Syncopoli] launcher icon is designed by Armin Moradi and distributed under Creative Commons Attribution-ShareAlike 4.0 International Public License

https://gitlab.com/fengshaun/syncopoli/blob/master/LICENSE.icons

5 Comments

> And equally hopefully, I will soon switch over to, and report on, using rsync and Syncopoli over SSH.

SimpleSSHD allows you to ssh to your phone and basically just rsync from the comfort of your PC terminal. However, the setup is painful, albeit I believe it is less complicated compared to this whole ordeal. It is actively developed too, last update 2021/01/17.

> no-nonsense approach to file syncing on an increasingly nonsense platform

Could not put it better myself, I just want to one-way rsync photos out of my phone, automatically when I connect to a WiFi. This should not be such a painful task. I was able to ssh to my phone quite easily and rsync a test file within 10 minutes with SimpleSSHD, but I can’t really make this work automatically and continuously for the photos. Plus, you can really realizably SSH to your phone on home WiFi, since you cant exactly bind phones MAC address to certain IP on public WiFi.

BTW I landed here looking for someone who reliably made the rsync work. So far it is rocky. Cheers!

Glad to hear you got it working. I’m not overly fond of the idea of continuoulsy running an SSH daemon on my phone, partly for battery but I guess mostly because it feels like an open invitation? Do you know if they’re using Dropbear of OpenSSH? Does it use passwords or certificates?

Switching to a new phone and I wanted to add a few things.

First: If you’re seeing mkstemp error with some files and not others when syncing remote-to-local, it’s likely that your phone is using a FAT filesystem and so objects to certain filename characters. My former phone (it is no older than the current, so… not old?) seems to have been more tolerant, so I’m guessing Samsung uses a different filesystem to OnePlus. I don’t think there’s any other fix for it than to be more restrictive in the filenames used on the server.

Second: Local-to-remote works fine. Just use the localpath for “Origins” and the Rsync daemon module on your server for destination. It’s a nice and simple way to get a lot of data off of the phone. I would suggest moving –delete from a global rsync parameter to a sync job specific one and not using it for uploading. The just dump a folder into your phone’s upload folder and press run job. Also make sure your server rsync user has permission to write to the the server path, or – if running as root – enter the relevant uid/gid settings for the job.

Hi,

after a lot of trouble I did it! Syncopoli isn’t well documented (now) and not very handy (now). So it looks like a bad combination, BUT when it works it do fine, really fast and with a lot less implications than e. g. Syncthing. For other users here my troubles and solutions for Local β†’ Remote (e. g. rsyncing from mobile to your NAS):

1) I’m using the latest version v0.6 of Syncopoli.
2) Establishing rsync (NOT via ssh!) was thanks to this HowTo here pretty easy.
3) Tap on your established folder for syncing (not the play button) will showing you the debug screen.
4) But ssh droves me nuts. First of all it looks like Syncopoli accepts RSA Keys only, which you have to convert into the Dropbear format. (The tool dropbearconvert is part of the main package). You have to generate a key pair with openssh, convert the secret key and move them to your phone at any place. The public key (not converted) belongs to your remote ~/.ssh/authorized.keys. (But I’m not sure if this is really necessary.) After that you DON’T need dropbear at all, so you don’t have to establish a dropbear server instead you can keep your openssh. (Of course some maschines prefer dropbear due to the smaller size e. g. RasPi, OpenWRT). WITHOUT converting it wont work!
4a) I tried an Ed25519 cypher and it wont work, too.
4b) When generated the RSA key pair you have to use the ‘-m PEM’ as additional flag, which specifies the PEM-format. Otherwise dropbearconvert won’t converting your key. That way e. g. ssh-keygen -t rsa -b 4096 -o -a 100 -m PEM -f /tmp/android_rsa -N “”
4c) When your using a different key type with sshd on your server (e. g. Ed25519) you have to add/uncomment the follwoing line in your /etc/ssh/sshd_config: HostKey /etc/ssh/ssh_host_rsa_key
Otherwise you getting error like “no matching algo hostkey”.
4d) The converted and to your phone moved secret key will stay there and potentially readable by other Apps. There is no solution since four years.
4e) When you establish your connection the first time and you made everything right it may won’t work despite. Have a look in the debug screen it will/should show you something like: “The device with fingerprint xxx try to access your machine. Do you confirm this fingerprint? (y/n)” In other words you won’t ask interactivly for confirmation. You have to go to your Syncopoli settings, tap on “Private Key File” and there you can confirm (or not). BUT it will show the old md5 format. So you have to convert the fingerprint of your public SERVER key. You should find them here: /etc/ssh/ssh_host_rsa_key.pub To convert is easy with: ssh-keygen -l -E md5 -f /etc/ssh/ssh_host_rsa_key.pub
5) Now it should really work πŸ™‚
6) You can add “Additional rsync options” in each profile. This is useful for –exclude. E. g. when you’re rsyncing your WhatsApp folder under /storage/emulated/0/WhatsApp but wont rsync the stickers and voice notes your can exclude them with: –exclude ‘WhatsApp Sticker’ –exclude ‘WhatsApp Voice Notes’
6a) –exclude={‘WhatsApp Sticker’,’WhatsApp Voice Notes’} wont work here.

Enjoy!

Resources:
https://stackoverflow.com/questions/49392607/backup-android-to-nas-with-rsync
https://stribika.github.io/2015/01/04/secure-secure-shell.html
https://linux-audit.com/using-ed25519-openssh-keys-instead-of-dsa-rsa-ecdsa/
https://gitlab.com/fengshaun/syncopoli/-/issues/40
https://www.unixtutorial.org/how-to-inspect-ssh-key-fingerprints/
https://linuxize.com/post/how-to-exclude-files-and-directories-with-rsync/

Hi Knut, I am really glad to hear that the post got you the first part of the way. And thank you so much for sharing the second part! That was super-detailed and really helpful. And I guess there was a reason I never moved on to it, because, man alive, that sounds like a pain. I will try to see if I can replicate it and if I can, I’m absolutely stealing your solution for a new post πŸ™‚

Leave a Reply to brokkr Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.