How can I automate an SFTP transfer between two servers?

SSH Configuration

To begin with, most of this relies on a the configuration of sshd, the SSH (Secure SHell) daemon running on the server you're attempting to connect to (we'll call it "server2.com"). Check the "sshd_config" on that server, typically in /etc/ssh. In some cases, these settings are not always present or set the way we need:

RSAAuthentication yes
PubkeyAuthentication yes

This enables the public/private key authentication mechanism we're about to use.

Public/Private Key Generation

We'll generate the keypair on the Linux box that you want to connect from. We'll call that "server1.com". It's that box on which you plan to run ssh, sftp or rsync.

ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/user1/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
c1:21:e 3:01:26:0d:f7:ec:52:0e:0c:90:9b:6e:d8:47 user1@server1.com

What I've done with the command above is generated a public/private key pair. I responded to each prompt by hitting Return.

Note that I did NOT enter a passphrase. That's kind of important, because if you do enter a passphrase you'll need to enter it in order to use the private key. Since we're looking for an automated solution, the private key must not have a passphrase.

This is important: by not placing a passphrase on your private key, the security implication is that mere possession of the private key is sufficient to gain access to what ever resources into which you've placed the corresponding public key. Safeguard your private key.

My private key was placed in /home/user1/.ssh/id_rsa. This needs to be kept secure, because of the security implication above, but also needs to be available to the process attempting to make an ssh, sftp or rsync connection. If these tools are run under the 'user1' account, the tools will automatically look in the ".ssh" directory and I won't need to specify the private key location. Otherwise, command line options will need to point to the right place and key.

My public key is in /home/user1/.ssh/id_rsa.pub. This is the key that gets distributed to those places that want to grant you access.

Planting the public key

On the "remote" server, server2.com, pick an account - ANY account - that you want to connect as. In that account's home directory, create a ".ssh" subdirectory, and in that directory create a new text file called "authorized_keys". If it already exists, that's fine, use the existing file.

If you create the file and/or directory, I recommend that the directory be chmod 700, and the file 600. In other words, only the owner can access the directory, and the file within it.

Add to that file the contents of the id_rsa.pub file created above. That would be a *single line* that looks something like this:

ssh-rsa <lots of characters> user1@server1.com

Once saved anyone in possession of the private key that matches this public key can now login as this account.

sftp

I planted the public key in the account user2 on server2.com. So now, on my server, server1.com, logged in as user1, and where the private key is stored as described above, an sftp session looks like this:

sftp user2@server2.com

"user2" specifies the remote account on server2.com to login as.

That's it. Magic happens, and I'm authenticated. That magic? The private key is matched to the public key, which indicates you are authorized to login to that account. An sftp session is born. No interactivity required.

(IF you did enter a passphrase on the private key, you would have been prompted to enter it here. NOTE that this is the passphrase to unlock the private key, which is local. It has nothing to do with any passwords on the remote site.)

rsync

For file copy operations, rsync rocks. It does things like intelligent compression, copy only if needed, and a whole host of other operations.

So, assuming all the keys are set up as above, this rsync command copies a file from the local machine to the remote:

rsync -e ssh file user2@server2.com:/home/user2/

Local file "file" is copied to the remote /home/user2/file after logging in as "user2" using ssh as the transport (hence the "-e ssh" option), and with that, using the private/public key pair we created for authentication. Again, no interactivity required.

Rsync supports an incredibly rich set of options for recursion, compression attribute retention, date/time stamp and so on. Well worth a look see if you're copying anything of any significant volume.

SSH

Since we've gone this far, it's worth noting that SSH itself just works as well to open up a remote shell once the keys are in place. Example:

ssh user2@server2.com

and *poof* - a remote shell on server2, logged in as user2.