Ask Leo!

How can I automate an SFTP transfer between two servers?

Home » General Computing

I'm trying to set up an automated sftp transfer from one Linux box to another. I understand that you have to create a key with ssh-keygen, then put the key file on the other machine. But sftp still prompts me for the password. I read that the users on both machines must be the same... is that correct?

No, not correct.

As it turns out, this is something I do regularly with ssh, as well as both sftp and rsync, as part of my backup and load balancing approaches for Ask Leo! Let me walk you through what I've done.

SSH Configuration

To begin with, most of this relies on a the configuration of sshd, the SSH (Secure SHell) daemon running on the server you're attempting to connect to (we'll call it "server2.com"). Check the "sshd_config" on that server, typically in /etc/ssh. In some cases, these settings are not always present or set the way we need:

RSAAuthentication yes
PubkeyAuthentication yes

This enables the public/private key authentication mechanism we're about to use.

Public/Private Key Generation

We'll generate the keypair on the Linux box that you want to connect from. We'll call that "server1.com". It's that box on which you plan to run ssh, sftp or rsync.

ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/user1/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
c1:21:e 3:01:26:0d:f7:ec:52:0e:0c:90:9b:6e:d8:47 user1@server1.com

What I've done with the command above is generated a public/private key pair. I responded to each prompt by hitting Return.

"...mere possession of the private key is sufficient to gain access to what ever resources into which you've placed the corresponding public key."

Note that I did NOT enter a passphrase. That's kind of important, because if you do enter a passphrase you'll need to enter it in order to use the private key. Since we're looking for an automated solution, the private key must not have a passphrase.

This is important: by not placing a passphrase on your private key, the security implication is that mere possession of the private key is sufficient to gain access to what ever resources into which you've placed the corresponding public key. Safeguard your private key.

My private key was placed in /home/user1/.ssh/id_rsa. This needs to be kept secure, because of the security implication above, but also needs to be available to the process attempting to make an ssh, sftp or rsync connection. If these tools are run under the 'user1' account, the tools will automatically look in the ".ssh" directory and I won't need to specify the private key location. Otherwise, command line options will need to point to the right place and key.

My public key is in /home/user1/.ssh/id_rsa.pub. This is the key that gets distributed to those places that want to grant you access.

Planting the public key

On the "remote" server, server2.com, pick an account - ANY account - that you want to connect as. In that account's home directory, create a ".ssh" subdirectory, and in that directory create a new text file called "authorized_keys". If it already exists, that's fine, use the existing file.

If you create the file and/or directory, I recommend that the directory be chmod 700, and the file 600. In other words, only the owner can access the directory, and the file within it.

Add to that file the contents of the id_rsa.pub file created above. That would be a *single line* that looks something like this:

ssh-rsa <lots of characters> user1@server1.com

Once saved anyone in possession of the private key that matches this public key can now login as this account.

sftp

I planted the public key in the account user2 on server2.com. So now, on my server, server1.com, logged in as user1, and where the private key is stored as described above, an sftp session looks like this:

sftp user2@server2.com

"user2" specifies the remote account on server2.com to login as.

That's it. Magic happens, and I'm authenticated. That magic? The private key is matched to the public key, which indicates you are authorized to login to that account. An sftp session is born. No interactivity required.

(IF you did enter a passphrase on the private key, you would have been prompted to enter it here. NOTE that this is the passphrase to unlock the private key, which is local. It has nothing to do with any passwords on the remote site.)

rsync

For file copy operations, rsync rocks. It does things like intelligent compression, copy only if needed, and a whole host of other operations.

So, assuming all the keys are set up as above, this rsync command copies a file from the local machine to the remote:

rsync -e ssh file user2@server2.com:/home/user2/

Local file "file" is copied to the remote /home/user2/file after logging in as "user2" using ssh as the transport (hence the "-e ssh" option), and with that, using the private/public key pair we created for authentication. Again, no interactivity required.

Rsync supports an incredibly rich set of options for recursion, compression attribute retention, date/time stamp and so on. Well worth a look see if you're copying anything of any significant volume.

SSH

Since we've gone this far, it's worth noting that SSH itself just works as well to open up a remote shell once the keys are in place. Example:

ssh user2@server2.com

and *poof* - a remote shell on server2, logged in as user2.

Related:

Article 10287 | Posted May 13, 2006

Recent Comments
35 Comments

Hi Leo,

I generated rsa keys for system a and b and put both keys in authorized_keys for my user on system c.

system a connects using sftp and does not prompt for a password.

system b prompts for a password - when I provide it the connection is made.

on all 3 systems all files in .ssh are writable only by the user.

what should I be looking at to trouble shoot this?

Posted by: mike huber at September 18, 2007 8:07 AM

Hi Leo, Thanks for the tip, great info!

Posted by: George Schweizer at October 2, 2007 2:41 PM

Leo, excellent article!! But it has worked for me only when i try to ssh/sftp to root account on server machine (server2 in the ex.). Any other user (making all the steps for each one) fails.
Is there any trick related to that ??
Thanks in advance.

Posted by: Leonardo at November 13, 2007 10:30 AM

I am having same problem as "Manoj Das".

here is part of the post and your comment on it:

script:
sftp ncc_b2b@blrsun27
get file1.txt
exit

I have followed the steps given by you for automated connection. Once I am executing this, it's automatically connecting to the remote system and I am getting the SFTP prompt. But it's not executing the get command.

Can you please guide me how to do this now, as the -b option is not available.


Posted by: Manoj Das at October 3, 2006 07:36 AM
YOu might need to have

get file1.txt
exit

in one file (commands.txt) and then use that as input for the sftp command:

sftp

*********************

Here is my commannds.txt

commands.txt
pwd
quit

my script:

my_test_1
#!/bin/ksh
#set -x

sftp user1@pontoon

exit_status=$?

return $exit_status

For the same setup I can do scp

anju_test
#!/bin/ksh

scp user1@pontoon:$1 $2

exit_status=$?

return $exit_status

But we need to list file before copying them so I have to make sftp work for us.

thanks a lot for your help Leo.

Posted by: Anju at December 5, 2007 8:48 AM

This is followup to my earlier post. I made it work sort of:

#!/bin/ksh

sftp -B commands.txt user1@pontoon

in commands.txt I have
get 1.dat 2.dat ( I want to get 1.dat and rename it to 2.dat)
quit

It gets 1.dat and complains can not find 2.dat on remote server.

Here is some other info about my system:

sftp -V
sftp: F-Secure SSH 3.1.0 (build 12) on sparc-sun-solaris2.8

thanks for reading my posts.

Posted by: Anju at December 5, 2007 11:09 AM

Can this be applied to Windows 2003 to Solaris?
That is automated connection from Windows 2003 to Solaris?

Thanks

Posted by: Francis at February 24, 2008 12:58 PM

Hi Leo, I tried to follow the steps you posted here but I still cannot make it work. I am trying to use SFTP to send files from HP-UX to Windows Server 2000 with a domain account setup on the Window Server. No matter what I did, I always get promoted with password. Is the public key authentication not going to work for me since this is an AD account? What's the best way I can troubleshoot this issue? Thanks.

Posted by: Yvonne at March 12, 2008 3:46 PM

This article helps me greate.
I have one problem.
I can sftp without password from my account to abc@srvr1, but it asks fro password while sftp from my account to xyz@srvr3 even though I have copied the same public file in .ssh directory on both these severs.These both servers have same sshd_conf files.

Posted by: Vasant at April 3, 2008 5:20 AM

Very nice document. First I want to thanks U.
1. Is there any way to write script which copy files from remote server.

Posted by: Gurdeep Singh at July 7, 2008 5:14 AM

Hi,

I'm currently using the -b Batch mode reads a series of commands from an input batchfile
eg: -b batchfile user@host

Right now I have to add in the switch -C for the compresion. How I can do that with the using the above code as well with the batch file contain script like - Put command to upload the file from local to remote system.

Posted by: sftp compression at September 24, 2008 4:01 AM

Post a comment on "How can I automate an SFTP transfer between two servers?":






(Email Address will not be published.)

Remember Me?

By popular demand...
my tip jar
Cuppa Joe
Buy Leo a Latte!

(you may use HTML tags for style)

New!

RSS feed Subscribe to the RSS Feed specifically for comments on this article.

Before commenting, please...

Please wait. Your comment is being processed ...


Ask Your Question:


ask-leo.com
Web

Stay Informed

Weekly Newsletter

Archives

By Category
By Date

Advertisers

Advertise on Ask Leo!

««   »»

Question? - Ask Leo!
Who is Leo?
Link to Leo!

Terms, Conditions & Privacy