Helping people with computers... one answer at a time.

I'm trying to set up an automated sftp transfer from one Linux box to another. I understand that you have to create a key with ssh-keygen, then put the key file on the other machine. But sftp still prompts me for the password. I read that the users on both machines must be the same... is that correct?

No, not correct.

As it turns out, this is something I do regularly with ssh, as well as both sftp and rsync, as part of my backup and load balancing approaches for Ask Leo! Let me walk you through what I've done.

SSH Configuration

To begin with, most of this relies on a the configuration of sshd, the SSH (Secure SHell) daemon running on the server you're attempting to connect to (we'll call it "server2.com"). Check the "sshd_config" on that server, typically in /etc/ssh. In some cases, these settings are not always present or set the way we need:

RSAAuthentication yes
PubkeyAuthentication yes

This enables the public/private key authentication mechanism we're about to use.

Public/Private Key Generation

We'll generate the keypair on the Linux box that you want to connect from. We'll call that "server1.com". It's that box on which you plan to run ssh, sftp or rsync.

ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/home/user1/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
c1:21:e 3:01:26:0d:f7:ec:52:0e:0c:90:9b:6e:d8:47 user1@server1.com

What I've done with the command above is generated a public/private key pair. I responded to each prompt by hitting Return.

"...mere possession of the private key is sufficient to gain access to what ever resources into which you've placed the corresponding public key."

Note that I did NOT enter a passphrase. That's kind of important, because if you do enter a passphrase you'll need to enter it in order to use the private key. Since we're looking for an automated solution, the private key must not have a passphrase.

This is important: by not placing a passphrase on your private key, the security implication is that mere possession of the private key is sufficient to gain access to what ever resources into which you've placed the corresponding public key. Safeguard your private key.

My private key was placed in /home/user1/.ssh/id_rsa. This needs to be kept secure, because of the security implication above, but also needs to be available to the process attempting to make an ssh, sftp or rsync connection. If these tools are run under the 'user1' account, the tools will automatically look in the ".ssh" directory and I won't need to specify the private key location. Otherwise, command line options will need to point to the right place and key.

My public key is in /home/user1/.ssh/id_rsa.pub. This is the key that gets distributed to those places that want to grant you access.

Planting the public key

On the "remote" server, server2.com, pick an account - ANY account - that you want to connect as. In that account's home directory, create a ".ssh" subdirectory, and in that directory create a new text file called "authorized_keys". If it already exists, that's fine, use the existing file.

If you create the file and/or directory, I recommend that the directory be chmod 700, and the file 600. In other words, only the owner can access the directory, and the file within it.

Add to that file the contents of the id_rsa.pub file created above. That would be a *single line* that looks something like this:

ssh-rsa <lots of characters> user1@server1.com

Once saved anyone in possession of the private key that matches this public key can now login as this account.

sftp

I planted the public key in the account user2 on server2.com. So now, on my server, server1.com, logged in as user1, and where the private key is stored as described above, an sftp session looks like this:

sftp user2@server2.com

"user2" specifies the remote account on server2.com to login as.

That's it. Magic happens, and I'm authenticated. That magic? The private key is matched to the public key, which indicates you are authorized to login to that account. An sftp session is born. No interactivity required.

(IF you did enter a passphrase on the private key, you would have been prompted to enter it here. NOTE that this is the passphrase to unlock the private key, which is local. It has nothing to do with any passwords on the remote site.)

rsync

For file copy operations, rsync rocks. It does things like intelligent compression, copy only if needed, and a whole host of other operations.

So, assuming all the keys are set up as above, this rsync command copies a file from the local machine to the remote:

rsync -e ssh file user2@server2.com:/home/user2/

Local file "file" is copied to the remote /home/user2/file after logging in as "user2" using ssh as the transport (hence the "-e ssh" option), and with that, using the private/public key pair we created for authentication. Again, no interactivity required.

Rsync supports an incredibly rich set of options for recursion, compression attribute retention, date/time stamp and so on. Well worth a look see if you're copying anything of any significant volume.

SSH

Since we've gone this far, it's worth noting that SSH itself just works as well to open up a remote shell once the keys are in place. Example:

ssh user2@server2.com

and *poof* - a remote shell on server2, logged in as user2.

Article C2653 - May 13, 2006 « »

Share this article with your friends:

Share this article on Facebook Tweet this article Email a link to this article
Leo Leo A. Notenboom has been playing with computers since he was required to take a programming class in 1976. An 18 year career as a programmer at Microsoft soon followed. After "retiring" in 2001, Leo started Ask Leo! in 2003 as a place for answers to common computer and technical questions. More about Leo.

Not what you needed?

65 Comments
An
July 4, 2006 12:45 PM

Hi!! I'm doing all the steps, but the sftp always asks me for the password. I don't know what is happning, do you?

Please, help!

An

Joe
July 19, 2006 12:58 AM

your public/private keys are probably not compatible with the ssh server. you are probably using either OpenSSH or Putty and your server is probably using an ssh.com based package. If this is the case drop putty and use openssh, create the public and private keys then run ssh-keygen -e -f key.pub > key2.pub (where key is the name of your key) this creates a new copy of the public key which can be used on an ssh.com server. Rename the key2.pub file to key.pub (which probably means you will have to delete existing key.pub) then use the private key (which has not changed) with the new public key (ie upload the new public key). Also don't forget to change the authorisation file on the server (/home/username/.ssh or .ssh2) to accept your new key.

Manoj Das
September 23, 2006 4:12 AM

This article is incredible Leo. Thank You.

I have seen the sftp usage options given below,
sftp [-1vC] [-A Password] [-b batchfile] [-L logfilename] [-osshopt=value] [user@]host[:file [file]]

But in my SunOS 5.9 , I am not getting the options -L, -b, -A. What could be the reason. I wanted to get a file transferred from the remote server to my server after I connect through my automated connection script. My automation script connects and gives me the SFTP prompt. It's not identifying the commends given in the batch file.

Thanking you once again for all the help.

Leo Notenboom
September 23, 2006 8:46 AM

There are slightly different versions of sftp on different OS's. Does "man" work on your system? If so I'd try a "man sftp" and see if the documentation includes the features you need.

Manoj Das
October 3, 2006 7:36 AM

Thank you Leo.

I have checked the man pages for sftp. As you have mentioned -L, -b, -A keys are not available.

I just wanted to connect automatically to a remote system and fetch a file from that machine.
Please see the script I am having.

sftp ncc_b2b@blrsun27
get file1.txt
exit

I have follwed the steps given by you for automated connection. Once I am executing this, it's automatically connecting to the remote system and I am getting the SFTP prompt. But it's not executing the get command.

Can you please guide me how to do this now, as the -b option is not available. Should I go for SSLFTP or any other alternative to this.

Thanking You Leo.

Manoj.

Leo Notenboom
October 4, 2006 3:39 PM

YOu might need to have

get file1.txt
exit

in one file (commands.txt) and then use that as input for the sftp command:

sftp < commands.txt

Manoj Das
October 4, 2006 11:47 PM

Thank You Leo.

It's working fine now. I have managed to Automatically connect to the remote machine and transfer the required file. All because of your help.

Once Again Thanking You Leo,
Manoj.

Rajeswari
October 6, 2006 3:27 AM

Hi,
I have tried all the steps but still the automation is not working with one machine.Kindly help me out in resolving this issue

Thanks,
P.Rajeswari

Manoj Das
October 8, 2006 9:57 PM

Rajeswari,

I have followed the steps given by Leo and it's working fine. It's not clear from your message what's the actual problem. Can you please clarify, where you are facing the problem.

Bashir
October 12, 2006 4:17 PM

I have this normal ftp script that works fine, ftping files from the unix box to a windows box.
#!/usr/bin/ksh
ftp -v -n hostname ftp-gw
Type:xxx@xxx.xxx.com
Type in password when prompted
Ftp>quit
Put C:\temp\filename.txt

Can you help me with this sftp script.

Amit Gupta
October 18, 2006 1:30 AM

Hi Leo,
I am also following all the steps but SFTP is still asking me the password. I am having SunOs and Global Scape OS and I am trying to do automation from SunOS to GlobalScape. Is there any compatible problem. Please guide

Mike McGinn
November 17, 2006 7:35 AM

I'm trying to write a script to connect to a server via sftp that has publickey+password authentication. Even when I put the password in a file and use the -b option, it still prompts me for a password. Is there any other way to pass the password to the remote server?

Ian McDermid
December 27, 2006 3:34 AM

Well written article. More of this needed on the net. My batch job now works!!!

kumar
January 3, 2007 2:47 AM

Hi Leo,
Nice article. I have one question though.
If I want to put a file from server1 to server2 - automatically - How do I do that?

If I do sftp user2@server2.com then it goes into interactive mode of sftp. But my problem is I want to put a file from server1 to server2 - using sftp command not from interactive prompt.

any clues on that please?

Regards
Kumar

Leo Notenboom
January 3, 2007 9:12 AM

Check out the gazillion command line options for sftp. You can bypass the interactive prompt by specifying everything on the command line.

Minesh Shah
March 2, 2007 4:24 AM

Hi, We need to create a kshell script to FTP files to the destination server. We have to use sftp to do so.

My concern is, how do I avoid getting password prompt when using sftp command. Please help, it is very urgent.

Thanks in advance.

Regards, Minesh Shah

James
March 29, 2007 5:26 PM

Hi Leo,
Your instructions to automate my sftp transfers worked great! I've only run into one problem. When I try to change user1 on server1 to point to user3 on server2, it starts prompting for a password again. i.e. sftp user2@server2.com works fine but sftp user3@server2.com doesn't work automatically. I copied the public key from the .ssh folder on user2 over to the .ssh folder for user3 but that doesn't seem to help. What am I missing here?

Regards,
James

Leo Notenboom
March 29, 2007 7:16 PM

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Nine times out of ten, the permissions on the copy of the .ssh files and the
authorized_keys file that you copied are wrong. It needs to be owned by the
account who's .ssh directory it resides in.

Leo
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)

iD8DBQFGDHLeCMEe9B/8oqERAureAJ9CmSJMmSetKZZV8UaFGH2JVXRBswCbBd4I
xuTKYBGphXOrpANg7P3CqMY=
=izE9
-----END PGP SIGNATURE-----

Rajesh Acharya
April 2, 2007 5:39 AM

We have the lprng daemon running. When we put a job on a queue, it calls a shell filter script that is suppose to sftp the file to the remote server.

The problem is, the daemon tries to connect to the remote host even though we specify the user@host2 (user is qadmin and has the keys set up both sides).

ie: scp -b qadmin@host2

sai
April 4, 2007 6:31 AM

Hi Leo,great advice. One additional question, if the account doesn't have a standard login shell will this automat process work?

Leo A. Notenboom
April 4, 2007 2:13 PM

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I don't believe so. I vaguely recall having to enable login ability for
accounts with which I want to do this.

Leo
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)

iD8DBQFGFBU3CMEe9B/8oqERAtkwAJ4t3CuIMp7Ld/5D+2CElHl7TeP+qQCfXJX4
8P8DihofejdoYIYiwuWF1rQ=
=7nkv
-----END PGP SIGNATURE-----

Cowboy
April 5, 2007 7:43 AM

This was a superb article. I am going to try getting it incorporated into our IT procedures. Thank you very much.

macmd
June 15, 2007 4:59 AM

If you want to automate sftp because for some reason scp doesn't work in you situation you can try "expect" You need to install it. The script would look something like this:
Replace:
DIR=local and remote directories.
IP-adress=remote server address
user=user set up on remote sftp server.


#!/usr/local/bin/expect -f -d
# Expect script that automates sftp
###################################

spawn /usr/bin/sftp user@"IP-adress"
expect "user@IP-adress's password: "
sleep 7
send Password\r"
expect "sftp> "
send "lcd DIR\r"
expect "sftp> "
send "cd DIR\r"
expect "sftp> "
send "cd DIR\r"
expect "sftp> "
sleep 10
send "mput *.pgp\r"
sleep 10
expect "sftp> "
send "bye\r"

Andy
July 13, 2007 5:23 AM

Leo, I put your steps to use and they work. Mostly.
Here's my situation:
3 servers, I'll call A, b and c.
A: RHEL4
b: Solaris 8 on Sparc
c: Solaris 8 on Sparc.

If I send from A to b or c, the automation works fine. If I send from b or c to A, I am challenged with a password request. b and c have A's public RSA key. A has both b and c's RSA public keys. In every case, I am the user who is also attempting the SCP connection. It is most expedient to auto send from b to A, but I can rewrite b's script to run on A, if I must. Just wondering why its only working in one direction.

Andy
July 13, 2007 6:01 AM

Leo, it turns out that on server A, the authorized_keys file had permission 664. Apparently if that file is writable by anyone other than the owner, you will be challenged for a password, regardless. Once I CHMODed it to 644, the process worked flawlessly.

mike huber
September 18, 2007 8:07 AM

Hi Leo,

I generated rsa keys for system a and b and put both keys in authorized_keys for my user on system c.

system a connects using sftp and does not prompt for a password.

system b prompts for a password - when I provide it the connection is made.

on all 3 systems all files in .ssh are writable only by the user.

what should I be looking at to trouble shoot this?

George Schweizer
October 2, 2007 2:41 PM

Hi Leo, Thanks for the tip, great info!

Leonardo
November 13, 2007 10:30 AM

Leo, excellent article!! But it has worked for me only when i try to ssh/sftp to root account on server machine (server2 in the ex.). Any other user (making all the steps for each one) fails.
Is there any trick related to that ??
Thanks in advance.

Anju
December 5, 2007 8:48 AM

I am having same problem as "Manoj Das".

here is part of the post and your comment on it:

script:
sftp ncc_b2b@blrsun27
get file1.txt
exit

I have followed the steps given by you for automated connection. Once I am executing this, it's automatically connecting to the remote system and I am getting the SFTP prompt. But it's not executing the get command.

Can you please guide me how to do this now, as the -b option is not available.


Posted by: Manoj Das at October 3, 2006 07:36 AM
YOu might need to have

get file1.txt
exit

in one file (commands.txt) and then use that as input for the sftp command:

sftp

Anju
December 5, 2007 11:09 AM

This is followup to my earlier post. I made it work sort of:

#!/bin/ksh

sftp -B commands.txt user1@pontoon

in commands.txt I have
get 1.dat 2.dat ( I want to get 1.dat and rename it to 2.dat)
quit

It gets 1.dat and complains can not find 2.dat on remote server.

Here is some other info about my system:

sftp -V
sftp: F-Secure SSH 3.1.0 (build 12) on sparc-sun-solaris2.8

thanks for reading my posts.

Francis
February 24, 2008 12:58 PM

Can this be applied to Windows 2003 to Solaris?
That is automated connection from Windows 2003 to Solaris?

Thanks

Yvonne
March 12, 2008 3:46 PM

Hi Leo, I tried to follow the steps you posted here but I still cannot make it work. I am trying to use SFTP to send files from HP-UX to Windows Server 2000 with a domain account setup on the Window Server. No matter what I did, I always get promoted with password. Is the public key authentication not going to work for me since this is an AD account? What's the best way I can troubleshoot this issue? Thanks.

Vasant
April 3, 2008 5:20 AM

This article helps me greate.
I have one problem.
I can sftp without password from my account to abc@srvr1, but it asks fro password while sftp from my account to xyz@srvr3 even though I have copied the same public file in .ssh directory on both these severs.These both servers have same sshd_conf files.

Gurdeep Singh
July 7, 2008 5:14 AM

Very nice document. First I want to thanks U.
1. Is there any way to write script which copy files from remote server.

sftp compression
September 24, 2008 4:01 AM

Hi,

I'm currently using the -b Batch mode reads a series of commands from an input batchfile
eg: -b batchfile user@host

Right now I have to add in the switch -C for the compresion. How I can do that with the using the above code as well with the batch file contain script like - Put command to upload the file from local to remote system.

Venkat
November 6, 2008 10:00 PM

Hi Leo, I wanted to know abouty sftp, and stumbled at your site, you are a great teacher.
Accept my humble respects.
Regards,
Venkat

Bill
November 7, 2008 6:43 PM

Hey Leo,

When I pasted the address you used before into my terminal (sftp user2@server2.com) to modify it, the CR somehow got into my copy buffer, so when I pasted it I apparently ftped into that actual address. It then locked up my system for awhile and I was unable to exit.

Do you happen to have any idea what server2.com actually is, and whether having ftped into it could have somehow compromised my system?

Thanks, Bill

Dickster
February 7, 2009 9:56 PM

I didn't see a response to Vasant's post of 4/3/08, so thought this might be useful. Each "from" account must create a key-pair and append the public key to the .ssh/authorized_keys on the "to" account side. You can't use one public key generated by one account to cover two source "from" accounts. But one "to" account can have multiple public keys in its .ssh/authorized_keys file to allow multiple "from" accounts to access that one "to" account.

Saurabh
February 21, 2009 4:16 AM

Hi Leo
I have a problem here. I have planted the keys in the remote server but when I use the scp command in a script, it does not work.
The login to the remote server does not create a problem but the scp itself does, in a sense that the file is not getting copied to the remote server. Do I have a hope or will have to look for alternatives.

Thanks in advance
Saurabh

Terry Porter
April 3, 2009 11:26 AM

Hi Leo,
Great article. I thought I would not be able to use this, as in the secure government environment I work in using Red Hat Enterprise Linux 5.1, the sshd_config file has both RSAAuthentication and PubkeyAuthentication commented out with #'s.

However, as I really needed this functionality from User A on Host A to User oracle on Host DB, I gave it a try and it's working like a charm. Not sure "why" but I'm glad it's working.

Many thanks for a great, useful, well-written article.

Srinivasan
April 7, 2009 6:06 AM

Hi Leo,
I have a simillar requirement.
I would like test the functionality of sftp connectivity with keys from User A on Host A to User oracle on Host B.

Pls confirm whether
We can connect from HOST A USER A to HOST B Oracle
Using passworldess authentication
when we copy the public keys of USER A to oracle's home directory on Host B.

Please help.

veeru
May 20, 2009 3:00 AM

Hi, HOw can we connect using user id and password without changing the keys in sftp server.

Kenny
January 24, 2010 10:04 PM

I am trying to automate scripts using pageant for winscp. but i need to manually input the passphrase. as the file will be retrieved over the night and server reboot is done daily, is there a wasy to hardcode the passphrase in the script?

sftpuser
March 2, 2010 11:37 AM

Thanks, Leo. Just ran what you wrote in the article and works perfect. Your suggestions run smooth as usual.

Tracy
March 2, 2010 12:48 PM

I setup 2 linux servers with the information provided and it works like a charm.

However I'm also trying to setup a Windows server using EFT server by Globalscape. Where would the authorized_keys file be created on the Windows box to allow access?

Rohit
May 10, 2010 12:19 AM

Leo, your article works for me too, beautifully written

vams
July 12, 2010 6:13 AM

I followed the process but still password is prompted.

from teh below text
"If these tools are run under the 'user1' account, the tools will automatically look in the ".ssh" directory and I won't need to specify the private key location. Otherwise, command line options will need to point to the right place and key."

could you please let me know the "command line options will need to point to the right place and key." so that password is not prompted.

Adenwala
July 28, 2010 11:23 AM

How can I make this work with an ftp server where there is no unix login available? I can ftp to that server and I have login there, but want to use sftp to automate my file transfers. Thanks!

Prashant Singh
August 4, 2010 9:46 AM

Dear Friend ,
I have followed the process given above .I have 3 servers . On one server the process worked fantastically and SSH is authenticated without password , but on one server it is again asking for password , not the passphrase , the actual password for login on server but on the other server the process given on the sit is working fantastically . what to do on the server which is not responding ?

Best Regards
Prashant

shivkumar
October 14, 2010 3:12 AM

how can i put time format so that only the latest file gets fetched(csv) when i make sftp to server
thanks
shiv

Bob Luebbe
November 1, 2010 3:10 PM

You may also want to check out GoAnywhere from Linoma Software. They let you automate SFTP transfers with their built-in scheduler. Check it out at www.GoAnywhereMFT.com

Priyanka
November 30, 2010 10:28 PM

The article is well described and very easy to follow.
I have followed the steps mentioned above but i still get a password prompt while making a sftp connection. I believe that the problem is because the files are generated under user linus and the .ssh is under root as mentioned by you ""If these tools are run under the 'user1' account, the tools will automatically look in the ".ssh" directory and I won't need to specify the private key location. Otherwise, command line options will need to point to the right place and key."

Could you please help me as to how to make the command line options point to the right place and key.

Marcus
March 26, 2011 9:47 AM

Thanks Leo. Worked perfectly.

kaladhar
May 6, 2011 1:54 PM

I have a query regarding the keys.I generated the public and private keys with the userid X on server 1 and asked server2 team to add the publickey to the user id Z on server 2.I tried to SFTP using the private key with the user Y(which have access to the private key) from Server1 to Server2 with the user id Z.Iam not able to login ..Its asking for password by saying that permissions are too open.When i try to SFTP using the User X(keys generated on this id) from server 1 to Server 2.its logging in without asking for password.can u explain what the issue is..?

Siva
July 15, 2011 4:48 AM

Hello Leo,

The document that you given is very superb. Its very simple and useful to follow. Thank for the document.

I followed the same with out giving the passphrase while generating the key.Its working. But my requirement is to pass the passphrase while generating keys. When I pass it, the automated process is not working. Its asking me to pass the passphrase interactively.

Could you please help me in this.

Thanks for ur help in advance.

Siva

Ken
September 1, 2011 7:25 AM

Sorry rushing too fast - should have read before posting....

Document is fantastic but I transfer files across multiple boxes how can I set up this process so that ALL boxes are able to transfer files without password notification. Process works well with box 1 & 2 but when I put 3 in, I lose one.
Box 1 talks to box 2. I then set up box 1 then talks to box 3 but I lose interactive connection with box 2 ?!?!
Can you please suggest something (I have 6 boxes that interact with each other).


Murali K
September 2, 2011 4:22 AM

Hi,
Thanks for the detailed artcile. I have one clarification though.
"Once saved anyone in possession of the private key that matches this public key can now login as this account."
Can any other user, besides user1, initial a SFTP connection to server2 as user2?
Thanks.

Josephine
October 10, 2011 12:07 PM

Thanks so much for this great article. It has helped me immensely. I can run rsync just fine without passwords from a bash script, but now I'm now trying to execute rsync as a cron task and I am getting the following errors:
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.8]
rsync returned 12

I've tried using keychain (http://www.gentoo.org/doc/en/articles/openssh-key-management-p2.xml) and get the same errors. Any ideas? Thanks, Josephine

B. Williams
April 4, 2012 6:45 AM

Excellent! It worked perfectly!

kris laks
May 16, 2012 12:43 PM

Hi, still I'm getting the prompt for password in the remote server when I try to sftp...
any clues please...
it is uix to unix box...

Raj
May 24, 2012 1:39 AM

Perfect!!! this is how things should be explained :)

Kiran Kola
June 6, 2012 4:42 AM

Very nice explanation !!! :)

A. Tuscano
August 5, 2012 10:21 PM

I'm using GlobeScape's EFT as my sftp server and the remote server is using another kind of SFTP. will this code work?

Varun Verma
August 28, 2012 12:01 PM

This is an excellent post. Works flawless.

Swapnil Patel
April 4, 2013 6:42 AM

What configuration required to automate SFTP script to connect windows server above details are from Unix to Unix what if the target server is windows
please help me i m in huge trouble....

Comments on this entry are closed.

If you have a question, start by using the search box up at the top of the page - there's a very good chance that your question has already been answered on Ask Leo!.

If you don't find your answer, head out to http://askleo.com/ask to ask your question.