Debugging SSH Setup for two RHEL systems

ISSUE: SLAVE1 IS NOT ABLE TO CONNECT WITH SLAVE2 VIA SSH WITHOUT PASSWORD PROMPT.

IP AND HOSTNAME MAPPING:
  10.74.19.50 ​   MASTER
  10.138.22.103  SLAVE1
  10.85.62.107   SLAVE2

COMMANDS FOR DOING SSH SETUP:

  1) sudo iptables -A INPUT -p tcp --dport ssh -j ACCEPT
  2) sudo reboot
  3) ssh-keygen -t rsa -f ~/.ssh/id_rsa -P ""
  4) ssh-copy-id -i ~/.ssh/id_rsa.pub projadmin@SLAVE2
  5) ssh-copy-id -i ~/.ssh/id_rsa.pub admin@SLAVE1
  6) ssh-copy-id -i ~/.ssh/id_rsa.pub admin@MASTER

~~  ~~  ~~

Command "ssh-copy-id" is used to copy your SSH public key to remote server for password less authentication.

MESSAGE YOU GET WHEN SSH KEYS ARE ALREADY PRESENT ON THE REMOTE SYSTEM:

  (base) [projadmin@SLAVE2 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub admin@SLAVE1
    /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/projadmin/.ssh/id_rsa.pub"
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    
    /usr/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                    (if you think this is a mistake, you may want to use -f option)

~~  ~~  ~~

MESSAGE YOU GET WHEN SSH KEYS ARE COPIED ON THE REMOTE SYSTEM:

  (base) [admin@SLAVE1 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub projadmin@SLAVE2
    /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/admin/.ssh/id_rsa.pub"
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    IT IS AN OFFENSE TO CONTINUE WITHOUT PROPER AUTHORIZATION.
    projadmin@SLAVE2's password:
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh 'projadmin@SLAVE2'"
    and check to make sure that only the key(s) you wanted were added.

LOGIN ATTEMPT 1:
  (base) [admin@SLAVE1 ~]$ ssh 'projadmin@SLAVE2'
    The authenticity of host 'slave2 (10.85.62.107)' can't be established.
    ECDSA key fingerprint is SHA256:+BqTUw27qVUgqcYRErYL8nksgX4XX9cimu/sgk2IkRs.
    ECDSA key fingerprint is MD5:27:41:cd:39:f2:97:a9:29:6b:e8:8b:f3:e6:aa:cd:8e.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'slave2' (ECDSA) to the list of known hosts.
    IT IS AN OFFENSE TO CONTINUE WITHOUT PROPER AUTHORIZATION.
    projadmin@slave2's password:
    Last login: Fri May  1 11:56:39 2020 from SLAVE1
    W A R N I N G
    THIS IS A PRIVATE COMPUTING SYSTEM FOR USE ONLY BY AUTHORIZED USERS.
  (base) [projadmin@SLAVE2 ~]$

LOGIN ATTEMPT 2:
  
  (base) [admin@SLAVE1 ~]$ ssh 'projadmin@SLAVE2'
    IT IS AN OFFENSE TO CONTINUE WITHOUT PROPER AUTHORIZATION.
    projadmin@slave2's password:
    Last login: Fri May  1 12:01:02 2020 from SLAVE1
    W A R N I N G
    THIS IS A PRIVATE COMPUTING SYSTEM FOR USE ONLY BY AUTHORIZED USERS.
	
  (base) [projadmin@SLAVE2 ~]$

~~  ~~  ~~

SSH PUBLIC KEYS ARE STORED IN REMOTE SERVER'S FILE "~/.ssh/authorized_keys". NEXT, WE DISPLAY THAT:

  (base) [projadmin@SLAVE2 ~]$ cat ~/.ssh/authorized_keys
    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDAaG5d2wcXeVGQkTtiHr5EQD5nYPugU1upCAnsei8vuZ1LpoUdrCiFq0jkvnQCOa... admin@MASTER
    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDckTRtXhW3JlQ/dgR3cEn70MGUNU29DT438ItypXh+BRnGTSuFayGLLb7XfgR4Fg... projadmin@SLAVE2
    ssh-rsa	AAAAB3NzaC1yc2EAAAADAQABAAABAQCb/WDdGt0abaEI9aTljhgtRYtzrjjAJu3+GK3wbmjFMTvvlb5729l4kcUwg3IeAv... admin@SLAVE1

NEXT, WE CHECK OUR PUBLIC KEY OF SLAVE1:
  (base) [admin@SLAVE1 ~]$ cat ~/.ssh/id_rsa.pub
    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCb/WDdGt0abaEI9aTljhgtRYtzrjjAJu3+GK3wbmjFMTvvlb5729l4kcUwg3IeAv... admin@SLAVE1

~~  ~~  ~~

NEXT, WE CHECK SSH CONFIG ON SLAVE2:
  (base) [admin@SLAVE1 ~]$ cat ~/.ssh/config
    cat: /home/admin/.ssh/config: No such file or directory
  (base) [admin@SLAVE1 ~]$

  (base) [projadmin@SLAVE2 ~]$ ssh-agent
    SSH_AUTH_SOCK=/tmp/ssh-1ULxPy4vidVX/agent.5929; export SSH_AUTH_SOCK;
    SSH_AGENT_PID=5930; export SSH_AGENT_PID;
    echo Agent pid 5930;

NEXT, WE CHECK SSH CONFIG ON SLAVE2:
  (base) [projadmin@SLAVE2 .ssh]$ ls
    authorized_keys  id_rsa  id_rsa.pub  known_hosts
  
  (base) [projadmin@SLAVE2 .ssh]$ cat known_hosts
    SLAVE1,10.138.22.103 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTIt...rObCVOxrV5XaKARNHQA=
	slave1 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTIt...rbCVOxrV5XaKARNHQA=
    SLAVE2,10.85.62.107 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoY...Xg6hmPnlGbfIiVmVPNdU=
	slave2 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbm...ArObCVOxrV5XaKARNHQA=
    10.74.19.50 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYT...rObCVOxrV5XaKARNHQA=
    MASTER ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTIt...ArObCVOxrV5XaKARNHQA=
    master ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTI...hmPnlGbfIiVmVPNdU=
    localhost ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoY...mPnlGbfIiVmVPNdU=
      
CREATING 'CONFIG' FILE SINCE IT DOES NOT EXIST:
  (base) [projadmin@SLAVE2 .ssh]$ touch config
  (base) [projadmin@SLAVE2 .ssh]$ ls
    authorized_keys  config  id_rsa  id_rsa.pub  known_hosts
  (base) [projadmin@SLAVE2 .ssh]$ vi config
  (base) [projadmin@SLAVE2 .ssh]$ cat config
    Host *
      UseKeychain yes
      AddKeysToAgent yes
      IdentityFile ~/.ssh/id_rsa

  (base) [projadmin@SLAVE2 ~]$ ssh-add -A
    Could not open a connection to your authentication agent.
  (base) [projadmin@SLAVE2 ~]$

~~  ~~  ~~

  (base) [projadmin@SLAVE2 ~]$ eval `ssh-agent -s`
    Agent pid 5182

  (base) [projadmin@SLAVE2 ~]$ ssh-add -K ~/.ssh/id_rsa
    unknown option -- K
    usage: ssh-add [options] [file ...]
    Options:
      -l          List fingerprints of all identities.
      -E hash     Specify hash algorithm used for fingerprints.
      -L          List public key parameters of all identities.
      -k          Load only keys and not certificates.
      -c          Require confirmation to sign using identities
      -t life     Set lifetime (in seconds) when adding identities.
      -d          Delete identity.
      -D          Delete all identities.
      -x          Lock agent.
      -X          Unlock agent.
      -s pkcs11   Add keys from PKCS#11 provider.
      -e pkcs11   Remove keys provided by PKCS#11 provider. 
	  
  (base) [projadmin@SLAVE2 ~]$ ssh-add -k ~/.ssh/id_rsa
    Identity added: /home/projadmin/.ssh/id_rsa (/home/projadmin/.ssh/id_rsa)
    (base) [projadmin@SLAVE2 ~]$

  (base) [projadmin@SLAVE2 .ssh]$ eval `ssh-agent -s`
    Agent pid 5611
  (base) [projadmin@SLAVE2 .ssh]$ ssh-add ~/.ssh/id_rsa
    Identity added: /home/projadmin/.ssh/id_rsa (/home/projadmin/.ssh/id_rsa)
    
~~  ~~  ~~

LAST RESORT:

  Deleting the everything in the directory: projadmin@SLAVE2:/home/projadmin/.ssh
  Copying public key only again from slave1: (base) [admin@SLAVE1 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub projadmin@SLAVE2
  The issue still exists. Not able to SSH slave2 from slave1 without password prompt.

~~  ~~  ~~

STATUS: ISSUE UNRESOLVED

References:

1. SSH Setup: Remote machine still asking for password (Stackoverflow)
2. Copying SSH key to remote server

Tags: Technology,Linux,SSH