Q:

SSH "connection refused" on Raspberry pi - cannot find why

I have the following problem SSH "connection refused". It would work fine, but while I am working on the SSH, I would get "connection refused" and would have to re-connect (which does not occur for good 5-10 minutes). I have tried everything (restart linux, restart ssh) yet still does not help. I've checked var/log as well but nothing helpful in there...
NOTE: Interesting note I found is, when I try ssh to my external ip(277...), it goes to black cmd screen, I close this and try my 192.168.0.13 address and it lets me sign in.. but short while, again connection refused... Of course if I open my server and sudo sshd restart, my ssh "connection refused" goes away...
My ssh port is opened at 23, and I have it set on modem/router as well
sudpi@raspberrypi:~$ sudo netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1817/apache2
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      2227/vsftpd
tcp        0      0 127.0.0.1:3350          0.0.0.0:*               LISTEN      1784/xrdp-sesman
tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN      3115/sshd
tcp        0      0 0.0.0.0:3389            0.0.0.0:*               LISTEN      1781/xrdp
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      2397/mysqld

When I get disconnected, and log back in and check "WHO" I only see my previous connection, my root and the current connection
whopi@raspberrypi:~$ who  
pi       tty1         2013-01-28 09:20
pi       pts/0        2013-01-28 10:28 (192.168.0.12)
pi       pts/1        2013-01-28 10:32 (192.168.0.12)

This is what I have in my ssh_config that I have modified (SSH to 23 because rogers use 22...)
#   Port 23
...    
#Keep my damn connection alive!
KeepAlive yes
ServerAliveInterval 60

This is what I get when I run arp-scan
pi@raspberrypi:/etc$ sudo arp-scan -I eth0 -l | grep 192.168.0.13
192.168.0.13    a0:6c:ec:ec:bb:5b       (Unknown)

This part I found online that told me I should post the following two information
pi@raspberrypi:~$ ps afxu | grep sshd
root      2838  0.0  0.7   9800  3168 ?        Ss   10:27   0:00 sshd: pi [priv]                                                                                   
pi        2845  0.0  0.3   9800  1628 ?        S    10:28   0:00  \_ sshd: pi@pt                                                                                   s/0
root      2854  0.0  0.7   9800  3168 ?        Ss   10:32   0:00 sshd: pi [priv]                                                                                   
pi        2861  0.0  0.3   9800  1628 ?        S    10:32   0:00  \_ sshd: pi@pt                                                                                   s/1
root      2900  0.3  0.7   9800  3168 ?        Ss   10:44   0:00 sshd: pi [priv]                                                                                   
pi        2907  0.0  0.3   9800  1628 ?        S    10:44   0:00  \_ sshd: pi@pt                                                                                   s/2
root      2934  0.0  0.2   6204  1060 ?        Ss   10:45   0:00 /usr/sbin/sshd
root      2954  0.7  0.7   9800  3164 ?        Ss   10:45   0:00  \_ sshd: pi [p                                                                                   riv]
pi        2961  0.1  0.3   9800  1624 ?        S    10:46   0:00      \_ sshd: p                                                                                   i@pts/3
pi        2970  0.0  0.1   3536   796 pts/3    S+   10:46   0:00              \_                                                                                    grep sshd

EDIT: pi@raspberrypi:~$ ssh -vvv 192.168.0.13 (THIS might be long... Thank you Bart Friederichs)
pi@raspberrypi:~$ ssh -vvv 192.168.0.13 -p 23
OpenSSH_6.0p1 Debian-3, OpenSSL 1.0.1c 10 May 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.0.13 [192.168.0.13] port 23.
debug1: Connection established.
debug1: SELinux support disabled
...
debug1: match: OpenSSH_6.0p1 Debian-3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-3
debug2: fd 3 setting O_NONBLOCK
debug3: put_host_port: [192.168.0.13]:23
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
...
debug1: Next authentication method: publickey
debug1: Trying private key: /var/www/.ssh/id_rsa
debug3: no such identity: /var/www/.ssh/id_rsa
debug1: Trying private key: /var/www/.ssh/id_dsa
debug3: no such identity: /var/www/.ssh/id_dsa
debug1: Trying private key: /var/www/.ssh/id_ecdsa
debug3: no such identity: /var/www/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
pi@192.168.0.13's password: - I TYPED PASSWORD -
...
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Linux raspberrypi 3.6.11+ #350 PREEMPT Mon Jan 7 21:51:11 GMT 2013 armv6l

A:

Your sshd is listening on port 23, yet you are connecting to port 22 (ssh's default).
Try this:
  pi@raspberrypi:~$ ssh -vvv 192.168.0.13 -p 23

A "connection refused" error means the TCP layer in the kernel is not accepting any connections on that port. It has nothing to do with keeping alive or anything.
Also, you are using arp to find out stuff, but a better tool would be nmap. arp only tells you if the IP is in your broadcast domain, not if any ports are open.

A:

Quoting the documentation:

As of the November 2016 release, Raspbian has the SSH server disabled by default. You will have to enable it manually. 

To enable SSH on machines through console:

Enter sudo raspi-config in the terminal, first select advanced options, then navigate to ssh, press Enter and select Enable or disable ssh server.

To enable SSH for headless machines:

For headless setup, SSH can be enabled by placing a file named ssh, without any extension, onto the boot partition of the SD card.

Copy-pasted from @techraf's answer in this related question: SSH not working with fresh install.

