[net] Add Interrupt Process and Abort Output telnet protocol handling

[ghaerr]

Enhances telnet and telnetd to properly handle telnet protocol "Interrupt Process" (^C) and "Abort Output" (^O). Previously, these characters were sent to the remote system. Now, ^C sends telnet IAC IP, which is then converted to ^C by telnetd. ^O sends IAC AO, which currently does nothing in telnetd.

Telnet continues to discard output while ^C or ^O processing is ongoing. Typing any character will reenable output, as will also after a period of network input timeout.

The telnet protocol and how other systems work, particularly GNU inetutils, is discussed in detail in Mellvik/TLVC#215.

@Mellvik, these are my minimal changes based on our long discussion. Everything seems to be working well testing using QEMU and ELKS localhost, and macOS. However, strangely enough, my debug shows that macOS never sends a DM (DataMark) response. Thus, it seems a network timeout is required in order to make this work. I have debug code writing "TO" for timeout and "DM" when DataMark received, and DM is never displayed.

[Mellvik]

Nice @ghaerr.

However, strangely enough, my debug shows that macOS never sends a DM (DataMark) response. Thus, it seems a network timeout is required in order to make this work. I have debug code writing "TO" for timeout and "DM" when DataMark received, and DM is never displayed.

Interesting - you definitely have a different telnetd (probably telnet/telnetd pair) than I have (I'm using home-brew). Your comment sent me testing the MacOS versions again, and (there is no end to it) I made a set of new discoveries. Including the fact that there is a 'DO TIMING MARK/WILL TIMING MARK/WONT TIMING MARK' (RFC 860) that may be useful in order to synchronize between client and server. Also I'm surprised to find that the macOS telnetd server initiates option negotiations all the time, every return, every ^C. Very chatty to say the least (see below). Other servers I've tested don't do this.

What I immediately found valuable in this is the observation that when the telnet server processes an IAC command that is supposed to return a DM, it does not return that immediately, but waits - until - and then sends it. What that is, is unclear and probably needs a peek into the sources. What it looks like though, is that the DM is held back until there is more output to send back. Then the DM is pushed ahead of that data. If this is the case it seems really smart because it solves the lingering data problem (discard after IAC_IP). It is also possible that the server is programmed to wait to see if there is a DO TIMING MARK coming, which apparently is supposed to be handled before the DM:

macOS% ^CSENT IAC IP                  <---------- typed ^C
SENT DO TIMING MARK
RCVD WILL TIMING MARK
RCVD IAC DMARK
                      <------ this is the newline echoed from the server after the ^C
more options processing... 

[ In the example I'm connecting the macOS telnet client to localhost, then issue the set options command to telnet to see what's going on. ]

This is of course speculation until verified from the sources, but I find it really interesting. I tested this (that is, moving the DM reply) on TLVC telnetd (no TIMING MARK), it works well - as would be expected. I never thought of that before.

When MacOS connects to a Big Linux server, the frequent options negotiations are gone and the Time Mark is nowhere to be seen.

[Mellvik]

I'm assuming the syntax "\003" will create a string occupying 2 bytes. A literal 3 would also be two bytes (int), right? What about '\03'?

[ghaerr]

A "\003" will add two bytes somewhere in the data segment: 3, 0, and returns the address of the start byte. write then writes the first byte only - exactly what's wanted.

A literal 3 is an 'int' and if cast to (char *) would return whatever is at address 3 in the data segment (this would be a big error).
A '\03' is a 'char' and if cast to (char *)(int) would also return whatever is at address 3 in the data segment (error).

[Mellvik]

Sorry, I obviously wasn't thinking for a moment (?), I guess the extra byte is less than an initialized char on the stack.

[ghaerr]

Yes, you could say "char c = '\03'" then write(1, &c, 1), but don't forget then you're adding code to initialize the variable c, which will certainly be more than a single byte of code.

[Mellvik]

This is effectively disabling ^O for most hosts, is that the purpose?

[ghaerr]

and (there is no end to it) I made a set of new discoveries.

There really is no end to the madness of telnet protocol implementations - you've got that right! Seriously, I'm pretty sure that the deeper you go, the more telnet will end up like GNU telnet, and having to implement an output ring buffer in order to meet the spec. BTW, I also saw in the GNU source that it negotiates the same additional TELNET options like you're seeing, but I didn't dig into it because my plan is not to implement a standards-based telnet - far too big for ELKS.

when the telnet server processes an IAC command that is supposed to return a DM, it does not return that immediately, but waits - until - and then sends it.

Yes - I think I kind of thought something like that after reviewing the GNU sources. All select I/O is time-stamped and put in a huge input and output ring buffer to be processed and managed at a later time. Very complicated. I imagine they're doing all this to fully meet "the spec".

If this is the case it seems really smart because it solves the lingering data problem (discard after IAC_IP).

No doubt perfection can be achieved with lots of code.

This PR seems to work extremely well with almost no overhead. telnet and telnetd now handle ^C -> IAC IP and ^O -> IAC AO properly and a small timeout manages to (in my case) handle discarding output from both satisfactorily, since having any output discarded from our previous version is a huge step forward. I won't be implementing a full TELNET protocol, as I don't see much benefit in trying to achieve perfection from a ^C typed occasionally.

This is effectively disabling ^O for most hosts, is that the purpose?

Well, looking at GNU telnet, it used VDISCARD (^O) to abort output by sending IAC AO, which is where I got this idea from. In other words, rather than passing ^O (discard output) passthrough to the remote system, I'm implementing the GNU telnet approach of handling ^O locally and sending AO. While my version doesn't check termios.c_cc[VDISCARD] like GNU does, the result would be the same - VDISCARD is used to locally override. Note that from my previous discussion on your thread, one can use a telnet setting to not handle special characters locally, in which case the ^O would be interpreted by the local OS if VDISCARD is set locally to ^O. If not, then it would be passed through to the remote telnetd unfiltered.

In summary, for my systems, DM is never seen, and this PR discards output very well, all things considered, for both ^C and ^O. I will leave the DM processing in ("discard = 0") for later testing with other hosts, but won't likely try to setup fancy telnet negotiations for such things, as then this would all have to be added to ELKS telnetd... and the question is: for what, really? I suppose the answer is whether one wants a "spec" telnet, or just wants Interrupt Process to discard output and "mostly" work in "all" cases.

[Mellvik]

when the telnet server processes an IAC command that is supposed to return a DM, it does not return that immediately, but waits - until - and then sends it.

Yes - I think I kind of thought something like that after reviewing the GNU sources. All select I/O is time-stamped and put in a huge input and output ring buffer to be processed and managed at a later time. Very complicated. I imagine they're doing all this to fully meet "the spec".

If this is the case it seems really smart because it solves the lingering data problem (discard after IAC_IP).

No doubt perfection can be achieved with lots of code.

Adding a static state variable doesn't really count as 'lots of code', does it? As I recall the goal was to add functional ^C and ^O support, not perfection.

This PR seems to work extremely well with almost no overhead. telnet and telnetd now handle ^C -> IAC IP and ^O -> IAC AO properly and a small timeout manages to (in my case) handle discarding output from both satisfactorily, since having any output discarded from our previous version is a huge step forward.

This is effectively disabling ^O for most hosts, is that the purpose?

Well, looking at GNU telnet, it used VDISCARD (^O) to abort output by sending IAC AO, which is where I got this idea from. In other words, rather than passing ^O (discard output) passthrough to the remote system, I'm implementing the GNU telnet approach of handling ^O locally and sending AO.

As it turns out, Linux (my testbed is Ubuntu latest) doesn't support discard/^O in the tty drive or any other context that I've found. It's present in the configurations (like stty -a and display at the telnet prompt). Linux telnet simply discards (!) ^O, does not turn it into AO and does not pass it along. Finally, in telnetd, and AO simply returns a DM, nothing else. So the GNU telnet approach turns out to be to ignore ^O, really disappointing. Going back to the old Linux version on my RapberryPi 2, it's the same.

On the Mac it's different - supported in the tty driver and in telnet. AO (in telnet) returns a DM and nothing else, like Linux.

So in order for telnet ^O (discard) support to be meaningful, it either has to be implemented locally in the telnet client (which I am doing) or passed on to the server, which will work with MacOS (possibly other Unix) hosts, but not with Linux servers.

In summary, for my systems, DM is never seen,

It would be interesting to know what systems that is, I'd like to add them to my test cases! All systems I can get my hands on, emit DMs at the same (and expected) places.

[ghaerr]

No doubt perfection can be achieved with lots of code.
Adding a static state variable doesn't really count as 'lots of code', does it?

Oops - I was referring to the perfection achieved with the GNU telnet with lots of code!

As I recall the goal was to add functional ^C and ^O support, not perfection.

Yes. I mentioned this because I thought you might have said you were still working on eliminating some "tail end" data being received after ^C from certain telnetd systems... is that still the case? Does the dm_flag fix in your own telnetd finally remove that data from being displayed, or does it still sometimes happen with some systems?

Linux (my testbed is Ubuntu latest) doesn't support discard/^O in the tty drive

Interesting - so ^O/discard is macOS only after all. I just looked this up and apparently ^O/discard is not Posix and the Linux termios(7) man page also says it's not supported.

So the GNU telnet approach turns out to be to ignore ^O, really disappointing.

Well, the GNU telnet source does show that after sending an AO, which is does on ^O, auto flush is called:

  else if (c == termFlushChar)
    {
      xmitAO ();		/* Transmit Abort Output */
      return 0;
    }

void
xmitAO (void)
{
  NET2ADD (IAC, AO);
  printoption ("SENT", IAC, AO);
  if (autoflush)
    {
      doflush ();
    }
}

I'm not sure whether this contradicts what you're seeing, or whether the auto flush mechanism doesn't seem to be doing much on the Linux telnet? I haven't studied the GNU telnetd source (yet) but perhaps it doesn't do anything on AO, sends no DM, and the GNU telnet ring buffer finally decides to start displaying data after a short timeout?

On the Mac it's different - supported in the tty driver and in telnet. AO (in telnet) returns a DM and nothing else, like Linux.

Oh I see - Linux returns DM on AO, but doesn't actually flush any data. I think I understand now. Yes, so we have perform discard ourselves in the local telnet client.

In summary, for my systems, DM is never seen,

It would be interesting to know what systems that is, I'd like to add them to my test cases! All systems I can get my hands on, emit DMs at the same (and expected) places.

My telnetd seems to be from homebrew, from 2018. I don't have it running under automatic start, instead I start it on port 2424 using:

$ telnetd -debug 2424 &

I've attached it here so you can play with it. I've never seen a DM from it, but it does try to send DO TIMING-MARK, which of course our telnet respond with WONT. If you can get it to send a DM, that'd be great, since then I could test the DM handling in this PR. Nonetheless, this PR still seems to work quite well with this macOS telnetd and the updated ELKS telnetd with regards to discarding data appropriately.
telnetd-63.zip

[ghaerr]

So the GNU telnet approach turns out to be to ignore ^O, really disappointing.

I looked further at the GNU telnetd code and found it does actually "try" to flush its (PTY) buffers, reinitializes its terminal buffer, try inserting the "AO" character into the PTY stream (not sure whether that's ^O?) and then sends DM:

	      /*
	       * Abort Output
	       */
	    case AO:
	      {
		DEBUG (debug_options, 1, printoption ("td: recv IAC", c));
		ptyflush ();	/* half-hearted */
		init_termbuf ();

		if (slctab[SLC_AO].sptr
		    && *slctab[SLC_AO].sptr != (cc_t) (_POSIX_VDISABLE))
		  pty_output_byte (*slctab[SLC_AO].sptr);

		netclear ();	/* clear buffer back */
		net_output_data ("%c%c", IAC, DM);
		set_neturg ();
		DEBUG (debug_options, 1, printoption ("td: send IAC", DM));
		break;
	      }

It appears what you're saying is that either this code doesn't result in much of any output discarding, or perhaps this isn't the code running on Linux telnetd?