Serial ports are hard

There might be sonething wrong with me: I keep finding situations where I need to use serial ports. They're not always RS-232 ports, either. Sometimes they're TTL serial to microcontrollers or other wacky stuff. Sometimes I modify firmware to add a USB serial port.

Serial communication boils down to bits – ones and zeroes – transmitting high and low voltages on one wire, receiving high and low voltages on a second wire. It's an extremely common way to interface with peripherals because it's extremely simple.

At least, it's simple until you get into the details.

Signalling details

How long do you send each bit before sending the next bit? (This is usually specified in terms of a baud rate which is related to but not exactly equal to this value.) How many ones and zeroes are sent per character? Are you transmitting extra bit(s) for error correction purposes, and if so, how are they calculated? How do you mark the start of a character? What about the end?

Familiar configuration strings like 9600,8,N,1 can answer few questions, but that's not the end of it. See… besides data, devices often need to communicate additional information.

You might want to reset the device somehow. Why not just hold the transmit line high (sending a constant stream of ones) for the length of several characters? This is the "break" sequence; it's not data (it's encoded in a way that cannot be confused as data), but it's transmitted on the data lines.

Another such need has to do with buffering, wherein devices can't accept more data until they've processed data they already have. A receiving device can:

Hope no new data arrives. What if it does? Well… throw it away, I guess.
Send a special "don't send me more data right now" character. This would be with the XOFF character… but now you have to worry about what to do if the XOFF bit sequence legitimately appears in the data stream.
Use a separate electrical connection to indicate that whether or not it's ready. This would be the CTS line… but now instead of two wires, you have three.

There's other signalling needs too. The host might want to know if there's a device plugged in (the DSR line), and the host might want to tell the device that it is plugged in (the DTR line). The host might want to tell the device that it wants to send data (the RTS line). The device might be acting as a bridge between the serial link and something else – say, a phone line or radio link – and might need to indicate that it has a connection to the other side (CD) or that something on the other side is trying to establish a connection (RI).

All this sucks, of course, because by now you've either ballooned from 2 wires to 9 wires (DB-9) to 25 wires (full RS-232) to handle everything or you've thrown up your hands in frustration and completely given up on trying to communicate most of this information.

Implementation details

It's woefully inefficient to force general-purpose processors to sit there and transmit and receive individual bits, so instead, people make dedicated hardware called UARTs – universal asynchronous receiver/transmitters – to do this.

The key advantage of a UART is that you can send and receive whole characters at your convenience. If you want to send a character, give it to the UART. If you want to send more, check to see if it's done, then send more. When it receives a character, it'll say so, and you can process it as a unit.

Of course, character-at-a-time operation isn't hugely more efficient than bit-at-a-time, so the new hotness 20 years ago was the 16550 UART which had onboard buffers. You could send and receive up to 16 characters at a time! Amazing!

Nowadays, 16550-like functionality is included in larger I/O chips alongside real-time clocks and other decidedly unsexy functions. But… remember how data isn't the only consideration? Yeah, well, UART buffers only deal with data.

If you want to send a break, well, poke at a register saying to send a break. If you want to signal things or receive signals on all those status lines, poke at another register, or check a register to see what lines are set. Catch is, since UARTs are asynchronous and include a buffer by design, it's very difficult to line up "I received this character" with "that status line changed"; you can be sure both happened, but rarely are you quite sure in which order.

Also, good luck changing baud rates or flow control settings at a specific point in a data stream. Those internal buffers? Not so great if you have a need for precision at the electrical level.

Unix

Serial ports are character devices on UNIXy systems, which means they look just like a file for reading or writing. Simple!

Of course, if you want to do anything with it, you need to specify the serial port configuration. That sounds like an I/O control, right? So, like, ioctl()?

Well… kinda. Serial ports and UNIX go way back – back to a time when people primarily used computers through teletype machines – so a lot of serial functionality is mixed in with terminal functionality. Terminals have a separate set of control functions (see <termios.h>) that are may or may not be implemented with ioctl() under the hood. Still, even with these functions specifically aimed at serial control, setting 9600,8,N,1 is far from trivial.

Sending and receiving data works like any other character device. Well, mostly. The operating system may or many not buffer things in ways you'd expect, and your data likely flows through a UART that may or may not buffer things in ways you'd expect, but it works.

What about all that other stuff? Like, how would you send a break? Call tcsendbreak()! What about receiving a break? Well, if the file handle in question is not your controlling terminal, you can ask to get breaks delivered as \x00 or \xff\x00\x00, your preference! But, wait… how would you distinguish breaks from normal data you might read with those patterns? You can't. Have fun!

How would you wait for data? Easy, use any of the normal ways you'd wait for data on a file descriptor. How would you wait for a status line change? Surprise, you can't! Poll ioctl(fd, TIOCMGET) several times a second and watch for changes yourself. Let's not even worry about detecting parity errors, truncated characters, or things like that.

See? Simple.

TCP/IP

It gets worse when people try to expose serial devices over a network, because most of the time, people just pump characters back and forth. Simple!

This might even work, but it's extremely naïve: all that other stuff is just glossed over, and anyone that needs a serial port to support anything besides transmitting and receiving characters will be very disappointed.

Enter RFC 2217. This is a specification that extends Telnet to include serial port controls. (Some people use Telnet as a dumb pipe for characters, but in fact it includes an extension mechanism.) Hooray! Now our serial port is back to being a single byte channel, there's a clear escape mechanism to distinguish control signals from data, and all the 9-wire serial port functions are available.

(Okay, it's really only almost all of those functions. For example, the UNIX API allows you to transmit with one baud rate and receive with another, while RFC 2217 does not. Also, while you can clear the buffers, you can't synchronize command execution with flushing the buffers, but in practice anything you need to do like that would suck using a local ports anyway.)

Programs such as ser2net can be used to expose physical serial ports as intelligent, functional RFC 2217 services. Perfect.

…unless you want to use software that assumes you have an actual serial port, or software that assumes a physical serial device. Aagh!