AMQP: In­tro­duc­ing the Advanced Message Queuing Protocol

Services must be able to com­mu­ni­cate with each other. In­for­ma­tion is exchanged and data trans­mit­ted. To many people it sounds easier than it actually is, because there are several dif­fi­cul­ties to overcome with such con­ver­sa­tions between different ap­pli­ca­tions: In IT, for example, there are language barriers (in the form of various pro­gram­ming languages) and you have to stick to a protocol so that it does not end up in chaos.

The Advanced Message Queuing Protocol, or AMQP for short, promises a solution: a common protocol that can transport the in­for­ma­tion through a mediator. How does it work? And what else can AMQP do?

The Advanced Message Queuing Protocol has been developed since 2003 and, by way of exception, does not originate from a dedicated tech company, but was first conceived by JPMorgan Chase, a US bank. There, they decided to continue designing the project together with others. However, in addition to IT companies such as Cisco, it was the financial sector that continued to be primarily in­ter­est­ed in AMQP. Why is that? Because, there more than anywhere, time is money! The trans­mis­sion of in­for­ma­tion plays a major role at a bank, a credit card company or an exchange. Several hundred thousand messages are exchanged per second there. If news arrives too late or not at all, it can be expensive.

Since no com­mer­cial product was able to cope sat­is­fac­to­ri­ly with the re­quire­ments at that time, project manager John O'Hara chose to conceive his own, separate protocol. O'Hara’s focus was on open standards such as TCP/IP and he decided to also make AMQP freely available, in order to advance the de­vel­op­ment of the protocol. Meanwhile, the OASIS working group, a non-profit or­ga­ni­za­tion, was working to develop the protocol.

AMQP single-handedly solves several problems: the protocol (in co­op­er­a­tion with a messaging broker) ensures a robust data transfer, while also allowing messages to be stored in a queue. This in turn enables an asyn­chro­nous com­mu­ni­ca­tion: sender and receiver do not have to work around each other. The recipient (consumer) of the message does not need to directly accept the in­for­ma­tion, process it, and confirm receipt to the sender (producer). Instead, they collect the message from the queue when they have capacity available. Meanwhile, this gives the producer the op­por­tu­ni­ty to continue working - so there is no idle time.

The success of the rel­a­tive­ly recent protocol also has to do with in­ter­op­er­abil­i­ty. The Advanced Message Queuing Protocol es­tab­lish­es a common basis of its own. This even allows the different ap­pli­ca­tions to be written in different pro­gram­ming languages. In this way, even programs in different or­ga­ni­za­tions can easily com­mu­ni­cate with each other. And as AMQP is freely available, any company can use the protocol at no extra cost.

In the OSI model AMQP acts on the ap­pli­ca­tion layer, meaning it has direct contact with the various programs. IMAP (for email), FTP (for trans­fer­ring files) and IRC (for instant messaging) are also active on this layer. To transmit the messages, the protocol relies on mediators, or messaging brokers. These take over the dis­tri­b­u­tion of messages to different re­cip­i­ents according to defined rules. AMQP also controls the behavior of these mediation servers.

The Advanced Message Queuing Protocol therefore refers both to the com­mu­ni­ca­tion between various par­tic­i­pants and to the behavior of the brokers them­selves. These contain their in­struc­tions from the messages.

In the cosmos of AMQP there are three par­tic­i­pants and one item:

• The message is the core element of the whole com­mu­ni­ca­tion.
• The producer creates a message and sends it.
• The messaging broker dis­trib­utes the message to different queues according to defined rules.
• The consumer takes the message out of the queue that they can access, and edits it.

The mediation of the message is paused again with the broker. The exchange accepts the messages and routes the data to the correct queue. The exchange learns which queue the message belongs to from the binding. There are four different ways in which the exchange forwards messages.

The first kind, the direct exchange, sends messages to exactly one recipient and works with routing keys. One of these keys is given to the message. A queue in turn has a binding key. This iden­ti­fies the queue to the exchange. If the routing key and binding key match, the message can be forwarded to the queue and therefore to the recipient of the message. It is also possible that a queue has several binding keys and therefore also comes into question for several routing keys. Con­verse­ly, multiple queues can also share one binding key, which is called multiple binding. The exchange repli­cates the message and sends it to multiple re­cip­i­ents.

The fanout exchange works similarly. However, here the broker com­plete­ly ignores the routing key. Instead, the exchange routes a message to all available queues and thereby du­pli­cates the in­for­ma­tion. The topic exchange works in a different way. Similar to the direct exchange, routing key and binding keys are matched against each other, but there doesn’t have to be an exact match present. Instead, you use wildcards. This makes it possible to specif­i­cal­ly provide messages for multiple queues.

Finally, the headers exchange acts with the header of a message, instead of using a routing key. This contains values that are matched with the binding. The argument with the name x-match de­ter­mines if all values must match (value: all) or only one must match the binding (value: any). While the former cor­re­sponds to the direct exchange, the latter can produce the same effect as a topic exchange.

A frame is the basic unit with AMQP. A con­nec­tion consists of the ordered sequence of frames. Order in this case means that the last frame must not arrive at the receiver until all other frames have first reached their des­ti­na­tion. Each frame can be divided into three segments (in version 1.0):

• Frame header: This mandatory header has a size of 8 bytes. Here you will find in­for­ma­tion that de­ter­mines the routing of the message.
• Extended header: This area is optional and has no set scope. It serves to expand the header in the future with further in­for­ma­tion.
• Frame body: The body contains the actual data to be trans­ferred. The size is freely se­lec­table. However, this area can also be left empty, then the frame only serves to maintain the con­nec­tion.

The body of a frame, in turn, can take nine different forms:

• open: Ne­go­ti­ates the con­nec­tion pa­ra­me­ters between broker and client.
• begin: Indicates that a con­nec­tion is starting.
• attach: The message is appended with a link that is necessary in order to use the data transfer.
• flow: Changes the status of a link.
• transfer: The actual message is trans­mit­ted with the transfer frame.
• dis­po­si­tion: A dis­po­si­tion frame provides in­for­ma­tion on changes to the in­for­ma­tion delivery.
• detach: Removes the link.
• end: Indicates that the con­nec­tion will be ter­mi­nat­ed.
• close: Ter­mi­nates the con­nec­tion and declares that no further frames will be sent.

Each queue has its own name, which iden­ti­fies it to the other par­tic­i­pants. Either a client or the broker de­ter­mines the name. A queue is realized by a memory, which can either per­ma­nent­ly sit on a hard disk or briefly in RAM. The permanent variant guar­an­tees that the queue will continue to exist even after restart­ing a broker. But it does not guarantee that messages are also per­ma­nent­ly secured: here, it depends on the message as to whether they are still available after a restart.

What happens when a consumer cannot recall the message in queue, because, for example, the client or the con­nec­tion has broken down? It is possible to decide whether a consumer must properly ac­knowl­edge the receipt of a message or whether the mere delivery is suf­fi­cient for success. If the first variant is chosen and the consumer does not need to send back a message, the broker tries to send the message to another consumer or to reach the actual recipient again. However, if the variant is activated without con­fir­ma­tion and the consumer does not retrieve the message, it will be lost.

But it is also possible for a client to de­lib­er­ate­ly reject the ac­cep­tance of a message. This may be useful if message pro­cess­ing does not work. The feedback from the consumer causes the broker to either com­plete­ly delete the message or in­cor­po­rate it back into the queue.

There are currently two com­plete­ly in­de­pen­dent versions of Advanced Message Queuing Protocols. Version 1.0 was developed by the OASIS group. However, the slightly older version 0-9-1 is often used, es­pe­cial­ly with RabbitMQ. They are not com­pat­i­ble with each other. Version 1.0 differs primarily by the reduced im­por­tance of brokers, bindings and exchanges. 0-9-1 doesn't need this, but neither does it prohibit the use of such middlemen.