It's based on the Noise Protocol Framework in the outermost layer, which encrypts a compressed XMPP stream. The end-to-end encryption is done within various XMPP message payloads using the Signal Protocol, which encrypts message data serialized using Protocol Buffers, with different formats depending on the message type (text, image, video, sticker, etc).