MQTT turns 25(andypiper.co.uk) |
MQTT turns 25(andypiper.co.uk) |
My experience with the eclipse clients (paho, etc) is similar to yours in both python and C++, - it makes it overly complicated and seems very low level. It also is somewhat buggy because of the architecture.
I believe (perhaps wrongly), it's all basically maintained by one person or close to it (only one person has contributed to the C++ client in the past 6 months, for example, and nobody has contributed in the past 3), so i kind of understand how it's gotten this way over time.
I filed a simple PR (1 word change) to fix an obvious bug and it took 2 years to review it and accept it with no changes. Again not a complaint, just trying to portray how the state feels - a lot of libraries, bugs, and work, and few overworked people helping get it all done.
I moved to other clients and the experience is much much better in python, rust, C#, and C++.
Most of them have a good combination of high and low level API's, so if you just want to send a message on a topic, you don't have to worry about acks, retries, etc.
But if you need control, you can get it.
Honestly, i worry at this point that keeping paho/etc alive in this state is doing more harm than good - if they were officially dead it would at least force the issue. Right now you end up with users who have an experience like yours, and then either give up on MQTT or assume they are doing it wrong :)
Everything about it rubs me the wrong way. The API design, the poor documentation, even the way they seem to not adhere to typical python conventions.
It looks like it's easy to start using it, but then the breadth of protocols and implementations start getting to you. At a point in time, I remember the only reliable way we had of knowing if we had successfully connected to a server was to try to subscribe to a topic twice and catch a specific error code on the on_connect message (which actually was documented as a success code at the time). I know how crazy it sounds and maybe there was a better way to do it, but, if there was, I don't think it was that easy to find.
It's easy to complain, though. I'm very grateful so many people have worked to create this library. If they hadn't, I wouldn't have built what we've built with it. I really admire people who take on these huge projects.
The only place where I was unable to do this effectively was with the last will and testament message (and I might not have tried hard enough.) It also doesn't work with microcontrollers that don't run something like Linux.
It tries to be the simplest way to send and receive events.
I'm not sure what the sweet spot is, unless it's just interfacing with projects that have already committed to MQTT.
If you're open to using Lua, you might find this client library more straightforward: https://realtimelogic.com/ba/doc/en/lua/MQTT.html. It’s designed to be simple to use and works with the BAS runtime, which supports many platforms, including microcontrollers like the ESP32: https://realtimelogic.com/downloads/bas/ESP32/
I totally disagree, your expectations for the protocol seems unusual. They are not frameworks, and need to game many options.
MQTT specification is like 3 pages.
For comparison try to read specification of HTTP3, or AMQP - those are hideously complicated.
10+ years ago we added it to Kepware (OPC server) and streamed tag values to "the cloud". I was at a conference giving a presentation on it when Arlen Nipper, one of the creators of MQTT, came up after the presentation and said I did a "decent job". It was humbling :). Fast forward to today, and we have a new company (HighByte) modeling factory data at the edge and sending it via MQTT, SparkplugB (protocol over MQTT), direct to S3, Azure Blob, etc, etc.
All that to say, MQTT is a big driver in Industry 4.0, and it's cool to see it so heavily used all these years later.
https://www.bbc.co.uk/blogs/technology/2009/06/things_that_t...
The core protocol, devised in the days when 1 byte over a satellite link cost $1, is incredibly efficient and simple to implement.
I once had a client where the only port available for me to use on their firewalls was for MQTT (1883) because that's how we were getting sensor data from them. They would not open anything else for us no matter how we implored them so I wrote a live TCP wrapper over MQTT to get around it. It was a local multithreaded TCP daemon that listened for outbound requests on a certain port, wrapped them in MQTT and then published them using a unique topic. The server daemon would detect these topics and unwrap them before forwarding to our server processes. So the client machine thought it was making a live TCP connection to our server but in the middle was a funky invisible MQTT wrapper. It was really elegant once it worked but my goodness was it a pain to debug - a couple of months before I got the whole thing right because of all the blind alleys I went down.
edit: boost.org is a blast from the past, still looks exactly like I remember it from like 2008. Down to the "Get Boost" shopped on an emergency off button!
Apart from that, Boost has quite a few goodies such as Boost.Json, Boost.Program_options, Boost.Interprocess, Boost.Lockfree, Boost.Unordered, Boost.Dynamic_Bitset, etc.
Some libraries, like Boost.Atomic, Boost.Thread, Boost.Chrono and Boost.Filesystem, simply became obsolete with modern C++ versions. In fact, they served as the blueprint for the corresponding C++ standard libraries.
Personally, I have been wary of using Boost in the past because it's such huge library, but CMake integration is actually quite good these days and I found it pretty easy to use. Documentation is also quite good IMO.
[1]: https://engineering.fb.com/2011/08/12/android/building-faceb...
I guess it's nice that MQTT is a standard and so you have a way to connect to off-the-shelf devices that are enabled. I still think there are better options for pub/sub and message queues, esp. if you need to failover on the consumer side.
Sincerely, what are these options?
If I needed to sniff or debug something it was as easy as putting the device on the network and using MQTT Explorer to record or inject messages. I could even let coworkers that were working remotely work with the system by opening the port outside the LAN.
(If we ignore retained messages) MQTT brokers are not designed to store data, so there's no need to be concerned about durability because there simply isn't any...(!)
It is (just) a messaging protocol.
The nature of my project was that a missed message wouldn't be fatal, and message retention was good enough protection for a process that crashed and restarted.
Some of these servers have stat pages that show how much data they transfer. Usually phenomenal amounts of data. It's pleasing to me as someone doing a lot of stuff with open source networking to have such an establish protocol (instead of say -- 10 crappy ones.) MQTT works and it works well.
https://digitransit.fi/en/developers/apis/4-realtime-api/veh...
https://www.mqtt-dashboard.com/
But a quick Google search will get you some others.
It helped me to understand pubsub philosophy and I ended up using it for my personal development - to the point where I was misusing it (which is the typical path with technologies one discovers)
It is a great protocol for home automation but quickly gets annoying for more complex scenarios.
We have shipped many products that use MQTT for telemetry transfer to various different vendor servers and it always just worked. It was never a pain-point in stark contrast to CoAP or some proprietary protocols.
Works great for my smart home needs while also letting me explore latency and throughput load testing if I want to have a little fun.
Also since MQTT is TCP-based you can't natively speak the protocol from a webbrowser without the broker also listening on a websocket.
Ended up switching to WAMP-proto [0], which checked all my boxes and just did everything over websockets by default.
Sounds like a very good idea on paper, which can even be brokered if needed
I mean it was almost designed for this use case.
Also if using NATS things become very simple to setup.
My personal gripe with LwM2M is that it uses CoAP (Constrained Application Protocol) as its primary transport protocol. Since CoAP is based on UDP, you’re forced to rely on DTLS for authentication and encryption. Managing sessions with DTLS can be quite frustrating, and unlike TCP, there’s no reliable indication of when a session is closed by your peer.
Another issue is that it doesn't strictly follow the client-initiated communication pattern and can receive unsolicited messages from the server. I have found this to be very unreliable in real-world conditions for Device Management use cases.
Do/Did you work at Kepware?
Then again the async stuff they’re doing now is so over-engineered and terrible as well.. I took part in their teams meetings some time, only to find they had never read the mqtt spec (less than 50 easy to read pages) and didn’t understand what the headers are for for example (they wanted to put stuff in there that actually belongs in the payload). Some microsoft guy also took offense at my suggestion to first look at what competitors are doing with mqtt, because he wanted to create something new rather than copy. My company will just treat opc ua at the very edge and isolate it from our tech as much as possible on my suggestion.
For binary and JSON payloads, and with encryption and group key management features.
I see quite a bit of unsubstantiated bashing lately. Especially advocating 20% solutions that solve the easy parts with less effort.
What are you missing from OPC UA?
This is where I first encountered it too -- chemical production. Also saw a lot of it in aviation and railroad control systems.
That said, seeing a lot more Kafka and RabbitMQ crowding into MQTT's marketspace.
https://www.ifm.com/de/de/product/AL1332
Pretty wild times.
PS: You can web GET/POST set and get ios with these. Robot experience that tragic irony POST {NOOO} and he just dropped it like its hot.
It works in simple cases, but the lack of transactions & ability to control what and how often you're reading from the client side, can create challenges.
Its a C89 MQTT library intended for embedded devices.
Hopefully most people understand that I meant that you don’t need a stand-alone broker, which on embedded systems means one more thing to deploy and worry about the license for.
Edit: Also:
> The philosophy of ZeroMQ starts with the zero. The zero is for zero broker (ZeroMQ is brokerless)