Ask HN: How do you test and develop time-dependent/flow-dependent applications?

3 points by thomgo 5 years ago | 2 comments

Hi all,

I find myself running into difficulties when testing and developing for time-depdendent applications. What I mean by that is that certain actions in the flow of command result in states that depend on previous interactions.

For example, how does Uber test their driver dispatch system, and corresponding notifications on the part of users when the driver is getting close to their pickup location? In here, the behavior of the application depends on the driver accepting a request, starting his ride, etc.

I find it difficult to reproduce situations where the previous 'flow' affects the current state of the application and it's expected behavior.

Hope I was able to explain my difficulties clearly.

Thank you!

sethammons 5 years ago |

In unit tests, we use configurable timeouts and/or mock time (dependency injection usually). This lets us keep tests fast.

In integration tests, we black box test our service from the edges. Docker Compose ftw. We validate logs and other output such as metric endpoints. If we read from a queue, we run that queue; same with all our dependencies. Sometimes we will use fake services with mock/default responses if running the dependency is too onerous.

System testing. We run a smaller clone of production as a staging environment. Automated tests kick off customer actions (say, using an API or submitting a file for processing) and monitors logs and sinks (customer webhooks, user notification, emails, files generated, etc). These test suites exercise the entire system from the customer point of view.

Production. Metrics, metrics, and logs (and alerts). Deployment dashboards of metrics and error logs, slo/sli graphs, and automatic rollback on some metric violations. We usually canary a single node, the do a multi-tier roll out.

When a bug shows up, we determine which level of testing makes the most sense and add a test case to prevent future regression. New alerts may be set up.

Jugurtha 5 years ago |

This is where modular code and dependency injection are valuable. Example: one project I had interfaced with a Bluetooth Low Energy device. It didn't follow the usual profiles and the communication protocol was sent by the manufacturer (send this sequence of bytes, this header, payload, and checksum, and you'll get this response).

What I did was to turn the protocol specified in pages and pages of a PDF into a tiny YAML file containing a list of functions, their headers, request parameters, number of bytes, padding bytes. Then wrote a codec to encode a function function call with keyword arguments into a byte sequence corresponding to the function call, and vice versa: decode a byte sequence from the BLE device into a dictionary.

This allowed me to be able to test for several devices simply by feeding the same codec a different YAML file, and then feeding that instance of a codec to a device class. It also made the upgrade to newer models trivial, as in add entries to a YAML file for new functions like blood pressure, etc. I had a decorator that allowed me not to write code. The decorator used the codec in order either to transform a sequence of bytes into a dictionary, or a dictionary of keyword arguments into a sequence of bytes.

In addition to taking a codec instance, the device took another argument: bledevice. This device had an interface, which allowed me either to use a real BLE connection with the actual device, or a stub with the same interface as a BLE connection.

Which meant I could test without even having a real physical device. Which meant I could test all kinds of scenarios and transitions.

The gist of this is that the code was very modular. The bluetooth code handled all that was BLE related (connecting, disconnecting, scanning, reset, etc). It could be tested.

The codec to make parameters into byte sequences and byte sequences into parameters was isolated as well.

Given that, I was able to capture weird behavior that happened with a physical watch, and write a test for it.

The system used the actor model which was useful when facing weird exceptions and bringing new actors up again, as the devices were on customer premises and distributed geographically.

  from silverwatch import BLEDevice, Codec, Watch
  
  conn = BLEDevice('AA:BB:CC:DD:EE:FF')
  codec = Codec()
  watch = Watch(conn, codec)
  watch.start_ecg()

What follows is part of the documentation for "Testing". I spent a lot of time thinking about ways so colleagues did not have to get dirty with hardware problems. There was a binary file containing real data they could load, for exampe and a nice repr function to see the state of data collection:

>Toying with the watch is great, but what if you want to play with this code without necessarily having a BLE dongle or a physical watch? You can simply import a mock BLEDevice and inject it to Watch as you would do with a real connection.

  >>> from silverwatch import MockBLEDevice, Codec, Watch
  >>> codec = Codec()
  >>> conn = MockBLEDevice('XX:XX:XX:XX:XX:XX')
  Started
  Connect:  XX:XX:XX:XX:XX:XX
  >>> watch = Watch(conn, codec)
  >>> watch
  -------------------------
  Queues sizes:
  -------------------------
  Heart:            0
  Activity:         0
  Current Activity: 0
  Total Activity:   0
  Sleep:            0
  Battery:          0
  -------------------------
  Raw Packets:      0

>You can also mimick the behavior of the watch receiving packets. You can load the captured packets from packets.bin and feed them to the watch as if it had received them. Assuming you have a watch(real or mock):

  >>> from silverwatch import dispatch_packets, load_raw
  >>> packets = load_raw('packets.bin')
  >>> dispatch_packets(watch, packets)
  >>> watch
  -------------------------
  Queues sizes:
  -------------------------
  Heart:              1487
  Activity:           26646
  Current Activity:   99
  Total Activity:     85
  Sleep:              1688
  Battery:            58
  -------------------------
  Raw Packets:        0