Verification

Verification is the process of checking that the simulation model correctly implements the intended conceptual model. It involves checking that the model’s logic, structure and parameters are implemented as planned and free from coding errors.

Things I can definitely do

Desk checking

• Systematically check code.
• Keep documentation complete and up-to-date.
• Maintain an environment with all required packages.
• Lint code.

Debugging

• Write tests - they’ll help for spotting bugs.
• During model development, monitor the model using logs - they’ll help with spotting bugs.
• Use GitHub issues to record bugs as they arise, so they aren’t forgotten and are recorded for future reference.

Assertion checking

• Add checks in the model which cause errors if something doesn’t look right.
• Write tests which check that assertions hold true.

Special input testing

• If there are input variables with explicit limits, design boundary value tests to check the behaviour at, just inside, and just outside each boundary.
• Write stress tests which simulate worst-case load and ensure model is robust under heavy demand.
• Write tests with little or no activity/waits/service.

Bottom-up testing

• Write unit tests for each individual component of the model.
• Once individual parts work correctly, combine them and test how they interact - this can be via integration testing or functional testing.

Regression testing

• Write tests early.
• Run tests regularly (locally or automatically via. GitHub actions).

Execution tracing

• Trace individual calls or ambulances.

Things that require some discussion

Desk checking

• Get code review.

To discuss:

• Who can / should / best to do it?
• Technical planning can base on https://pythonhealthdatascience.github.io/des_rap_book/pages/guide/sharing/peer_review.html

Mathematical proof of correctness

• For parts of the model where theoretical results exist (like an M/M/s queue), compare simulation outputs with results from mathematical formulas.

This requires some thought as to whether there is anything we could compare against for even the most basic model. To discuss with Tom. Have put comparison to regression under comparison testing validation