Verification

Verification is the process of checking that the simulation model correctly implements the intended conceptual model. It involves checking that the model’s logic, structure and parameters are implemented as planned and free from coding errors.

Things I can definitely do

Desk checking

• Systematically check code.
• Keep documentation complete and up-to-date.
• Maintain an environment with all required packages.
• Lint code.
• Get code review
  • Primarily using LLM - will provide code and prompt to review and identify any potential issues and improvements - we can then manually review
  • Also, can explore other more occassional/one-off review from colleagues to check over (e.g., tests, docs, generally).
  • https://pythonhealthdatascience.github.io/des_rap_book/pages/guide/sharing/peer_review.html

Debugging

• Write tests - they’ll help for spotting bugs.
• During model development, monitor the model using logs - they’ll help with spotting bugs.
• Use GitHub issues to record bugs as they arise, so they aren’t forgotten and are recorded for future reference.

Assertion checking

• Add checks in the model which cause errors if something doesn’t look right.
• Write tests which check that assertions hold true.

Special input testing

• If there are input variables with explicit limits, design boundary value tests to check the behaviour at, just inside, and just outside each boundary.
• Write stress tests which simulate worst-case load and ensure model is robust under heavy demand.
• Write tests with little or no activity/waits/service.

Bottom-up testing

• Write unit tests for each individual component of the model.
• Once individual parts work correctly, combine them and test how they interact - this can be via integration testing or functional testing.

Regression testing

• Write tests early.
• Run tests regularly (locally or automatically via. GitHub actions).

Execution tracing

• Trace individual calls or ambulances.

Things that require some discussion

Mathematical proof of correctness

• For parts of the model where theoretical results exist (like an M/M/s queue), compare simulation outputs with results from mathematical formulas.

This requires some thought as to whether there is anything we could compare against for even the most basic model. To discuss with Tom. Have put comparison to regression under comparison testing validation