Telemetry Test Plan: Lockbox-extension
Given the importance the Lockbox team places on actionable user telemetry, ensuring confidence in the fidelity of that data is vital. Coverage aspects are broken into two distinct areas: test automation and manual testing.
When a new feature is developed or an additional user action needs to be recorded, the developer implementing the new telemetry event will add a unit test to ensure the action creates a ping.
New telemetry events
Once a new telemetry event is added to a feature and that event has a unit test, the telemetry ping will be manually inspected by either Leif Oines, Product Data Scientist or Product Integrity. The goal of this exercise it to ensure the new event and accompanying unit test are capturing the correct user behavior.
A small handful of smoke tests will manually be run prior to each release. Note, the team will rely on the underlying unit tests for comprehensive testing of event verification.
Product Integrity will investigate options for automating these tests into the integration test suite.
Metrics being gathered
For a comprehensive list of metrics being gathered review the Lockbox Telemetry Plan.