Don't get me wrong. Running the predefined tests on all releases is good to ensure that there is no regression. But test engineers need to go beyond that be creative and think like attackers and think of multiple kinds of deployments. Creating test cases is continuous process.
I have written one blog entry on "advice to network equipment testers". I hope that was useful.
What are the test Engineers need to know from their counter parts in development and product management. What questions you need to ask yourself as test Engineer.
- Do I know the purpose of the feature which I am testing.
- What are deployment scenarios where this feature is applicable.
- What are configuration parameters exposed by this feature to the end user.
- How do I map these configuration parameters to the deployment scenarios.
- Is this feature associated with any protocol processing?
- If there is any protocol processing, do I know the protocol.
- Go through the standard document if it standard protocol.
- Go through your internal protocol documentation in case if it is proprietary protocol.
- Go through the messages and fields AND possible values of protocols.
- Find out if there are any tools in the public domain related to this feature.
- Think like an attacker and note down the messages and fields for possible negative test cases.
- What are the hardcoded parameters for this feature
- How many configuration records can it take.
- How many run time blocks can it create.
- Think of deployments where the traffic can't be predicted. These will help in creating test cases to test the robustness of the feature.
- If there is any standard based protocol processing, do I need to ensure that it is inter-operating with other products in the market.
- What are different ways this feature is used and interoperability testing should cover those.
- What is the expected performance of this feature in low load and in high load conditions?
- What is other traffic that is typically run when this feature is enabled.
- Find out all kinds of traffic that is possible in deployment scenarios. Ensure that this feature works along with other features.
- Can the test cases automated?
- If so, automate them. So that next time you don't have to spend time on doing manual testing.