Software Test Automation has been there in one form or another for decades. Organizations had mixed successes with Test Automation even though few have been immensely successful in automating tests as part of their application development process. The benefits to these organizations have been huge in terms of costs, time to market and quality.
Why is it that many teams are still continuing with their manual tests or struggling as they embark on Test Automation? Can every manual test be automated? How to get started with a Test Automation? What are some of the strategies and best practices? These were some of the questions that were answered in this webinar. We also covered how AI/ML has been helping in implementing autonomous testing.
Here is the recording of the webinar-
Below you will find answers to the questions asked during the webinar-
We always find it difficult to find the right test data and many times we had to cancel the testing. this is because of many factors such as, no access to prod data, not able to mock the data, not able to scrub..etc. How do you handle such situations or can automation play any role in such cases?
Test Data is key in evaluating systems and ensuring they are working as expected. So setting up and maintaining test data is critical for testing. There are many ways this can be done-
- Have separate QA servers and ensure all the test data needed for testing is set up here. We need to ensure the test data resembles data in production as much as possible, to mimic true behavior while testing. This QA server should not be touched by anyone except for testers. Procedures need to be put in place such that only the admin can change the test data to ensure effective testing
- Have separate config files or databases that houses all the data. This makes editing and maintaining test data really easy compared to having it stored in different locations of the system
- Use automation to test different modules with different data sets. You can have one test that runs through different data sets. This will be otherwise hard to do manually
- If possible mock API requests for getting faster feedback instead of waiting for API’s to be implemented
- If there are situations where there are no test data available, take the help of developers to come up with a plan to have mock test data or test these modules via developer testing from the code
I want to know more about handling changing requirements in an agile world?
Scope creep in requirements is one of the biggest problems in software development. There are various ways to prevent scope creep and ensure features are developed and tested on time. Here is an article which describes how do this – https://www.infoq.com/articles/quick-guide-atdd/
What is price of Testim and does the tool have manual or tutorials?
Our pricing is based on the number of test runs. It varies based on the number of browsers, grid set up needed, whether you are choosing the cloud, tunneling or On prem solution and a number of other factors. So, it is best to contact us here -> https://www.testim.io/contact-us or get in touch with our 24/7 customer support for more accurate pricing information.
We created a robust help documentation repository that has answers to the majority of questions people may have, when they initially start using Testim. Check out the help docs here – https://help.testim.io/docs.
Also, the below blog posts describes how we differentiate ourselves from other tools-
Are there any open source tools for AI/ML automation that is well known?
Currently there is none. But, I won’t be surprised if something comes up in the upcoming months/years.
I am new to automation and still learning can you recommend where to find more information about Design Patterns and Principles Please?
When it comes to automation design principles, we need to ensure the tests are-
- Reusable
- Have single responsibility
- Maintainable
- Extensible
The different strategies that go along with this are Data Driven Testing, Parameterization and following Page object pattern to structure your automation code.
If you google for any one of these keywords you will find more details about them. There are numerous resources on the web.
Assuming JIRA is used, how do you link test cases to requirements? Assuming Stories are used and not typical requirements?
There are numerous ways JIRA can be used along with stories. It depends on the context. So, it is hard to give one single answer to this question. Based on my experience, here are some possible options-
- Have the requirements in Wiki Pages. Then add the link to the requirement in the story to show the connection between the two
- Have the requirements in JIRA. Then have an Epic story that relates to this requirement. Then, have smaller stories linked to the epic story focusing on different parts of the requirement
- You could use different tools like Test Rail, Zephyr etc to help in the process
Do we assure to automate 100% of the functionality of some app? I always knew any app can’t reach 100% of automation coverage, however, the CEO of a company is very confident that automation would solve all problems, finding bugs and going forward with all functionality?
Saying you can automate 100% of the functionality is like saying you know everything about computers. This is impossible. How can we know every user action a user would perform on our application and automate those scenarios before hand? There are always going to certain functionalities or user flows we did not think about and probably has not been automated.
The above being said, it is important to understand the scenarios automated testing could be a good option. Here are some use cases for it-
- To automate mundane/repeatable tasks that are really time consuming to do manually
- To operate in a DevOps environment where we need automation right from the start of the SDLC; till the release and production monitoring phase
- To get quick feedback about the system when new code is checked in (new feature is implemented). We will have automated tests that get automatically triggered for every code check in
- To run several tests on a daily basis to ensure the older functionalities of the system are still working as expected
- To create Test Data that will be used for doing manual exploratory testing which is otherwise time consuming to do manually
- To test different fields with hundreds of data sets using data driven testing
- To do load/stress testing where we simulate thousands of users using the application simultaneously which would otherwise be hard to do manually
Automation is not a one size fits all solution. It is not a silver bullet that can solve all the testing problems. It is just an aid in the overall testing process. This being said, it is important to remember that, not everything can be automated and in trying to do so, we will move the focus away from actual testing and finding bugs.
There are many scenarios that cannot be automated or if automated may be unstable and provide less value. Some of them are-
- Using automation for catching rendering issues in the application (look and feel) is a bad idea. There are few tools out there which does visual validation but it is really difficult to replace humans in this aspect. For example – I had a scenario where my mobile web page looked white on one Mobile phone, whereas it looked dark grey in another mobile phone. Yes, we can try to automate this but I think humans would be better at finding these subtle differences in look and feel of the application
- Using automation to figure out an element location on a page is a bad idea. The automated tests will become unstable if we start writing tests based on x,y coordinates of elements, as the web page could be viewed in different browsers, devices and OS’s and the coordinates are going to change based on the screen size.This means the automated tests are going to be really flaky.
- Using automation to test integrated systems which involve software, hardware, web services, API’s and cloud services all communicating in real time with each other would be a bad idea. For example – How would we write an automated test which tests all the end to end scenarios of fitness trackers like Fitbit? We can try as hard as possible to simulate real human movements and mock services, but it is going to be a really difficult task to automate the entire process of a fitness tracker. We could rather have real humans do exploratory testing in parallel to some automated tests
Can you suggest me any best tool can be used for Multimedia device (multimedia device in Car) testing?
It is hard to answer this question without knowing the context of what you are trying to test in the multimedia device. Are you testing the hardware or software portion of it? Is it possible to use simulators do that or everything has to be done on a physical device?
Can we use data driven testing using testim for local run?
You can set up your test data with different data sets and can ensure the test is able to interact with these data sets by running locally. But, when run locally, the test will go through only the first data set in the list. If you want to run through all the data sets, then you would run it on the grids using our CLI. The reason we designed it this way was, people were trying to run hundreds of datasets locally and it was consuming time and space of the local machine. So, to prevent this from happening, we encourage users to run the tests with datasets on the cloud as it is much faster.
Can testim locate canvas object?
Currently we do not support canvas testing but it is something you can expect Testim to support in the future.
