|
Backoffice & OSS
Industry View
Bridging the divide between testing and operations
Automated diagnostics is a logical next step for improving product quality
by David Gehringer, Fanfare
Dictionary.com defines diagnostics as “serving to identify or characterize; being a precise indication“—in other words, something done to find the source of a problem. In the technology industry, people who use diagnostic tools tend to be end users who cannot figure out why their device or software is not working. Why are so many problems being diagnosed after the product has been released—and not while the product is being built? Why not try to identify and solve these problems earlier?
Testing, Diagnostics, and Automation
Given the complexity of today’s products and the increasing demand for quality, we need to consider a new approach to diagnostics. It’s time to think about moving diagnostics upstream to the testing phase and fully integrating it into the QA process. In some ways, we are already headed in this direction, with the features and use models of testing and diagnostics becoming increasingly similar. By automating diagnostics, we can facilitate a convergence between testing and diagnostic technologies, and create standardized processes that can be leveraged by all groups, at all stages of production. More importantly, we can bridge the gap between testing and operations to maximize the productivity of both groups
Why Automate Diagnostics?
In many ways, the reasons why we need to automate diagnostics are very similar to the reasons why we needed to automate testing. For years, developers had the benefit of automated tools to help them more efficiently conquer everyday tasks. Meanwhile, testers were left struggling to keep pace with development and rising testing requirements. Their manual processes could not scale, and their attempts at home-grown automation solutions proved to be maintenance nightmares. Today, organizations can leverage advanced automated testing tools to test faster and more effectively, while enabling all groups to contribute to automation. Automated testing also has allowed testers and developers to work together more efficiently by enabling testing to take place earlier in the workflow, sometimes even during development.
Like earlier generations of testing tools, most diagnostic tools rely mainly on manual processes. As product complexity soars, so does the number of diagnostic tests that need to be run. As with testing, manual diagnostic tools simply cannot scale, and automation offers not only the promise of efficiency through standardization, but also enhanced productivity. Automated diagnostics is therefore a logical next step for further streamlining production and improving product quality. And once these tools and processes are adopted, they can be integrated at any point in the workflow, including during the testing phase.
How Automated Diagnostics Works
To understand how automated diagnostics would work, let’s first look at sophisticated automated testing, since my claim is that they are very similar. Among its many capabilities, automated testing software can run tests unattended, schedule test runs, determine whether the test cases passed, provide a report, and, in some cases, notify the tester of the conclusions. With advanced automated testing, we can insert a significant amount of logic into the tests. For example, we can specify that if the value of the target data is out of range, the test should grep that device’s log. At the same time, these tools follow the same logical processes that a tester would if he discovered a problem during manual testing. If a test fails, automated testing tools attempt to determine the source of the problem and the conditions that prompted it. This data is then sent to engineering to help them locate and resolve the bug.
In many ways, automated diagnostics would take a similar approach to problem solving—from troubleshooting to the repair process. Imagine we are on a live system, and e-mail goes down. We run our automated e-mail diagnostics, and seconds later it shows a problem in the firewall. It offers to execute a set of actions to reconfigure the firewall, which it does, and then reruns it to verify that the e-mail problem has been fixed.
Contrast this with what usually happens with manual diagnostic tools. If engineers are tending to a live system and an issue occurs, they look at the alerts, reboot the system, and later sift through the log files to find the cause. Or, they run one test at a time, manually analyze the results, and hopefully select the right tests to help identify the source of the issue. By automating diagnostics, you can run multiple tests simultaneously, speeding execution and analysis, while following the same fundamental steps of manual diagnostic tools.
A Modular Approach to Diagnostics
Today, the increasing complexity of applications and their corresponding network infrastructures is driving the need for more basic diagnostics. Engineers must juggle numerous diagnostic tool sets because each tool can perform only one task. Like testers, they could work more efficiently if they had a single tool that could tackle many problems at the same time, such as: Why is e-mail slow? Why is the server down again?
That’s why I think that automating diagnostics is only the first step. Creating diagnostic tools that can multitask is the second step. Rather than having individual diagnostic tools, I envision engineers working with a stack of diagnostic modules or procedures focused on specific tasks. An engineer could use one module to automate the discovery of the issue and another module to pinpoint the cause--from hundreds of devices and thousands of configuration options.
This automated, modular approach to diagnostics would save time and provide the foundation for a consistent methodology.
Bridging the Divide Between Testing and Operations
To create powerful automated diagnostic tools, we need the same features found in powerful automated testing tools: logic, the ability to interface at the application and device hardware levels, transferability, easy maintenance, abstraction, and robust analysis. It makes sense, then, to bring together all of these features into a single set of automated tools that can be used by both testing and operations teams. They can use the same tools for pre-deployment and quality testing as well as for post-deployment diagnostics.
To further bridge the divide between testers and operators, we need to ensure that automated testing and diagnostic tools are built on the same platform. This will enable developers, testers, and operators to share automated test and diagnostic modules. By sharing critical test assets throughout the workflow, these groups will be able to improve communication and speed problem resolution.
For example, when testers finish running black box and interoperability tests they can share these test assets with the operations team. The operations team can then leverage those tests as they deploy in an interoperability lab. Moreover, the operations team can layer on its real-world operation models and then pass those back to testing and development. This creates an even more robust quality process, improving quality at development and throughout the rest of the process.
Looking Ahead
By automating diagnostics along with testing, organizations stand to increase their overall efficiency and catch problems before they escape to customers. In addition, testers and operators will have processes and tools to help them maximize their productivity at every stage of the quality workflow.
With organizations actively seeking ways to improve their efficiency, automated diagnostics might not be too far off in the future. It’s possible that this will result in only a new set of automated tools. But I suspect that automating diagnostics will actually expand the role of testers, providing them with a way to further their careers, expand their skill sets, and increase their value to the company.
David Gehringer is the Vice President of Marketing for Fanfare. He holds bachelor's degrees in mechanical engineering and aeronautical engineering from the University of California, Davis.
|
|
|
