Tag Archives: troubleshooting

devops

Everything Sucks – Managing IT Risks: Strategies for IT Professionals.

As someone who has worked in the IT industry for many years, I have realized that technology is far from perfect. In fact, I would go so far as to say that everything sucks when it comes to technology.

IT professionals constantly deal with a never-ending barrage of issues, from unexpected hardware failures to software bugs and infrastructure breakdowns. It is Murphy’s Law all the way.

And while we often joke about the shortcomings of operating systems like Windows, even the most reliable and robust systems like Linux are not immune to bugs and glitches. The sheer complexity of software development means that dozens of bugs are likely lurking in every thousand lines of code, making it impossible to catch them all.

It is everything

But it’s more than just problematic software. Even the best hardware can fail unexpectedly, despite companies spending large sums on the latest and greatest equipment. Mean Time Between Failures (MTBF) might offer some guidance, but it’s often a source of delusion rather than certainty.

And when it comes to infrastructure, the fragility of the Internet can be mind-boggling. For example, one broken fibre cable in Egypt caused widespread disruption to millions across Africa, the Middle East, and South Asia. Given the countless potential points of failure and the constant threat of cybercriminals, it’s a miracle that the Internet works at all.

And let’s not even go into all the problems around Border Gateway Protocol (BGP), which is a fundamental protocol that helps keep the Internet running. It is based on trust rather than security. This means that every network operator must trust the information provided by others, even if they have no direct relationship with them. What could possibly go wrong, right?

But not all is lost

Despite all these challenges, there are ways to mitigate the risks and prepare for the worst.

It’s important to perform risk analyses and prioritize resources accordingly. While protecting against every potential threat is impossible, it’s crucial to focus on the most significant risks and allocate resources accordingly.

Performing risk analysis is a critical step for any IT professional in preparing for the worst. It involves identifying potential risks and evaluating the likelihood of those risks occurring, as well as the potential impact they could have. By conducting a risk analysis, IT professionals can better understand where their systems and infrastructure are vulnerable and prioritize resources accordingly.

Risk Matrix

One common risk analysis method uses a risk matrix, which assigns likelihood and impact scores to various risks to determine their overall risk level. Once the risks have been identified and evaluated, IT professionals can develop strategies to mitigate them and prepare for the worst.

Wikimedia

For example, if a company relies heavily on a particular system, it might identify the failure of that system as a significant risk. They could then develop a backup plan, such as having redundant systems or backup servers, to minimize the impact of a potential failure.

It is a continuous process

It’s important to note that risk analysis is an ongoing process. Risks can change over time, and new ones can emerge, so it’s essential to regularly review and update risk analyses to ensure that IT professionals are always prepared for the worst.

IT professionals must acknowledge technology’s flaws and take action to prepare for potential risks. By performing risk analyses and prioritizing resources, we can develop effective strategies to minimize the impact of unexpected challenges and ensure critical systems remain operational. Let’s make risk analysis and mitigation strategies a priority in our work and ensure technology works for us.

troubleshooting

Troubleshooting with the OODA Loop

The OODA Loop, a decision-making technique developed by the US Air Force for combat operations, is a helpful framework for troubleshooting. By following the four phases of the OODA Loop – Observe, Orient, Decide, Act – and repeating the process if necessary, you can break down complex problems into manageable steps and make informed decisions to find a solution.

Here’s how to apply the OODA Loop to troubleshooting:

Observe: The first step is to observe the problem. Gather as much information as possible about the issue, including symptoms, errors, and messages. What else do you know about the problem? When did it start? What is the scope of the problem? What other pertinent information can you gather at this early stage? During this phase, it’s essential to focus on gathering quantity over quality. The more information you have in the beginning, the better. Leave to the decision phase on what to do with all this information.

Orient: Once you have observed the problem, the next step is to orient yourself to the expected or desired state of the system. This phase may be straightforward if you are troubleshooting something familiar. You can use your knowledge of the system to determine the normal state. However, you will need to gather information if the troubleshooting is in an unfamiliar system. The system’s end-users are often the best source of information, but documentation can also be helpful. In some cases, you may need to infer some things.

Decide: With the information gathered during the observation and orientation phases, it’s time to decide on an action plan. Keep in mind that, at this point, it is not a 100-step action plan that is sure to solve the problem. Don’t forget that we are working on a loop here. Plan something straightforward and easy to implement, just a single step to help you move further in the process. Make incremental changes, and reevaluate before moving forward.

Act: With the plan in place, it’s time to take action. Implement the plan and take the necessary steps. This could involve changing a setting, restarting a service, enabling more detailed logging in the application, or adding a breakpoint somewhere. Whatever you decided in the previous phase, do it.

Repeat: If the problem persists, enter the loop again and go through all the phases. As you observe the problem again, ask yourself if anything has changed. What is different at this moment? During the orientation phase, you need to understand the cause and effect of the problem. How did your changes during the last interaction cause the observed behaviour? Does that make sense? Have you seen similar problems? Can you infer any conclusions from what you observe? Based on your findings, decide what to do next. You may already know enough to try to implement a solution, or you may need to gather more information. Act again and return to the loop as necessary until you find a solution.

Remember, the OODA Loop is a continuous process that allows you to quickly and effectively tackle complex problems by breaking them down into manageable steps. Using this framework, you can make informed decisions based on the information you gather, act on those decisions, and continually re-evaluate until you find a solution. So, next time you’re faced with a troubleshooting challenge, try applying the OODA Loop to guide your decision-making. You can improve your problem-solving skills and become more effective at finding solutions with practice. Give it a try and see how it works for you!