setback in a two-root-cause bug

(Updated: )
Setback

A recent setback in a two-root-cause bug investigation due to ignoring one of the root causes.

The Issue

One of the module’s services relies on a global configuration that should be set by a script after the system boots. If the configuration isn’t loaded correctly, it will cause the service to run incorrectly.

The Root Causes

    1. The script is not robust. It leverages a third-party tool to query some information from the system, which may not be 100% correct if we query it right after the system boots.
    1. The script is not able to execute before the next reboot. Either the script is too slow to load and execute, or the system reboots so quickly that it terminates the script. However, this root cause was found too late.

Background

    1. We had a long holiday shortly after this issue was reported, meaning everyone would take time off soon.
    1. The issue was reported at the last minute.
    1. A very tight schedule and the green light before the holiday meant working under stressful conditions.

Result

    1. I didn’t analyze this issue deeply.
    1. I trusted and relied heavily on AI and didn’t give myself enough time to analyze it.
    1. I tried several rounds of optimization on the script, but they only fixed root cause 1. Therefore, I kept receiving unexpected results due to root cause 2.
    1. Luckily, I figured out another workaround to unblock the bug.

Timeline

Date Events
9.29 The issue was found during validation.
9.30 The last working day before the holiday. In the morning, I had a discussion with a coworker and figured out a solution. At noon, I tried optimized script 1 but received an unexpected result. It was due to root cause 2, but I didn’t have a chance to double-check with my coworker again. I also neglected to analyze it deeply, so I didn’t figure out root cause 2.
10.1 I requested that optimized script 2 be tried. I didn’t follow up on the issue, thinking it should have fixed the problem 100%.
10.2 At night, I got an unexpected result from script 2 and started to analyze it.
10.3 I received the log from script 2 in the morning and tried follow-up scripts 3 and 4, which only hardened the fix for root cause 1. However, I still didn’t realize there was a root cause 2. In the afternoon, I tried another workaround, leaving the script issue unfixed and using the workaround to unblock the green light.
10.4 Sunday, a rest day for everyone, including the factory.
10.5 The build started with the workaround.
10.6 I took a rest with only two days of the holiday left.
10.7 I took a rest with only one day of the holiday left.
10.8 Everyone returned to work. I got a chance to talk with my coworker again. He thought the first optimized script should have fixed the problem permanently, so I started to review the log again and then figured out root cause 2.

Conclusion

I am sure I could have handled this better if I had had more time. The lessons learned are:

  • Analyze every issue calmly and deeply.
  • A problem may have multiple root causes rather than a single one.