How to Triage a Bug
A bug report rarely includes enough information to pinpoint the issue at first glance. Most of the time you get "invoices aren't sending" or "the dashboard is slow for some customers." It turns you into a detective as you triage the issue.
Triage is the work between the report and the eventual fix. Your job is to figure out what's going on, how many users are affected, and what the fix should be.
The problem is that most software engineers never learn how to triage an issue and escalate it appropriately. Worse, most engineers never learned to connect the dots across systems like stack traces, job queues, and vendor dashboards to solve the mystery.
So, how can you improve your triage skills and make sure everyone is informed during your next SEV incident?
1. Start with specific examples
You need real examples to understand what's going on. "Invoices aren't sending" gives you nothing to search for. You end up searching for a needle in a haystack. "Invoice 48213, created for ABC customer on Tuesday at 9:14 am Central never sent" helps you start identifying the issue.
Ask whoever reported the bug for:
- One specific record, such as a user ID, order number, or request ID. Anything the system stores will work.
- A timestamp with the time zone. "This morning" doesn't help when your logs are in UTC.
- The exact error message or a screenshot. A summary of the error loses the details you'll search for.
- Where it happened: production or staging, which region, web or mobile.
Support and product teams usually have these details or can get them in minutes. If you get several examples, start with the most detailed one and check the others against it later.
2. Figure out how big the problem is
The number of affected users determines how fast you need to move and who needs to know. Answer these, even roughly:
- Who's affected? It could be one customer, one segment, or everyone.
- When did it start? Find the first occurrence, not just the most recent one.
- Is there a workaround customers can use today?
- Is it getting worse, or holding steady?
- Is anything at risk that can't be undone? Corrupted data, missed payments, and security exposure fall into a different category.
Google's SRE book argues that your response should match the impact. One customer with a workaround can wait for normal sprint planning. Spreading data corruption requires stopping the damage now and finding the cause later. If you're in the second group, escalate first.
If you can only reproduce it on your own machine or test account, rule out your own setup before you go further.
3. Follow one example through the stack
Start at the beginning. Follow it one hop at a time until something doesn't look right. For the invoice example, the path looks like this:
- The application logs show the invoice was generated and an email job was queued.
- The job queue shows the job ran and finished without errors.
- The email provider's activity log shows the message was received and then dropped.
By the third hop, you know the invoice code works. The problem is somewhere between your system and the customer's inbox.
The systems you'll need are often ones you rarely open:
- Log search and tracing tools show what each service did with the request.
- Background job and queue dashboards show whether async work ran, retried, or failed quietly.
- A read-only database connection confirms what was actually saved.
- Third-party dashboards for payments, email, auth, or search show what happened after the request left your system.
- Configuration, environment variables, and feature flag settings show how the system was set up when it failed.
- The support history for the same customer shows whether this has happened before.
Correlation IDs
Many systems tag each incoming request with a unique ID and pass it to every service that handles it. If yours does, search each tool for that ID instead of matching timestamps by hand.
OpenTelemetry's overview of traces explains how this works.
4. Find what changed, including outside the code
Google's SRE team found that roughly 70% of outages come from changes to a live system. The git log only shows code changes, so check the rest too:
- Deploys from other teams whose services yours depends on.
- Config and environment variable updates.
- Feature flag changes and new targeting rules.
- Vendor changes, such as a new API version or a deprecated endpoint.
- Data changes, like migrations, bulk imports, or a new customer with unusual records.
- Things that expire, including certificates, API keys, tokens, and permissions.
In the invoice example, the code hadn't changed in a month. Marketing sent a re-engagement campaign Monday morning from the same email account. The bounces and spam complaints put a batch of addresses on the provider's suppression list, and that list blocks invoices the same way it blocks newsletters.
Compare these changes against when the problem started. A change right before the first failure is a strong lead. Confirm it before you call it the cause, since two things happening at the same time doesn't mean one caused the other.
|
|
Where can AI save you time?
My friends at Big Creek Growth put together a quick survey to spot the repetitive work you can hand off to automation.
|
|
5. Dig on your own, then ask the owner
I recently wrote about how long to dig before asking for help in The Goldilocks Rule for Asking for Help. In triage, dig through anything you can read on your own. How long you keep going depends on what you found in step 2.
Look through logs and dashboards, and run read-only queries yourself. The system's owner can give you a better answer once you've done that. Ask when you need:
- Access you don't have. Request it through the normal channel, and don't borrow someone else's credentials.
- To change anything in production, even if you're "just testing" a theory. The SRE book lists unsafe changes made while testing a hypothesis as a common way troubleshooting goes wrong.
- To know what normal looks like.
Owners know what normal looks like, and you don't. An error that fires 2,000 times a day might be harmless and unrelated to your bug. A 400ms response time might be slow or perfectly fine.
Compare two versions of the same question:
"Is the email service broken?"
"I'm tracing a missing invoice email. The provider's log shows it was dropped because the address is on the suppression list, and the list grew by about 1,200 addresses on Monday. Is that normal for us, or does it line up with something on your side?"
The second one shows what you've already checked and asks the one question only the owner can answer.
6. Take notes as you go
Keep notes in the ticket or a shared doc. Each entry needs the time, what you checked, a link or the exact query, and what you found.
A sample triage trail
9:40 Reproduced with invoice 48213. Invoice created, email job queued. (log link)
9:52 Job completed with no errors. (queue link)
10:05 Provider log: message dropped, recipient on suppression list. (link)
10:20 Checked deploys and config changes since Friday. Nothing in billing or email. Ruled out.
10:35 Suppression list jumped Monday afternoon. Asked marketing ops about Monday sends.
Include what you ruled out. The SRE book argues that negative results are worth recording because they keep the next person from repeating dead ends.
Notes also show you where you left off after an interruption. They let someone else take over without starting from scratch, and they give you most of your summary when you're done.
7. Write a summary someone can act on
Write the summary so your lead, a PM, or the team that owns the system can approve a fix without redoing your work.
A triage summary
What's happening: the observed behavior, with links to the evidence.
Impact: who's affected, how many, since when, and whether there's a workaround.
Likely cause: what you think is going on, and how confident you are.
Ruled out: what you checked that wasn't the cause.
Options: what you can do now and later, with the tradeoffs of each.
Recommendation: what you'd do and who should own it.
Open questions: what you still don't know.
Keep what you saw separate from what you think. "The provider dropped 340 invoice emails since Monday" is something you confirmed. "The marketing campaign caused it" is your best explanation. Label each point as confirmed, likely, or a hunch, so readers know which parts they can rely on.
Separate the immediate fix from the lasting one. Triage often ends with a mitigation, such as a rollback, a disabled flag, or a manual fix to the affected records. The proper fix can follow on a normal schedule, sometimes owned by another team. In the invoice example, the immediate fix is to clear the affected addresses and resend this week's invoices today. The lasting fix is separating transactional and marketing email so one can't block the other. You can do the first today. The second belongs to whoever owns the email setup.
Every bug you triage teaches you another part of the system. Next time a ticket lands, you'll already know where to look in the logs, how the queue dashboard works, and who owns the email settings. Do that enough times, and you become the person people pull in when a bug crosses three teams.