Namespace

Namespace / History

History

Aug 2026-Nov 2026
November
04
Wed
Increased queue time
2:15 PM

All systems are operational.

Increased queue time for GitHub jobs
11:55 AM

All systems are operational.

October
30
Fri
Elevated queue time for GitHub Jobs
2:34 PM

All systems are operational.

29
Thu
Elevated control plane error rate
10:20 AM

We confirm seeing full recovery across the board.

28
Wed
Increased error rate on Registry and Storage operations in Virginia region
2:16 PM

All systems are operational.

A subset of dependency fetches see stalling connections
10:57 AM

All systems are operational.

27
Tue
Elevated queue time for a subset of GitHub runner jobs
10:53 AM

All systems are operational.

26
Mon
Degraded network performance for a subset of routes
1:22 PM

The team has addressed the networking issue and traffic is again allowed into the European compute shards. All systems are operational.

23
Fri
Ingress disruption in iad
8:15 AM

All systems are operational.

22
Thu
Elevated queue time for GitHub Jobs
12:02 PM

All systems are operational.

21
Wed
Elevated queue time on GitHub jobs
4:58 PM

All systems have recovered, and jobs are being picked up normally again.

The influx of jobs once GitHub events recovered lead to increased pressure on our systems and fleet,
resulting in a longer impact time than expected.

Increased error rate when starting Remote Builders, Bazel Clusters and Devboxes
2:35 PM

The rollback is complete, and the error rate fully recovered.

20
Tue
Increased queue time for jobs using macOS runners
6:47 AM

We confirm seeing full recovery.

19
Mon
Delayed events from GitHub leading to increased queueing time
1:29 PM

We confirm seeing full recovery.

Degraded registry and artifact performance in Western Europe region
10:21 AM

We've applied a mitigation and are seeing full recovery.

Increased queue time for subset of jobs
6:41 AM

All jobs have been picked up, queue time is back to normal.

17
Sat
Failing instance lists and a subset of RPCs
3:26 AM

All systems are operational. The orchestrator in affected regions was not able to retrieve update configuration from our configuration sub-system, leading to failures.

15
Thu
Increased queue time
7:10 PM

All systems are operational.

Delayed processing of instance logs
12:36 PM

We've applied a mitigation and are deploying additional capacity, processing is back to normal.

09
Fri
GitHub is rejecting a subset of job acquisitions
5:24 PM

All systems are operational.

07
Wed
DNS resolutions failing across all instances in linux/arm64 in iad3-a (North Virginia)
1:33 PM

All systems are operational.

06
Tue
Degraded networking performance
7:26 PM

All systems are operational.

Increased queue time on mac instances in a limited set of workspaces
9:56 AM

Queue time has recovered.

04
Sun
Devbox access is failing on tenant resolution
10:19 PM

All systems are operational.

02
Fri
A subset of devboxes is failing to start
3:51 PM

All systems are operational.

01
Thu
Degraded registry and artifact storage performance
3:28 PM

All systems are operational.

SAML SSO logins are failing
6:53 AM

All systems are operational.

September
30
Wed
Delayed runner startup
9:36 AM

All jobs are being processed normally.

24
Thu
Small subset of artifact and cache requests in Illinois region failing
2:45 PM

Error rates have recovered, all systems operational.

23
Wed
Elevated instance creation time
9:54 AM

All systems are operational.

Elevated instance creation time
5:30 AM

We've confirmed applied mitigations have lead to full recovery of the affected compute shard.

22
Tue
Elevated instance creation latency
11:23 AM

All systems are operational.

16
Wed
Subset of container image pulls from external registries failing
5:15 PM

The fix addressed the root cause and all images are being served correctly.

14
Mon
Ongoing maintenance
8:52 AM

All systems are operational.

10
Thu
Increase in queue time of jobs in Virginia (iad-2 and iad-3)
7:57 PM

All systems are operational.

09
Wed
Extended GitHub queue time
2:02 AM

All systems are operational.

06
Sun
Instance pages failing to load
3:20 PM

All systems are operational.

03
Thu
We're investigating reports of GitHub connectivity issues
3:22 PM

All systems are operational.

Network disruption in iad2
2:41 AM

All systems are operational.

02
Wed
A subset of control plane calls were failing, including job starts
11:21 AM

All systems are operational.

August
26
Wed
Elevated queue time for GitHub action jobs
2:20 PM

All systems are operational.

Network availability
10:49 AM

All systems are operational.

19
Wed
Elevated queue time for a small set of GitHub Action jobs
8:00 AM

All systems are operational.

17
Mon
Elevated error rate from GitHub API
4:38 PM

All systems are operational.

16
Sun
Some IAM calls are failing
9:38 PM

All systems are operational.

12
Wed
Increased queue time for subset of GitHub jobs
3:01 PM

We've processed the backlog of jobs.

All systems are operational.

Elevated error rate in our control plane
12:15 PM

We've confirmed the fix resolves the root cause and we're seeing full recovery.

10
Mon
Elevated errors during logs serving
8:14 PM

All systems are operational.

Powered by

This page lists incidents by Namespace and known upstream issues.

Privacy policy

·

Terms of service