Our report on the major outage on August 10, 2025 in EYERCORD services
On August 10, EYERCORD's IT infrastructure experienced a major outage, which temporarily made most of our products unavailable. We have fixed the consequences of the outage and now all our products are working as usual again, but we also want to explain to you - our users - what exactly caused the interruption in service and what we have done to prevent similar outages in the future.
This article will contain some technical terms that will be referenced by external authoritative sources such as Wikipedia to explain these terms for a general audience to understand.
What's happened?
During the night of August 10, 2025, our IT infrastructure encountered an issue affecting one of its critical components — the Core APIs¹. As a result, most of our products became unavailable, since the majority of them depend on the Core APIs¹ for operation.
What the Core APIs¹?
Core APIs¹ are one of the main components of our IT infrastructure, accessed by the majority of our software products. This component is responsible for common functions used across many of our services, such as paid subscriptions, spam database checks, determining a user’s region by IP, certain parts of the authentication system, and more.
Because we operate a wide range of products with different purposes and complex architectures, we are required to develop numerous auxiliary modules to support their operation. As we strive to continually improve our products and release updates more quickly, we must also focus on accelerating software development.
In addition, some of our services use similar functions, such as authentication or handling paid subscriptions and purchases. We aim to unify all our modules to reduce the number of potential points of failure and to enable rapid, efficient implementation of feature and security updates.
To achieve this, we created the Core APIs¹ — a consolidated collection of shared APIs¹ that can be used across many of our products. Some of these APIs¹ may also be made available to third-party developers if we determine they can help in building and improving external applications.
This approach has both strengths and weaknesses. One of the weaknesses is the significant impact on operations if the Core APIs¹ fail.
Impact on our services
Given the explanation above of how and why our services interact with our Core APIs¹, a reasonable question may arise: how significant is the impact of this component on our products, and why, when it fails, are most of our services unable to continue operating?
There are several reasons for this. One of them is that our products are developed entirely from scratch by our own engineers, from the lowest level to the highest. When we have a task to implement specific functionality, in order to follow best coding² practices and minimize duplicate code, we first utilize functions from the Core APIs¹ whenever possible. Only if the required functionality is not available there do we develop a custom solution from scratch.
As a result, our products rely on Core API¹ functions at various levels — including critical ones such as the core of the product or essential microservices³ required for its operation. When the Core APIs¹ experience an outage, the product’s code encounters an error when attempting to send a request and receives no response. This prevents further execution of the command, ultimately resulting in an error visible to the end user.
What happened to the Core APIs¹?
Our Core APIs¹ are a highly complex system that play a central role in the operation of many of our products. While they are designed to handle significant workloads, some parts of the codebase were originally built for different usage patterns and lower traffic volumes, which can make them less efficient under certain conditions.
In this incident, an unexpected surge in traffic triggered increased resource consumption in one of the Core APIs¹’ processing paths. This inefficiency led to a depletion of available server resources, ultimately causing the Core APIs¹ to become unavailable.
Because the current Core APIs¹ operate in a predominantly monolithic⁴ architecture, an issue in one area can impact the availability of the entire system. As a result, many of our dependent services also experienced disruptions during this event.
What have we done to prevent similar outages in the future?
At EYERCORD, we have always aimed to build reliable, useful, fast, secure, and user-friendly services for all our users. Following this outage, we have made it a top priority to rewrite the outdated and inefficient parts of our Core APIs¹, conduct a comprehensive end-to-end review and testing of the entire system, and redesign its architecture to prevent a single module failure from affecting the entire platform.
This is a significant undertaking, but we have already begun the work and are committed to keeping you informed as we move toward a more stable and resilient Core APIs¹ in the future.
Conclusion
Our products experienced service outage due to a outage in one of the most critical components of our IT infrastructure. We sincerely apologize to all our users who were affected by this outage. We are actively working to prevent similar outages in the future and will share updates in this blog once the work is complete.
Glossary
- API - A connection or fetching, in technical terms, between computers or between computer programs. It is a type of software interface, offering a service to other pieces of software.
- Coding (or Computer Programming) - Computer programming or coding is the composition of sequences of instructions, called programs, that computers can follow to perform tasks.
- Microservice - In software engineering, a microservice architecture is an architectural pattern that organizes an application into a collection of loosely coupled, fine-grained services that communicate through lightweight protocols.
- Monolithic application - In software engineering, a monolithic application is a single unified software application that is self-contained and independent from other applications, but typically lacks flexibility.