FACEIT Derek (@FaceitDerek) header
FACEIT Derek (@FaceitDerek) profile picture

@ me your thoughts about FACEIT Matchmaking. Lead of Matchmaking & League @ FACEIT

Joined May 2024
Photo by FACEIT Derek (@FaceitDerek)Photo by FACEIT Derek (@FaceitDerek)Photo by FACEIT Derek (@FaceitDerek)Photo by FACEIT Derek (@FaceitDerek)
NA: New York, Atlanta, Seattle, Los Angeles EU: Istanbul, Warsaw, Bucharest, Madrid This is our final list of new locations coming with Season 10. Which are you looking forward to having the most?
Announcing the final two new FACEIT server locations coming to Europe, joining Istanbul and Warsaw. Live December 09 with Season 10. Follow each update as it is announced at fce.gg/s10-x
32
1
126
9,047
More EU locations to be announced... but excited to finally get to share that Istanbul will be live with Season 10. What locations will be next?
Announcing the first new FACEIT server location coming to Europe. Live December 09 with Season 10. Follow each update as it is announced at fce.gg/s10-x
6
7
566
FACEIT Derek retweeted
Announcing the four new FACEIT server locations coming to North America. Live December 09 with Season 10. Follow each update as it is announced at fce.gg/s10-x
75
29
26
446
99,705
FACEIT Derek retweeted
Darwin genuinely gives a damn about FACEIT and the community. Been grinding for years, checking reports even outside work hours. He might not be making the final calls, but trust me, he does his best to push for the right ones.
Replying to @intex62
Hey, I saw the tags about this case late last night, and I can see a colleague had also raised it internally around the same time. First of all, my thoughts are with the victims and everyone affected by what happened in Kraków. I'm genuinely sorry that people have also had to deal with others mocking such a terrible tragedy while playing on FACEIT. I completely understand why this has upset so many people and why stronger action is being called for. Just to clarify my own role a little, I work within the marketing team, and while I can raise cases internally and share my opinion on what I think should happen, I don't make the final decisions on punishments or ban lengths. That's handled separately by our Trust & Safety team. I've already raised the negative sentiment surrounding this particular case, and I'll continue to pass on the concerns being shared by players. While I'm often the person communicating with you publicly, and I can raise concerns and share my own thoughts internally, I'm not necessarily the person responsible for the decisions being made behind the scenes.
8
5
132
17,112
Do you want to get faster into the game or keep it traditional?
Announcing the knife round vote, where you decide whether we keep the knife round or pick sides at random. The winning option goes live as a test in Season 10 on December 09. Vote by November 12 fce.gg/knife-vote-tw
16
26
3,327
FACEIT Derek retweeted
Announcing the knife round vote, where you decide whether we keep the knife round or pick sides at random. The winning option goes live as a test in Season 10 on December 09. Vote by November 12 fce.gg/knife-vote-tw
96
21
27
405
204,520
FACEIT Derek retweeted
Introducing automatic server matching, a new way FACEIT will pick the server for your match by rating each location on your and your party’s live and historical ping, jitter and packet loss. Live December 09 with Season 10. fce.gg/s10-x
69
40
23
619
143,317
FACEIT Derek retweeted
Season 9 has been out for a little while now, so I want to hear your thoughts on the updates 👀 What are your thoughts so far on: • Seasonal reset • Seasonal stats • Reloaded • Sound alerts • Commendations Let me know👇
217
6
1
285
52,512
FACEIT Derek retweeted
Introducing Seasonal Stats, a new profile section that shows your stats and Elo milestones from previous FACEIT seasons. We save your highest Elo when you reach it, so you can keep playing without ever losing it. Launches August 05 with Season 9.
34
33
10
469
131,518
FACEIT Derek retweeted
Announcing the half-time voice chat vote, your call on whether voice between the two teams returns to FACEIT as a limited test in Season 9's first week, from August 05. Vote now at fce.gg/Half-Time-Vote
320
222
182
7,610
1,583,067
FACEIT Derek retweeted
Introducing the updated Soft Elo Reset, lighter than Season 8 and now factoring in your recent win rate and FACEIT Rating. Most players start close to where they finished, and strong placements are your chance to climb higher than before. Live with Season 9 on August 05.
50
17
4
343
63,922
FACEIT Derek retweeted
Replying to @1bird_d @FACEITcs
The Season 9 reset will be lighter and closer to the Season 7 reset, which mainly targeted the top players on the platform. Season 8 was the heaviest soft reset we've done. While we will still have big resets like that in the future when they benefit the ecosystem, we do not need one of that size every few months. We will be sharing more details and additional changes next week to the reset for Season 9.
13
6
64
12,885
We're considering adding a sound that plays in the matchroom 1. When the connect info is ready 2. When the knife round begins If we did this, what sounds do you think we should play for each of these?
7
1
13
3,312
With it defaulted on, that means most matches should have Cache as an option to play on. Do you think Cache could overtake Mirage & Dust2 for the most played map by tomorrow?
The updated version of Cache is now live on FACEIT. Cache is now selected by default in map preferences and can be deselected without needing to subscribe or change your other map bans.
4
23
4,828
FACEIT Derek retweeted
The updated version of Cache is now live on FACEIT. Cache is now selected by default in map preferences and can be deselected without needing to subscribe or change your other map bans.
36
40
6
1,217
93,768
FACEIT Derek retweeted
I wanted to take this opportunity to share some background information on what happened during the FACEIT outages over the last 2 days. My goal here is to provide more information about how the platform responded under record demand. I hope some of you find these insights interesting or can learn something from them for yourselves. I’m trying to bridge the gap between non-technical and technical folks, so some things might be too nuanced for some, while other things might be too high-level for others. On Wednesday, April 22, the amount of traffic to our website far exceeded even our highest forecast, especially so early in the day at 1PM CEST. Like most cloud native software companies, we have more servers provisioned than necessary, including auto-scaling mechanisms to handle traffic spikes. Yet, your enthusiasm for our platform outgrew even these generous buffers, making us push the upper safety auto-scaling threshold. As we recognized the increased traffic, our engineering team started to increase the ceiling of our auto-scaling configuration in terms of physical cloud limits and Kubernetes configuration, practically provisioning more web servers to handle the load. Unfortunately, we had already reached a critical point at which existing servers (Kubernetes Pods, for the techies here) were failing faster than new ones could spawn. What happened in this moment is that the extremely spikey traffic caused people to get errors, making them refresh continuously, which sent our servers into a death spiral that they couldn’t recover from without intervention. As we stabilized the website, the spike in players queuing up led to our Matchmaker falling behind, resulting in much longer than usual queue times. The surge of matches spawning furthermore led to our game server architecture not being able to scale quickly enough. Our game servers are run on so-called bare-metal servers, which are not virtual cloud servers but actual machines, guaranteeing the best performance and latency. Bare-metal machines take longer to provision and deploy into our fleet, and they also sometimes have supply issues. Last week, we put in orders that maxed out available capacity in some of our regions and are waiting on more deliveries. During this period, the fallback is cloud scaling to ensure players are not waiting 10 minutes for a server. However, this scaling mechanism was unable to keep up with the demand. Since Wednesday, we've had additional deliveries, increasing server numbers to an all time high. On Thursday, April 23, as a follow-up to Wednesday's surge, we greatly increased the horizontal scaling of our Matchmaker. However, as we hit peak hours, this extra load pushed our proportionately scaled matchmaking database to its limit. The Matchmaker is using a Redis database, which should be operated in an environment where 20% of memory capacity should be reserved for the system memory, so that it can perform crucial operations like cleaning up stale data while keeping the database performing normally. Given the increased amount of matchmaking capacity, we had to basically double the amount of memory allocated to said database. There was a change made to our configuration files about a year ago that tipped the ratio of Redis and system memory below the 20% threshold. This hidden bottleneck never caused an issue during normal operations, but under the extreme pressure of Season 8's launch and our newly expanded queues, the database stalled. Write requests began timing out, which caused a cascading failure across our game queues. We eventually managed to scale the database up further, while keeping already active matches from cancelling and the platform as a whole operational. The “issue” here is that a zero downtime scaling of such a system basically requires a new replica to spawn and replicate all data from the old instance before connections can be rerouted, which takes time and system resources. Once that scaling safely concluded, we fully restored the service. We are now auditing all of our database configurations to ensure similar resource imbalances are not hiding anywhere else, and reinforced the Matchmaking database with extra system memory headroom.
26
10
4
242
93,196
FACEIT Derek retweeted
Season 8 launched this week and peaked at 12,459 concurrent matches, the highest volume in FACEIT history. Jan from our engineering team shared the technical context behind the platform and queue instability, and adjustments made to handle the increasing load this weekend.
I wanted to take this opportunity to share some background information on what happened during the FACEIT outages over the last 2 days. My goal here is to provide more information about how the platform responded under record demand. I hope some of you find these insights interesting or can learn something from them for yourselves. I’m trying to bridge the gap between non-technical and technical folks, so some things might be too nuanced for some, while other things might be too high-level for others. On Wednesday, April 22, the amount of traffic to our website far exceeded even our highest forecast, especially so early in the day at 1PM CEST. Like most cloud native software companies, we have more servers provisioned than necessary, including auto-scaling mechanisms to handle traffic spikes. Yet, your enthusiasm for our platform outgrew even these generous buffers, making us push the upper safety auto-scaling threshold. As we recognized the increased traffic, our engineering team started to increase the ceiling of our auto-scaling configuration in terms of physical cloud limits and Kubernetes configuration, practically provisioning more web servers to handle the load. Unfortunately, we had already reached a critical point at which existing servers (Kubernetes Pods, for the techies here) were failing faster than new ones could spawn. What happened in this moment is that the extremely spikey traffic caused people to get errors, making them refresh continuously, which sent our servers into a death spiral that they couldn’t recover from without intervention. As we stabilized the website, the spike in players queuing up led to our Matchmaker falling behind, resulting in much longer than usual queue times. The surge of matches spawning furthermore led to our game server architecture not being able to scale quickly enough. Our game servers are run on so-called bare-metal servers, which are not virtual cloud servers but actual machines, guaranteeing the best performance and latency. Bare-metal machines take longer to provision and deploy into our fleet, and they also sometimes have supply issues. Last week, we put in orders that maxed out available capacity in some of our regions and are waiting on more deliveries. During this period, the fallback is cloud scaling to ensure players are not waiting 10 minutes for a server. However, this scaling mechanism was unable to keep up with the demand. Since Wednesday, we've had additional deliveries, increasing server numbers to an all time high. On Thursday, April 23, as a follow-up to Wednesday's surge, we greatly increased the horizontal scaling of our Matchmaker. However, as we hit peak hours, this extra load pushed our proportionately scaled matchmaking database to its limit. The Matchmaker is using a Redis database, which should be operated in an environment where 20% of memory capacity should be reserved for the system memory, so that it can perform crucial operations like cleaning up stale data while keeping the database performing normally. Given the increased amount of matchmaking capacity, we had to basically double the amount of memory allocated to said database. There was a change made to our configuration files about a year ago that tipped the ratio of Redis and system memory below the 20% threshold. This hidden bottleneck never caused an issue during normal operations, but under the extreme pressure of Season 8's launch and our newly expanded queues, the database stalled. Write requests began timing out, which caused a cascading failure across our game queues. We eventually managed to scale the database up further, while keeping already active matches from cancelling and the platform as a whole operational. The “issue” here is that a zero downtime scaling of such a system basically requires a new replica to spawn and replicate all data from the old instance before connections can be rerouted, which takes time and system resources. Once that scaling safely concluded, we fully restored the service. We are now auditing all of our database configurations to ensure similar resource imbalances are not hiding anywhere else, and reinforced the Matchmaking database with extra system memory headroom.
14
12
1
320
41,904
FACEIT Derek retweeted
The moment @FaceitDerek tells @Faceit_Mikey that Season 8 just broke every traffic record in FACEIT history 🚀 (and that our infra team is about to earn their stripes)
45
17
1
655
79,907
We've pushed out an update to @FACEITcs Matchmaking to prefer matching parties who have Cache selected.
7
5
2
256
64,898