The OneQode Blog • Stories & Resources from OneQode

2021-11-22T00:00:00Z

As it turns out, it’s not easy.

First up, why would you put a server on a Pacific island, anyway?

Imagine this: you’re a gamer living in Sydney.

You want to play games with your friends Alice and Bob, who live in Singapore and Japan respectively — but there’s a problem:

Choosing a game server location that works for everyone is hard.

If you put a game server in someone’s home city, the other two players would have a delayed connection, and it wouldn’t be as fun for them.

So what’s the solution? Easy! Put a server on an island somewhere in the middle. That way, everyone can play together on a level playing field.

How hard could it be? That’s what we asked when Ben Cooper (whose idea it was) brought it up.

If the game plan worked, it could finally unlock cross-regional gaming in the Asia-Pacific (APAC) region.

As a team of gamers — we couldn’t resist.

The easiest part: choosing a Pacific island

First thing’s first, the island had to be well connected, and, when you open up a submarine cable map, one place in Asia-Pacific lights up like a Christmas tree:

A casual browse of SubmarineCableMap.com

The Pacific island of Guam.

On the surface, it looks like the perfect spot for an international server hub:

it’s highly connected
it’s a submarine cable hub for the region
it’s right in the middle of the Asia-Pacific region
so a perfect spot for a cross-regional server facility
latency to nearby countries is excellent
internet packets can do a return trip to nearby countries faster than the blink of an eye
it’s a US territory
with a familiar legal environment
it’s a beautiful place to work

So why hasn’t it been done before?

We figured most companies hadn’t even thought about doing it. Those who knew it was possible, didn’t know there was a need for it.

And the people who needed it, didn’t know it was possible.

Plus, creating something new takes time and money.

So we figured we’d give it a shot.

Over the last 18 months, we’ve been building a pool of servers on the island of Guam. We’ve got the scars, the tears, and a high-performance cloud platform to prove it.

Here’s what we learned on the way.

A tropical beach in Guam

Getting space & submarine connectivity is hard

The first thing we found was that there aren’t many places in Guam where you can put servers.

There are plenty of buildings, but most of them aren’t in reach of the cables that carry internet traffic across the Pacific.

After many dead-ends, we found a potential location in a new cable landing station that was being built.

And then we struck gold.

An undersea cable company, RTI, was finalising a new subsea cable which went from Japan to Australia (where we are) — with a stop-off halfway in the exact same building in Guam.

An RTI submarine cable coming ashore • Watch on YouTube

Even better, instead of only landing in Sydney (where cables on the east coast of Australia usually land), it was also landing in the Sunshine Coast, Queensland — our backyard.

Normally, subsea operators don’t deal with small startups (which we were).

But RTI were no normal company and Russ Matulich (CEO) was no normal leader. He loved the idea of what OneQode was doing and took the time to help us through the process of getting capacity on their cables.

Thanks to them, we were soon preparing to deploy our equipment in the Sunshine Coast, Tokyo and Guam to leverage our new capacity.

What does “preparing to deploy” mean anyway?

To turn the lights on at a location, there are a few things we need to do:

negotiate legal agreements
arrange data centre space
hire staff
get transit connectivity
(internet for data centres)
organise peering
(connections to last-mile internet providers)
order and set up equipment
routers, switches, servers, and DDoS protection hardware

All this means mountains of paperwork, jurisdictional legal headaches, language translations and endless email threads — before we even got to hiring staff.

There aren’t many people around

It’s hard to find qualified network engineers in Guam — and there’s good reason:

With a population of ~170,000, 3 internet providers, and only a handful of companies employing IT professionals, there aren’t a lot of technical specialists looking for work.

We put some feelers out and luckily managed to snag the excellent Matt Ziobro, an IT specialist (and keen pilot), to take charge of our operations in Guam. Without him, it’s unlikely our Guam project would have taken off the ground at all.

So — we now had a place to put servers and the beginnings of a team.

Next, we needed to order the equipment.

Shipping takes forever

When you decide to send a 100kg server to Guam, it’s most likely getting there via boat — which takes 25 days at the best of times.

So if something is broken or missing, that’s at least another month on the clock.

We found ourselves up late at night praying that our suppliers had triple-checked orders before shipping them out — and included the right cables. There aren’t many places on the island where you can buy replacements.

Decent shipping is like a lottery. Sometimes, the stars and your wallet align, and 2-day air freight can be available. It’s rare, but when it happens, it’s a great day.

Red tape plays a role, too. As a U.S. territory, the Jones Act applies. It effectively bans non-US vessels from carrying cargo to Guam. That means you can order an item from Japan, and if there’s no U.S. ship to take it to Guam, it’s probably going via Alaska.

Don’t ask.

Even when your shipment gets to Guam, tragedy can strike.

More than once, we’ve had large shipments arrive on the island only to be sent back because the delivery company didn’t think the data centre existed. Maybe they went to the beach instead of delivering our equipment.

One last thing — if you’re using anyone except USPS for delivery, there’s a 4% “Use tax” on the value of your delivery. At this point in the process, you just sigh and pay up.

Finally, your delivery has arrived — so you plug things in.

Electricity is expensive

On the island, electricity costs at least twice as much as it does in the United States.

Why so pricey? Like most things in Guam, the energy sector is at the mercy of the shipping industry too. With 90% of the power coming from fossil fuels and limited alternatives to choose from, electricity is precious.

Not only is it more expensive, but power distribution to the data centres we use is also precarious, so there are strict limits to how much power we can draw.

As if that’s not enough, because we play life on hard mode and operate high-performance servers, we’ve had to dance on the knife-edge between power and performance. That means choosing very specific components and making trade-offs where we can. It’s a delicate balancing act.

Can’t they just build more power plants?

Well, they did — but it blew up back in 2015.

How? Best to ask an islander.

The good news though, is that a new power plant is on its way, due for completion in 2024 💪

You can’t do it from behind a computer screen

Doing this during a pandemic was odd, to say the least.

Every single step of the project was affected — from factory shutdowns and global supply shortages to customs delays and interstate travel.

Thing is, you can’t build a global network from behind a computer screen. You need boots on the ground to build servers, package and ship things, install equipment and plug stuff in, as well as replace broken hardware in the middle of the night.

With travel restrictions preventing key staff from travelling interstate and overseas, we had to improvise.

Remote telesurgery: OneQode CEO Matt Shearing installs new hardware in a data centre, guided 500 miles away by Chief Architect Ben Cooper via TeamViewer’s augmented reality app

Things would have been 10 times more difficult without the last 5 years of technology.

Augmented reality support apps like Teamviewer Pilot have made remote data centre operations much easier. You can have a colleague assist you from the other side of the world, drawing arrows and annotations in real-time over objects in front of you.

A game-changer • TeamViewer Pilot demonstration video • Watch on YouTube

Teamwork tools have improved, too.

Most of our team have never met in person — we’re spread across 10 cities and 6 time zones.

We used to live on Microsoft Teams and Zoom, but they didn’t scratch our itch. Sometimes you want the spontaneity of being in an office instead.

So we switched to Gather. And it is good.

We’re lucky. We haven’t encountered a typhoon... yet

One thing we haven’t come across — thankfully — is a typhoon.

Guam is often rocked by tropical storms, and occasionally a typhoon. Some have been severe category 4 and 5 storms with winds reaching close to 300 km/h, and they leave devastation in their wake.

Typhoon Mangkhut passes through Guam in 2018 [Source: NOAA]

Much of the island’s infrastructure (including the data centres we’re in) is built to withstand the most extreme conditions. But, up against nature, anything can happen.

All we can do is be prepared and build resilience into our systems. 🤞

In summary: it’s harder than it looks — but worth it

Bringing it all together, we’ve encountered delays, travel restrictions, chip shortages, hardware failures, electricity calculations, import duties and a truckload of paperwork.

But we did it.

We were the first company to put servers on the island designed to bring the region together, and while it’s been a hard road, it was worth it.

What would that look like for gaming with your friends Alice & Bob?

A marked improvement, with a very playable latency for everyone.

And how did it turn out in the real world?

Earlier this year we hosted a $17,000 cross-regional CS:GO tournament to put our brand new network to the test.

96 players from 9 countries took part over 5 days to fight for the prize pool, streamed live to a global audience on Twitch.

While we had a rocky start (solving scenic internet provider routing in real-time… story for another day), the tournament ended on a high — with an epic final between two excellent teams.

During the tournament, there was a defining moment when we knew we’d done something special.

We watched live as an Australian team battled a Mongolian team 5,000 km away — with less than 90ms of latency.

It was huge. Unthinkable before we put servers there.

We could hardly believe it.

Guam (APAC Central) is a brand new hub for cross-regional gaming. Players across 3 continents can now connect, introducing completely new strategies and levels of competition that haven’t been possible until now.

This is beyond exciting — and we are proud to declare Guam as the first region for our high-performance cloud that’s built for game studios, developers, publishers, and tournament operators.

We’re grateful to our families, teammates, suppliers and friends for supporting us up till now. Without your motivation, we would have given up long ago.

Finally, we’ve gained a new respect for the people who pave the way for their industry — in spite of the naysayers.

If that’s you, carry on 👏

2021-12-15T00:00:00Z

As OneQode heads into 2022 with the goal of network expansion and taking our products to the market (and a successful funding round!), I feel that it’s of utmost importance to clearly dictate where we’re going, and why we’re going there.

We’re a team of 20 people right now, but we’ll soon be a lot larger — so having a clear vision, mission and strategy is important.

The great thing about authoring a piece like this in 2021 is that a lot of founders have walked this path before. As such, I’ve taken significant inspiration for this framework from Brian Armstrong, founder and CEO of Coinbase, and his regular pieces on this topic.

His framework is as follows:

Vision: the why

Why are you doing what you’re doing? How do you want the world to be different in ten years?

Mission: the what

What are you going to do to help create that future? What single broad statement sums up what you doing?

Strategy: the how

How are you going to accomplish that mission? What different steps will you take to get there year over year?

By dictating this, we can know what we’re doing, why we’re doing it and how we’re going to achieve it.

Vision: the why

The evolving digital economy

We are increasingly living online. This sounds obvious, but it’s easy to underestimate just how much our lives have changed in a relatively short period of time.

Twenty years ago the thought of physical storefronts shuttering while online stores thrive, face-to-face meetings going extinct and moving online, or entertainment being accessible on-demand to anyone with an internet connection would have seemed almost impossible, but they are now norms in our daily lives.

These examples (and many more) are evidence of the gargantuan shift in our species’ interactions brought on by the internet’s proliferation. They’re coupled with an increasingly voracious appetite for fidelity and emphasis on direct connection.

As a result, mainstream use-cases for the internet are trending towards real-time applications and demanding higher levels of detail and data than ever before.

Digital is accelerating

The kicker? This isn’t slowing down. Connectivity is becoming the currency of our time.

Like so many things, we can look to the world of multiplayer gaming for an idea of what’s coming.

Gamers these days are playing less small matches (i.e. Starcraft 2, CS:GO) or huge scale MMORPGs (WoW, Guild Wars) with each other. Rather, we’ve seen the emergence of large-scale fast paced multiplayer, such as battle royale games (Fortnite, Apex Legends) and mobile arenas (Free Fire, PUBG:Mobile).

The trend is to feature more players, larger worlds, higher fidelity and more real-time interactions between them.

But even non-gamers (a declining population) are benefiting as the technological advancements bleed through to video conferencing, virtual events and collaborative experiences.

Soon, we’ll see this in a whole new tranche of applications, including augmented and virtual reality workplace solutions.

In short, people are becoming comfortable with having meaningful connections online. And while our infrastructure has worked passably for most current applications, there are some big issues on the horizon as we move into this next epoch.

The problem: the infrastructure isn’t ready!

The genesis of these coming challenges is not new.

I experienced this personally more than 15 years ago, running an international gaming organisation which started in Freelancer — a multiplayer space sim made by Chris Roberts.

Boasting a massive star system brimming with human and AI pilots, Freelancer servers were self-hosted and drew players from all over the world. Different organisations had virtual wars, trade routes, treaties and events with each other, and we participated in this heavily. Or at least, we did our best to participate.

An epic battle in Freelancer // Video by Chronicron

Most of the player base on our servers were from North America, Australia and Asia. With the servers hosted in the US, this brought big challenges for those in other places — huge pings, rubber banding and server capacity issues.

Freelancer had ‘twitch based’ combat — meaning milliseconds mattered — and battles could be happening in flashpoints across many star systems. Add to this hundreds of players and NPCs going about their daily tasks across a huge universe, and the servers themselves struggled to keep up.

This cocktail of demanding requirements meant players were generally fighting the servers/latency as much as they were fighting each other.

In the end, it got so bad that most PvP combat tactics would revolve around accounting for the lag. Soon, these frustrations boiled over and inter-continental communities fractured, with players preferring more ‘local’ servers.

My organisation pivoted to a mainly Aussie focus as a result, predominantly playing Battlefield 2 and Counter Strike: Source.

To this day, I’m left with the thought of what could have been if there was a genuine ‘middle ground’ connecting the US, Australia and Asia — somewhere that facilitated cross-border connectivity and had high-performance servers up to the task of running a virtual universe.

The experience like I had above is not rare. Nor has it improved markedly with time.

Sure, there have been incremental improvements (Discord, chat applications, last-mile connectivity), but the larger issue — cross-continental, real-time interactions — has not been solved.

It’s not just in gaming. Multinational teams are on the rise, too.

Why isn’t the infrastructure ready?

Put simply, the internet isn’t built the way it needs to be. There are some key ingredients holding us back from realising our real-time connected future:

Latency optimisation
Current carriers do not optimize for latency, they optimize for cost. Their philosophy is to take a connection and to hand it off to someone else’s network as cheaply and quickly as possible — regardless of where that is or how long it takes to get there. That’s OK for your Netflix — but not if you’re trying to have a shared experience from thousands of kilometres apart!
Stability
Many of these experiences will require stable connectivity, with minimal packet loss and jitter — something not prioritised by most modern carriers.
Strategic centrality
To connect billions, you need to host infrastructure in locations that are conducive for that purpose. Until now, this has not been a priority nor a focus for infrastructure providers.
High-performance
The kinds of applications we envision for our future (and the many thousands that are still yet to be thought of) will have a consistent thread — the trend towards more detail, more fidelity and more demanding operational requirements. It will require fit-for-purpose infrastructure built for high-performance.

So that begs the question — where do we want things to go and what are we doing about it? This is the backdrop for which our Vision, Mission and Strategy form the manifesto for us to address what we believe is one of the greatest needs of our time.

Our three core ‘imagine’ themes:

We’ve developed three ‘imagine’ themes as a thought exercise to communicate how we want things to change with our connectivity infrastructure.

Imagine if people could have meaningful connections with each other, anywhere in the world, without needing to worry about distance or latency
Imagine if cloud infrastructure was powerful, accessible and made sure connections got where they needed to go as quickly as possible.
Imagine if innovators, creators and studios could provide billions of people with the ability to connect with each other on the same platform, on-demand.

Our vision statement

This can be distilled into a single vision statement:

Imagine a world where the difference between physical and digital connectivity are imperceptible.

In a world where our three imagine themes and vision statement are a reality,

a game studio can build a game like Freelancer and expect it to just work
young gamers anywhere in the world can start global organisations and not feel like they’re fighting with the technology
infrastructure enables virtual connections that are indistinguishable from physical connections
borders mean nothing and people connect like never before

And that’s a world worth building.

Mission: the what

If our vision is for billions of people to connect together in high-fidelity on digital applications, how do we achieve this vision? This is our mission — what we’re trying to achieve in the world and how we can help bring this about.

This went through a number of iterations, but ultimately settled on this:

Our mission is to build the foundation for the next decade of connected experiences.

This captures the essence of what we’re trying to do as a company. We are building foundational infrastructure.

Everything we’ve built so far, network, servers and cloud, are all focused on getting data where it needs to go as quickly as possible. We’ve unlocked low-latency gaming for millions of players across three continents, and are providing high-performance-by-default servers for cutting-edge workloads.

Our first use case, gaming, will clearly benefit from this design philosophy. But gaming is just one of a myriad of future technologies that will harness real-time virtual experiences, and our infrastructure will be uniquely positioned to enable these kinds of emergent applications.

That is our mission — to build the foundation for those businesses, and those experiences. If we achieve this mission, we’ll make our vision become reality.

Serving billions

When can we say we’ve achieved this mission? For us, that will be when OneQode’s infrastructure has connected at least one billion end-users, through all the various businesses that build on our platform.

The biggest limiting factor in meaningful online experiences is the amount of people that can share those experiences.

Currently the design of the internet is, ironically, limited by locality. Carriers and providers, even those with an international presence, only make an effort to serve their local area well. A complex web of incentives has assured this, but our goal is to disrupt that ingrained thinking.

As our reliance on deeply connected technology grows, the underlying design of the internet itself must change to support it. Without changing that structure, truly mass-scale, intercontinental experiences will remain out of reach.

OneQode’s current reach // About the network

Our model was built to enable this. Even now, we have the framework in place for people to build applications connecting billions, at high fidelity. This in turn makes it possible for people anywhere — particularly developing economies — to access opportunities they never would have had prior. It will also allow those same people to genuinely participate in new emergent ecosystems as they mature.

Gaming first

Short term, we are focusing on gaming. Our technology will enable people to interact and play with a far larger player base than they are used to.

We’ve already shown the prototype for this with our recent tournament, which had players from Mongolia to Australia compete against each other for the very first time — something impossible before the OneQode platform.

Gaming is really just the first step though, preparing for what’s to come. Longer term, the platform we’re building has far wider applications and implications.

From virtual, global offices to VR shopping malls and many, many other usecases, we’re creating an open foundation for online experiences which connect not millions, but billions.

If we can do that in a manner that promotes entrepreneurial innovation, open commerce and value exchange, the benefits will be revolutionary.

We’ll explore the potential implications for developing economies in a later post, but this increase in opportunity is central to everything we’re doing.

Strategy: the how

We’ve dealt with why we’re building OneQode and what’s driving us. Now we turn to the fun part — how we’re going to get this done.

1. Build a high-performance IaaS framework

The first step is to build a high performance Infrastructure as a Service (IaaS) framework. We’ve already been working at this for the last 18 months, and while we’ve completed what we’re calling ‘stage 1’, expanding this framework is critical for the long-term achievement of our mission.

So what is this framework?

It starts with our private international carrier network, which is designed to pick up data as close as possible to the user and deliver it quickly, optimising for reduced latency by default (unlike existing carriers, who optimize for cost).

Our regional backbone uses network hubs we locate at ‘crossroads’, where sub-sea cables meet and we can serve the maximum number of people with acceptable latency. Our first connectivity hub is located in Guam.

From there, we establish local Points of Presence (PoPs) in other key locations, designed to pick up user data and serve more latency sensitive applications.

Once those are established, we install high-performance servers at each location and put them behind our premium DDoS protection, which shields our network and servers from malicious denial-of-service attacks known to target gaming and other real-time applications.

Finally, we integrate it all into our cloud platform, providing a portal and API layer for on-demand access. This allows developers to access all the benefits of our network, hubs and servers in one place.

How our points of presence are architected

So what do we need to do from here to achieve our mission?

Expand our framework by deploying regional backbone locations in the Americas, India and the sub-continent, Europe, Africa and the Middle-East.
Bolster our cloud presence to complement each location and grow to meet increasing demand.
Continue to build strong relationships with last-mile network providers, orchestrators, publishers, studios and vendors.

If we’re achieving the above, OneQode’s footprint will grow from Asia-Pacific to global, with increasing capacity on our cloud and more last-mile carriers interconnecting with us at our various locations.

2. Become the leading gaming IaaS provider

As we’ve built the first wave of our high-performance IaaS platform, we’re now ready to enter the second phase of the strategy: adoption. For us, this starts with gaming.

Gaming is one of, if not the most, performance-sensitive application verticals on the internet.

This market is massive with a direct need. Right now, game servers (in Asia-Pacific particularly) are hyper-localised. It’s difficult for gamers to play across regions and the infrastructure is far behind North America and Europe.

Despite that, it is the largest region by gaming revenue in the world — and this is only growing.

+17.5% year on year spend growth

1.5 bn gamers in the region

+4.9% annual player growth

Our most immediate goal is to address this. If we are to lay it out in mission form, it is:

To be the pre-eminent gaming Infrastructure as a Service Provider for Asia Pacific

And on a slightly longer timeframe:

To be the pre-eminent gaming Infrastructure as a Service Provider for the world.

Apart from our marketing and sales strategy, a key component for achieving this is something truly profound — playing games!

Having games, particularly competitive games, playing on our servers and connectivity hubs (like Guam) is important social proof to demonstrate why we’re the pre-eminent gaming infrastructure provider.

It’s why we’re investing heavily into partnerships with tournament organisers and game studios at the moment, and will continue striving for new ways to support the gaming ecosystem with our technology.

Ultimately, if we’re achieving this mission, game studios, orchestrators and providers will be able to:

keep queue times low and off-peak latencies acceptable in far smaller margins;
serve a larger user pool than ever before;
build games supporting more players, higher tickrates, larger maps/universes and more server-side features in general; and
never feel like their infrastructure is letting them, or their players, down.

Overall, gamers should be feeling like they’re queuing faster, waiting less time for a game, playing against a more varied range of opponents and playing more expansive games. Our eventual vision for this is that studios will use our infrastructure to build games which are unparalleled in immersion and player reach.

3. Become the leading on-demand platform for connected experiences

To make sure we can meet our customers’ evolving needs, a key part of our strategy will be iterating heavily on both infrastructure and product.

Just like any solid foundation, we’ll be looking to both build horizontally on that foundation and also add features vertically which can help serve our customers better, increase the depth and utility of our products, and cement our position in the market.

Some key aspects we’ll be focused on as part of this strategy:

Enterprise solutions
We’ll increasingly build an enterprise arm which can take our world-class infrastructure profile and build fit-for-purpose solutions for businesses pushing boundaries.
Product improvement and iteration
Both horizontal and vertical improvement of our IaaS platform, from increased cloud features through to expanded network reach and presence. Will include a raft of unique software-side developments to serve key customer types.
Strategic acquisitions
Acquiring businesses and software which are complementary for our mission. Primary targets include niche cloud solutions, diverse networks or products which can be significantly improved with access to our platform.
Wholesale and white-labelling
We’ll work with large companies who already move in the IaaS space to supplement their offerings via whitelabelling or offerings on their marketplace.
Partnerships
We work with anyone and everyone and this won’t change anytime soon. We’ll continue to be hyper-focused on forming partnerships which improve our products and incentivise win-win outcomes.

In summary

OneQode is not only focused on improving connected experiences but enabling new types of real-time applications. This starts for us with gaming, but the foundation we’re building has far greater potential.

As OneQode begins its global expansion and rollout, we’ll focus on serving businesses in complementary verticals, particularly those which can benefit from the unique network and infrastructure that we’ve built.

Gaming is the harbinger of what’s coming and it needs high-performance connectivity and compute across the board.

We’re building the foundation for this now, so people everywhere can benefit in the future.

We hope you join us on this journey, we couldn’t be more excited to see where it takes us.

Matt Shearing, CEO

2022-02-22T00:00:00Z

Latency. Lag. Rubberbanding.

Call it what you like, there isn’t a gamer on the planet who hasn’t experienced it.

For anyone lucky enough to not have experienced it: it sucks - a lot.

Your in-game senses become unreliable. Your reaction speed delayed.

It becomes a huge roadblock to being competitive, or even just enjoying the game.

Imagine trying to catch a ball, but your reaction speed is twice as long.

Your focus? On your network and not the game.

For gamers, this is the difference between virtual life and death.

So we started OneQode

A wireless last-mile internet provider for gamers. Well, that’s what we thought we were starting.

The important part was the mission: better internet for gamers.

We eventually found ourselves launching an international carrier network, with reach far beyond our borders. The path had changed but the mission was the same.

But before we get to that, we need to go back to the beginning.

The elephant in the network

Look: Australian internet sucked.

Our internet speeds ranked 67th globally — and providers weren’t getting any better. They prioritised cost over everything.

The worst thing? None of them cared about latency.

It seemed like the only people who could fix gamers’ internet, were gamers.

Once upon a time, there was hope for Australia, the national broadband network (NBN). The biggest infrastructure commitment in Australia’s history. A fibre to the premises promise. A huge vision.

But as it rolled out, it failed in scope and speed.

Gamers were left out in the cold. We were on our own.

It was clear if we wanted help, we had to help ourselves.

We wanted to create another option that would offer better connectivity than traditional last-mile internet — and eliminate unnecessary latency for gamers.

Last mile: the gift that never gives

For most people, last mile is “the internet”.

This has always been the pain point. The final portion of the network that connects to your home, where the resources to ensure speed are spread the thinnest.

To welcome gamers in from the cold, we’d just need to fix last-mile — or so we thought.

Traditional last-mile infrastructure requires three key resources:

Money
Time
Effort

A typical last-mile setup involves laying underground cables to each individual premises. At scale, it’s a massive operation.

As a small startup, the cost to compete on these terms would be huge.

So we didn’t.

Wireless last-mile technology had been quietly improving, so we threw some dishes on roofs, and took our first steps as a provider.

How hard could it be?

As soon as the data left our hands (aka the last-mile) the quality plummeted (just like that pizza). We couldn't guarantee a good experience, especially when gamers connected abroad.

So we ventured upstream.

Going upstream: the network problem

“It was like an onion. The more we peeled back (to improve internet for gamers), the deeper we had to go into carrier land to deliver the service for last mile”

— Matt Shearing, OneQode CEO

Internet infrastructure is like a river. The widest portion of the river carries the most data, splitting into smaller streams closer to its destination.

We started wireless last-mile to clean up the streams, but for a startup, this was a lot of ground to cover. Worse? The river was being polluted upstream.

When we went upstream, we realised that the situation was worse than we thought.

You see, internet providers have to connect to the wider internet, too. To do this, they buy connectivity from larger “carriers” upstream, who have massive global networks.

But a familiar problem reared its ugly head. Carriers weren’t bothered about quality either. They:

Didn’t prioritise low latency
Weren’t focused on speed
Have over congested networks
Were expensive for what you got

All our efforts downstream would be pointless unless this was fixed. It was out of our control.

To deliver great internet for gamers, we needed end-to-end control of the infrastructure.

And if you want something done right, you’ve got to do it yourself.

This all felt a bit like this scene from Malcolm in the Middle:

Hal from Malcolm in the Middle fixing a light bulb

Until our future partner RTI Cables landed right in our backyard.

The sunny coast cable: a new hope

What do a giant pineapple and the fastest cable to Asia from Australia’s east coast have in common?

As of 2020, you’ll find both on the Sunshine Coast.

Before RTI landed the Sunshine Coast cable, most international-bound Australian data travelled through Sydney.

See, the internet isn’t magic. To travel from continent to continent, your data joins billions of other bits and bytes on a vast network of subsea cables.

When we decided we had to connect internationally, it looked like we had to go through Sydney.

But thanks to RTI we could connect to the cable that gave us the fastest pathway overseas. The best part? We were the first to connect to it.

We still needed the data to be handled with care throughout the entire journey. Sadly you can’t label data “this side up”... yet.

The Sunshine Coast landing station connected us with the United States, Tokyo, Hong Kong, and Singapore, with one pit stop between us and them. But without presences on all sides, we would have to hand off our data.

Once again we ran into a familiar problem.

Quality control.

That one stop for all our data? Guam.

I think you can see where this is going.

Enter Mr. Guam

Moving upstream meant we needed someone with experience in “carrier land”.

Meet Ben Cooper.

Ben’s spent his career building carrier networks — the fundamental layer the internet is built on. Carrier network architects are a rare breed in the industry with only a handful of them in Australia.

These days he’s Chief Architect at OneQode.

Ben’s a scientist at heart, and scientists love to experiment. A couple of years before, following a whim, Ben had tested the latency between Guam and APAC regions. In two years he’d receive the results via email. You know, the kind of thing we all do on our lunch breaks.

He promptly forgot about the whole thing.

In 2018, he struck gold.

Google Taiwan held a 15ms ping to the University of Guam for two months.

Guam had been an interchange. But if it could become a meeting point, the possibilities were massive.

Ben realised that with Guam as the backbone of an international network, you could create a latency-neutral location to serve billions of gamers.

Matt still refers to Ben as “Mr. Guam” sometimes. Since his experiments, Ben had gotten to know the island like the back of his hand.

So once RTI connected us to Guam, the opportunity was clear.

Guam could be APAC Central.

A server in Guam, however, was still easier said than done. To take full advantage of Guam, we would need presences on the other ends.

By setting up in Guam, we’d begun a global network. The next steps would allow us full control of our data.

End-to-End.

Across the whole region.

...Just to handle game traffic.

If you love something, let it go

We were at a crossroads.

Ultimately we wanted better internet for gamers.

But by trying to root out the problem at its source, we’d moved away from last-mile.

Two OneQode’s existed. Which was going to serve gamers better?

So we took stock.

While we’d gone upstream, bigger players entered last-mile. Giants of telecommunications meeting the demand of skyrocketing needs in a pandemic.

So what had we done in that time?

Set up points of presence in three continents
Began to grow the company (entirely remotely)
Built a high-performance international network

4 countries

6 points of presence

10 staff employed

100GbE core network

During Covid, all without “squillions” of dollars (as Matt puts it)

So instead of going toe-to-toe with giants, we stood on their shoulders.

There was no one else upstream addressing the issue at its core.

“It’s best to sail with the wind,” says Daniel McGregor, OneQode’s Chief Financial Officer, on our move away from last-mile connectivity.

What we had built had the potential to be huge for not only Australian gamers, but the entire APAC region.

So we ditched last-mile to focus on our high-performance, latency-optimized international carrier network. That meant that other providers — and their gamers — could benefit from our work.

None of this made it easy to leave last-mile behind, it was the relatable issue for gamers.

But we’d built a kickass international network. And we were doing things nobody had done before.

For game developers, it would reach more people, quicker. Maintaining a healthy population of players, especially during non-peak hours.

For gamers, this would mean better connections, quicker matches, and most importantly a fairer game.

Decision time. Increasingly, that decision was looking like this:

To best serve gamers, it was time to play to our strengths. Connecting more people through Guam.

We’d found a way to help gamers across the world — far beyond our backyard.

Now we had to make it happen.

Launching the rocket (going global)

Guam was deployed, so we took the lessons we learnt and expanded our points of presence.

To Singapore. And Hong Kong. And Los Angeles.

With these locations deployed, we’d honed our skills and perfected the recipe.

Deploying any point of presence is a massive undertaking.

Logistics range from navigating legal jurisdictions, to hiring local staff, to communications, shipping and receiving.

Our team was ready for the challenge though, and successfully deployed international points of presence during the pandemic.

Then, we needed gamers. Not just any gamers, but the kind of gamers who would recognise the difference we were making.

So we hosted a $17,000 CS:GO tournament — the most latency sensitive game — in Guam.

It was kind of a big deal.

96 players. 9 countries. Matches that were impossible before Guam. All live-streamed globally.

An Australian player single-handedly takes on a Mongolian team (and almost wins)

The end of the beginning

OneQode has evolved and transitioned into a serious operation.

We’re working with game studios and server providers to increase their reach throughout APAC.

We’re also helping enterprises build networks that span continents, giving them a regional backbone unlike anything they’ve had before.

The most important thing is this: we’re improving the experience for end-users across the Asia-Pacific — connecting more people at lower latency.

And this is just the beginning.

By building to a gaming standard, our platform is ready for anything. But it’s not just for gamers.

We’ve built it for the technology that doesn’t exist yet. Technology that’ll push the boundaries and connect more people than ever before, in real-time.

To bravely go

Pioneering isn’t easy. You have to forge your own path.

Prakash V, founding investor and our chief of operations, always reminds us to “stay curious”.

We always knew the destination: better internet for gamers across the region. The route we took wasn’t the one we expected, but now that we’ve carved our way forward we wouldn’t have it any other way.

The freedom to adapt was central to this journey.

Our mission is clear: make latency imperceptible, and break down barriers to the virtual worlds that are being created.

It’ll mean expanding. More people. More points of presence. More servers. More everything.

Most importantly, it’ll mean uniting gamers across the world.

We hope you’ll join us on the road ahead.

2022-05-27T00:00:00Z

Ben Cooper is the lead architect and designer behind OneQode's network. He sat down with Austin Gallagher to discuss his day to day, Wolfenstein Enemy Territory, and his hidden talents.

What is your role here at OneQode?

Chief Architect. My core job is to identify our customer’s needs and then built the solution.

I’m responsible for the design and implementation of the design for all our infrastructure, and I’m heavily involved in what OneQode sells and where.

Day to day what does that look like?

My day to day is very vague because I work across so much of the company. I talk with suppliers, I meet internally with Scott (Wisely, Head of Cloud) and Joe (Swinn, Head of Marketing) to spec out product suites.

With Scott, I’m the one on the whiteboard working out what we need to achieve - and Scott makes it happen.

A lot of our product starts with me saying “I think we should sell this”, and then I'm with it every step of the way until billing.

So a lot of the daily operations go through you at some point?

I act as a bit of a node between customers, engineers, billing, and design.

At the end of the day, my job as Chief Architect is to make sure we make good products. Hopefully, the customers are happy, and the investors are happy.

Upcoming regions for OneQode’s Cloud Platform

What previous roles led you to this position at OneQode?

I’ve worked as a consulting network architect and engineer, a strategic partnership manager for a gaming company, various levels of IT support consultant, and I’ve jumped around a bunch of jobs.

I don’t think one role trained me for OneQode, but I think a variety of jobs have prepared me to work at OneQode.

After moving through those roles, how did you end up here?

Initially, Matt needed someone for some consulting, which lasted about a week! Then he brought me back on, in a more hands-on capacity.

The day I joined there was an unofficial offer at midday, and I was signed on by 4 pm.

What’s the most exciting thing you’ve achieved since starting at OneQode?

I don’t think there is any one thing.

In the past, I always had a problem getting across what Guam could be for Asia Pacific. Understanding an idea deeply and being able to take it and have someone else understand it as deeply as you is completely different.

I think I’ll have achieved my mission when people stop needing to ask “why Guam?”

What else is part of that mission?

In the early days, the mission was a big network. Now I think just a network is a bit useless.

Game developers don’t want to manage servers. They want simplicity so they can focus on their product and games. We want to focus on infrastructure, so they can click a few times and rely on us to have built the framework around them.

This came together with a conversation with Scott, where he was asking just the right questions to make me realise what our customers needed.

What was that?

Well, we had a network, but providing for our customers meant doing more.

We talked and talked and finally decided on the thermonuclear option: building our own cloud.

That first night we got four services done of the thirty we needed. We thought we’d finish it by the end of the week!

Nine weeks later we high-fived (virtually) and celebrated that we’d finished.

Where do you see the company going in the future? What sort of opportunities that OneQode will enable excite you the most?

It’ll be a sad day for me personally, but when OneQode can enable more things beyond just gaming. We want to give small carriers and wholesalers the capacity to build cool and new things to change paradigms.

We’ve jumped through a lot of hoops to negotiate contracts, and hopefully, we’ll be able to enable people to build their network without going through that.

We want to make a portal that gives people the ability to choose their destiny.

What is your first memory of playing games online? Any communities or friends you made/remember?

Wolfenstein Enemy Territory. The people I know through Wolfenstein is a long list, which funnily includes Scott.

I think it’s the best multiplayer shooter ever made, the skill ceiling still hasn’t been reached. It requires a lot of the player.

A Wolfenstein Enemy Territory screenshot from the Steam page.

It was meant to be a DLC for Return To Castle Wolfenstein, but something happened behind the scenes and it was released free with just multiplayer. This was in 2003, way before stuff like that went free-to-play.

In Asia-Pacific, OneQode actually hosts a lot of the Enemy Territory servers that still run today.

After Enemy Territory, I went to Battlefield 2, Team Fortress 2. I played a lot of Counter-Strike as well. But I much prefer the dynamic nature of a game with more objectives and team play.

All of those games are quite competitive, any favourite games to relax?

My idea of kicking back with games is a match of Dota... so, no (laughs).

Don’t get me wrong, Stardew Valley or even the GTA can be fun, but I’m very competitive by nature.

For me, those competitive games are relaxing.

Any games that you wish still had an active player base for one more day?

Battlefield 2. I think it was everything that defines the Battlefield series in one game. There wasn’t any one thing that makes it excellent, but everything is working together so perfectly in Battlefield 2.

There’s a Jeremy Clarkson quote about the Lexus LFA that the creators made something so good that even they don’t know how they did it, and that’s Battlefield 2 in my opinion.

A big part of me also wishes that I could go back to Wolf ET (Wolfenstein Enemy Territory) because those big matches are hard to come by these days.

Any hidden talents?

No (laughs).

Where can people hear more from you?

Twitter is my best place to see my rantings @BenHyperionOQ.

2022-06-07T00:00:00Z

Key takeaways

Multiplayer tailspin is a phenomenon where a declining player count in a game causes a feedback loop where additional players abandon the title due to the declining population.
Various games have experienced this tailspin, resulting in catastrophic outcomes for the development studio.
Post-launch strategies that focus on monetization, post-launch content, community management, network stability and other aspects are key to retaining players.
Modern multiplayer games are reliant on much more than just gameplay.
Retaining players is easier than drawing old players back, and retaining players is hard.

I love Titanfall 2.

There’s nothing like it.

On foot, you’re a fast and nimble pilot. Jet assisted jumps, wall running, and slides mean you can speed across the battlefield gunning down your foes with ease.

Then, you initiate Titanfall.

A two-storey mech drops from orbit for you to command. In the mech, you’re slower, heavier, and harder hitting. A perfect merge of mechanics and man.

Gameplay from Titanfall 2’s Angel City gameplay trailer

Titanfall is a game of two halves, and much like pilot and mech, they fuse to make a stronger whole.

But, I don’t play Titanfall 2 anymore. Not many people do.

Circling the (player) drain

One of the reasons I stopped playing was, ironically, a lack of players.

But then I realised, I was part of the problem.

This isn’t unique to Titanfall 2.

We’ve all played a game we love, only to see it die as players return to heavy-hitting industry giants.

For every Fortnite that captures a huge player base, there are ten or more Titanfall 2s.

So what separates the two?

Tailspin: The Feedback Loop

“You are not stuck in traffic. You are traffic.”

— TomTom SatNav Advertisement

What is Tailspin?

Tailspin is a phenomenon where a declining player count in a multiplayer game causes a feedback loop where additional players abandon the title due to a low population.

We’ve all been part of it, whether we know it or not. We’ve all been partly responsible for killing our favourite games. Each player who leaves in turn fuels further losses, worsening over time.

Tailspin affects both players and developers.

The feedback loop of losing players, resources, and support for a game

For developers, fewer players mean less income

This reduces the resources available to the developer, especially if their game is free to play and relies on frequent microtransactions. Without a huge publisher willing to push for the revival of a title, a declining population can bleed cash from the development studio.

So without those resources, support for the game becomes less frequent.

Games without frequent updates risk becoming ‘stale’, especially as multiplayer games are increasingly built on the service and seasonal models.

For players, fewer players mean longer wait times.

Waiting isn’t fun.

After this wait, it is common for games with low populations to struggle to create a fair match. Newer or less skilled players are then pitted against highly experienced players.

Those newer players are likely to then move on to less punishing games.

Players may decide to leave due to a lack of content or a lack of other players, but both fuel the loop.

Players consolidate to fewer, more popular titles.

How can you avoid key causes of tailspin?

I'm not a game developer, but from research, I've found a few key themes that can kickstart the feedback loop of player decline. This is not an exhaustive list, and there are countless other elements to take into account.

Market your game without misleading players

Marketing a new game is difficult, especially when enticing players from other multiplayer games. That is why setting reasonable expectations for what the content schedule, gameplay, and monetization model will be is key.

Remember to communicate the best possible image of your game, without overpromising and underdelivering.

Avoid busy launch windows

Launching your title in a busy season, amongst other games or expansions, can immediately place your game on the back foot. Be wary of release windows near games with existing large audiences.

A successful launch helps buffer the initial drop games will experience as some players don't stick with your game. Plan carefully on when to best launch your game.

Take a fair approach to monetization

For a multiplayer game, deciding whether your game will be free-to-play or paid is one piece of the complicated puzzle of monetization. The industry standard for some genres has shifted, so if you're launching a multiplayer game you should look to see what monetization is popular.

Further, clearly communicate this monetization strategy. A strategy that is perceived as overtly greedy may net some short term cash, but the long term life of your game can suffer.

Create a transparent and regular content timeline

Gone are the days of releasing a game and supporting with updates months apart.

Seasons, weekly events, special daily items, the industry has shifted to a constant slew of updates. A multiplayer title with a slower content timeline can be dead in the water. If you're launching a multiplayer game, make sure to communicate to players what they should expect if they're investing time in your title.

Manage your community's expectations post-launch

Alongside the importance of having a clear content timeline, communicating accurately to your player base when to expect updates is also key. When there's no update on the game itself, continue to communicate what is upcoming to the players.

Open, direct, and frequent community engagement gives your player base more faith in your commitment to the game.

Invest in a stable network and server architecture

It may seem like a no-brainer, but multiplayer games need a strong network and solid servers. Without stable connections and players being unable to get into games, people will quickly drop off a game.

Having comprehensive server locations, a resilient network, and a good netcode will pay dividends for years to come.

Luckily, you're already in the right place to take care of this.

Build enticing long term progression systems

This is another sign of the times changing. Many multiplayer games without strong progression systems struggle to maintain an audience, even if that progression is just for cosmetic items.

Modern multiplayer games, regardless of genre, are expected to have a meta progression system outside the matches. Between the intense action of the game, these systems should be designed to continue to engage the players and give them a reason to return over and over.

Don’t neglect the new player experience

Once your game has launched, ensuring new players have a smooth on-ramp is very important. As players find they love it, you want them to be able to easily recommend the game to their friends.

Make your game easy to pick up, and give options for players of all skill levels. Over time, this will help build your audience.

Pay attention to these elements

So what happens when developers and publishers struggle to stay on course, and we see tailspin kick in?

How Tailspin killed Titanfall 2

Whenever I do venture back to play Titanfall 2, I’m often met with two major issues.

First: the waiting.

When a game’s population spirals like this, the wait for a match or activity can be agonising. Time is valuable, so sitting for ten minutes just watching a timer tick up is not my idea of fun.

Second: the skill gap.

With a population this low, most of the people still playing are good. Really, really good.

A combination of fast-paced movement, very low health, countless weapons, maps, and pieces of equipment to learn means the new player experience in Titanfall 2 can be especially harsh.

Even as a returning player I become frustrated as I am outpaced by players with far more hours of practice.

A massive skill barrier like this isn’t a good incentive to get players to return to a game after a long break, or for new players to want to try it out.

An example of how a player entering late in a game’s lifespan can struggle against seasoned players.

Titanfall 2 is also frequently subject to denial of service attacks on their servers, making the game unplayable for many. This even led to an interesting, but unrelated, conspiracy regarding the game.

With an already low population, frequent server outages are devastating to the remaining players.

Nothing is worse than launching a game and finding it unplayable because of network instability.

When Titanfall 2 does work, the low population makes it a struggle to quickly find a match, especially from Australia.

All this means after trying to play, usually, only for an evening or two, I put the game back down.

I hope that one day a larger player base will return.

But if I stop playing, I am once again part of the problem.

Titanfall 2 is one example, but numerous other games lost their players far more quickly.

Players escaped Hyper Scape

Take Ubisoft’s Hyper Scape, a late addition to the crowded Battle Royale genre, attempting to distinguish itself with a city setting and fast pace.

Hyper Scape gameplay by T2CA

Hyper Scape initially drew large numbers of Twitch viewers due to a limited release to streamers only. But without exact numbers from Ubisoft, it isn’t known how big the player base was at launch.

What is clear is Hyper Scape suffered from two massive issues.

First, is the new player experience.

Both Hyper Scape and Titanfall 2 are fast-paced, which makes them brutally unforgiving for players trying to learn the game.

This is exacerbated in Hyper Scape, as Battle Royales can be notoriously punishing, where one mistake can often mean multiple loading screens.

Second, long term progression.

Between matches in Hyper Scape, the game lacked long term goals for players to aim for. Without that feeling of progression, the struggle of each match wasn’t being rewarded.

Battle Royale games also need a high population, and that population is difficult to maintain.

When Hyper Scape launched, lobbies were initially one hundred players before dropping to sixty after declining player counts.

Hyper Scape’s city setting of “New Arcadia”

Only three months after its launch in October 2020, Ubisoft published a blog on Hyper Scape, stating that they “had not achieved the high expectations” they had set for the game.

Ubisoft also addressed the clear issues of long term progression, and the difficulty new players have in picking up the title. Updates attempted to fix these issues, but the initial zeitgeist created by the Twitch rollout had moved on.

There wasn’t any huge misstep from here on out that drove the population down, it simply started the feedback loop. Hyper Scape just couldn’t maintain an active player base.

Ubisoft seemingly lost faith in the game, without even mentioning it at E3 the following year. This lack of faith likely further contributed to declining player counts.

In January 2022, Ubisoft announced the future of Hyper Scape.

There wouldn’t be one.

The image accompanying Ubisoft’s blog post detailing the end of Hyper Scape

In January 2022, Ubisoft announced it had decided to permanently shut down Hyper Scape, making it impossible to play. This, sadly, isn’t the worst result for games that suffer this fate.

Disintegration lived up to its name

V1 Interactive was founded by Marcus Lehto, an industry veteran with over twenty years of experience. Marcus was integral to Bungie and worked on the Halo series for over ten years.

The banner on Marcus Lehto’s LinkedIn showing games he was involved in

The first game from his new studio was Disintegration, a Sci-Fi shooter with Real-Time Strategy elements.

On release, it received mixed reviews, but the launch is hopefully just the start for a multiplayer title.

Unfortunately, despite industry pedigree behind the scenes, the game never captured an audience.

On Steam Charts, the peak during its launch in June 2020 was 120 concurrent players. By September, it was averaging less than 30 peak players a day.

While online matches needed substantially fewer players than a Hyper Scape match, ten to Hyper Scape’s sixty post-launch, this was abysmal.

The vast majority of popular multiplayer shooters are free-to-play, but Disintegration was released at full price, likely hurting its potential player base from the outset.

Free weekends and large sales attempted to entice new players, but the market for shooters has shifted. Disintegration was full price when the most popular multiplayer games have moved to free-to-play models.

A screenshot of Disintegration’s gameplay from PC Mag

The end of the tailspin feedback loop came six months after release.

Multiplayer was removed completely from the game, likely due to the cost to maintain inactive servers.

This small studio couldn’t weather the storm and shuttered the following year.

The announcement from V1 Interactive as shared on Twitter.

But neither Disintegration nor Hyper Scape were critically acclaimed.

Maybe if you release a game with an initially warm reception, the rest will work itself out?

Sadly, sometimes great ideas still fall short.

Survival of the fittest killed Evolve

That was the case with Evolve. An asymmetrical 4v1 shooter from Turtle Rock Studios, who also developed Left 4 Dead under the name Valve South.

The gameplay of Evolve from the Steam Store Page

Evolve launched with solid player numbers, but quickly declined for a few reasons.

The monetization strategy was increasingly confusing and placed a lot of content behind DLC paywalls.
The slow release of content meant it was updated far slower than many other multiplayer titles.
Gameplay issues weren’t patched quickly enough.

DLC available for Evolve on the day of release, credit to nerdSlayer Studios on Youtube

Despite a huge amount of hype before release, and a warm critical reception, players weren't as satisfied with the game in terms of design and variety.

Updates may have addressed these issues, but a policy at the publisher apparently prevented this.

The policy, according to lead writer Matt Colville, was to only update the game once every three months. This infrequent update schedule did not match player expectations.

“If you release a game that lives and dies on online competitive play, maybe let the DevTeam update the game more often than once every three months.” - Matt Colville

A month in, Evolve was haemorrhaging players in the thousands.

Evolve was rebranded as Evolve Stage 2, but it could only stem the loss of players

Turtle Rock attempted a recovery, with the form of updates that came as season passes, but paying for the game and then these updates was not well received.

So Evolve entered “Stage 2”.

The game shifted to free to play on PC and saw a substantial bounce in numbers. Sadly, this was not enough.

The tailspin had been reverted once, but without a strong strategy following the launch of Stage 2 — player counts again declined drastically.

Evolve’s entire lifespan, from launch to the discontinuation of dedicated servers

Evolve Stage 2, was taken offline in September 2018. As it required an online connection and the dedicated server binaries were not released publicly, Stage 2 is unplayable.

The original game, should anyone want to revisit what once was, limps on with peer to peer networking available.

Author’s note: as of July 23rd 2022, Evolve: Stage 2 has been updated with working matchmaking, and peer-to-peer servers available in dedicated Discord servers.

Evolve in another life: Dead by Daylight

Evolve was unique on launch, but a year following its initial release another 4v1 multiplayer game hit the market, Dead By Daylight.

Dead by Daylight promotional image

With an initially mixed launch, Dead by Daylight showed that you can build a player base over time if you play your cards right.

March 2022 Dead by Daylight player counts, with a steady on and off-peak rhythm

At the time of writing, Dead By Daylight maintains a solid player base almost eight years later, peaking in mid-2021.

In early 2022, Dead by Daylight hit 50 million total players.

This recovery has been attributed to a solid content schedule, updates to networking that moved away from only peer to peer connections, and more robust social features.

Player retention: aiming for a moving target

Multiplayer games have unique hurdles to overcome.

The gameplay itself is only a piece of the puzzle. To retain players over a long time developers need constantly adjust and aim for a shifting target.

Games are expected to have a longer tail than ever with the live service business model, and the launch of new titles can always attract players away from your game.

So to retain players, and avoid falling into this tailspin, games must be reactive to the demands of the community and industry.

Above I’ve outlined games that I feel fell short on this, and how they missed the mark.

If you’re a game developer and looking to launch a multiplayer game, consider every aspect of your post-launch strategy.

It isn’t enough to be good. New multiplayer games have to be firing on all cylinders.

But, if you as a developer manage all this, gamers will fill your servers for years to come.

Maybe, long after release, when you have moved onto a new project. The community will maintain your legacy.

Afterword: hope is on the horizon

Luckily for my personal favourite on this list, there is hope.

Titanfall 2’s Northstar mod has revitalised the community around the game.

Titanfall 2 has a particularly passionate community who refuse to let the game die. Some of the members of that community have rallied together and launched a mod called Northstar, allowing players to host custom Titanfall 2 matches on privately hosted servers.

These servers are safer from DDoS attacks, and will hopefully keep players piloting titans and wall running in that game for years to come.

Remember, it is easier to retain a player than to bring a player back, and retaining a player is hard.

2022-06-28T00:00:00Z

Every swing of a sword, every spell, every headshot, every critical hit — it all relies on servers.

Fractions of a second can change everything.

High ping stacks the game against you.

You want the best for your players. For high-stakes fast-paced games, you need to minimise the chance of a lost packet, or a desynchronised game. No matter what, players need to have faith in reliably low ping.

But that’s not always possible.

If you’re lucky, the majority of your player base might be close to one server. But players might be spread out across entire continents.

How do you make sure there’s an option for every player?

Central servers.

What are central servers?

Central servers are designed to walk the tightrope between acceptable latency and maximum reach, primarily built in locations with high-speed connectivity such as capital cities.

They’re for gaming groups spread across regions.

They’re for gamers who live off the beaten track.

They’re for the late-night gamer when the other servers are winding down.

When considering a central server location, the average ping matters more than the lowest ping. To give a larger group of users good latency, most users will experience slightly higher latency than if the server they were connecting to was local.

In an ideal network, distance is the main limiting factor, and central servers aim to eliminate the longest distances.

From personal experience, there are games that I have put down forever because there wasn’t an option to play with my friends without someone on unplayable ping.

For competitive games, someone was always at a disadvantage, and if we were playing cooperatively they’d be playing with a severe handicap.

While this isn’t right for every game, countless games would benefit from having a central server.

What are the benefits of a central server location?

Benefit #1: Extensive reach

Central servers reach a wider audience of players

With a central server location, developers get massive reach across a given region. Players can play with friends, old and new, and connect with people that would be impossible to reach otherwise.

Take APAC Central for instance. It can reach 1.45 billion gamers in twenty countries. If you compare that to an average server in Tokyo, the difference in reach is orders of magnitude.

For each dollar spent by the developer you’re reaching more people, and you’re giving them each a more comprehensive experience.

Benefit #2: Better latency

Player skill over geography.

When those players jump online, it isn't just a wider range geographically, but in terms of skill.

This means that distant players ‘meet in the middle’ with similar ping, avoiding unfair advantages.

By investing in a central server, developers provide a fair playing field for gamers to hone their skills.

Central servers help build a more competitive and engaged community across the entire region.

Benefit #3: Bigger player pool

Connecting players beyond borders.

By investing in central servers, you can unite your game's population on fewer servers. This makes matchmade activities much easier to fill with fewer active players overall.

A consolidated player population with quick searches keeps players in the gameplay loop and off menus.

A game that you spend time playing instead of waiting? That is a game where players keep coming back for more.

Central servers change the game

When these benefits work together in tandem, it greatly improves the multiplayer of any game.

It means that no matter the time of day, people can jump on your game and know that they will be able to find a match quickly and at their skill level.

They won’t have to wait for prime time, and they won’t be stuck in a queue for a match.

Centrality isn’t a silver bullet

No tool is right for every job.

For the best ping, a local server will always beat a central one.

If you live in the same city as a data centre for your game of choice, the connection delay will be almost imperceptible.

At the highest level of play, there would probably be players who would camp inside a data centre if it shaved a few milliseconds off their ping.

At the expense of everyone else.

Some players would live here, if they could.

This is why you need central servers to combat ping inequity.

As a game becomes more popular, more players connecting from across the world will need to be supported

The choice every developer will need to make becomes:

Faster connections for some, with higher ping for everyone else
An increased average ping, but that is equitable across many players

Choosing the second option means more players honing their skills against a variety of opponents. It means more gamers playing with far-flung friends. It means a healthier game population united on one server. It means a game that they can consistently play, regardless of whoever is online.

With all these benefits in mind, central servers are a powerful tool to support an already extensive network or lower costs without sacrificing reaching players.

But where are the world’s central servers?

Multiplayer games need server coverage across the globe.

Take Riot Games’ Valorant as an example.

(for reference, we're not connected with, or related to Riot Games or Valorant)

Riot Games has central servers in North America (Iowa) and Europe (Frankfurt) but lacks total coverage in other regions.

South and Latin American players have Santiago, Mexico City, Miami, and Sao Paulo, none of which are central.

African players only get one option in Istanbul.

Lastly, the geography of Asia-Pacific (APAC) makes the most common regions poor choices for a central server, separating the players substantially.

Valorant’s Asia-Pacific server locations

Functionally, Valorant’s many server locations probably work for a majority of players, especially given its high player count.

Splitting your players too much can kill your game

But for smaller games, separating the population into so many server locations may damage the game, as each individual location may suffer from a lack of a viable population. Not to mention that having this many server locations may be financially unviable for many studios.

Without the support of a huge publisher, or a massive amount of players, a central server is an invaluable tool for any multiplayer game.

But in APAC, the region with the most gamers, there simply wasn’t an alternative.

Where would you put a central server in APAC?

APAC had been left behind, despite being the largest games market globally.

Servers are located across Asia, but they don’t serve a potential player base equally.

The best option used to be Singapore or Hong Kong. While that was great for a player there or in Bangkok, their competitors in Sydney facing a huge ping was at a massive disadvantage.

So where would an APAC central server go?

Some Australian locations had potential.

Darwin had geographical benefits, but lacked the infrastructure. Perth was considered but is too far south to give benefits to all but a handful of users.

So we looked further afield, and instead established our central server on an island in the middle of the Pacific, Guam.

We built it from the ground up, Asia-Pacific’s first truly central server location. It connects Asia, Australia, and even the east coast of the USA.

We call it APAC Central.

The promise of central servers is now a reality in Asia-Pacific, Guam (APAC Central).

The existing options in APAC weren’t good enough for the number of gamers trying to connect across the region.

So we took a risk and established what is a true central server location for APAC.

The island of Guam happened to be a perfect location with access to a web of subsea cables making it the crossroads of APAC connectivity.

Now APAC can finally take advantage of all the benefits of a central server.

Further reach

Better latency

More players

The only thing that stood in the way? Putting servers on a Pacific island.

2022-07-18T00:00:00Z

What is a game server?

A game server is a computer that acts as a central authority to define the current state of a multiplayer game.

Players send the server updates about what they’re doing in-game — running, jumping, taking or receiving damage, fishing, anything. The server translates those inputs and returns the information back to the players.

So how does the server communicate that in-game “reality”?

The server processes all inputs, settles inconsistencies, and transmits regular updates to players, providing a stream of snapshots of all elements in the game, updating the game state every tick.

Sounds complicated? To explain, we need to cover two key concepts: game state and tickrate.

Game state: the single source of truth

The game state (or “tick”) is a snapshot in time of the properties of every object in the game world.

With lots of players doing things at the same time, often near to each other in-game, a server has to keep track of what happens and when — to make sure everyone’s on the same page.

When the game state is calculated, the server transmits it to each connected player’s device — after this, each player’s game world is refreshed.

And this happens many times each second.

Tickrate: the frequency of game state updates

Tickrate is the number of times an updated game state is calculated and broadcast by a game server to players every second.

The higher the tickrate, the more precise a game simulation is.

If a server sends 64 updates to players each second, it’s referred to as a “64-tick”, or “64 Hz” server (because Hertz, or Hz, is the unit of frequency).

Here’s what that looks like in action:

Visualising a 64-tick server

A server with a tickrate of 64 — every second, the server sends 64 state updates

Visualising a 32-tick server

Compare that with a 32-tick server, which sends updates half as often:

A server with a tickrate of 32 — every second, the server sends 32 state updates

What tickrate means for gamers

When more game states are calculated every second, more movement and action can be accounted for — so a higher tickrate produces a more precise, high-fidelity gaming experience.

That means split-second actions in the game can be factored in — and it can make the difference between life or death in-game — especially in a first-person shooter.

When does tickrate matter?

Not all games need game servers with a high tickrate.

Turn-based games, like Hearthstone, don’t need a high tickrate since the game does not update multiple times a second.

Real-time games like DOTA, Counter-Strike, or Valorant do need higher tickrates to maintain synchronised game states.

Games with real-time action usually feature higher-tickrate servers (64 or 128-tick) for the best gameplay experience.

How does tickrate affect a game server?

As tickrate increases, more processing power and more bandwidth are required.

When you double the tickrate from 32 to 64 ticks, you increase the load on the server, as players send data more frequently, and the server needs to calculate the game state more often.

So to game on a high tick server, the server needs to be fast enough to handle the extra workload.

What type of hardware is in a game server?

Dedicated game servers

A dedicated game server is the most common type of game server.

Under the hood, a dedicated game server is a powerful computer designed to be run 24/7. Like any other computer, it features a motherboard, CPU (processor), RAM (memory for short-term storage), power supply, fans, and a case.

The standard components in a game server

The single most important component in a game server is the CPU, which processes all the data in an online game.

By having a faster CPU, a game server can run more calculations every second — so it can handle higher tickrate gaming workloads.

While processing data is important, a game server is also optimized for high-throughput network connectivity. Because lots of people connect and send data to the server, a game server will typically have a powerful network card (or cards) to support lots of simultaneous connections.

Generally OneQode has four network cards running at 10 gigabits per second on our servers. For some perspective, your home computer probably has a one gigabit per second network card.

Peer-to-Peer game servers

Peer-to-peer game servers are less common — and games that use them have fewer concurrent players in a match (e.g. Super Smash Bros. Ultimate).

In peer-to-peer games, the game assigns one player’s device to also host the match, so that player’s device runs the game server in the background, while the player is also playing on the same device.

Yep — your console can be a game server.

While peer-to-peer games are cheaper for the developers (as they don’t have to buy dedicated servers), it usually isn’t great for players.

Running both the game client and game server at the same time can be taxing on the device — and all the players in the game are at the mercy of the “host” player’s internet connection. If the host’s connection is poor, everyone in the game suffers.

Peer-to-peer game servers can’t handle a large number of players because they’re limited by the connection quality, not built for handling lots of connections at once, and can’t focus all their power on hosting the game.

No matter the kind of server, it probably won’t run on one of these.

Where are game servers located?

Most dedicated servers are in data centres

Dedicated servers are usually located in purpose-built data centres in major cities, designed for 24/7 uptime and operation.

Data centres have built-in cooling systems (to keep hardware running optimally), insanely-fast internet connectivity, and most have two separate power sources to ensure round-the-clock availability.

For gaming, where connection quality is critical, data centres are the best place for game servers.

A row of server racks in a data centre

Other servers are in people’s homes

Some people host their own dedicated game servers at home, although this isn’t a recommended approach.

People who do this need to organise a good quality internet connection with large upload and download speeds, as well as backup power and cooling.

And lastly, peer-to-peer game servers run on the hosting player’s game console or device — so these can be anywhere — usually in their house.

How are game servers set up?

To set up a game server, a server needs to have the game’s “server binary” installed.

This is the specialised software that runs the game world, keeps track of in-game actions and object properties, and controls the calculation of the game state in real-time.

Players run a separate “client” version of the multiplayer game that sends information to (and receives information from) the game server.

For many games, the developer organise servers

Many of the most popular multiplayer games like Fortnite, Valorant, Overwatch, and Apex Legends have servers that are set up by the game developer — and they don’t release the server binaries.

So you can’t host your own game server for games like this.

Why don’t they release server binaries? There are a range of reasons, including:

Privacy
keeping server binaries private can stop cheating or game modification
Quality control
managing servers and server hardware in-house means it’s set up exactly how they want it
Ease of update
when the developer updates the server software, they can quickly roll the update out

The downside of this is that if the developer decides to shut down the servers or stop working on the game, the multiplayer community dies with it.

For some games, you can host your own servers

Some developers release the server binary for their games, which often leads to the game having a much longer life span.

Games that let you host your own servers include:

Minecraft
Team Fortress 2
Counter-Strike: Global Offensive (CS:GO)
Valheim
ARK: Survival Evolved

To set up a server, there are two ways to do it:

Option #1: Manually set up your own server

To do this, you’ll need some technical capability, preferably experience with Linux. And you’ll need access to a server.

The easiest way to set up a game server is to use a game server management tool like:

LinuxGSM
Free, open-source command-line application supporting 125+ games
Pterodactyl
Free, open-source server management panel supporting 46+ games.
TCAdmin
Paid server management panel supporting 86+ games
SteamCMD
Command-line application for Steam-based game servers. The above tools use this for Steam games.

Option #2: Using a game server hosting provider

If you don’t want to handle the technical setup, you can purchase ready-to-go servers from a game server hosting provider — and there are loads to choose from.

One example of a game hosting provider is PacificES, which we recently acquired. They offer superfast 128-tick CS:GO servers for less than a dollar a day.

Well, that was an introductory guide.

Want to learn more?

There’s some brilliant thinking that goes on behind the scenes with multiplayer games.

How does a server make sure everyone’s on the same page? How do developers make games feel less laggy?

Now we’re in the territory of netcode and game networking.

If you’re interested to read more about how game developers solve these challenges, check out these excellent article series:

Be advised: the rabbit hole is deep.

🕳🐇

2022-07-28T00:00:00Z

Are your teammates tired of you blaming it on the lag?

You miss at the pivotal moment, and your team is wiped out, defeated, and sent back to the lobby with nothing but bruised egos.

Avoid this happening every drop in Apex Legends

But you didn’t choke. Your internet did.

“Sure,” they all say. But you can hear the doubt creeping into their voice.

If only you could fix it.

But that’s something for your provider to deal with, there’s nothing you can do, right?

Wrong.

There’s plenty that can be done to improve your connection and your game.

The first step is to get the best connection between your console or PC and your router.

Get off Wi-Fi and reach for the ethernet cable

If you can only do one thing, our engineers recommend grabbing an ethernet cable.

Wired connections offer better throughput, lower latency, and more stable connections than wireless.

This will make the single biggest difference to your ping

Don't be fooled by 'gaming routers'

Your Wi-Fi might be fine for browsing YouTube & Netflix, but when it comes time to boot up your favourite game, consider plugging in the ethernet cable.

Wi-Fi can be interfered with or disrupted by other electronics, metal, or even just small barriers.

A hardwired ethernet cable, particularly one of high quality, will have an almost lossless and instant connection.

This Nord VPN blogpost for example, where they outline the specific benefits of using an ethernet cable clearly.

To put it simply?

Ethernet is more reliable. Interference from other networks, and other devices, won’t affect the hard-wired ethernet connection.
Ethernet is more secure. An ethernet cable is less likely to degrade the signal over the same distance.
Ethernet is faster. Depending on the network, an ethernet connection could be more than twice as fast as wireless.
Ethernet will lower your ping.

Ethernet beats Wi-Fi for latency and reliability

The distance between your computer or console and the router is fully within your control, so you might as well take advantage of the best possible connection.

It isn’t the most exciting option, but an ethernet cable is the best solution.

But my router is in another room!

The first and easiest suggestion is to just buy a long ethernet cable and some cable organisation clips.

It isn’t necessarily the best-looking option, and if your doors don’t have a lot of clearance around the frame you may run into some issues.

Money no object? If you want to hide it fully, you can pay an electrician to run an ethernet cable through the walls.

Can’t run ethernet? There are some backup options.

You don’t want to run a cable through the entire house.

You’ve got a dog, a cat, a kid, or a housemate who would take issue with that.

Luckily, we’ve got options for you to not become the most hated housemate.

Powerline: Internet over existing wiring

Powerline products use the existing wiring in your house to transmit data. Any existing outlet can be turned into powerline port with their adaptors, which lets you connect an ethernet cable into the outlet.

Plug one in next to your router, and another in the room where you want the port, and the two will connect through your home wiring.

A TP-Link branded Powerline adaptor.

A powerline solution’s effectiveness will differ based on the wiring in your home.

Newer homes with high-quality wiring will allow powerline connections to offer lower latency, higher stability, and higher throughput than Wi-Fi.

Older homes with poor existing circuitry may have a worse overall connection through powerline than Wi-Fi.

Unless you have far more tools or have wiring diagrams for your property, it is hard to gauge if a powerline connection will solve your networking issues.

So if you want to check out powerline but don’t know if your home wiring will be good enough? Keep the receipts.

Mitigate your risk and make sure there is a return policy just in case!

Hot tip: opt for the premium option if you can afford them. Either they work and you’ve got a great one, or you return them.

MoCA: Repurpose your TV Connection

An alternative to powerline is MoCA, standing for Multimedia over Coax Alliance, which was created to stream video.

If you have a coaxial (TV aerial) connections between the two rooms that you need to connect, MoCA could work for you.

If you have these in multiple rooms, you can use them to extend your physical network.

Unlike power cables, Coax was designed with data transmission in mind. In some cases, this will make it a faster connection than powerline.

MoCA will still introduce a small amount of latency to your connection over Ethernet, and you may not have the connections in the rooms you need to connect.

Keep in mind that both Powerline and MoCA require some investment, and quality may vary. So if you go with either be prepared that it might not improve the connection.

Now that you’ve gotten yourself on a wired connection (hopefully), there are a few other tips and tricks to optimize your set-up.

Try a gaming network optimizer without swapping ISP

We’ve all heard the advertisements for VPNs.

VPNs sned your data over a private tunnel to secure your connection.

Companies like WTFast, or Exitlag, do a similar thing, but to lower latency in games.

Instead of sending your game data entirely over your ISPs network, you connect to the nearest accelerator server.

After your data has reached the accelerator, it travels via their gaming-optimized network, lowering your ping.

You can check the games WTFast supports, or try Exitlag’s list before you download.

The best part? Both offer a free trial so you can see which one works better for you.

For transparency, WTFast is a OneQode partner, and they use our high-end designed for gaming connectivity across Asia-Pacific.

Lower your system latency

This suggestion is slightly niche.

If you play on a PC with an Nvidia graphics card, you can take advantage of Nvidia Reflex.

This is the only suggestion on this list that isn’t reliant on your internet connection but is limited by your platform and GPU.

Image from Nvidia's system latency webpage showing the latency between your inputs and your display.

The full list of supported games, mice and monitors primarily features popular shooters, where the difference will be most noticeable.

If you’ve got an Nvidia card, and you’re playing games that are taking advantage of the SDK, you can hopefully shave a few milliseconds off your internal latency.

But if you’ve already optimized your hardware as best you can, and you’re still struggling against a poor network, here are some options to improve your set-up.

Move house (joking... unless?)

If all else fails, there’s one thing to keep in mind.

You could move house.

This is obviously not a reason to move, but internet connectivity is something to keep in mind if your lease is up and you’re looking for a new place.

If you live in Australia you might remember our ill-fated NBN program, after a botched rollout there are now a variety of potential connections that can drastically alter the quality of your internet.

We don’t recommend you move house to improve your gameplay... but if you’re already moving.

If you live in Australia, you can use this NBN address checker to see if places you’re looking at are going to offer you the best connection possible for your online play.

There are a variety of overlapping technologies that make up the networks across Australia. Here’s how to decode the tech talk.

Fibre to the premises: the best option.
Fibre to curb, or fibre to the building: good, but not the best.
Hybrid Fibre Coaxial: far from the best.

Is there anything to avoid?

If you see fibre to the node, “run away screaming” (according to our engineers). This connection is still bottlenecked by the old copper wiring that made up connections before fibre was the new standard.

Basically, that option will cause you more headaches than anything going forward, so avoid it if possible.

The final step

Your computer is directly connected via ethernet. System latency is as low as possible. You've moved to a fibre connected house.

But you’re still dying. You’re still letting the team down.

There’s one more option we can recommend. It won’t be easy, but if you’ve tried every other step along the way, this might be the only thing you can do.

You could try getting better at the game.

It’s worth a shot.

If that doesn’t work, don’t tell your friends how hard you worked to get your internet to work, and keep blaming it on the lag.