An actual intrusion case bypassing two factor verification

Tuesday, March 1, 2016

Last week, cloud service provider linode posted a blog post describing how its cloud service client pagerduty's server was hacked.

Network intrusion events have been rampant, but it is worth noting that the attacker enters pagerduty's server through the management panel of linode. To do this, the attacker must bypass the highly secure two factor authentication (2fa).

In a typical 2fa application, user authentication generally follows the following process:

When the user enters the user name and password on the website or other servers, the user's terminal device will run TOTP algorithm to generate a one-time server password, and then the user will enter this password into the server. The server will also run TOTP to verify the one-time password. In order to verify successfully, the clock of the user's device needs to be roughly synchronized with that of the server. A single key used for all subsequent authentication sessions must be shared between the server and the user device through a secure channel before.

Google authenticator TOTP screenshot

The security of TOTP algorithm depends on the security of shared key. Once the key is stolen, TOTP will lose all its power whether it flows from the server or the client, because the attacker has the ability to generate a one-time password by holding the key. Linode uses a time-based one-time password (TOTP) as the second authentication factor.

Let's see how attackers can use the configuration problem on linode to invade the client:

In this intrusion event, the attacker obtained the linode credentials of a pagerduty employee, including 2fa, which can log in to the linode management interface. Through this interface, an attacker logs into the server where pagerduty is hosted in linode.

But oddly, pagerduty was convinced that the source of the stolen credentials was not the client (for example, some mobile malware), because the employee had no 2fa credentials at all, and his mobile phone was emptied months before the incident. Later, pagerduty informs linode of the blackout event and submits the IP address of the server used for the attack. Surprisingly, the attack server is also hosted on linode, so linode can thoroughly check the full image of the server.

As a result, it is found that all the tools and data used to crack the TOTP algorithm of linode are stored on the server. It includes: software that can generate TOTP password with TOTP key, software and key that can decrypt TOTP key protection mechanism. Commands that successfully generate a one-time password are also found in the command line history.

It's usually not a good idea to use a cloud service provider's own hosted server to attack, because then the cloud service provider can thoroughly check the attack host (as linode ultimately did). If the attacker's machine is hosted by another provider, such a thorough investigation is unlikely. Therefore, it can be speculated that the attacker has to launch an attack on the server hosted by linode for some reasons.

One possible explanation is that an internal vulnerability in linode can only be triggered from within linode. The linode security team has found a vulnerability in the SSH gateway of its lish shell, which may be used to obtain the information found on the attacker's machine image. The details of the vulnerability were not disclosed. However, the blog pointed out that the first improvement made by linode after the blackout was to restrict lish's direct access to the credential information: "many of our applications (such as linode manager and lish) perform user authentication. Previously, these applications accessed the database directly to obtain credential information. "

To summarize: a reasonable technical explanation for the event of linode / pagerduty being hacked is that the attacker took advantage of lish's direct database access function to obtain other user credentials stored in the same database (such as through SQL injection). However, to take advantage of lish, an attacker has to operate inside the linode system to launch an attack from the server hosted by linode.

When two factors combine into one

In most cases, the management account voucher is the weakest link in this kind of security event. Linode two factor authentication only makes sense on the client side, because it requires that what the client knows (password) and holds (mobile app with embedded key) are valid at the same time. However, on the server side, the two factors merge into one, that is, the password and the TOTP key can be accessed by a single application, and it is likely to be stored in the same row of the same database.

The cloud environment introduces opportunities as well as risks. Cloud service providers must put security first, and cloud customers should consider security as a key factor when purchasing services.

It is reported that pagerduty has changed another cloud service provider due to the blackout.