In my last post, I’ve discussed the basics of the secure Web protocol HTTPS (or HTTP-over-SSL), which gives you some idea what HTTPS is and why it is important. In this post, though, I would like to go into details how the SSL (or TLS, I will use the latter in this post so that the new name sticks with you) part of HTTPS works, what are cipher suits, and why are those important. This post is targeted more towards software developers or IT professionals, although I will try to use language simple enough for everybody to understand.

The first thing I would like to ask you to do is to go to https://www.ssllabs.com/ssltest/analyze.html and get the SSL report for your website (or any other site for that matter). Keep the page open because I will refer to portions of it throughout the post. If scroll down the page, you will see a section Configuration and subsection about cipher suites – this is what I will explain. I have analyzed my own domain toddysm.com on the SSLLabs site and will use it as an example. One of the cipher suites that you will see for my domain (and most probably for yours) is this one TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, which will be the concrete example that we will decipher.

Now, let’s look at the so-called TLS handshake that happens when you try to load an HTTPS page from a website. Although the actual TLS 1.2 RFC-5246 has a pretty detailed explanation of the handshake protocol, I find IBMs Knowledge Center overview of TLS handshake much simpler and clearer to understand. Let’s look deeper into the steps one by one (italicized steps text is a credit to IBM’s Overview of SSL or TLS Handshake):

TLS Protocol Handshake Sequence

Step 1: The client sends a “client hello” message that lists information like TLS version and the cipher suites supported by the client in the client’s order of preference. The message also contains a random byte string that is used in subsequent computations.

The most important thing to note is that the information sent as part of the “client hello” message is not encrypted (i.e. it is in plain text). Thus, if somebody is sniffing on your traffic, they will know two pieces of information: 1.) the cipher suites your browser supports (in order of preference) and 2.) the random byte string used to create the master secret. It is also unfortunate that not all modern browsers (or TLS clients) tell you what is their “order of preference”, which is a shame because the negotiated cipher suite may not be the most secure one and having the ability to change the order of preference may increase your security.

Step 2: The server responds with a “server hello” message that contains the cipher suite chosen by the server from the list provided by the client, the session ID, and another random byte string. The server also sends its digital certificate.

Once again, the message exchanged in this step is not encrypted. For somebody listening to your traffic, all this information may be of some value.

Step 3: The client verifies the server’s certificate.

This is done through a trusted third party called Certificate Authority. Another option is if the server certificate is already added as a trusted certificate on the client’s machine.

Step 4: The client sends the random byte string that enables both the client and the server to compute the secret key to be used for encrypting subsequent message data. The random byte string itself is encrypted with the server’s public key.

The first thing to note here is that this is the first message that is encrypted (eventually) – in this case with the server’s public key and only the server can decrypt it. The next thing to note is the importance of the yet another random byte string. This random byte string is also called pre-master secret and it is used together with the previous two random values to generate the so-called master secret. Depending on the sophistication of the algorithm used to generate this pre-master secret, your connection with the server may or may not be vulnerable.

Let’s look back at our  SSLLabs scan and the different ciphers that your server supports (see picture below).

List of supported cipher suites

The example I took above is the fifth from the list and means the following (credits to Daniel Szpisjak for his explanation of TLS-RSA vs TLS-ECDHE-RSA vs static DH on StackExchange):

  • Ephemeral Elliptic-Curve Diffie-Hellman (aka ECDHE) is the algorithm used to generate key pairs on both the client and the server. The public keys are the random values exchanged in steps 1 and 2 above.
    One important thing to note here is that the use of the ECDHE algorithm provides the so-called forward secrecy, which means that if future communication channels between those parties are compromised, the keys cannot be used to decrypt previous conversations.
  • The pre-master secret is this case is NOT the shared secret key as per the Diffie-Hellman algorithm as some may think. The shared secret key is never sent over the wire. This random number is encrypted using the server’s RSA public key (according to IBM’s explanation) and is used to generate an uniform shared secret using a hashing algorithm. The encryption, in this case, is not so important because this is not the actual key used for encryption.
    The RSA part of the cipher suite also denotes two more things:

    • The public key type used to authenticate the server in step 3 (and the client if client authentication is required – part of step 4)
    • And the public key type used to sign the ECDHE public keys during the exchange

In this step the client uses the following information to generate the master secret:

  • Clients private key
  • Server’s public key

Then the client uses the random value to generate an uniform hash of the master secret that is used to encrypt the traffic. In our example, the hash is generated using SHA384 (the last part of the cipher suite).

On the other side, the server uses the following information to come up with the same master secret:

  • Client’s public key
  • Server’s private key

Then the server uses the random value to generate the same uniform hash of the master secret that is used to encrypt the traffic.

Important to note is that using the ECDHE algorithm for key exchange, both the client and the server generate the master secret independently and do not share it over the wire, which limits the opportunities for sniffing the secret.

Now, let’s move forward and look at the remaining steps in the handshake and decipher the rest of the cipher suite…

Step 5: If the TLS server sent a “client certificate request”, the client sends a random byte string encrypted with the client’s private key, together with the client’s digital certificate, or a “no digital certificate alert”.

This step is used only for authentication purposes for the server to make sure that it communicates with the correct client. When browsing this step is not required because the browsers are not authenticated, however, if you develop services that need to authenticate their clients using certificates, this is an important step.

Note that the cipher suite is already agreed upon and in this step, the client should send a public key in the previously agreed format. In our case RSA.

Step 6: The TLS server verifies the client certificate.

This step is exactly the same as step 3 but on the server side.

Step 7: The TLS client sends the server a “finished” message

Step 8: The TLS server sends the client a “finished” message

Those two steps just confirm from both sides that the handshake is complete and both parties can start exchanging traffic securely.

Step 9: For the duration of the TLS session, the server and the client can now exchange messages that are symmetrically encrypted with the shared secret key.

The communication between the TLS server and the TLS client is now encrypted using the symmetric key (the master secret hash) that both parties generated independently. For the purpose of this communication, AES-256 symetric key encryption algorithm is used in our example. This is contrary to the popular belief that the traffic is encrypted with the public key of the server on the client side and decrypted with the private key on the server side.

To complete the example, we need to explain two more parts:

  • GCM is the mode of operation for the symmetric key cryptographic block ciphers like AES. Another mode of operation is HMAC. Both are used to provide both data authenticity and confidentiality but GCM is proven to be quite efficient and it is widely used.
  • The SHA384 is the Secure Hashing Algorithm used to hash every message in the above mode of operation in order to ensure the integrity of the message.

Keep in mind that the flow above describes the superset of steps for establishing TLS secure channel. Depending on the TLS version and agreed upon cipher suite some information may be ignored during the exchange, which may make your communication vulnerable.

If you want to get really technical how the handshake works, you can read Moserware’s post that dates back to 2009 http://www.moserware.com/2009/06/first-few-milliseconds-of-https.html (not a lot have changed though) or watch the Khan Academy video on Diffie-Hellman Key Exchange.

With all the cybersecurity reports that we hear about lately, increasing our awareness and knowledge of online security raises higher on the task list. Unfortunately, by using crazy acronyms and fancy words cybersecurity experts do not make it easy for normal people to understand online security. Hence, I thought, a series of posts that explain the online security concepts can be beneficial not only for normal people but also for some IT professionals.

In this post, I would like to explain the basics of secure communication on the Web and the terminology around them as well as give you some guidance why you should communicate securely on the Internet.

Let’s start with the acronyms and some terminology:

  • A protocol is a procedure or sequence of steps used to communicate over the Internet (or in general). For example, it is a “protocol” when you introduce yourself in a business meeting in Western countries to 1.) say your name 2.) do a handshake and (optional) 3.) hand over your business card.
  • HTTP stands for HyperText Transport Protocol and is the main protocol used for the exchange of information on the Web. Note that the Web is not the Internet! Although more and more of the communication on the Internet goes over HTTPS, there are other applications on the Internet (like email for example) that do not use HTTP as a protocol for communication. Data sent over HTTP is not secure and everybody who can sniff (listens) to your traffic can read what you are exchanging with the server. It is like chatting on the street with your friend – everyone passing by can hear and understand what you are talking about.
  • HTTPS stands for Secure HTTP (or also called HTTP-over-SSL, which is the correct name) and is (as the name implies) the secure version of HTTP. Data sent over HTTPS is encrypted and only the sender (your browser) and the receiver (the server) can understand. It is like chatting on the street with your friend but in a language that two of you have invented and only you two can understand. Keep in mind that everybody else can still hear you. If you don’t want everybody else to understand what you are talking about, you use this language.
  • SSL stands for Secure Socket Layer and it is used to secure the communication for many different protocols on Internet (not only HTTP, which is used for browsing). Using the street chat analogy, imagine that instead only you and your friend, you have two more friends with you. You (HTTP) whisper a message to one of your friends in plain English but only she can hear it. She (SSL) then uses a special language that she and the second of your friends have invented (and only two of them can understand! Note that even you don’t understand the language they are speaking) and communicates your message to that second friend of yours (SSL again). The second friend then translates the message from the special language into plain English and then whispers it to your third friend (HTTP) so quietly that only he can hear. Here a visual of that process:
    HTTPS Analogy
  • TLS stands for Transport Layer Security and it is the new, better, more secure, more advanced (and many more superlatives you will hear) version of SSL.

From the explanations above, it may be obvious to you why you should use HTTPS when you browse the Web but if it is not, think about the following. Sites are collecting all kind of information about you. They use it to provide more targeted information (or advertisement) but they also integrate with third-party sites including advertisement sites, Facebook, Twitter, and Google (the latter may also be used for authentication purposes). The information they collect includes but is not limited to things like your location, IP address, browsing patterns, laptop details and quite often information that is used to automatically sign you into certain services. This information is automatically exchanged between your browser and the website you are visiting without your knowledge. Thus, if the website you are visiting doesn’t use HTTPS protocol, your information will be easily readable by every hacker that monitors your Web traffic.

If you own a website, you should care even more about HTTPS and make sure you configure your site to use only the HTTPS protocol for communication. The reason is that the browser vendors are starting to explicitly notify users if the site they are visiting doesn’t support HTTPS and mark it as insecure. In addition, Google will start ranking sites that do not support HTTPS lower in their search results, which may have a significant impact on your business.

With this, I hope you understand the importance of HTTPS and the implications of not using it. In the next post, targeted more to IT professionals and software developers, I plan to go more technical and explain how the TLS encryption works.

Since the sprawl of mobile apps and web services began, the need to create new usernames and passwords for each app or service started to become annoying and as it proved out decreases the overall security. Hence we decided to bet our authentication on the popular social media platforms (Facebook, Twitter, and Google) but wanted to make sure that we protect the authentication tokens on our side. Maybe in a later post I will go into more details what the pros and cons of this approach are but for now, I would like to concentrate on the technical side.

Here are the constraints or internal requirements we had to work with:

  • We need to support multiple social media platforms for authentication (Facebook, Twitter, and Google at minimum)
  • We need to support the web as well as mobile clients
  • We need to pass authentication information to our APIs but we also need to follow the REST guidelines for not maintaining state on the server side.
  • We need to make sure that we validate the social media auth token when it first reaches our APIs
  • We need to invalidate our own token after some time
The flow of events is shown on the following picture and the step-by-step explanations are below.

Authenticate with the Social Media site

The first step (step 1.) in the flow is to authenticate with the Social Media Site. They all use OAuth, however, each implementation varies and the information you receive back differs quite a lot. For details how to implement the OAuth authentication with each one of the platforms, take a look at the platform documentation. Here links to some:

Note that those describe the authentication with their APIs but in general the process is the same with clients. The ultimate goal here is to retrieve an authentication token that can be used to verify the user is who she or he claims to be.

We use Tornado Web server that has built-in authentication handler for the above services as well as generic OAuth handler that can be used for implementing authentication with other services supporting OAuth.

Once the user authenticates with the service the client receives information about the user as well as an access token (step 2. in the diagram) that can be used to validate the identity of the user. As mentioned above, each social media platform returns different information in the form of JSON object. Here are anonymized examples for the three services:

It is worth mentioning some differences related to the expiration times. Depending on how you do the authentication you may receive short-lived or long-lived tokens, and you should pay attention to the expiration times. For example, Twitter may respond with an access token that never expires ("x_auth_expires":"0"), while long-lived tokens for Facebook expire in ~60 days. The expiration time is given in seconds and it is approximate, which means it may not be exactly 60 mins or 60 days but a bit less.

Authenticate with the API

Now, that the user has authenticated with the Social Media site we need to make sure that she also exists in our user database before we issue a standardized token that we can handle in our APIs.

We created login APIs for each one of the Social Media platforms like follows

GET https://api.ourproducturl.com/v1.0/users/facebook/{facebook_user_id}
GET https://api.ourproducturl.com/v1.0/users/google/{google_user_id}
GET https://api.ourproducturl.com/v1.0/users/twitter/{twitter_user_id}

Based on what Social Media service was used to authenticate the user, the client submits a GET request to one of those APIs by including the authorization response from step 2 as part of the Authorization header for the request (step 3 in the diagram). It is important that the communication for this request is encrypted (ie. use HTTPS) because the access token should not be revealed to the public.

On the server side, few things happen. After extracting the Authorization header from the request we validate the token with the Social Media service (step 4).

Here are the URLs that you can use to validate the tokens:

  • Facebook (as well as documentation)
    https://graph.facebook.com/debug_token?input_token={token-to-inspect}&access_token={app-token-or-admin-token}
  • Google (as well as documentation)
    https://www.googleapis.com/oauth2/v3/tokeninfo?access_token={token-to-inspect}
  • Twitter (as well as documentation)
    https://api.twitter.com/1/account/verify_credentials.json?oauth_access_token={token-to-inspect}

If the token is valid, we compare the ID extracted from the Authorization header with the one specified in the URL. If any of the above two fail we return a 401 Unauthorized response to the client. If we pass those two checks, we do a lookup in our user database to find the user with the specified Social Media ID (step 5. in the diagram) and retrieve her record. We also retrieve information about her group participation so that we can do authorization later on for each one of the functional calls. If we cannot find the user in our database we return a 404 Not found response to the client.

Create API Token

For the purposes of our APIs, we decided to use encrypted JWT tokens. We include the following information into the JWT token:

  • User information like ID, first name and last name, email, address, city, state, zip code
  • Group membership for the user including roles
  • The authentication token for the Social Media service the user authenticated with
  • Expiration time (we settled on 60 minutes expiration)

Before we send this information back to the client (step. 8 in the diagram) we encrypt it (step. 7) using an encryption key or secret that we keep in Azure Key Vault (step. 6). The JWT token is sent back to the client in the Authorization header.

Call the functional APIs

Now, we replaced the access token the client received from the Social Media site with a JWT token that our application can understand and use for authentication and authorization purposes. Each request to the functional APIs (step 9 in the diagram) is required to have the JWT token in the Authorization header. Each API handler has access to the encryption key that is used to decrypt the token and extract the information from it (step 10).

Here are the checks we do before  every request is handled (step 11):

  • If the token is missing we return 401 Unauthorized to the client
  • If the user ID in the URL doesn’t match the user ID stored in the JWT token we return 401 Unauthorized to the client. All API requests for our product are executed in the context of the user
  • If the JWT token has expired we return 401 Unauthorized to the client. For now, we decided to expire the JWT token every 60 mins and request the client to re-authenticate with the APIs. In the future, we may decide to extend the token for another 60 mins or until the Social Media access token expires, so that we can avoid the user dissatisfaction from frequent logins. Hence we overdesigned the JWT token to store the Social Media access token also
  • If the user has no right to perform certain operation we return 403 Forbidden to the client denoting that the operation is forbidden for this user

Few notes on the implementation. Because we use Python we can easily implement all the authentication and authorization checks using decorators, which make our API handlers much easier to read and also enable an easy extension in the future (like for example, extending the validity of the JWT token). Python has also an easy to use JWT library available at Github at https://github.com/jpadilla/pyjwt.

Some additional resources that you may find useful when implementing JWT web tokens are: