GitHub comments - Part 3 - Overcoming GitHub REST API v3 restrictions

comments

GitHub Octocat Logo GearHost Logo

Year ago I've wrote simple github-issue-comments component using vue.js. It was interesting to me to play with vue-component in sandbox. In this post I'll explain why the component was not ready to production and what I need to overcome the limitations.

Other posts in the series

  1. Creating reusable components with Vue.js - Part 1 - Tooling overview
  2. Creating reusable components with Vue.js - Part 2 - Viewing GitHub Issue Comments
  3. GitHub comments - Part 3 - Overcoming GitHub REST API v3 restrictions This post

Table of contents

GitHub REST API v3 restrictions

The component had used GitHub REST API v3 via Octokat library to retrieve issue comments.

All the requests to API made was unauthenticated. GitHub REST API has rate limiting. There is quota of 60 queries per hour for unauthenticated requests and 5000 requests per hour for authenticated.

Quota for unauthenticated requests simply is not enough for production usage. At the same time, requiring visitors to authenticate in GitHub just to see comments is overkill for blog :)

Also, in this blog I've post's comments counter below each post's header. It's not possible to update all counters with single request to GitHub REST API v3. Imagine list of blog posts page should send 20 requests just to update comments counters!

GitHub GraphQL API v4 restrictions

GraphQL API can retrieve comments counters for all posts in single request. It is very promising but unfortunately GraphQL API requires all requests to be authenticated. Again, requiring visitors to authenticate in GitHub just to see comments is overkill for blog.

Also, rate limits for GraphQL API is hard to understand at first glance. Fortunately, we could retrieve current rateLimit counter along other data in our GraphQL request:

query {
  # ...put here GraphQL request you need...

  # request current limits:
  rateLimit {
    limit
    cost
    remaining
    resetAt
  }
}

will return json rateLimit json:

{
  "data": {
    "rateLimit": {
      "limit": 5000,
      "cost": 1,
      "remaining": 4977,
      "resetAt": "2019-01-16T20:16:20Z"
    }
  }
}

In practice, on simple queries remaining counter decreases by one on each query. The upper limit for requests is the same 5000 requests per hour as in GitHub REST API v3. This should be enough for production usage.

Need for backend

Initial goal was to make serverless component for viewing github issue comments. Unfortunately, rate limits for unauthenticated requests makes GitHub REST API v3 unusable in production. I need to use GitHub GraphQL API v4, but it needs authentication.

What if I'll create Web API backend with simple actions get comments and get comments counters? Then I'll could create authenticated OAuth requests to GitHub from server. The downside of this approach is single rate limit for all visitors. But this downside is advantage too. If rate limit is not enough I could add caching - comments are rare subject to change and same for all visitors. To fill the requirement "let unauthenticated users to view the comments" it should be more than enough.

Also, I can securely configure ClientID and Client Secret for OAuth2 on backend. But I need to find appropriate hosting provider...

Actually, I could leave with GitHub REST API v3 on authenticated backend, but with GitHub GraphQL API v4 I can fetch more data with just one request. So I've decided to use latter.

Backend hosting selection

For this blog I'd like to find free hosting for ASP.NET Core or Node.JS applications.

GearHost is the best choice I've found (get $10 in credit by referral link):

  • has free plan
    • 256Mb Application pool,
    • 1Gb bandwidth,
    • 100Mb storage,
    • supports custom domains,
    • 60 minutes of CPU time per day,
  • supports both ASP.NET Core and Node.JS (I can choose :) )
  • supports publishing via git push or FTP

Actually, I've found GearHost more than year ago. And it is not perfect. As stated in supported technologies page, GearHost supports a bit outdated node.js versions. An year ago, at 30th March of 2018, latest supported node.js version on their page was v5.1.1, although latest LTS version on nodejs.org was v8.11.1 and current version was v9.10.1.

I've asked for supporting newer version of node.js in support. They answered that there is no such plans and then I've postponed experiments with node.js hosting for a year.

And today little has changed. Latest supported node.js stated is v8.1.4 (released in 11 July of 2017!), although latest LTS version now is v10.15.0 and current version is v11.6.0.

To be clear, GearHost can host any binaries without dependencies and I've successfully published ASP.NET Core 2.0 self-contained application and keep within 96Mb of binaries on hello-world application, although GearHost doesn't declared its support a year ago. The bad thing is, although GearHost now states it supports ASP.NET Core 2.0 out of the box, in practice there is no ASP.NET Core 2.0 runtime on the servers, you are left with self-contained applications only and only 4Mb disk space for logs and your features on free plan. This is not acceptable.

Backend implementation details

So, for backend I've decided to use node.js application based on express.js web framework.

Actually I've epxerimented with koa web framework too. I've created GitHub repository where you can compare the expressjs and koa implementations of same stuff on GitHub.

Node.js v8.1.4 supports async/await and many stable ES2015-ES2017 features out of the box. You can write and host modern javascript applications right now!

Unfortunately, node.js v8.1.4 doesn't support spread operator for objects:

ctx.response.headers = {
  ...ctx.response.headers,
  ...responseHeaders,
}

will throw syntax error without --harmony flag (which we can't turn on in GearHost):

      ...ctx.response.headers,
      ^^^

SyntaxError: Unexpected token ...

I ended up using plain-old Object.assign(target, ...sources) to do the same thing instead:

ctx.response.headers = Object.assign({}, ctx.response.headers, responseHeaders)

IISNode port passing

GearHost actually uses iisnode for hosting node.js processes in IIS. iisnode passes port number your application should listen in process.env.PORT. Here is part of app.js for example:

const express = require('express')
const github = require('./github-api')
const app = express()

app.get('/page-comments/:number', (req, res) => github.getPageComments({
  headers: req.headers,
  query: req.query,
  number: req.params.number,
}, logger)
  .then(({ responseData, responseHeaders, responseStatus }) => {
    res.set(responseHeaders)
    res.status(responseStatus).send(responseData)
  }))

app.listen(process.env.PORT, () => console.log(`Application started on port ${process.env.PORT}`))

GitHub GraphQL queries

To run GitHub GraphQL queries, you should:

  • get personal access token as described here
  • use Authorization header with value bearer <token> to be authorized
  • use HTTP POST of JSON object with query to https://api.github.com/graphql like in github-api.js:
const fetch = require('axios')
const settings = require('./settings')

// your actual GraphQL query here:
const repositoryQuery = `
issue(number: ${number}) {
  comments(first:100${afterKeyFilter}) {
    totalCount
    nodes {
      bodyHTML
      createdAt
      author { login avatarUrl url }
    }
  }
}`
const graphQlQuery = `query {
  repository(owner: "${settings.owner}", name: "${settings.repository}") {
    ${repositoryQuery}
  }
  rateLimit { limit cost remaining resetAt }
}`
const body = JSON.stringify({ query: graphQlQuery })
const request = {
  method: 'POST',
  url: 'https://api.github.com/graphql',
  data: body,
  headers: { 'Authorization': `bearer ${settings.authToken}` }
}
fetch(request).then(response => console.log(response))

GearHost's git push deployment

The tricky part in iisnode is to create web.config for IIS. Hopefully, GearHost generates web.config for you when you use Local Git Deployment right after you call git push. GearHost deployment scripts searches for app.js or server.js in repository to execute as starting point.

GearHost provides generated login and password. Password is long and hard to type and remember. But you can store credentials in local repository using:

git config credential.helper store
git pull website master

git pull will prompt username and password and they will be stored locally.

Proxify requests to backend from Apache WebServer

I'd like that in future requests to backend follows to /api path.

My domain hosted under Apache WebServer. Hopefully, my hosting provider supports mod_rewrite and mod_proxy http modules and I can modify my .htaccess file with rewriting rules (P flag is for proxy):

RewriteEngine on
RewriteRule ^api/(.*)$ http://put-your-gearhost-cloud-site-name-here.gearhostpreview.com/$1 [P]

If you have direct access to httpd.conf file, using ProxyPass and ProxyPassReverse solution also might be helpful.

One of possible cautions for me is CloudFlare caching. I've had used CloudFlare recently and it turns out that CloudFlare will cache only on-site static content resources based on the file extension. So, it wouldn't cache my WebAPI calls.

Summary

I've found node.js hosting with free plan called GearHost. There is enough free disk space, bandwidth and CPU power on free plan for my purposes. Moreover, I love git push deployment option, it shines.

Btw it's not ideal in terms of support for the latest versions of node.js. Also, ASP.NET Core Runtime is not installed by default and you left out with self-contained deployment of ASP.NET Core applications.

You can view backend source code on GitHub. Now I've integrated with GitHub using GraphQL API v4. It requires requests to be authenticated with OAuth2 and provides reasonable rate limits.

So, here we go.

Now I need to update vue.js controls to use this backend API.

Happy hosting!

Comments

You can post comment for this post directly on github