Host a Website on GitHub with Jekyll – Part 2


Hi! I’m Dennis and I’ve been working at Linkbal as a software engineer since October 2017. In Part 1, we took a quick look at Jekyll and how to create and deploy a website to GitHub Pages. In this second and final part, let’s explore some of the customization features Jekyll provides us.

Layouts

In the previous example, you may have noticed the layout definition below in the front matter:

layout: post

While the markdown file provides the content of a page, the structure and appearance are defined in the layout file. We can create our own layout files, but because we are using an existing theme, we can customize the theme’s layout files instead.

First, let’s copy the layout files from the theme’s folder to our project (the specific folder to copy from will probably be different for you as it depends on the version of the packages installed).

cp -r vendor/bundle/ruby/2.3.0/gems/minima-2.5.0/_layouts .

If we take a look at the _layouts folder, we can see the 4 files below:

  • default.html
  • home.html
  • page.html
  • post.html

If we open the default.html file, we can see contents like the below:

<!DOCTYPE html>
<html lang="{{ page.lang | default: site.lang | default: "en" }}">

  {%- include head.html -%}

  <body>

    {%- include header.html -%}

    <main class="page-content" aria-label="Content">
      <div class="wrapper">
        {{ content }}
      </div>
    </main>

    {%- include footer.html -%}

  </body>

</html>

It is an ordinary HTML file, but we can see a couple of the features provided by Jekyll and the Liquid template language it uses.

“Include” tags, like the below, allow us to include content from another file stored in the _includes folder:

{%- include header.html -%}

While the line below specifies where the content of a markdown file that uses the layout file will be injected:

{{ content }}

If we take a look at the contents of page.html:

---
layout: default
---
<article class="post">

  <header class="post-header">
    <h1 class="post-title">{{ page.title | escape }}</h1>
  </header>

  <div class="post-content">
    {{ content }}
  </div>

</article>

We can see that a layout itself can use another layout and that this layout can be used to display an article with a title in the top.

Now, we can also open the about.md file that ships with Jekyll. We can see in the front matter that this file uses the page layout.

---
layout: page
title: About
permalink: /about/
---

Also, the front matter defines a title and this title is injected in the line below in the page layout:

<h1 class="post-title">{{ page.title | escape }}</h1>

Now that we have an idea of how layouts work, let’s customize our website a little bit.

Includes

In the default.html layout file, we can see the line below:

{%- include header.html -%}

Because the default.html layout includes the header and all other layouts uses the default layout, it is obvious that all pages in our website includes the content in header.html.

Let’s take a look at the contents of the header.html file. Because it is also part of the minima theme, we need to copy this file as well:

mkdir _includes
cp vendor/bundle/ruby/2.3.0/gems/minima-2.5.0/_includes/header.html _includes

This include file is responsible for displaying the website header and shows many variables that Jekyll make available for us, as well as filters provided by the Liquid template language.

{%- assign default_paths = site.pages | map: "path" -%}

The line above, for example, shows the site.pages variable. It is an array of Jekyll’s Page variables (you can check the list of attributes a Page variable contains in Page Variables). Any .html or .md file in the root folder or a new subfolder that contains a front matter will be part of the site.pages variable. Then, we execute Liquid’s map filter on the site.pages to get an array of the paths of those pages.

{%- assign page_paths = site.header_pages | default: default_paths -%}

The next line creates a variable called page_paths. It is assigned the value in site.header_pages, but can also be assigned the value in default_paths in the case the former does not exist, what is possible with the use of Liquid’s default filter. We are going to come back to site.header_pages later.

{%- for path in page_paths -%}
  {%- assign my_page = site.pages | where: "path", path | first -%}
  {%- if my_page.title -%}
  <a class="page-link" href="{{ my_page.url | relative_url }}">{{ my_page.title | escape }}</a>
  {%- endif -%}
{%- endfor -%}

This last snippet iterates over the page_paths variable created before and creates a link in the header for each page that has a title (defined in the front matter).

To test what was just shown, let’s create a new page notes.md in the project’s root folder. Now, let’s add the content below to the new file:

---
layout: page
title: Notes
permalink: /notes/
---

If we refresh our website, we are going to see a new “Notes” link in the header. Add another page called contact.md to the root folder with the content below:

---
layout: page
title: Contact
permalink: /contact/
---

A new link is created as well. The problem is that the links are created in the alphabetical order, but we may want a different order to put the “Contact” page last, for example. In that case, we can use the site.header_pages mentioned before. This currently does not exist, but we can easily create this variable by adding the following lines to the _config.yml file:

header_pages:
  - about.md
  - notes.md
  - contact.md

You will notice the change will not be reflected immediately and will require the server to be restarted. After that, the header will look as expected.

The Resulting Header

The Resulting Header

Collections

Now, we are going to explore Collections that are a great way to group related content. Let’s use this feature to implement the “Notes” section in our website.

First, add the following content to _config.yml:

collections:
  - notes

Then, let’s create a _notes folder and add 2 files in it. The /_notes/note1.md:

---
menu_title: Note 1 Menu Item
title: Note 1 Title
---

Note 1 content

> A nice blockquote

A **formatted** line of *text*.

And the /_notes/note2.md:

---
menu_title: Note 2 Menu Item
title: Note 2 Title
---

Note 2 content

> A nice blockquote

A **formatted** line of *text*.

Now, let’s create a custom layout listing all our notes. It will be the /_layouts/note.html file and will contain the following content:

---
layout: default
---
<style>
.note {
  margin: 2rem 0;
  padding: 1rem 2rem;
}

.yellow {
  background-color: #fdf475;
}

.blue {
  background-color: #cbf0f8;
}

.note-date {
  font-size: 0.8rem;
  font-style: italic;
  color: gray;
}
</style>
<div class="notes-container">

  <header class="notes-header">
    <h1 class="notes-title">{{ page.title | escape }}</h1>
  </header>

  {%- for note in site.notes -%}
  <article class="note {{ note.background }}">

    <header class="note-header">
      <h2 class="note-title">{{ note.title | escape }}</h2>
      <p class="note-date">{{ note.date }}</p>
    </header>

    <div class="note-content">
      {{ note.content }}
    </div>

  </article>
  {%- endfor -%}
</div>

I added the styles directly in the layout, to keep things simple, as that is not the main focus of the article. As Jekyll has built-in support for Sass, please refer to the Docs to know how to manage your stylesheets.

You can see we were able to access our notes through the site.notes variable and even use a custom variable background we added to the front matter.

Restart the server and we can now access those notes in the /notes/ path.

The Notes Page

The Notes Page

The files we created in the _notes folder does not generate corresponding files in Jekyll’s build process. If you need, you can change this behavior as shown in Add content. This would generate files such as /_notes/note1.html in the build output. In our example, we could, for example, display only part of the content of the notes in the list and link each note to a page where the user can see their whole content.

Data Files

In this final section, we are going to take a quick look at Data Files. It allows us to create files with custom data to be used in our templates. Those can be YAML, JSON, or CSV files. Let’s do an example to illustrate its use.

First, create a file named /_data/contacts.yml:

- type: Email
  value: me@denakitan.com

- type: Phone
  value: (00) 0000-0000

- type: Website
  value: http://github.com/denakitan

This is enough to make our data accessible in our templates in the site.data.contacts variable. So, we can now edit our /contact.md file with the following content:

---
layout: page
title: Contact
permalink: /contact/
---

<table>
  {%- for contact in site.data.contacts %}
  <tr>
    <td>{{ contact.type }}</td>
    <td>{{ contact.value }}</td>
  </tr>
  {%- endfor %}
</table>

This will generate the below output:

The Contact Page

The Contact Page

As we could see, we can use Data Files to remove repetition in our pages by separating data and configuration. A practical use is illustrate in Navigation where it is used to create custom navigation for a website.

Conclusion

In these article, we could learn and practice basic Jekyll concepts such as Layouts, Collections and Data Files. There are plenty of content was not covered in the articles, so I highly recommend reading the documentation, as it is very well written and easy to understand.


元PHPエンジニアがPHPとRubyを比較してみた


はじめに

はじめまして。去年の6月に入社したチャン・ゾアン・タンと申します。

私はリンクバルに入る前にPHPエンジニアでした。RubyもRailsも未経験でした。入社してからRubyやRailsを勉強し始めました。以下の方法で勉強しました。

まず、Rubyを学ぶ

もともとPHPエンジニアだったので、Rubyを学んた時に、PHPと比較しながら学びました。

両方の言語はオブジェクト指向プログラミング言語(実は、Rubyはpureオブジェクト指向プログラミング、PHPはオブジェクト指向サポート言語と言われている)であるので、同様なものがあるかと思いました。この学び方でPHP知識に参照出来ると思います。

同様

まず、PHPとRubyは以下の通り同様なところがあると思います。

PHPとRubyとも型の指定せず変数を宣言できます

  • php
$hoge = 'linkbal';
echo $hoge;

$arr = [1,2,3];
print_r($arr);
  • ruby
hoge = 'linkbal'
puts hoge

arr = [1,2,3]
puts arr

オブジェクト指向プログラミング言語なので、両方PHPとRubyはpublic, protected, privateというアクセス修飾子を付けてクラスを宣言できる

配列

一次元配列は大体同じ、両方とも動的な配列がサポートされています。シンタックスも同様です。
Rubyで二次元配列がhashになります。

文字列補間

両方は文字列補間という文字列内に変数を埋め込むことができます。シンタックスも同様です。

  • php
$a = 'hogehoge';
echo "hello {$a}";
  • ruby
a = 'hogehoge'
puts "hello #{a}"

パッケージ管理ツール

両方とも盛んな言語なので、コミュニティーが広いし、ライブラリーも豊富なので、パッケージ管理ツールが必要です。

PHPではComposerを使っています。
RubyではGemです

いくつか同じ関数

  • eval: ローコードスクリプトを実行する関数
  • spintf: 文字列をフォーマットするための関数

違い

シンタックス

  • 基本のシンタックス

PHPはC言語と同様、セミコロンでコードステートメントを終了にします。semicolon terminatedと言います。
クラス関数ループなどというブロックは{}が包まれます
RubyはPythonと同様、改行でコードステートメントを終了にします。newline terminatedと言います。
クラス関数ループなどというブロックは{}が要らない

def method(a, b)
  if a &amp;amp;gt; b
    puts 'hoge hoge'
  end
end

以下の通り{}を使用するケースもあります。

eachメソッドのshorthand

[1,2,3].each { |elm| puts(elm) }

コードブロック

def hoge
puts 'welcome to'
yield
end
hoge { puts 'Linkbal engineer blog' }

・Rubyでは全部オブジェクトで、シンタックスはわかりやすいし、すっきりだと思います。

文字列の長さを取得する

PHP

echo strlen("welcome to linkbal engineer's blog");

Ruby

puts "welcome to linkbal engineer blog".length

・Rubyのシンタックスはかなりは柔軟です。
関数を呼び出す時、括弧があるかないかどちらもいけます。
Rubyの関数は最後の変数または表現がreturnの値になります。

def hoge
1 + 1 # return 2
end

If-else文は自然言語に近いので、結構わかりやすいです。

# これは
# if a == b
# hoge = 10
# end
# と相当です
hoge = 10 if a == b

Rubyでは逆の条件の場合、unlessが使える

#「if a != 10 hoge = 'test'」と相当です
hoge = 'test' unless a == 10

こういうかき方もよく使われています。

def hogehoge
'test' if a == 10
end

Truthy/Falsy

Rubyではfalse, nilのみfalseを返します。0、空文字列、空配列がtrueです。
PHPではfalsenullfalseであることはもとより、0、空文字列、空配列もfalseです。

Rubyはabstractクラスとinterfaceともありません。

文字列の連結

PHPではdot(.)を使います。
Rubyでは+を使います。

変数種

PHPでは変数が1種類しかありません。

Rubyでは4種類があります。

$variable # グローバル変数
@@variable # クラス変数
@variable # インスタンス変数
variable # ロカール変数

コード規約

開発するには言語を学ぶことだけではなくて、コード規則を学ぶべきです。

PHPのコード規約は「PHP Standard Recommendation(PSR)」が一番盛んなものだと思います。

RubyではRubocopという静的解析ツールが結構有名で、うち会社のプロジェクトも使っているので、Rubycopのコード規約に参考しました。

Railsを学ぶ

プログラミング言語を学んだあと、Railsフレームワークを学びます。

PHPを開発した時、Laravelフレームワークを使っていました。
LaravelはRailsのようにモダンな「MVC」Webフレームワークであります。それに、Laravelの作者はLaravelを作ったときに、Railsに参考したということを言われています。そのため、LaravelとRailsは構造や機能の中に同様なものがあるかと思いました。

Model

Laravelでは「Eloquent」と言います。
Railsでは「ActiveRecord」と言います。

両方ともパワーフルORM(Object Relational Mapping)で、Model連携も便利です。

View

LaravelではBladeというテンプレートエンジンがあります。これで簡単にレイアウトを作れ、再利用もできます。

RailsではERBがあります。LaravelのBladeと同様です。

Controller

他の「MVC」フレームワークと同じで、RailsとLaravelはリクエストがルーターで処理したあと、コントローラーにいって、ロジックを処理されます。

その他

Rails Laravel
コマンドライン rails console php artisan
マイグレーション サポート(実行, ロールバック, リセット、初期データ生成) サポート (実行, ロールバック, リセット、初期データ生成)
ストレージ Active Storage File Storage
Websocket Action Cable Broadcasting
多言語 i18n Localization
ユニットテスト rspec PHPUnit

終わり

最初、Railsに触った時に、こういう方法でRubyやRailsを勉強しました。
そして、私の部署ではPull Requestは必ず二人以上のApproveがないとマージできないようになっているので、ソースレビューでチームメイトのコメントからもいろいろ勉強できました。だんだんRailsの開発に慣れました。

現在、うちの会社ではエンジニアを募集しています。ご応募お待ちしております!

参考

https://guides.rubyonrails.org/index.html
https://docs.ruby-lang.org
https://laravel.com/docs
https://github.com/rubocop-hq/ruby-style-guide


Progressive Web Apps – Part 2


Introduction

Hi! I’m Dennis and I’ve been working at Linkbal as a software engineer since October 2017.

In this second part on the series (Part 1 here), we are going to take a deeper look on application manifests and start talking about service workers.

The Web App Manifest

Browser Support

A line like the below, mentioned before, is used activate the application manifest.

<link rel="manifest" href="/manifest.json">

If the application is accessed by unsupported browsers, this line will simply be ignored, so the app can still be used without problems.

The current state of browser support for the web app manifest can be found at Web App Manifest – Can I Use.

Demo Application

For the remaining of this series of articles, I prepared a very basic project that can be used to practice what was learned. At this point, it has the minimum we need to start building upon: the main screen with an example journal entry and a button that opens a modal to add new entries, navigation and a help page. Also, the project already contains a web app manifest.

You can find the project in the link below and the only requirement is to have Node.js installed.

https://github.com/denakitan/progressive-journal

Clone the project to your machine and run:

$ npm install
$ npm start

This will install the http-server package and start the development server. You can open the application by accessing the address that the server outputs and you can probably access http://localhost:8080 by default.

The Demo App

The Demo App

Chrome Dev Tools

After starting the application as described in the previous item, we can now check the validity of our app manifest file by opening the “Developer Tools” in Chrome. In the tools, we can check the “Application” tab and click the “Manifest” option in the sidebar, as shown below:

The App Manifest section in Chrome's Developer Tools

The App Manifest section in Chrome’s Developer Tools

This section should display errors and warnings in the case of issues with the manifest.json file.

Android Emulator

During the development process, we will probably want to test the app on an actual device. This way, we can add a shortcut to the app on the device’s homescreen, for example.

But, instead of a real device, we can use the Android emulator that comes with Android Studio. We are not going to cover the installation of Android Studio, but it should be straightforward to install it with the default options.

After starting Android Studio, we want to start a virtual device to test our application. Unfortunately, it is only possible to do that within a project. So, we need create a new project and it is OK to do that using the default settings.

With a new project opened, we can access the virtual devices manager by selecting “Tools” > “AVD Manager” or by clicking the “AVD Manager” icon in the toolbar. After that, it is necessary to create a new virtual device and the article Create and manage virtual devices explains how to do so. Preferably, we want to use a virtual device with the most recent version of Android, as it should contain a quite recent version of Chrome installed, so we don’t need to install a more recent version of it manually.

After creating and starting the virtual device, let’s open Chrome. To access our application, we need to type http://10.0.2.2:8080 in the address bar. This is the address that is mapped to the localhost in the virtual device. From there, we can use the “Add to Home screen” on Chrome to add a shortcut in our device.

Add to Home screen

Add to Home screen

Service Workers

Now, it is time to work on service workers, the most important piece that makes an app a PWA. We already introduced the subject in Part 1, but we are going to go much deeper in the subject from now on.

The browser uses a single thread to run all the JavaScript in a page, as well as to perform layout, reflows, and garbage collection [1].

But, as mentioned before, service workers run in a separate thread from the JavaScript that runs on the web pages (the main thread). They run in the background, are decoupled from the HTML pages and have access to a different set of features compared to the main thread. They cannot access the DOM on the application pages directly, but because they run in the background, service workers can listen to events emitted by those pages and even listen to messages from other servers, to display push notifications, for example.

Events

As mentioned above, service workers stays in the background listening for events. Below, some of the events service workers can react to are listed:

  • Fetch – service workers can listen to fetch events, that are triggered every time a resource (an image file or a CSS file, for example) needed by the page is requested to the server by the browser. Thanks to the fetch event, service workers can intercept and manipulate those network requests, blocking them or returning cached assets, for example.
  • Push Notifications – service workers can listen to push notification events even if the application is already close, helping bringing the users back to the application by showing useful notifications on their phones, for example. We can also react to the user interaction to those notifications through service workers, allowing us to program what happens when the user taps a push notification on their devices.
  • Background Synchronization – on a situation of unstable internet access, we can schedule a user action to be synced to the remote server once the internet connection is re-established. This task can be performed by the service worker once it receives an event indicating the connection is back.
  • Lifecycle Events – as we are going to see in the next section, we can program certain actions to be executed during the lifecycle of a service worker.

The Service Worker Lifecycle

Service workers live in its own JavaScript file and need to be registered by the JavaScript code running in the main thread. Through the registration process, we request the browser to run our service worker code as a background process.

Registering a service worker will cause the browser to start the service worker install step in the background [2]. This triggers the install event, when we have the opportunity to cache some static assets.

Also part of the registration process is the service worker activation, that triggers the activate event, when we can cleanup resources used in previous versions of a service worker script [3]. Once it is activated, the service worker transitions to an idle state, when it is ready to handle functional events.

Because the code for the service worker registration runs in the main thread, it is run every time the application reloads. But, it is not the case for the installation process, as it is only run again if the service worker code changes.

Finally, when not in use, service workers are terminated what does not mean they are uninstalled, as they are automatically re-enabled when an event is triggered.

Diagram of the states in the Service Worker Lifecycle

States in the Service Worker Lifecycle. Image: Service Workers

Browser Support

As we can check in Is service worker ready?, browser support is already very good.

According to Can I use…, browser support currently covers more than 89% of the users.

Registering a Service Worker

Now, it’s time to register a service worker for the demo app mentioned before.

The first thing to do is to create a JavaScript file for the service worker. The location of this file is very important because it determines the scope of the service worker. In many cases, we want our service worker to be able to affect our entire application. To do so, we need to create the new file in the root folder of our project, that is the public folder in the case of our demo app.

So, we are going to create a new file called sw.js, although it can have any name we want.

Before implementing our service worker, let’s register it in our app.js file. Because we don’t know which page the user will access first, it is important to guarantee that the code to register the service worker run in all pages of our application. We can do so by executing the registration process in the app.js file because it is imported in both the main index.html and the help page index.html.

Let’s add the code below to /public/src/js/app.js:

if ('serviceWorker' in navigator) {
  navigator.serviceWorker.register('/sw.js').then(() => console.log('Service worker registered'))
}

In the code above, we are checking if the browser supports service workers to avoid trying to run code that will not work. After that check, we are asking the browser to register the /sw.js file as a background process.

Now, let’s start the development server if it’s not started already and refresh our application. The console in the Dev Tools should output Service worker registered as expected. We can also check the “Service Worker” section in the “Application” tab, where it show that our service worker is activated and running.

Service Worker Status in the Chrome Dev Tools

Service Worker Status in the Chrome Dev Tools

Reacting to Lifecycle Events

In this section, we are going to handle some events in the service worker lifecycle. Let’s edit the /sw.js file we created previously:

self.addEventListener('install', e => {
  console.log('[Service Worker] Installing the service worker...', e)
})

self.addEventListener('activate', e => {
  console.log('[Service Worker] Activating the service worker...', e)
  return self.clients.claim()
})

The line return self.clients.claim() ensures the service worker is activated correctly. It may become unnecessary in the future, but for now this helps make the registration process more robust. Now, let’s refresh our application and we will see that the registration and installation console.log() are displayed in the “Console”, but the activation output is not shown.

This is expected behavior and, if we take a look at the “Application” tab in “Chrome Dev Tools”, we can see that our service worker shows a “waiting to activate” state. When your application is already open, service workers will be installed but not activated. The reason is: the application may still be interacting with the old service worker and activating a new version of the service worker which may contain breaking changes, can make the application misbehave. Therefore, we can make our service worker activate by closing the application (all the tabs/windows running the application) and reopening it.

If we do that, we will see the activation log will be displayed in the “Console” and the “Application” tab will no longer show the “waiting to activate” status. That’s the behavior of the activation process of service workers, but we have a couple of ways to make a new version activate, as shown below:

  • check the “Update on reload” option
  • click “Update”
  • click “Unregister” and then “Register”
  • click “skipWaiting” when the service worker is in the “waiting to activate” state

Reacting to Non-Lifecycle events

As the last step in this article, let’s write an event handler to deal with the “Fetch” event, that is triggered whenever an asset is needed by the application.

self.addEventListener('fetch', e => {
  console.log('[Service Worker] Fetching asset...', e)
  e.respondWith(fetch(e.request))
})

The method e.respondWith allow us to customize how to respond to each fetch event. For now, it was just placed here to call attention to it as we are not doing any manipulation.

Conclusion

This is it for now. In the next article, we will introduce caching with service workers.

References

  1. Intensive JavaScript – MDN
  2. The Service Worker Life Cycle – Google Developers
  3. Using Service Workers – MDN

Automating security updates on AWS and pushing the output to Slack


Introduction

Hello there! My name is Adam, an aspiring DevOps engineer who joined the Infrastructure team of Linkbal this year in February. My main focuses are optimizing the AWS environments and the server middleware. Occasionally I will make something cool and useful, most likely involving AWS, which I would like to share with you.

Today, I will be detailing how to schedule a security update script to be run on an EC2 Linux instance on a time schedule, then push the output logs to a Slack channel for everyone to see.

Objectives

First, let’s consider what we want to achieve, so as to determine what the end result should be:

  1. Automatically run security updates on an EC2 Linux instance on a pre-determined schedule.
  2. Be notified of the update and see the output in a Slack channel.
  3. Be able to change the time schedule or turn on/off without having to access the instance.

The 3rd point is important as it adds a layer of convenience to the method. If it ever becomes necessary to stop the updates, we want to be able to do so quickly and easily, without having to remove what we put in place.

Pre-requisites

  • AWS account with privileges to create a CloudWatch rule and use Systems Manager.
  • EC2 Linux-based instance with AWS CLI installed.
  • IAM role attached to the EC2 instance. 
  • Permission to access the EC2 instance via SSH connection and use root credentials.
  • A Slack account with a channel for receiving the output logs and the permission to create a webhook.

Plan and script

The source code originally came from here: Slack notification via curl
The diagram was drawn using a free account in LucidChart.
#!/bin/bash

# This script runs a security update then pushes the output to Slack.

VAR1=$(sudo yum update --security -y)
IP=$(curl ifconfig.me)

function post_to_slack () {
    SLACK_MESSAGE="\`\`\`$1\`\`\`"
    SLACK_URL={Slack web hook} # Replace this with your own web hook, without the braces

    case "$2" in
      INFO)
        SLACK_ICON=':slack:'
        ;;
      WARNING)
        SLACK_ICON=':warning:'
        ;;
      ERROR)
        SLACK_ICON=':bangbang:'
        ;;
      *)
        SLACK_ICON=':slack:'
        ;;
    esac

curl -X POST --data "payload={\"text\": \"${SLACK_ICON} ${SLACK_MESSAGE}\"}" ${SLACK_URL}
}
post_to_slack "Result of security update for $IP: $VAR1" "INFO"
exit 0

Procedure

  1. Create a Slack web hook for the channel you want to receive the log output in. Paste the web hook into the script as the value for SLACK_URL (without the {} braces).
  2. Upload script to the instance (take a note of the path). This can be done from the command line as such:
    $ scp -i /path/to/key.pem ec2-user@{IPv4 DNS}:path/to/file
    Or you can SSH connect to the instance, make a new file and paste the code in there:
    $ ssh -i /path/to/key.pem ec2-user@{IPv4 DNS}
    $ vi /path/to/file/update_security.sh
    It doesn’t matter were the file goes, so long as you know where it is.
    Be sure to change the permissions on the script so that it can be executed.
    $ chmod 775 /path/to/file/update_security.sh
    Install the AWS SSM agent: (installed by default on AWS Linux 1 AMIs since 2017.9 and on all AWS Linux 2 AMIs)[Manually install SSM agent on Amazon EC2 Linux instances]
    Add the policy “AmazonEC2RoleforSSM” policy to the role being use by the EC2 instance.
  3. Go to CloudWatch, select “Rule” on the left toolbar and then “Create rule”. Set the time expression by either a fixed rate or a cron expression. Note that the time will be in UTC, so depending on your timezone you will need to adjust the hour and possible day values. As an example, to update at 2am UTC Monday-Friday, the expression would be:
    0 2 ? * MON-FRI *
    For help with cron expressions in AWS, please see: AWS CloudWatch – How to schedule events
    Select “SSM Run Command”.
    Select the document “AWS-RunShellScript (Linux)”
    Target key = InstanceIds (can choose a different target, if you prefer)
    Target values = {Instance ID} i.e. i-a1b2c3d4e5
    Click on the “Configure parameters” drop down.
    Commands = ./update_security.sh
    WorkingDirectory = /path/to/file/
    ExecutionTimeout = 100
    Select “Create a new role for this specific resource”
    Click on “Configure details”. Give it a name and a description.
  4. The CloudWatch Rule uses AWS Systems Manager to call the SSM Agent which runs the script.  The script will run the update, then pass the output to a post function which uses the Slack web hook to post the output in your channel.
    Even if there are no updates due, the message will still be sent.
  5. Success! You can customize the name and image of the web hook in the Slack settings menu.

 

Conclusion

We have now configured for security updates to run on the AWS EC2 Linux instance on a specific time schedule, with the result of the update being sent to a channel in Slack. The script can be turned on/off easily from the AWS console (or by the AWS CLI) and we can change the time schedule there as well.
Let us now consider the advantages and disadvantages of this method.

Advantages

  • Quick and simple to set up.
  • The script is reusable; can be used on most if not all Linux instances on AWS. Only the Slack web hook would have to be changed if a different channel was required, otherwise no need to edit the script.
  • Only a basic understanding of AWS is required.
  • Virtually zero cost.

Disadvantages

  • CloudWatch rule must be stopped manually. Could set up a trigger with SMS and Lambda to stop the rule without the need for human intervention.
  • Only works for Linux instances. Windows would require something different.
  • Root privileges are required to set up (possibly avoidable with AWS Systems Manager).
  • Would be easier to understand if more details concerning the instance were put into the script, such as the name tag. (Can use ec2-metadata to obtain a lot of this information, but tags are not included in this list. Could use AWS CLI to get the instance name tag.)

Overall we have achieved what we set out to do. There are ways we could improve the script, but for now this is more than suitable.

I hope you have enjoyed this post, it’s a simple start with many more things to come. See you next time!

References

  1. Source code: http://blog.pragbits.com/it/2015/02/09/slack-notifications-via-curl/
  2. LucidChart: https://www.lucidchart.com/
  3. AWS CloudWatch – How to schedule events: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
  4. AWS EC2 – Manually install SSM Agent on AWS EC2 Linux instance: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html
  5. Manually install SSM agent on Amazon EC2 Linux instances : https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html

 

 

 

 

 

 

 

 


Progressive Web Apps – Part 1


Introduction

Hi! I’m Dennis and I’ve been working at Linkbal as a software engineer since October 2017.

This is the first part in a series of articles about Progressive Web Apps or PWAs. I am currently taking the following course at Udemy and I am writing these articles to help me consolidate the earned knowledge: Progressive Web Apps (PWA) – The Complete Guide

What are Progressive Web Apps

As you can notice by reading articles such as Seriously, though. What is a progressive web app?, it is very difficult to describe what a PWA actually is, but I will give it a try.

Progressive Web Apps (PWAs) are web applications that behaves like native mobile applications. They do so by using a set of techniques and technologies to provide end-users with features they usually find on native apps, such as:

  • Background synchronization
  • Push notifications
  • Offline accessibility
  • Location based behavior
  • Device camera access
  • Application shortcut in the home screen

It’s difficult to define PWAs as there is not a single specific criteria to consider an app a PWA, so it is even more difficult (if not impossible) to call an app a “perfect” PWA. Because some of the recommended enhancements rely on modern web browser capabilities and new capabilities will surely create new recommended enhancements in the future, I believe it is not absurd to consider the goal of a “perfect” PWA unattainable. Not only that, but depending on the nature of your application, implementing some of the enhancements proposed for PWAs may not even make sense.

That said, as written in the article mentioned previously, there are three baseline criteria to qualify an app as a PWA:

  1. It must run under HTTPS.
  2. It must include a Web App Manifest.
  3. It must implement a service worker.

The first criteria is quite straightforward and we will cover the other two later.

I’m not sure if it’s completely accurate to consider an app a PWA once these three criterias are satisfied, but I like to think so, as it gives us an objective way to make this judgement. Even after we can call our app a Progressie Web App, we will always be striving to reach the goals proposed for PWAs, such as making our apps more reliable, faster and more engaging [1].

Core Building Blocks

Now that we have an idea of what Progressive Web Apps are, let’s take a look at their core building blocks, or the core techniques and technologies we need to use to build a PWA.

Service Workers

Service workers are basically JavaScript code that runs in a background process, separately from the main browser thread, even if the application is not open in the browser. It is a technology well supported by modern web browsers 2 and a building block that provides a lot of the features that PWAs can provide.

Service workers can intercept network requests and cache those requests to improve performance. This cache can then also be used to provide offline access when the network is not available [3].

Because they run in the background, they are capable of providing some features that make web applications work like native apps, such as:

  • Background Sync
  • Push Notifications

Other Characteristics

There are a couple more characteristics to service workers that are consequences of their role in a PWA and the fact that they are independent from the application they are associated with:

  • Because a service worker is not blocking, synchronous XHR and localStorage cannot be used.
  • Service workers can’t access the DOM directly, so they need to use a messaging system to communicate with the page.
  • As they can intercept network requests and modify responses, service workers only run over HTTPS to prevent “man-in-the-middle” attacks.
  • Service workers become idle when not in use and restart when needed.

Application Manifests

An application manifest makes it possible for a web application to be added to mobile device homescreens. It is a simple JSON file that also describes how it should behave when “installed”. Among other things, it specifies the icon to be used, the URL to access when the app is launched and the background color of the splash screen displayed while the app is loading.

Responsive Design

At this point, responsive web design is a topic that does not need introduction, but it can be considered a core bulding block of PWAs because one of the goals is to create an application that fits any form factor [4].

PWAs and SPAs (Single Page Applications)

There’s one important confusion that is frequently seen regarding SPAs and PWAs, that the techniques and technologies to progressively enhance web applications to turn them into PWAs can only be applied to single page applications. The reality is that any web application can become a progressive web app no matter it is a SPA or a traditional multiple pages application.

Application Manifests

An application manifest is a JSON file that describes how a web app should behave once it is “installed” in the user’s device. For example, the browser uses that information to determine the icon to display when the application is added to the device’s homescreen.

Adding a Manifest File

Adding a manifest file to a project is as simple as creating a file named manifest.json and linking it in the “ of all the pages of the web app:

<link rel="manifest" href="/manifest.json">

Content of the Manifest File

Below, there’s an example content for an application manifest file:

{
  "name": "PWApp - Progressively Enhanced App",
  "short_name": "PWApp",
  "start_url": "index.html",
  "scope": ".",
  "display": "standalone",
  "background": "#5cb4f7",
  "theme_color": "#5cb4f7",
  "description": "A web app progressively enhanced to become a PWA",
  "dir": "ltr",
  "lang": "en-US",
  "orientation": "portrait",
  "icons": [
    {
      "src": "/assets/images/icons/icon-36.png",
      "type": "image/png",
      "sizes": "36x36"
    },
    {
      "src": "/assets/images/icons/icon-48.png",
      "type": "image/png",
      "sizes": "48x48"
    }
  ],
  "related_applications": [
    {
      "platform": "play",
      "url": "https://play.google.com/store/apps/details?id=com.example.app1",
      "id": "com.example.app1"
    },
    {
      "platform": "itunes",
      "url": "https://itunes.apple.com/app/example-app1/id123456789"
    }
  ]
}
  • name: specifies a long name for the app, displayed in the app install prompt[5], for example.
  • short_name: short name for the app that may be used, for example, in a device homescreen where there is insufficient space to display the full name.
  • start_url: URL that is loaded when the app is launched through the icon added to the homescreen.
  • scope: determines the pages the application manifest applies to. “.” applies to all pages.
  • display: defines the preferred display mode for the website [6]. Using standalone will make the app look and feel like a standalone app, for example.

A PWA launched browser mode (left), standalone mode (middle) and fullscreen mode (right). Image: New in Chrome 58

  • background_color: sets the background color used in the splashscreen displayed while the app is loading.
  • theme_color: this color is shown, for example, in the browser toolbar and in the top bar on Android’s task switcher, what increases the sense of OS integration even more.
  • description: this field is used whenever the browser needs some description for an app.
  • dir: specifies the text direction for name, short_name and description.
  • lang: together with lang, it helps to display right-to-left languages.
  • orientation: sets the default orientation, but bear in mind users prefer selecting the orientation themselves.


Portrait and landscape orientation options for the app manifest. Image:The Web App Manifest

  • icons: icons to be displayed in the homescreen. Multiple icons with different sizes can be provided in the array and the browser will determine the “best” icon depending on the device.
  • related_applications: array of native applications that can be installed through the applications stores of the respective platforms.

Conclusion

In this first part of the series about the course on PWAs I am taking on Udemy, I tried to define what Progressive Web apps are, presented their building blocks and an introduction to application manifests. In the next part, we are going to go deeper into the subject of application manifests and start talking about service workers.

References

  1. Progressive Web Apps – Google Developers
  2. Service Workers – Can I use
  3. Introduction to Service Worker – Google Developers
  4. Your First Progressive Web App
  5. The Web App Manifest – Google Developers
  6. Web App Manifest #display – MDN

Host a Website on GitHub with Jekyll – Part 1


Hi! I’m Dennis and I’ve been working at Linkbal as a software engineer since October 2017. In Part 1 of this article, I will introduce Jekyll and show how to create a new website and make it available publicly.

Jekyll

Jekyll is an open-source static site generator. Its GitHub repository README shows a very simple description: “Think of it like a file-based CMS, without all the complexity.” With Jekyll, we can create our website structure using HTML, CSS and Liquid Templates. After the structure is in place, we can focus on updating the website’s content, by just creating files using Markdown.

It is an intermediate solution that sits between static websites and web applications. We can add a little bit of dynamic behavior when generating the website but, in the end, it is served as a completely static website.

Jekyll is also the engine behind GitHub Pages, which can be used to host websites from GitHub repositories. Because we are not dealing with a database, updating a website is a matter of editing files and pushing the changes to GitHub,  This article is an introduction on how to work with Jekyll to create a website and how to host it on GitHub Pages.

Repository Creation

Our final goal is to have our website hosted on GitHub Pages. To achieve this, we need to upload our Jekyll project to a GitHub repository. We can have a exclusive repository for our website, but we can also store our project source in a folder of an existing repository. This last option is particularly useful for creating documentation websites for open-source software projects as it allows us to manage both the project and its documentation sources as a single project, making it easier to keep the later updated.

In this article, I am going to create a repository called ‘jekyll_website’ to be used exclusively to host the website by following the steps in the following link: https://help.github.com/en/articles/create-a-repo

Jekyll Installation

In this article, I won’t cover Jekyll’s installation process. It is very well documented in the Jekyll documentation and I assume you are going to follow that, but I will try to present some guidelines. I also assume you have a working Git installation.

Jekyll is distributed as a Ruby Gem and can be installed on most systems. It is recommended to work on a Unix-based system such as macOS or Linux, but you should not have problems working on Windows as well. By the way, this article was written on a Windows machine with Windows Subsystem for Linux enabled. You can take a look at the official documentation if you choose to go this path too.

Whatever system you choose to work on, the requirements are the same. You are going to need to have Ruby, RubyGems and build tools (GCC and Make) installed. If you, like me, are a Rails developer and already have a Ruby development environment installed, you should be good to go after installing Jekyll with the following commands (we are also installing bundler, a gem that helps us manage other Ruby gems in our project):

$ gem install jekyll bundler

New Project

Now, we are going to create a new Jekyll project:

$ jekyll new jekyll_website --skip-bundle

We passed the option --skip-bundle because we don’t want to install our dependencies system-wide. Instead, we want the installation of our dependencies to be limited to our project. But first, as we know we are going to host our website on GitHub, we want to setup up our project accordingly. First, open the Gemfile file created on your project’s root folder.

And remove the line declaring the dependency of the jekyll gem. It may differ depending on the version of the software you are working with, but in my case, I removed the following line:

gem "jekyll", "~> 3.6.2"

And uncomment the following line:

# gem "github-pages", group: :jekyll_plugins

Dependencies Installation

We can now install or dependencies locally with the following command:

$ cd jekyll_website

$ bundle install --path vendor/bundle

Local Server Startup

After that we can serve our website with the following command:

$ bundle exec jekyll serve

But, it’s highly recommended to use the command below instead:

$ bundle exec jekyll serve --livereload

This way the website will automatically build and reload each time changes are saved. This is particularly useful when we are writing articles and want to see how it is looking. You can stop the jekyll process at any time by pressing “Cmd (Ctrl) + C”.

Just open a browser and access the following address:

http://localhost:4000

Great! We already have a working website. You can see Jekyll comes with a pre-defined look & feel. Jekyll supports themes and it uses a theme called minima by default. You can check this fact by opening the Gemfile file, where there is a line setting the minima gem as a dependency:

gem "minima", "~> 2.0"

In this article, I will show you some examples on how we can start customizing our website. Although we can start our website from scratch, without a theme at all, it is probably easier to have a starting point, at least when using Jekyll for the first time.

But, before that, let’s add our website to GitHub to manage the changes in our website. You will see that, in the case of a Jekyll website, adding the project to GitHub not only means that we are controlling its versions, but also means that we are deploying our website for public access.

Deployment

To start the deployment process, let’s first enable GitHub Pages. To do so, access the settings of your repository and scroll to the GitHub Pages section. Let’s change the Source option from “None” to “master branch”. By doing that, GitHub will run Jekyll to build the website and serve it whenever we push changes to the master branch.

Now we can push our project to GitHub. But, before doing that, let’s edit our .gitignore file. The Jekyll project already comes with a .gitignore to prevent unnecessary files to be commited. But, as installing the gems locally is an optional step, it does not include entries to ignore the installed packages. So, let’s add the following lines to the .gitignore file:

.bundle
vendor

 

Now we can push our files to GitHub as usual:

$ git init
$ git add .
$ git commit -m "First commit"
$ git remote add origin git@github.com:denakitan/jekyll_website.git
$ git push -u origin master

 

You can now access your website at the URL provided by GitHub. In my case, it is: https://denakitan.github.io/jekyll_website/. If you still can’t see your website, it may take some time for GitHub to build it.

Creating a Post

Now that we have a website running, we can create our first blog post. To do so, add a new file to the _posts folder. I will name it “2019-02-28-my-first-post.markdown”.

The first thing we need to do then is to add the front matter section to this file. It is YAML block containing some metadata about our post. Let’s add the lines below to the file:

---
layout: post
title: "My First Post!"
date: 2019-02-28 08:26:44 +0900
categories: first post
---

After that, we just need to write our post using markdown syntax. Jekyll uses kramdown as its markdown converter. Let’s just add some content to test it:

A **formatted** line of *text*.

> A nice blockquote

## Code

{% highlight ruby %}
puts ‘Hello Jekyll!’
{% endhighlight %}

You should be able to see the new posted listed in the home page and access its content.

Conclusion

In this article we could see how to create a new Jekyll website and make it available for the public. In the part 2 of the series, I am going to show some examples on how we can use some of Jekyll’s features to customize our website structure.


リンクバルのデータ系基盤


リンクバルにはデータ分析者が5名ほどと、機械学習のエンジニアが数名いる。社員90名弱という規模の中では、データ活用に大きいリソースを割いている方だと思う。全社員の3分の2程度が、彼らが作り出したダッシュボードや、分析レポートや、予測結果などを見ながら日々の業務を行っている。

もともと、それほどデータを活用する会社ではなかったが、2016年秋に街コンジャパンをフルリニューアルしてアプリケーション側のデータが綺麗になったのを機に、データ系の基盤を整備した。
続きを読む


いまご飯を食べるスキルと、10年後も生きるためのスキル


2018年の初頭、語学と数学の能力を上げようという目標を立てた。60歳で引退すると仮定した場合、僕の社会人人生の、ちょうど真ん中だったこともあって、人生に大きな方針を立てたかったのだ。年末になって振り返ると、自分が希望するよりは能力が上がらなかったなと、残念に思う。

それでも最近、英語面接にも少しは慣れてきたし、ほんの少しだけ、中国語で聞こえる音が増えてきたし、ベトナム語教室では発音がいいと褒められるので、進歩はしているようだ。ベトナム語の発音がいいといっても、あくまでも他の日本人と比べて、だが。

数学は、高校数学の微分積分を半分くらいしか復習できなかったものの、log x を微分すると 1/x になる、という程度は思い出せた。ちなみに、いまお世話になっている数学の教科書は、これだ。

初めから学べると評判の大学基礎数学微分積分キャンパス・ゼミ
https://www.amazon.co.jp/dp/4866150319/

書名も表紙も怪しいが、中身はいい。と思う。
続きを読む


GitLabの導入(Backlog + GitHub からの移行)


いま、社内に GitLab を導入中です。数ヶ月かけて、これまで利用していた Backlog と GitHub を、GitLab に置きかえていく予定です。

Backlog は課金額がユーザ数に依存しません。そこで、いろいろな人が関わる課題管理には Backlog を利用しています。過去の履歴を見ると、2013年から 5年以上もお世話になっています。現在のアクティブユーザー数は120名ほど。

一方 Backlog の Gitリポジトリ機能はそれほど魅力的ではないため、コード管理には GitHub を利用しています。しかし GitHub はユーザ毎にお金がかかるため、コードを書く人に限定して使っています。現在のアクティブユーザー数は50名ほど。

開発の流れ

当社の開発の流れは、だいたい次のようになっています。

  1. 誰かが「○○をやりたい」と提案する。誰が何を提案してもいい。要点をまとめて Backlog に proposal(提案)として登録する。
  2. やるか、やらないか、優先度はどうするか、を毎朝のミーティングで決める。
  3. やることになった案件は planning(企画)フェーズに移行し、ディレクター、デザイナー、エンジニアなど案件毎に適切な人が仕様を決める。
  4. planningフェーズの仕様をレビューして問題がなければ、dev フェーズに移行させる。課題の優先度は毎朝の会議で決める。
  5. 優先度高の課題の中から好きなものを、手が空いたエンジニアが自分で取っていく。
  6. 開発が終わったら、提案者やディレクターがテストする。問題がなければリリース待ちとなる。
  7. リリースする。「街コンジャパン」の場合、少ない日で 1件、多い日で 10件くらいのリリースをする。ただし、売上が一番多いのが土日祝であるため、休みの前日のリリースはなるべく避ける。
    続きを読む

開発組織の育て方 – エンジニア3人から30人へ


リンクバルの堀内です。技術部門の責任者をさせていただいています。リンクバルでは現在、業務委託やリモートワークも含めて約30名のエンジニアが活躍しています。ほんの3年ほど前は10名、5年前はたったの3名でした。当社の主力事業は、男女が出会うためのイベントのチケットを販売する、イベントECサイト「街コンジャパン」です。創業時からいままで、常にイベント掲載数が右肩上がりで増え続け、現在は月間2万件ほどのイベントを掲載している、日本最大のサイトです。

エンジニアが足りなくて困っているのは昔も今も同じですが、悩みの質はだいぶ変わりました。当時は、とにかく人がいない。増収増益を続けるマザーズ上場企業だったのに、人が採れない。今は優秀なエンジニアが揃っているので、人が足りないながらも、スケジュールのやりくりで、やりたいことはできるようになりました。

当時、うまくいかなかった要因はいくつもありました。年間数十億円を販売するECサイトなのに WordPress で作られているから関心を示すエンジニアが少ない。「チャラい若者で溢れる出会い系企業」だから自分には合わないとエンジニアから敬遠される。当時の人事部が問題を抱えていて人材紹介会社からの評判が悪かった。超絶技巧的に WordPress を使いこなしていた影響で、保守だけで精一杯になり、新しいことに目を向ける余裕がなかった、など。

ちなみに、リンクバルに「チャラい若者」はあまりおらず、中から見るとむしろ地味な人が多い印象です。
続きを読む