This article is the eighth in the DevOps series. In this issue, we shall learn to set up Docker in the host system and use it with Ansible.

Introduction

Docker provides operating system level virtualisation in the form of containers. These containers allow you to run standalone applications in an isolated environment. The three important features of Docker containers are isolation, portability and repeatability. All along we have used Parabola GNU/Linux-libre as the host system, and executed Ansible scripts on target Virtual Machines (VM) such as CentOS and Ubuntu.

Docker containers are extremely lightweight and fast to launch. You can also specify the amount of resources that you need such as CPU, memory and network. The Docker technology was launched in 2013, and released under the Apache 2.0 license. It is implemented using the Go programming language. A number of frameworks have been built on top of Docker for managing these cluster of servers. The Apache Mesos project, Google’s Kubernetes, and the Docker Swarm project are popular examples. These are ideal for running stateless applications and help you to easily scale them horizontally.

Setup

The Ansible version used on the host system (Parabola GNU/Linux-libre x86_64) is 2.3.0.0. Internet access should be available on the host system. The ansible/ folder contains the following file:

ansible/playbooks/configuration/docker.yml

Installation

The following playbook is used to install Docker on the host system:

---
- name: Setup Docker
  hosts: localhost
  gather_facts: true
  become: true
  tags: [setup]

  tasks:
    - name: Update the software package repository
      pacman:
        update_cache: yes

    - name: Install dependencies
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - python2-docker
        - docker

    - service:
        name: docker
        state: started

    - name: Run the hello-world container
      docker_container:
        name: hello-world
        image: library/hello-world

The Parabola package repository is updated before proceeding to install the dependencies. The python2-docker package is required for use with Ansible. Hence, it is installed along with the docker package. The Docker daemon service is then started and the library/hello-world container is fetched and executed. A sample invocation and execution of the above playbook is shown below:

$ ansible-playbook playbooks/configuration/docker.yml -K --tags=setup
SUDO password: 

PLAY [Setup Docker] *************************************************************

TASK [Gathering Facts] **********************************************************
ok: [localhost]

TASK [Update the software package repository] ***********************************
changed: [localhost]

TASK [Install dependencies] *****************************************************
ok: [localhost] => (item=python2-docker)
ok: [localhost] => (item=docker)

TASK [service] ******************************************************************
ok: [localhost]

TASK [Run the hello-world container] ********************************************
changed: [localhost]

PLAY RECAP **********************************************************************
localhost                  : ok=5    changed=2    unreachable=0    failed=0   

With verbose ’-v’ option to ansible-playbook, you will see an entry for LogPath, such as /var/lib/docker/containers//-json.log. In this log file you will see the output of the execution of the hello-world container. This output is the same when you run the container manually as shown below:

$ sudo docker run hello-world

Hello from Docker!

This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://cloud.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

Example

A Deep Learning (DL) Docker project is available (https://github.com/floydhub/dl-docker) with support for frameworks, libraries and software tools. We can use Ansible to build the entire DL container from the source code of the tools. The base OS of the container is Ubuntu 14.04, and will include the following software packages:

• Tensorflow
• Caffe
• Theano
• Keras
• Lasagne
• Torch
• iPython/Jupyter Notebook
• Numpy
• SciPy
• Pandas
• Scikit Learn
• Matplotlib
• OpenCV

The playbook to build the DL Docker image is given below:

- name: Build the dl-docker image
  hosts: localhost
  gather_facts: true
  become: true
  tags: [deep-learning]

  vars:
    DL_BUILD_DIR: "/tmp/dl-docker"
    DL_DOCKER_NAME: "floydhub/dl-docker"

  tasks:
    - name: Download dl-docker
      git:
        repo: https://github.com/saiprashanths/dl-docker.git 
        dest: "{{ DL_BUILD_DIR }}"

    - name: Build image and with buildargs
      docker_image:
         path: "{{ DL_BUILD_DIR }}"
         name: "{{ DL_DOCKER_NAME }}"
         dockerfile: Dockerfile.cpu
         buildargs:
           tag: "{{ DL_DOCKER_NAME }}:cpu"

We first clone the Deep Learning docker project sources. The docker_image module in Ansible helps us to build, load and pull images. We then use the Dockerfile.cpu file to build a Docker image targeting the CPU. If you have a GPU in your system, you can use the Dockerfile.gpu file. The above playbook can be invoked using the following command:

$ ansible-playbook playbooks/configuration/docker.yml -K --tags=deep-learning

Depending on the CPU and RAM you have, it will take considerable amount of time to build the image with all the software. So be patient!

Jupyter Notebook

The built dl-docker image contains Jupyter notebook which can be launched when you start the container. An Ansible playbook for the same is provided below:

- name: Start Jupyter notebook
  hosts: localhost
  gather_facts: true
  become: true
  tags: [notebook]

  vars:
    DL_DOCKER_NAME: "floydhub/dl-docker"

  tasks:
    - name: Run container for Jupyter notebook
      docker_container:
        name: "dl-docker-notebook"
        image: "{{ DL_DOCKER_NAME }}:cpu"
        state: started
        command: sh run_jupyter.sh

You can invoke the playbook using the following command:

$ ansible-playbook playbooks/configuration/docker.yml -K --tags=notebook

The Dockerfile already exposes the port 8888, and hence you do not need to specify the same in the above docker_container configuration. After you run the playbook, using the ‘docker ps’ command on the host system, you can obtain the container ID as indicated below:

$ sudo docker ps
CONTAINER ID        IMAGE                    COMMAND               CREATED             STATUS              PORTS                NAMES
a876ad5af751        floydhub/dl-docker:cpu   "sh run_jupyter.sh"   11 minutes ago      Up 4 minutes        6006/tcp, 8888/tcp   dl-docker-notebook

You can now login to the running container using the following command:

$ sudo docker exec -it a876 /bin/bash

You can then run an ‘ifconfig’ command to find the local IP address (“172.17.0.2” in this case), and then open http://172.17.0.2:8888 in a browser on your host system to see the Jupyter Notebook. A screenshot is shown in Figure 1:

TensorBoard

TensorBoard consists of a suite of visualization tools to understand the TensorFlow programs. It is installed and available inside the Docker container. After you login to the Docker container, at the root prompt, you can start Tensorboard by passing it a log directory as shown below:

# tensorboard --logdir=./log

You can then open http://172.17.0.2:6006/ in a browser on your host system to see the Tensorboard dashboard as shown in Figure 2:

Docker Image Facts

The docker_image_facts Ansible module provides useful information about a Docker image. We can use it to obtain the image facts for our dl-docker container as shown below:

- name: Get Docker image facts
  hosts: localhost
  gather_facts: true
  become: true
  tags: [facts]

  vars:
    DL_DOCKER_NAME: "floydhub/dl-docker"

  tasks:
    - name: Get image facts
      docker_image_facts:
        name: "{{ DL_DOCKER_NAME }}:cpu"

The above playbook can be invoked as follows:

$ ANSIBLE_STDOUT_CALLBACK=json ansible-playbook playbooks/configuration/docker.yml -K --tags=facts 

The ANSIBLE_STDOUT_CALLBACK environment variable is set to ‘json’ to produce a JSON output for readability. Some important image facts from the invocation of the above playbook are shown below:

"Architecture": "amd64", 
"Author": "Sai Soundararaj <saip@outlook.com>", 

"Config": {

"Cmd": [
   "/bin/bash"
], 

"Env": [
   "PATH=/root/torch/install/bin:/root/caffe/build/tools:/root/caffe/python:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", 
   "CAFFE_ROOT=/root/caffe", 
   "PYCAFFE_ROOT=/root/caffe/python", 
   "PYTHONPATH=/root/caffe/python:", 
   "LUA_PATH=/root/.luarocks/share/lua/5.1/?.lua;/root/.luarocks/share/lua/5.1/?/init.lua;/root/torch/install/share/lua/5.1/?.lua;/root/torch/install/share/lua/5.1/?/init.lua;./?.lua;/root/torch/install/share/luajit-2.1.0-beta1/?.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua", 
   "LUA_CPATH=/root/torch/install/lib/?.so;/root/.luarocks/lib/lua/5.1/?.so;/root/torch/install/lib/lua/5.1/?.so;./?.so;/usr/local/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/loadall.so", 
   "LD_LIBRARY_PATH=/root/torch/install/lib:", 
   "DYLD_LIBRARY_PATH=/root/torch/install/lib:"
], 

"ExposedPorts": {
   "6006/tcp": {}, 
   "8888/tcp": {}
}, 

"Created": "2016-06-13T18:13:17.247218209Z", 
"DockerVersion": "1.11.1", 

"Os": "linux", 

"task": { "name": "Get image facts" }

You are encouraged to read the ‘Getting Started with Docker’ user guide available at http://docs.ansible.com/ansible/latest/guide_docker.html to know more about using Docker with Ansible.

I wanted to attend TUG 2019 for two main reasons - to present my work on the “XeTeX Book Template”, and also to meet my favourite computer scientist, Prof. Donald Knuth. He does not travel much, so, it was one of those rare opportunities for me to meet him in person. His creation of the TeX computer typesetting system, where you can represent any character mathematically, and also be able to program and transform it is beautiful, powerful and the best typesetting software in the world. I have been using TeX extensively for my documentation and presentations over the years.

Day I

I reached the hotel venue only in the afternoon of Friday, August 9, 2019, as I was also visiting Mountain View/San Jose on official work. I quickly checked into the hotel and completed my conference registration formalities. When I entered the hall, Rishi T from STM Document Engineering Private Limited, Thiruvananthapuram was presenting a talk on “Neptune - a proofing framework for LaTeX authors”. His talk was followed by an excellent poetic narration by Pavneet Arora, who happened to be a Vim user, but, also mentioned that he was eager to listen to my talk on XeTeX and GNU Emacs.

After a short break, Shreevatsa R, shared his experiences on trying to understand the TeX source code, and the lessons learnt in the process. It was a very informative, user experience report on the challenges he faced in navigating and learning the TeX code. Petr Sojka, from Masaryk University, Czech Republic, shared his students’ experience in using TeX with a detailed field report. I then proceeded to give my talk on the “XeTeX Book Template” on creating multi-lingual books using GNU Emacs and XeTeX. It was well received by the audience. The final talk of the day was by Jim Hefferon, who analysed different LaTeX group questions from newbies and in StackExchange, and gave a wonderful summary of what newbies want. He is a professor of Mathematics at Saint Michael’s College, and is well-known for his book on Linear Algebra, prepared using LaTeX. It was good to meet him, as he is also a Free Software contributor.

The TUG Annual General Meeting followed with discussions on how to grow the TeX community, the challenges faced, membership fees, financial reports, and plan for the next TeX user group conference.

Day II

The second day of the conference began with Petr Sojka and Ondřej Sojka presenting on “The unreasonable effectiveness of pattern generation”. They discussed the Czech hyphenation patterns along with a pattern generation case study. This talk was followed by Arthur Reutenauer presenting on “Hyphenation patterns in TeX Live and beyond”. David Fuchs, a student who worked with Prof. Donald Knuth on the TeX project in 1978, then presented on “What six orders of magnitude of space-time buys you”, where he discussed the design trade-offs in TeX implementation between olden days and present day hardware.

After a short break, Tom Rokicki, who was also a student at Stanford and worked with Donald Knuth on TeX, gave an excellent presentation on searching and copying text in PDF documents generated by TeX for Type-3 bitmap fonts. This session was followed by Martin Ruckert’s talk on “The design of the HINT file format”, which is intended as a replacement of the DVI or PDF file format for on-screen reading of TeX output. He has also authored a book on the subject - “HINT: The File Format: Reflowable Output for TeX”. Doug McKenna had implemented an interactive iOS math book with his own TeX interpreter library. This allows you to dynamically interact with the typeset document in a PDF-free ebook format, and also export the same. We then took a group photo:

I then had to go to Stanford, so missed the post-lunch sessions, but, returned for the banquet dinner in the evening. I was able to meet and talk with Prof. Donald E. Knuth in person. Here is a memorable photo!

He was given a few gifts at the dinner, and he stood up and thanked everyone and said that “He stood on the shoulders of giants like Isaac Newton and Albert Einstein.”

< />

I had a chance to meet a number of other people who valued the beauty, precision and usefulness of TeX. Douglas Johnson had come to the conference from Savannah, Georgia and is involved in the publishing industry. Rohit Khare, from Google, who is active in the Representational State Transfer (ReST) community shared his experiences with typesetting. Nathaniel Stemen is a software developer at Overleaf, which is used by a number of university students as an online, collaborative LaTeX editor. Joseph Weening, who was also once a student to Prof. Donald Knuth, and is at present a Research Staff member at the Institute for Defense Analyses Center for Communications Research in La Jolla, California (IDA/CCR-L) shared his experiences in working with the TeX project.

Day III

The final day of the event began with Antoine Bossard talking on “A glance at CJK support with XeTeX and LuaTeX”. He is an Associate Professor of the Graduate School of Science, Kanagawa University, Japan. He has been conducting research regarding Japanese characters and their memorisation. This session was followed by a talk by Jaeyoung Choi on “FreeType MF Module 2: Integration of Metafont and TeX-oriented bitmap fonts inside FreeType”. Jennifer Claudio then presented the challenges in improving Hangul to English translation.

After a short break, Rishi T presented “TeXFolio - a framework to typeset XML documents using TeX”. Boris Veytsman then presented the findings on research done at the College of Information and Computer Science, University of Massachusetts, Amherst on “BibTeX-based dataset generation for training citation parsers”. The last talk before lunch was by Didier Verna on “Quickref: A stress test for Texinfo”. He teaches at École Pour l’Informatique et les Techniques Avancées, and is a maintainer of XEmacs, Gnus and BBDB. He also an avid Lisper and one of the organizers of the European Lisp Symposium!

After lunch, Uwe Ziegenhagen demonstrated on using LaTeX to prepare and automate exams. This was followed by a field report by Yusuke Terada, on how they use TeX to develop a digital exam grading system at large scale in Japan. Chris Rowley, from the LaTeX project, then spoke on “Accessibility in the LaTeX kernel - experiments in tagged PDF”. Ross Moore joined remotely for the final session of the day to present on “LaTeX 508 - creating accessible PDFs”. The videos of both of these last two talks are available online.

A number of TeX books were made available for free for the participants, and I grabbed quite a few, including a LaTeX manual written by Leslie Lamport. Overall, it was a wonderful event, and it was nice to meet so many like-minded Free Software people.

A special thanks to Karl Berry, who put in a lot of effort in organizing the conference, but, could not make it to due to a car accident.

The TeX User Group Conference in 2020 is scheduled to be held at my alma mater, Rochester Institute of Technology.

Day I

I had proposed a workshop titled “Shooting the trouble down to the Wireshark Lua Plugin” for the event, and it was selected. I have been working on the “Aerospike Wireshark Lua plugin” for dissecting Aerospike protocols, and hence I wanted to share the insights on the same. The plugin source code is released under the AGPLv3 license.

“Wireshark” is a popular Free/Libre and Open Source Software protocol analyzer for analyzing protocols and troubleshooting networks. The “Lua programming language” is useful to extend C projects to allow developers to do scripting. Since Wireshark is written in C, the plugin extension is provided by Lua. Aerospike uses the PAXOS family and custom built protocols for distributed database operations, and the plugin has been quite useful for packet dissection, and solving customer issues.

The workshop had both theory and lab exercises. I began with an overview of Lua, Wireshark GUI, and the essential Wireshark Lua interfaces. The Aerospike Info protocol was chosen and exercises were given to dissect the version, type and size fields. I finished the session with real-world examples, future work and references. Around 50 participants attended the workshop, and those who had laptops were able to work on the exercises. The workshop presentation and lab exercises are available in the aerospike-wireshark-plugin/docs/workshop GitHub repository.

I had follow-up discussions with the participants before moving to the main auditorium. “Using pod security policies to harden your Kubernetes cluster” by Suraj Deshmukh was an interesting talk on the level of security that should be employed with containers. After lunch, I started my role as emcee in the main auditorium.

The keynote of the day was by Bernd Erk, the CEO at Netways GmbH, who is also the co-founder of the Icinga project. He gave an excellent talk on “How convenience is killing open standards”. He gave numerous examples on how people are not aware of open standards, and take proprietary systems for granted. This was followed by flash talks from the audience. Jaskaran Narula then spoke on “Securing infrastructure with OpenScap: the automation way”, and also shared a demo of the same.

After the tea break, Shubham Mittal gave a talk on “OSINT for Proactive Defense” in which he shared the Open Source Intelligence (OSINT) tools, techniques and procedures to protect the perimeter security for an organization. The last talk of the day was by Shadab Siddiqui on “Running a successful bug bounty programme in your organization”.

Day II

Anant Shrivastava started the day’s proceedings with a recap on the talks from day one.

The first talk of the day was by Jiten Vaidya, co-founder and CEO at Planetscale who spoke on “OLTP or OLAP: why not both?”. He gave an architectural overview of vitess.io, a Free/Libre and Open Source sharding middleware for running OLTP workloads. The design looked like they were implementing the Kubernetes features on a MySQL cluster. Ratnadeep Debnath then spoke on “Scaling MySQL beyond limits with ProxySQL”.

After the morning break, Brian McKenna gave an excellent talk on “Functional programming and Nix for reproducible, immutable infrastructure”. I have listened to his talks at the Functional Programming conference in Bengaluru, and they have been using Nix in production. The language constructs and cases were well demonstrated with examples. This was followed by yet another excellent talk by Piyush Verma on “Software/Site Reliability of Distributed Systems”. He took a very simple request-response example, and incorporated site reliability features, and showed how complex things are today. All the major issues, pitfalls, and troubles were clearly explained with beautiful illustrations.

Aaditya Talwai presented his talk on “Virtuous Cycles: Enabling SRE via automated feedback loops” after the lunch break. This was followed by Vivek Sridhar’s talk on “Virtual nodes to auto-scale applications on Kubernetes”. Microsoft has been investing heavily on Free/Libre and Open Source, and have been hiring a lot of Python developers as well. Satya Nadella has been bringing in lot of changes, and it will be interesting to see their long-term progress. After Vivek’s talk, we had few slots for flash talks from the audience, and then Deepak Goyal gave his talk on “Kafka streams at scale”.

After the evening beverage break, Øystein Grøvlen, gave an excellent talk on PolarDB - A database architecture for the cloud. They are using it with Alibaba in China to handle petabytes of data. The computing layer and shared storage layers are distinct, and they use RDMA protocol for cluster communication. They still use a single master and multiple read-only replicas. They are exploring parallel query execution for improving performance of analytical queries.

Overall, the talks and presentations were very good for 2019. Time management is of utmost importance at Rootconf, and we have been very consistent. I was happy to emcee again for Rootconf!

Introduction

Erlang is a programming language designed by Ericsson primarily for soft real-time systems. The Open Telecom Platform (OTP) consists of libraries, applications and tools to be used with Erlang to implement services that require high availability. In this article, we will create a test Virtual Machine (VM) to compile, build, and test Erlang/OTP from its source code. This allows you to create VMs with different Erlang release versions for testing.

The Erlang programming language was developed by Joe Armstrong, Robert Virding and Mike Williams in 1986 and released as free and open source software in 1998. It was initially designed to work with telecom switches, but is widely used today in large scale, distributed systems. Erlang is a concurrent and functional programming language, and is released under the Apache License 2.0.

Setup

A CentOS 6.8 Virtual Machine (VM) running on KVM will be used for the installation. Internet access should be available from the guest machine. The VM should have at least 2 GB of RAM alloted to build the Erlang/OTP documentation. The Ansible version used on the host (Parabola GNU/Linux-libre x86_64) is 2.3.0.0. The ansible/ folder contains the following files:

ansible/inventory/kvm/inventory
ansible/playbooks/configuration/erlang.yml

The IP address of the guest CentOS 6.8 VM is added to the inventory file as shown below:

erlang ansible_host=192.168.122.150 ansible_connection=ssh ansible_user=bravo ansible_password=password

An entry for the erlang host is also added to the /etc/hosts file as indicated below:

192.168.122.150 erlang

A ‘bravo’ user account is created on the test VM, and is added to the ‘wheel’ group. The /etc/sudoers file also has the following line uncommented, so that the ‘bravo’ user will be able to execute sudo commands:

## Allows people in group wheel to run all commands
%wheel	ALL=(ALL)	ALL

We can obtain the Erlang/OTP sources from a stable tarball, or clone the Git repository. The steps involved in both these cases are discussed below:

Building from the source tarball

The Erlang/OTP stable releases are available at http://www.erlang.org/downloads. The build process is divided into many steps, and we shall go through each one of them. The version of Erlang/OTP can be passed as an argument to the playbook. Its default value is the release 19.0, and is defined in the variable section of the playbook as shown below:

vars:
  ERL_VERSION: "otp_src_{{ version | default('19.0') }}"
  ERL_DIR: "{{ ansible_env.HOME }}/installs/erlang"
  ERL_TOP: "{{ ERL_DIR }}/{{ ERL_VERSION }}"
  TEST_SERVER_DIR: "{{ ERL_TOP }}/release/tests/test_server"

The ERL_DIR variable represents the directory where the tarball will be downloaded, and the ERL_TOP variable refers to the top-level directory location containing the source code. The path to the test directory from where the tests will be invoked is given by the TEST_SERVER_DIR variable.

Erlang/OTP has mandatory and optional package dependencies. Let’s first update the software package repository, and then install the required dependencies as indicated below:

tasks:
  - name: Update the software package repository
    become: true
    yum:
      name: '*'
      update_cache: yes

  - name: Install dependencies
    become: true
    package:
      name: "{{ item }}"
      state: latest
    with_items:
      - wget
      - make
      - gcc
      - perl
      - m4
      - ncurses-devel
      - sed
      - libxslt
      - fop

The Erlang/OTP sources are written using the ‘C’ programming language. The GNU C Compiler (GCC) and GNU Make are used to compile the source code. The ‘libxslt’ and ‘fop’ packages are required to generate the documentation. The build directory is then created, the source tarball is downloaded and it is extracted to the directory mentioned in ERL_DIR.

- name: Create destination directory
  file: path="{{ ERL_DIR }}" state=directory

- name: Download and extract Erlang source tarball
  unarchive:
    src: "http://erlang.org/download/{{ ERL_VERSION }}.tar.gz"
    dest: "{{ ERL_DIR }}"
    remote_src: yes

The ‘configure’ script is available in the sources, and it is used to generate the Makefile based on the installed software. The ‘make’ command will build the binaries from the source code.

- name: Build the project
  command: "{{ item }} chdir={{ ERL_TOP }}"
  with_items:
    - ./configure
    - make
  environment:
    ERL_TOP: "{{ ERL_TOP }}"

After the ‘make’ command finishes, the ‘bin’ folder in the top-level sources directory will contain the Erlang ‘erl’ interpreter. The Makefile also has targets to run tests to verify the built binaries. We are remotely invoking the test execution from Ansible and hence -noshell -noinput are passed as arguments to the Erlang interpreter, as show in the .yaml file.

- name: Prepare tests
  command: "{{ item }} chdir={{ ERL_TOP }}"
  with_items:
    - make release_tests
  environment:
    ERL_TOP: "{{ ERL_TOP }}"

- name: Execute tests
  shell: "cd {{ TEST_SERVER_DIR }} && {{ ERL_TOP }}/bin/erl -noshell -noinput -s ts install -s ts smoke_test batch -s init stop"

You need to verify that the tests have passed successfully by checking the $ERL_TOP/release/tests/test_server/index.html page in a browser. A screenshot of the test results is shown in Figure 1:

The built executables, libraries can then be installed on the system using the make install command. By default, the install directory is /usr/local.

- name: Install
  command: "{{ item }} chdir={{ ERL_TOP }}"
  with_items:
    - make install
  become: true
  environment:
    ERL_TOP: "{{ ERL_TOP }}"

The documentation can also be generated and installed as shown below:

- name: Make docs
  shell: "cd {{ ERL_TOP }} && make docs"
  environment:
    ERL_TOP: "{{ ERL_TOP }}"
    FOP_HOME: "{{ ERL_TOP }}/fop"
    FOP_OPTS: "-Xmx2048m"

- name: Install docs
  become: true
  shell: "cd {{ ERL_TOP }} && make install-docs"
  environment:
    ERL_TOP: "{{ ERL_TOP }}"

The total available RAM (2 GB) is specified in the FOP_OPTS environment variable. The complete playbook to download, compile, execute the tests, and also generate the documentation is given below:

---
- name: Setup Erlang build
  hosts: erlang
  gather_facts: true
  tags: [release]

  vars:
    ERL_VERSION: "otp_src_{{ version | default('19.0') }}"
    ERL_DIR: "{{ ansible_env.HOME }}/installs/erlang"
    ERL_TOP: "{{ ERL_DIR }}/{{ ERL_VERSION }}"
    TEST_SERVER_DIR: "{{ ERL_TOP }}/release/tests/test_server"

  tasks:
    - name: Update the software package repository
      become: true
      yum:
        name: '*'
        update_cache: yes

    - name: Install dependencies
      become: true
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - wget
        - make
        - gcc
        - perl
        - m4
        - ncurses-devel
        - sed
        - libxslt
        - fop

    - name: Create destination directory
      file: path="{{ ERL_DIR }}" state=directory

    - name: Download and extract Erlang source tarball
      unarchive:
        src: "http://erlang.org/download/{{ ERL_VERSION }}.tar.gz"
        dest: "{{ ERL_DIR }}"
        remote_src: yes

    - name: Build the project
      command: "{{ item }} chdir={{ ERL_TOP }}"
      with_items:
        - ./configure
        - make
      environment:
        ERL_TOP: "{{ ERL_TOP }}"

    - name: Prepare tests
      command: "{{ item }} chdir={{ ERL_TOP }}"
      with_items:
        - make release_tests
      environment:
        ERL_TOP: "{{ ERL_TOP }}"

    - name: Execute tests
      shell: "cd {{ TEST_SERVER_DIR }} && {{ ERL_TOP }}/bin/erl -noshell -noinput -s ts install -s ts smoke_test batch -s init stop"

    - name: Install
      command: "{{ item }} chdir={{ ERL_TOP }}"
      with_items:
        - make install
      become: true
      environment:
        ERL_TOP: "{{ ERL_TOP }}"

    - name: Make docs
      shell: "cd {{ ERL_TOP }} && make docs"
      environment:
        ERL_TOP: "{{ ERL_TOP }}"
        FOP_HOME: "{{ ERL_TOP }}/fop"
        FOP_OPTS: "-Xmx2048m"

    - name: Install docs
      become: true
      shell: "cd {{ ERL_TOP }} && make install-docs"
      environment:
        ERL_TOP: "{{ ERL_TOP }}"

The playbook can be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/erlang.yml -e "version=19.0" --tags "release" -K

Build from Git repository

We can build the Erlang/OTP sources from the Git repository. The complete playbook is given below for reference:

- name: Setup Erlang Git build
  hosts: erlang
  gather_facts: true
  tags: [git]

  vars:
    GIT_VERSION: "otp"
    ERL_DIR: "{{ ansible_env.HOME }}/installs/erlang"
    ERL_TOP: "{{ ERL_DIR }}/{{ GIT_VERSION }}"
    TEST_SERVER_DIR: "{{ ERL_TOP }}/release/tests/test_server"

  tasks:
    - name: Update the software package repository
      become: true
      yum:
        name: '*'
        update_cache: yes

    - name: Install dependencies
      become: true
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - wget
        - make
        - gcc
        - perl
        - m4
        - ncurses-devel
        - sed
        - libxslt
        - fop
        - git
        - autoconf

    - name: Create destination directory
      file: path="{{ ERL_DIR }}" state=directory

    - name: Clone the repository
      git:
        repo: "https://github.com/erlang/otp.git"
        dest: "{{ ERL_DIR }}/otp"

    - name: Build the project
      command: "{{ item }} chdir={{ ERL_TOP }}"
      with_items:
        - ./otp_build autoconf
        - ./configure
        - make
      environment:
        ERL_TOP: "{{ ERL_TOP }}"

The ‘git’ and ‘autoconf’ software packages are required for downloading and building the sources from the Git repository. The Ansible Git module is used to clone the remote repository. The source directory provides an otp_build script to create the configure script. You can invoke the above playbook as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/erlang.yml --tags "git" -K

You are encouraged to read the complete installation documentation at: https://github.com/erlang/otp/blob/master/HOWTO/INSTALL.md.

At around 6:00 p.m. people had assembled for networking, snacks and beverage. I started the “Building Careers with FLOSS” presentation at 6:30 p.m. Prof. Dr. D. Janakiram, CSE, IIT Madras also shared his insights on the benefits of learning from source code available under a FLOSS license. The session was very interactive and the audience asked a lot of good questions. Dr. B. Govindarajulu, author of the famous “IBM PC and Clones: Hardware, Troubleshooting and Maintenance” book then presented his views on creating a Computer History Museum in Chennai. Dinner was served around 8:00 p.m. at the venue.

A review of my book has been published in Volume 14: No. 1, January-March 2019 IEEE India Council Newsletter. The excerpts of Chapter 4 of my book on “Project Guidelines” is also available. I had also written an article on “Seasons of Code” which is published in this edition of the IEEE newsletter.

Special thanks to Mr. H. R. Mohan, Editor, IEEE India and Chairman, ACM Professional Chapter, Chennai for organizing the event and for the logistics support.

Introduction

In this sixth article in the DevOps series, we will install Jenkins using Ansible and set up a Continuous Integration (CI) build for a project that uses Git. Jenkins is Free and Open Source automation server software that is used to build, deploy and automate projects. It is written in Java and released under the MIT license. A number of plugins are available to integrate Jenkins with other tools such as version control systems, APIs and databases.

Setting it up

A CentOS 6.8 Virtual Machine (VM) running on KVM will be used for the installation. Internet access should be available from the guest machine. The Ansible version used on the host (Parabola GNU/Linux-libre x86_64) is 2.3.0.0. The ansible/ folder contains the following files:

ansible/inventory/kvm/inventory
ansible/playbooks/configuration/jenkins.yml
ansible/playbooks/admin/uninstall-jenkins.yml

The IP address of the guest CentOS 6.8 VM is added to the inventory file as shown below:

jenkins ansible_host=192.168.122.120 ansible_connection=ssh ansible_user=root ansible_password=password

An entry for the jenkins host is also added to the /etc/hosts file as indicated below:

192.168.122.120 jenkins

Installation

The playbook to install the Jenkins server on the CentOS VM is given below:

---
- name: Install Jenkins software
  hosts: jenkins
  gather_facts: true
  become: yes
  become_method: sudo
  tags: [jenkins]

  tasks:
    - name: Update the software package repository
      yum:
        name: '*'
        update_cache: yes

    - name: Install dependencies
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - java-1.8.0-openjdk
        - git
        - texlive-latex
        - wget

    - name: Download jenkins repo
      command: wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo

    - name: Import Jenkins CI key
      rpm_key:
        key: http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
        state: present

    - name: Install Jenkins
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - jenkins

    - name: Allow port 8080
      shell: iptables -I INPUT -p tcp --dport 8080 -m state --state NEW,ESTABLISHED -j ACCEPT

    - name: Start the server
      service:
        name: jenkins
        state: started

    - wait_for:
        port: 8080

The playbook first updates the Yum repository and installs the Java OpenJDK software dependency required for Jenkins. The Git and Tex Live LaTeX packages are required to build our project, github.com/shakthimaan/di-git-ally-managing-love-letters (now at https://gitlab.com/shakthimaan/di-git-ally-managing-love-letters). We then download the Jenkins repository file, and import the repository GPG key. The Jenkins server is then installed, port 8080 is allowed through the firewall, and the script waits for the server to listen on port 8080. The above playbook can be invoked using the following command:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/jenkins.yml -vv

Configuration

You can now open http://192.168.122.120:8080 in the browser on the host to start configuring Jenkins. The web page will prompt you to enter the initial Administrator password from /var/lib/jenkins/secrets/initialAdminPassword to proceed further. This is shown in Figure 1:

The second step is to install plugins. For this demonstration, you can select the “Install suggested plugins” option, and later install any of the plugins that you require. Figure 2 displays the selected option:

After you select the “Install suggested plugins” option, the plugins will get installed as shown in Figure 3:

An admin user is required for managing Jenkins. After installing the plugins, a form is shown for you to enter the user name, password, name and e-mail address of the administrator. A screenshot of this is shown in Figure 4:

Once the administrator credentials are stored, a “Jenkins is ready!” page will be displayed, as depicted in Figure 5:

You can now click on the “Start using Jenkins” button to open the default Jenkins dashboard shown in Figure 6:

An example of a new project

Let’s now create a new build for the github.com/shakthimaan/di-git-ally-managing-love-letters project. Provide a name in the “Enter an item name” text box and select the “Freestyle project”. Figure 7 provides shows the screenshot for creating a new project:

The next step is to add the GitHub repo to the “Repositories” section. The GitHub https URL is provided as we are not going to use any credentials in this example. By default, the master branch will be built. The form to enter the GitHub URL is shown in Figure 8:

A Makefile is available in the project source code, and hence we can simply invoke “make” to build the project. The “Execute shell” option is chosen in the “Build” step, and the “make clean; make” command is added to the build step as shown in Figure 9.

From the left panel, you can click on the “Build Now” link for the project to trigger a build. After a successful build, you should see a screenshot similar to Figure 10.

Uninstall

An uninstall script to remove the Jenkins server is available in playbooks/admin folder. It is given below for reference:

---
---
- name: Uninstall Jenkins
  hosts: jenkins
  gather_facts: true
  become: yes
  become_method: sudo
  tags: [remove]

  tasks:
    - name: Stop Jenkins server
      service:
        name: jenkins
        state: stopped

    - name: Uninstall packages
      package:
        name: "{{ item }}"
        state: absent
      with_items:
        - jenkins

The script can be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/admin/uninstall-jenkins.yml

The following are the meeting minutes and notes from the Jitsi meetings held online in the months of February and March 2018.

February 2018

The February 2018 meetup was primarily focussed on using Emacs for publishing.

Using Emacs with Hakyll to build websites and resumes were discussed. It is also possible to output in muliple formats (PDF, YAML, text) from the same source.

I shared my shakthimaan-blog sources that uses Hakyll to generate the web site. We also discussed the advantanges of using static site generators, especially when you have large user traffic to your web site.

I had created the xetex-book-template, for creating multilingual book PDFs. The required supported features in the same, and its usage were discussed in the meeting.

Kushal Das asked about keyboards use in Emacs, and in particular for Control and Alt, as he was using the Kinesis. The best Emacs keyboard options available at http://ergoemacs.org/emacs/emacs_best_keyboard.html was shared. The advantage of using thumb keys for Control and Alt was obvious with the bowl shaped keyboard layout in Kinesis.

We also talked about the Emacs Web Browser (eww), and suggested the use of mu4e for checking e-mails with Emacs.

March 2018

At the beginning of the meetup, the participants asked if there was a live stream available, but, we are not doing so at this point in time with Jitsi.

For drawing inside Emacs, I had suggested ASCII art using Artist Mode.

Emacs has support for rendering PDFs inside, as the following old blog post shows http://www.idryman.org/blog/2013/05/20/emacs-and-pdf/. nnnick then had a question on “Why Emacs?”:

nnnick 19:23:00
Can you tell me briefly why emacs is preferred over other text editors

The discussion then moved to the customization features and extensibility of Emacs that makes it well suited for your needs.

For people who want to start with a basic configuration for Emacs, the following repository was suggested https://github.com/technomancy/emacs-starter-kit.

I had also shared links on using Org mode and scrum-mode for project management:

• https://github.com/ianxm/emacs-scrum

• https://gitlab.com/shakthimaan/operation-blue-moon

I shared my Cask setup link https://gitlab.com/shakthimaan/cask-dot-emacs and mentioned that with a rolling distribution like Parabola GNU/Linux-libre, it was quite easy to re-run install.sh for newer Emacs versions, and get a consistent setup.

In order to SSH into local or remote systems (VMs), Tramp mode was suggested.

I also shared my presentation on “Literate DevOps” inspired by Howardism https://gitlab.com/shakthimaan/literate-devops-using-gnu-emacs/blob/master/literate-devops.org.

Org entries can also be used to keep track of personal journal entries. The Date Trees are helpful in this context as shown in the following web page http://members.optusnet.com.au/~charles57/GTD/datetree.html.

Tejas asked about using Org files for executing code in different programming languages. This can be done using Org Babel, and the same was discussed.

Tejas 19:38:23
can org mode files be used to keep executable code in other languages apart from elisp?

mbuf 19:38:42
Yes

mbuf 19:39:15
https://orgmode.org/worg/org-contrib/babel/languages.html

The other useful tools that were discussed for productivity are given below:

• Magit for VCS: https://github.com/magit/magit.

• Workgroups for workspace management: https://github.com/pashinin/workgroups2.

• Winner mode to switch between sessions in windows: https://www.emacswiki.org/emacs/WinnerMode.

• Emacs Code Browser is like a full flown IDE-like environment: http://ecb.sourceforge.net/.

Tejas said that he uses perspective-el, but it does not have the save option - just separate workspaces to switch between them - for different projects basically.

A screenshot of the session in progress is shown below:

Arun also suggested using Try for trying out Emacs packages without installation, and cycle-resize package for managing windows.

Tejas and Arun then shared their Emacs configuration files.

Arun 19:51:37
https://github.com/aruntakkar/emacs.d

Tejas 19:51:56
https://github.com/tejasbubane/dotemacs

We closed the session with few references on learning Emacs Lisp:

Tejas 20:02:42
before closing off, can you guys quickly point me to some resources for learning elisp?

mbuf 20:03:59
Writing GNU Emacs Extensions.

mbuf 20:04:10
Tejas: Emacs Lisp manual

Tejas 20:04:35
Thanks 

The department were interested in organizing a scripting workshop, and I felt using a new programming language like Elixir with the power of the Erlang Virtual Machine (VM) will be a good choice. The syntax and semantics of the Elixir language were discussed along with the following topics:

• Basic types
• Basic operators
• Pattern matching
• case, cond and if
• Binaries, strings and char lists
• Keywords and maps
• Modules and functions

Students had setup Erlang and Elixir on their laptops, and tried the code snippets in the Elixir interpreter. The complete set of examples are available in the following repo:

https://gitlab.com/shakthimaan/elixir-scripting-workshop

A group photo was taken at the end of the workshop.

I would like to thank Prof. Karthik Myilvahanan J for working with me in organizing this workshop.

Although I announced the same only on Twitter and IRC (#emacs on irc.freenode.net), we had a number of Emacsers join the session. The chat log with the useful web links that were shared are provided below for reference.

We started our discussion on organizing Org files and maintaining TODO lists.

9:45 AM Suraj Ghimire: it is clear and loud :) video quality is good too 
                       yes. is there any session today on hangouts ?
                       wow thats nice thanks you for introducing me to emacs :). I am happy emacs user
                       should we add bhavin and vharsh for more testing
                       oh you already shared :)
	
9:55 AM Suraj Ghimire:  working on some of my todos https://i.imgur.com/GBylmeQ.png

For few Vim users who wanted to try Emacs Org mode, it was suggested to get started with Spacemacs. Other project management and IRC tools with Emacs were also shared:

  HARSH VARDHAN can now join this call.
  HARSH VARDHAN joined group chat.
  Google Apps can now join this call.
  Google Apps joined group chat.
	
10:05 AM Shakthi Kannan: http://spacemacs.org/
                         https://github.com/ianxm/emacs-scrum
                         https://github.com/ianxm/emacs-scrum/blob/master/example-report.txt
                         https://www.emacswiki.org/emacs/CategoryWebBrowser
                         ERC for IRC chat
                         https://www.emacswiki.org/emacs/InternetRelayChat

10:13 AM Shakthi Kannan: https://github.com/skeeto/elfeed
                         https://www.emacswiki.org/emacs/EmacsMailingLists
	
10:18 AM Suraj Ghimire: I started using emacs after your session on emacs, before that i used to 
                        get scared due to lot of shortcuts. I will work on improvements you told me.
	
10:19 AM Shakthi Kannan: M - Alt, C - Control
                         http://www.tldp.org/HOWTO/Emacs-Beginner-HOWTO-3.html

  Google Apps left group chat.
  Google Apps joined group chat.
  Sacha Chua can now join this call.
  Sacha Chua joined group chat.

We then discussed on key bindings, available modes, and reading material to learn and master Emacs:

10:27 AM Shakthi Kannan: http://shop.oreilly.com/product/9780596006488.do
	
10:31 AM Shakthi Kannan: https://www.masteringemacs.org/
                         http://shakthimaan.com/tags/emacs.html
                         http://shakthimaan.com/posts/2016/04/04/introduction-to-gnu-emacs/news.html

  Dhavan Vaidya can now join this call.
  Dhavan Vaidya joined group chat.
  Sacha Chua left group chat.
	
10:42 AM Shakthi Kannan: https://www.finseth.com/craft/
                         http://shop.oreilly.com/product/9781565922617.do

  Rajesh Deo can now join this call.
  Rajesh Deo joined group chat.

Users also wanted to know of language modes for Erlang:

10:52 AM Shakthi Kannan: http://www.lambdacat.com/post-modern-emacs-setup-for-erlang/

  HARSH VARDHAN left group chat.
  Aaron Hall can now join this call.
	
10:54 AM Shakthi Kannan: https://github.com/elixir-editors/emacs-elixir

Aaron Hall joined the channel and had few interesting questions. After an hour, we ended the call.

  Aaron Hall joined group chat.
	
10:54 AM Aaron Hall: hi!
10:54 AM Dhavan Vaidya: hi!
	
10:55 AM Aaron Hall: This is really cool!

Maikel Yugcha can now join this call.
Maikel Yugcha joined group chat.
	
10:57 AM Aaron Hall: Anyone here using Emacs as their window manager?
	
10:57 AM Suraj Ghimire: not yet :)
	
10:57 AM Shakthi Kannan: I am "mbuf" on IRC. http://stumpwm.github.io/
	
10:58 AM Aaron Hall: What about on servers? I just played around, but I like tmux for persistence and emacs inside of tmux.
	
10:59 AM Shakthi Kannan: https://github.com/pashinin/workgroups2
	
11:00 AM Aaron Hall: Is anyone compiling emacs from source?

Zsolt Botykai can now join this call.
Zsolt Botykai joined group chat.
	
11:00 AM Aaron Hall: yay, me too!

Zsolt Botykai left group chat.
	
11:00 AM Aaron Hall: it wasn't easy to start, the config options are hard
                     I had trouble especially with my fonts until I got my configure right

Maikel Yugcha left group chat.
	
11:03 AM Shakthi Kannan: https://github.com/shakthimaan/cask-dot-emacs
	
11:04 AM Aaron Hall anyone using Haskell? With orgmode? I've been having a lot of trouble with that...
                    code blocks are hard to get working
                    ghci
                    inferior?
                    not really sure
                    it's been a while since I worked on it
                    I had a polyglot file I was working on, I got a lot of languages working
                    Python, Bash, R, Javascript,
                    I got C working too
	
11:06 AM Shakthi Kannan: Rajesh: http://company-mode.github.io/
	
11:07 AM Aaron Hall: cheers, this was fun!

Aaron Hall left group chat.
Dhavan Vaidya left group chat.
Rajesh Deo left group chat.
Google Apps left group chat.
Suraj Ghimire left group chat.

A screenshot of the Google Hangout session is shown below:

We can try a different online platform for the next meetup (Monday, February, 19, 2018). I would like to have the meetup on the third Monday of every month. Special thanks to Sacha Chua for her valuable inputs in organizing the online meetup session.

Introduction

In this fifth article in the DevOps series we will learn to install and set up Graphite using Ansible. Graphite is a monitoring tool that was written by Chris Davis in 2006. It has been released under the Apache 2.0 license and comprises three components:

1. Graphite-Web
2. Carbon
3. Whisper

Graphite-Web is a Django application and provides a dashboard for monitoring. Carbon is a server that listens to time-series data, while Whisper is a database library for storing the data.

Setting it up

A CentOS 6.8 Virtual Machine (VM) running on KVM is used for the installation. Please make sure that the VM has access to the Internet. The Ansible version used on the host (Parabola GNU/Linux-libre x86_64) is 2.2.1.0. The ansible/ folder contains the following files:

ansible/inventory/kvm/inventory
ansible/playbooks/configuration/graphite.yml
ansible/playbooks/admin/uninstall-graphite.yml

The IP address of the guest CentOS 6.8 VM is added to the inventory file as shown below:

graphite ansible_host=192.168.122.120 ansible_connection=ssh ansible_user=root ansible_password=password

Also, add an entry for the graphite host in /etc/hosts file as indicated below:

192.168.122.120 graphite

Graphite

The playbook to install the Graphite server is given below:

---
- name: Install Graphite software
  hosts: graphite
  gather_facts: true
  tags: [graphite]

  tasks:
    - name: Import EPEL GPG key
      rpm_key:
        key: http://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6
        state: present

    - name: Add YUM repo
      yum_repository:
        name: epel
        description: EPEL YUM repo
        baseurl: https://dl.fedoraproject.org/pub/epel/$releasever/$basearch/
        gpgcheck: yes

    - name: Update the software package repository
      yum:
        name: '*'
        update_cache: yes

    - name: Install Graphite server
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - graphite-web

We first import the keys for the Extra Packages for Enterprise Linux (EPEL) repository and update the software package list. The ‘graphite-web’ package is then installed using Yum. The above playbook can be invoked using the following command:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/graphite.yml --tags "graphite"

MySQL

A backend database is required by Graphite. By default, the SQLite3 database is used, but we will install and use MySQL as shown below:

- name: Install MySQL
  hosts: graphite
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [database]

  tasks:
    - name: Install database
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - mysql
        - mysql-server
        - MySQL-python
        - libselinux-python

    - name: Start mysqld server
      service:
        name: mysqld
        state: started

    - wait_for:
        port: 3306

    - name: Create graphite database user
      mysql_user:
        name: graphite
        password: graphite123
        priv: '*.*:ALL,GRANT'
        state: present

    - name: Create a database
      mysql_db:
        name: graphite
        state: present

    - name: Update database configuration
      blockinfile:
        path: /etc/graphite-web/local_settings.py
        block: |
          DATABASES = {
            'default': {
            'NAME': 'graphite',
            'ENGINE': 'django.db.backends.mysql',
            'USER': 'graphite',
            'PASSWORD': 'graphite123',
           }
          }

    - name: syncdb
      shell: /usr/lib/python2.6/site-packages/graphite/manage.py syncdb --noinput

    - name: Allow port 80
      shell: iptables -I INPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT

    - name:
      lineinfile:
        path: /etc/httpd/conf.d/graphite-web.conf
        insertafter: '           # Apache 2.2'
        line: '           Allow from all'

    - name: Start httpd server
      service:
        name: httpd
        state: started

As a first step, let’s install the required MySQL dependency packages and the server itself. We then start the server and wait for it to listen on port 3306. A graphite user and database is created for use with the Graphite Web application. For this example, the password is provided as plain text. In production, use an encrypted Ansible Vault password.

The database configuration file is then updated to use the MySQL credentials. Since Graphite is a Django application, the manage.py script with syncdb needs to be executed to create the necessary tables. We then allow port 80 through the firewall in order to view the Graphite dashboard. The graphite-web.conf file is updated to allow read access, and the Apache web server is started.

The above playbook can be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/graphite.yml --tags "database"

Carbon and Whisper

The Carbon and Whisper Python bindings need to be installed before starting the carbon-cache script.

- name: Install Carbon and Whisper
  hosts: graphite
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [carbon]

  tasks:
    - name: Install carbon and whisper
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - python-carbon
        - python-whisper

    - name: Start carbon-cache
      shell: /etc/init.d/carbon-cache start

The above playbook is invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/graphite.yml --tags "carbon"

Dashboard

You can open http://192.168.122.120 in the browser on the host to view the Graphite dashboard. A screenshot of the Graphite web application is shown below:

Uninstall

An uninstall script to remove the Graphite server and its dependency packages is required for administration. The Ansible playbook for the same is available in playbooks/admin folder and is given below:

---
- name: Uninstall Graphite and dependencies
  hosts: graphite
  gather_facts: true
  tags: [remove]

  tasks:
    - name: Stop the carbon-cache server
      shell: /etc/init.d/carbon-cache stop

    - name: Uninstall carbon and whisper
      package:
        name: "{{ item }}"
        state: absent
      with_items:
        - python-whisper
        - python-carbon

    - name: Stop httpd server
      service:
        name: httpd
        state: stopped

    - name: Stop mysqld server
      service:
        name: mysqld
        state: stopped

    - name: Uninstall database packages
      package:
        name: "{{ item }}"
        state: absent
      with_items:
        - libselinux-python
        - MySQL-python
        - mysql-server
        - mysql
        - graphite-web

The script can be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/admin/uninstall-graphite.yml

References

1. Graphite documentation. https://graphite.readthedocs.io/en/latest/

2. Carbon. https://github.com/graphite-project/carbon

3. Whisper database. http://graphite.readthedocs.io/en/latest/whisper.html

Introduction

In this fourth article in the DevOps series, we will learn to install RabbitMQ using Ansible. RabbitMQ is a free and open source message broker system that supports a number of protocols such as the Advanced Message Queuing Protocol (AMQP), Streaming Text Oriented Messaging Protocol (STOMP) and Message Queue Telemetry Transport (MQTT). The software has support for a large number of client libraries for different programming languages. RabbitMQ is written using the Erlang programming language and is released under the Mozilla Public License.

Setting it up

A CentOS 6.8 virtual machine (VM) running on KVM is used for the installation. Do make sure that the VM has access to the Internet. The Ansible version used on the host (Parabola GNU/Linux-libre x86_64) is 2.2.1.0. The ansible/ folder contains the following files:

ansible/inventory/kvm/inventory
ansible/playbooks/configuration/rabbitmq.yml
ansible/playbooks/admin/uninstall-rabbitmq.yml

The IP address of the guest CentOS 6.8 VM is added to the inventory file as shown below:

rabbitmq ansible_host=192.168.122.161 ansible_connection=ssh ansible_user=root ansible_password=password

Also, add an entry for the rabbitmq host in the /etc/hosts file as indicated below:

192.168.122.161 rabbitmq

Installation

RabbitMQ requires the Erlang environment, and uses the Open Telecom Platform (OTP) framework. There are multiple sources for installing Erlang - the EPEL repository, Erlang Solutions, zero-dependency Erlang provided by RabbitMQ. In this article, we will use the EPEL repository for installing Erlang.

---
- name: Install RabbitMQ server
  hosts: rabbitmq
  gather_facts: true
  tags: [server]

  tasks:
    - name: Import EPEL GPG key
      rpm_key:
        key: http://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6
        state: present

    - name: Add YUM repo
      yum_repository:
        name: epel
        description: EPEL YUM repo
        baseurl: https://dl.fedoraproject.org/pub/epel/$releasever/$basearch/
        gpgcheck: yes

    - name: Update the software package repository
      yum:
        name: '*'
        update_cache: yes

    - name: Install RabbitMQ server
      package:
        name: "{{ item }}"
        state: latest
      with_items:
        - rabbitmq-server

    - name: Start the RabbitMQ server
      service:
        name: rabbitmq-server
        state: started

    - wait_for:
        port: 5672

After importing the EPEL GPG key and adding the EPEL repository to the system, the yum update command is executed. The RabbitMQ server and its dependencies are then installed. We wait for the RabbitMQ server to start and to listen on port 5672. The above playbook can be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/rabbitmq.yml --tags "server"

Dashboard

The RabbitMQ management user interface (UI) is available through plugins.

- name: Start RabbitMQ Management UI
  hosts: rabbitmq
  gather_facts: true
  tags: [ui]

  tasks:
    - name: Start management UI
      command: /usr/lib/rabbitmq/bin/rabbitmq-plugins enable rabbitmq_management

    - name: Restart RabbitMQ server
      service:
        name: rabbitmq-server
        state: restarted

    - wait_for:
        port: 15672

    - name: Allow port 15672
      shell: iptables -I INPUT 5 -p tcp --dport 15672 -m state --state NEW,ESTABLISHED -j ACCEPT

After enabling the management plugin, the server needs to be restarted. Since we are running it inside the VM, we need to allow the management user interface (UI) port 15672 through the firewall. The playbook invocation to set up the management UI is given below:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/rabbitmq.yml --tags "ui"

The default user name and password for the dashboard are ‘guest:guest’. From your host system, you can start a browser and open http://192.168.122.161:15672 to view the login page as shown in Figure 1. The default ‘Overview’ page is shown in Figure 2.

Ruby

We will use a Ruby client example to demonstrate that our installation of RabbitMQ is working fine. The Ruby Version Manager (RVM) will be used to install Ruby as shown below:

- name: Ruby client
  hosts: rabbitmq
  gather_facts: true
  tags: [ruby]

  tasks:
    - name: Import key
      command: gpg2 --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3

    - name: Install RVM
      shell: curl -sSL https://get.rvm.io | bash -s stable

    - name: Install Ruby
      shell: source /etc/profile.d/rvm.sh && rvm install ruby-2.2.6

    - name: Set default Ruby
      command: rvm alias create default ruby-2.2.6

    - name: Install bunny client
      shell: gem install bunny --version ">= 2.6.4"

After importing the required GPG keys, RVM and Ruby 2.2.6 are installed on the CentOS 6.8 VM. The bunny Ruby client for RabbitMQ is then installed. The Ansible playbook to setup Ruby is given below:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/rabbitmq.yml --tags "ruby"

We shall create a ‘temperature’ queue to send the values in Celsius. The consumer.rb code to receive the values from the queue is given below:

#!/usr/bin/env ruby

require "bunny"

conn = Bunny.new(:automatically_recover => false)
conn.start

chan  = conn.create_channel
queue = chan.queue("temperature")

begin
  puts " ... waiting. CTRL+C to exit"
  queue.subscribe(:block => true) do |info, properties, body|
    puts " Received #{body}"
  end
rescue Interrupt => _
  conn.close

  exit(0)
end

The producer.rb code to send a sample of five values in degree Celsius is as follows:

#!/usr/bin/env ruby

require "bunny"

conn = Bunny.new(:automatically_recover => false)
conn.start

chan   = conn.create_channel
queue   = chan.queue("temperature")

values = ["33.5", "35.2", "36.7", "37.0", "36.4"]

values.each do |v|
  chan.default_exchange.publish(v, :routing_key => queue.name)
end
puts "Sent five temperature values."

conn.close

As soon as you start the consumer, you will get the following output:

$ ruby consumer.rb 
 ... waiting. CTRL+C to exit

You can then run the producer.rb script that writes the values to the queue:

$ ruby producer.rb

Sent five temperature values.

The received values at the consumer side are printed out as shown below:

$ ruby consumer.rb 
 ... waiting. CTRL+C to exit
 Received 33.5
 Received 35.2
 Received 36.7
 Received 37.0
 Received 36.4

We can observe the available connections and the created queue in the management user interface as shown in Figure 3 and Figure 4, respectively.

Uninstall

It is good to have an uninstall script to remove the RabbitMQ server for administrative purposes. The Ansible playbook for the same is available in the playbooks/admin folder and is shown below:

---
- name: Uninstall RabbitMQ server
  hosts: rabbitmq
  gather_facts: true
  tags: [remove]

  tasks:
    - name: Stop the RabbitMQ server
      service:
        name: rabbitmq-server
        state: stopped

    - name: Uninstall rabbitmq
      package:
        name: "{{ item }}"
        state: absent
      with_items:
        - rabbitmq-server

The script can be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/admin/uninstall-rabbitmq.yml

You are encouraged to read the detailed documentation at https://www.rabbitmq.com/documentation.html to know more about the usage, configuration, client libraries and plugins available for RabbitMQ.

Introduction

In this third article in the DevOps series, we will install and set up Cacti, a free and open source Web-based network monitoring and graphing tool, using Ansible. Cacti is written in PHP and uses the MySQL database as a backend. It uses the RRDtool (Round-Robin Database tool) to handle time series data and has built-in SNMP support. Cacti has been released under the GNU General Public License.

Setting up Cacti

We will use a CentOS 6.8 virtual machine (VM) running on KVM to setup Cacti. Just for this demonstration, we will disable SELinux. You will need to set the following in /etc/selinux/config and reboot the VM.

SELINUX=disabled

When using in production, it is essential that you enable SELinux. You should then test for Internet connectivity from within the VM.

The Ansible version used on the host Parabola GNU/Linux-libre x86_64 is 2.2.1.0. The ansible/inventory/kvm/ directory structure is shown below:

ansible/inventory/kvm/inventory
ansible/inventory/kvm/group_vars/all/all.yml

The IP address of the guest CentOS 6.8 VM is provided in the inventory file as shown below:

centos ansible_host=192.168.122.98 ansible_connection=ssh ansible_user=root ansible_password=password

Add an entry for ‘centos’ in the /etc/hosts file as indicated below:

192.168.122.98 centos

The contents of the all.yml for use with the playbook are as follows:

---
mysql_cacti_password_hash: "{{ vault_mysql_cacti_password_hash }}"

mysql_username: "{{ vault_mysql_user }}"
mysql_password: "{{ vault_mysql_password }}"

The cacti.yml playbook is located in the ansible/playbooks/configuration folder.

Vault

Ansible provides the Vault feature, which allows you to store sensitive information like passwords in encrypted files. You can set the EDITOR environment variable to the text editor of your choice, as shown below:

$ export EDITOR=nano

In order to store our MySQL database credentials, we will create a vault.yml file as indicated below:

$ ansible-vault create inventory/kvm/group_vars/all/vault.yml

Provide a password when prompted, following which, the Nano text editor will open. You can enter the following credentials and save the file.

---
vault_mysql_cacti_password_hash: "*528573A4E6FE4F3E8B455F2F060EB6F63ECBECAA"

vault_mysql_user: "cacti"
vault_mysql_password: "cacti123"

You can edit the same file, if you wish, using the following command:

$ ansible-vault edit inventory/kvm/group_vars/all/vault.yml

It will prompt you for a password, and on successful authentication, your text editor will open with the decrypted file contents for editing.

Apache

Cacti has many dependency packages, and the first software that we will install is the Apache HTTP server.

---
- name: Install web server
  hosts: centos
  gather_facts: true
  tags: [httpd]

  tasks:
    - name: Update the software package repository
      yum:
	name: '*'
	update_cache: yes

    - name: Install HTTP packages
      package:
	name: "{{ item }}"
	state: latest
      with_items:
	- wget
	- nano
	- httpd
	- httpd-devel

    - name: Start the httpd server
      service:
	name: httpd
	state: started

    - wait_for:
	port: 80

A ‘yum update’ is first performed to sync with the package repositories. The httpd Web server and a few other packages are then installed. The server is started, and the Ansible playbook waits for the server to listen on port 80.

MySQL and PHP

The MySQL, PHP and RRDTool packages are then installed, following which the SNMP and MySQL servers are started as shown below:

- name: Install MySQL, PHP packages
  hosts: centos
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [database-web]

  tasks:
    - name: Install database/web packages
      package:
	name: "{{ item }}"
	state: latest
      with_items:
	- mysql
	- mysql-server
	- MySQL-python
	- php-mysql
	- php-pear
	- php-common
	- php-gd
	- php-devel
	- php
	- php-mbstring
	- php-cli
	- php-process
	- php-snmp
	- net-snmp-utils
	- net-snmp-libs
	- rrdtool

    - name: Start snmpd server
      service:
	name: snmpd
	state: started

    - name: Start mysqld server
      service:
	name: mysqld
	state: started

    - wait_for:
	port: 3306

Cacti

Cacti is available in the EPEL repository for CentOS. The GPG key for the CentOS repositories is enabled before installing the EPEL repository. A ‘yum update’ is performed and the Cacti package is installed. A ‘cacti’ user is then created in the MySQL database.

- name: Install Cacti
  hosts: centos
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [cacti]

  tasks:
    - name: Import EPEL GPG key
      rpm_key:
	key: http://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-6
	state: present

    - name: Add YUM repo
      yum_repository:
	name: epel
	description: EPEL YUM repo
	baseurl: https://dl.fedoraproject.org/pub/epel/$releasever/$basearch/
	gpgcheck: yes

    - name: Update the software package repository
      yum:
	name: '*'
	update_cache: yes

    - name: Install cacti
      package:
	name: "{{ item }}"
	state: latest
      with_items:
	- cacti

    - name: Create cacti database user
      mysql_user:
	name: cacti
	password: "{{ mysql_cacti_password_hash }}"
	encrypted: yes
	priv: '*.*:ALL,GRANT'
	state: present

Fixing a bug

The time zone data is missing in this MySQL version (5.1.73-8). In order to resolve this bug, the mysql_test_data_timezone.sql file needs to be imported and the ‘cacti’ user needs to be given the SELECT privilege to do this.

- name: For bug https://github.com/Cacti/cacti/issues/242
  hosts: centos
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [bug]

  tasks:
    - name: Import mysql_test_data_timezone.sql
      mysql_db:
	state: import
	name: mysql
	target: /usr/share/mysql/mysql_test_data_timezone.sql

    - name: Grant privileges
      mysql_user:
	name: cacti
	append_privs: true
	priv: 'mysql.time_zone_name:SELECT'
	state: present

It is a good practice to have a separate playbook for such exceptional cases. In future, when you upgrade to newer versions that have bug fixes, you can simply skip this step.

Configuration

The last step involves configuring Cacti.

- name: Configuration
  hosts: centos
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [config]

  tasks:
    - name: Create a database for cacti
      mysql_db:
	name: cacti
	state: present

    - name: Import cacti.sql
      mysql_db:
	state: import
	name: cacti
	target: /usr/share/doc/cacti-1.0.4/cacti.sql

    - name: Update database credentials in config file
      lineinfile:
	dest: /etc/cacti/db.php
	regexp: "{{ item.regexp }}"
	line: "{{ item.line }}"
      with_items:
	- { regexp: '^\$database_username', line: "$database_username = '{{ mysql_username }}';" }
	- { regexp: '^\$database_password', line: "$database_password = '{{ mysql_password }}';" }

    - name: Allow port 80
      shell: iptables -I INPUT 5 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT

    - name: Update access in cacti.conf for httpd
      replace:
	dest: /etc/httpd/conf.d/cacti.conf
	regexp: "{{ item.regexp }}"
	replace: "{{ item.replace }}"
      with_items:
	- { regexp: 'Require host localhost', replace: 'Require all granted' }
	- { regexp: 'Allow from localhost', replace: 'Allow from all' }

    - lineinfile:
	dest: /etc/cron.d/cacti
	regexp: '^#(.*)$'
	line: '\1'
	backrefs: yes    

    - name: Start mysqld server
      service:
	name: mysqld
	state: restarted

    - wait_for:
	port: 3306

    - name: Start the httpd server
      service:
	name: httpd
	state: restarted

    - wait_for:
	port: 80

A database called ‘cacti’ is created for the application, and the cacti.sql file is imported into it. The database credentials are updated for the Cacti application. The firewall rules are then updated to allow incoming HTTP requests for port 80. The periodic cron poller is then enabled in /etc/cron.d/cacti:

*/5 * * * *     cacti   /usr/bin/php /usr/share/cacti/poller.php > /dev/null 2>&1

The MySQL and HTTP servers are then restarted.

The result

The entire playbook can now be invoked as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/cacti.yml --ask-vault-pass

It will prompt you for the Vault password, following which all the playbooks will be completed. You can then open http://192.168.122.98/cacti to accept the GNU General Public License agreement. After you agree to the terms of the license, click ‘Next’. The Cacti installation wizard shows the pre-installation checks, which should not have any errors. This is followed by the selection of the installation type, binary location, version, and the directory permission checks. You can then decide on the templates you would like to set up, following which a user login is provided. The default user name and password is ‘admin:admin’, and you will be immediately prompted to change the password after logging in. You can then proceed to log in to the Cacti dashboard. Figures 1 to 8 give the screenshots of the Cacti Web UI installation for reference.

A screenshot of Cacti graphing for memory usage is shown in Figure 9.

I arrived at Belgaum on Friday, October 13, 2017 morning by train, and headed to the Sankalp Bhumi resort. I was given the Rock Cottage Dhyan (“meditation”) for my stay. After freshening up and having breakfast, I started to explore the resort. The place was an abandoned stone quarry that has been converted into a beautiful resort spread over an area of eight acres. It is a very quiet, peaceful place with trees, lawns, and water bodies. A number of fellow hackers and mentors also started arriving at the venue, and we were just getting ready for the day’s events.

After lunch, we headed to the large function hall where volunteers were to be trained on soldering. Jithin B. P. had prepared a simple PCB layout with LEDs and potentiometers for volunteers to practice soldering. These volunteers will help school students in the soldering workshops in the coming two days at the event.

In the evening, I had suggested to Praveen Patil and Hitesh Dharmdasani to have an informal BoF session. So, everyone got together under one roof, and Hitesh explained how and why they started SHD. There were a round of introductions followed by open discussions. After dinner, some of our fellow hackers headed to the hackerspace (“cafeteria”) to test the Wi-Fi, and prepare for the next day’s hacks.

Day 1

The volunteers arrived in the morning to ready the registration desk and prepare the name cards. Science Hack Day banners were placed at various locations in the premises. After breakfast, everyone gathered at the function hall for an informal inauguration. This was followed by a 30-minute introduction on Aeromodelling by Mr. Arjun Bilawar (retired trainer, NCC Karnataka and Goa Directorate). It had rained heavily last evening and hence a nearby playground was not suitable to demo the aeroplane. Mr. Arjun then took us to a nearby open area, and with a short runway, he was able to lift off the model plane, and flew it for few minutes to the loud cheers from the students.

We then headed to the hackerspace to start work on our projects. Siddhesh and other hackers began to build a 3D printer. Shreyas K was building an image detector to be used in his college to detect students jumping off the wall in his college premises. This was covered in the news, Tarun Bharat.

Students from Kerala were working on a Bat detector. Few students were creating 3D artefacts using the 3D pen from reserved-bit. Hitesh Dharmdasani had brought his company’s (Informant Networks) Banana Pi based-router for hacking. Vaishali Thakkar was working on the Tessel board and Rust. I was setting up the Icoboard with Raspberry Pi 3 Model B to demonstrate chip design on free and open source hardware.

The school students were divided into two groups. One group attended the soldering workshop in the function hall, and the other group participated in the science toys workshop in an open arena. The groups will swap the workshops the following day. In another sit out area, biology students had brought specimens from the labs to showcase them. A team of architects were also creating a geodesic dome that can be used as a makerspace.

In the evening at 1830 IST, we had lightning talks, and I had presented the latest version of “Nursery Rhymes”. There were other interesting talks on embedded hardware, security, product design, mechanics of a bicycle, biogas etc. It was a long day!

Day 2

The second day of the event began with a video conferencing session with Science Hack Day, San Francisco. Their event was scheduled on October 14-15, 2017 at the GitHub office. Ariel Waldman used the webcam to show around their hackerspace. The students then headed to their respective soldering and science toys workshops. Some of the toys and experiments were from Arvind Gupta. The hackers at the makerspace continued working on their projects. All the hacks had to be completed by noon, as they have to be demoed and voted for in the evening.

After lunch, there was a rocket session where students launched rockets prepared by experts. We also had a group photo session, and headed to the function hall. All the hacks were presented to the students, and parents were also invited to attend the evening session. There was also an audience voting poll to choose their favourite science hack. After demonstrating the Icoboard, I had to rush to catch the train back to Bengaluru, so I did miss the closing ceremony.

My hearty congratulations to the organizers of Science Hack Day India 2017, Belgaum for a wonderful event that motivates students to learn and experiment with science.

I have uploaded over 80 photos during the trip to my new gallery. Do browse through them leisurely!

This is the second article in the DevOps series, and covers the installation of a (G/)LAMP stack and WordPress, using Ansible.

Introduction

In this article, we are going to learn how to automate the deployment of a (G/)LAMP stack and install WordPress. (G/)LAMP stands for GNU/Linux, Apache (a web server), MySQL (a database) and PHP (server-side scripting). It is a technology stack on which you can deploy different web applications. We are also going to explore the installation of WordPress, which is free and open source software for creating websites and blogs.

Linux

A Parabola GNU/Linux-libre x86_64 system is used as the host system. An Ubuntu 15.04 image runs as a guest OS using KVM/QEMU. Ansible is installed on the host system using the distribution package manager. You should be able to issue commands from Ansible to the guest OS. For example:

$ ansible ubuntu -m ping

ubuntu | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

The /etc/hosts file already has an entry for the guest Ubuntu VM.

192.168.122.250 ubuntu

On the host system, we will create a project for our playbooks. It has the following directory structure:

ansible/inventory/kvm/
       /playbooks/configuration/
       /playbooks/admin/

An ‘inventory’ file is created inside the inventory/kvm folder that contains the following:

ubuntu ansible_host=192.168.122.250 ansible_connection=ssh ansible_user=xetex

Installing Apache

We will first install and test the Apache web server on the guest Ubuntu system. Let’s then create a playbooks/configuration/apache.yml file with the following content:

---
- name: Install Apache web server
  hosts: ubuntu
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [web]

  tasks:
    - name: Update the software package repository
      apt:
	update_cache: yes

    - name: Install Apache
      package:
	name: "{{ item }}"
	state: latest
      with_items:
	- apache2

    - wait_for:
	port: 80

On the Ubuntu guest system, the playbook runs apt-get update and then installs the apache2 package. The playbook finishes after the server has started, and is listening on port 80. Open a terminal, enter the ansible/ folder, and execute the playbook as shown below:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/apache.yml -K 
SUDO password: 

PLAY [Install Apache web server] ***********************************************

TASK [setup] *******************************************************************
ok: [ubuntu]

TASK [Update the software package repository] **********************************
changed: [ubuntu]

TASK [Install Apache] **********************************************************
changed: [ubuntu] => (item=[u'apache2'])

TASK [wait_for] ****************************************************************
ok: [ubuntu]

PLAY RECAP *********************************************************************
ubuntu                     : ok=4    changed=2    unreachable=0    failed=0   

The ’-K’ option is to prompt for the sudo password for the ‘xetex’ user. You can increase the level of verbosity in the Ansible output by passing ’-vvvv’ at the end of the ansible-playbook command. The more number of times ‘v’ occurs, the greater is the verbosity level.

If you now open http://192.168.122.250, you should be able to see the default Apache2 index.html page as shown in Figure 1:

Installing MySQL

The next step is to install the MySQL database server. The corresponding playbook is provided below:

---
- name: Install MySQL database server
  hosts: ubuntu
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [database]

  tasks:
    - name: Update the software package repository
      apt:
	update_cache: yes

    - name: Install MySQL
      package:
	name: "{{ item }}"
	state: latest
      with_items:
	- mysql-server
	- mysql-client
	- python-mysqldb

    - name: Start the server
      service:
	name: mysql
	state: started

    - wait_for:
	port: 3306

    - mysql_user:
	name: guest
	password: '*F7B659FE10CA9FAC576D358A16CC1BC646762FB2'
	encrypted: yes
	priv: '*.*:ALL,GRANT'
	state: present

The package repository is updated and the necessary MySQL packages are installed. The database server is then started, and we wait for the server to be up and running. A ‘guest’ user account with ‘osfy’ as the password is created for use in our web application. The chosen password is just for demonstration purposes. When used in production, please select a strong password with special characters and numerals.

You can compute the hash for a password from the MySQL client, as shown below:

mysql> SELECT PASSWORD('osfy');
+-------------------------------------------+
| PASSWORD('osfy')                          |
+-------------------------------------------+
| *F7B659FE10CA9FAC576D358A16CC1BC646762FB2 |
+-------------------------------------------+
1 row in set (0.00 sec)

An execution run to install MySQL is as follows:

$ ansible-playbook -i inventory/kvm/inventory playbooks/configuration/mysql.yml -K 
SUDO password: 

PLAY [Install MySQL database server] *******************************************

TASK [setup] *******************************************************************
ok: [ubuntu]

TASK [Update the software package repository] **********************************
changed: [ubuntu]

TASK [Install MySQL] ***********************************************************
changed: [ubuntu] => (item=[u'mysql-server', u'mysql-client', u'python-mysqldb'])

TASK [Start the server] ********************************************************
ok: [ubuntu]

TASK [wait_for] ****************************************************************
ok: [ubuntu]

TASK [mysql_user] **************************************************************
ok: [ubuntu]

PLAY RECAP *********************************************************************
ubuntu                     : ok=6    changed=2    unreachable=0    failed=0   

Note: The default MySQL root password is empty. You should change it after installation.

Installing PHP

PHP is a server-side programming language and stands for PHP: Hypertext Preprocessor (a recursive acronym). Although we have used PHP5 in ths example, it is recommended that you use the latest PHP for security reasons. The Ansible playbook for installing PHP is given below:

---
- name: Install PHP
  hosts: ubuntu
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [web]

  tasks:
    - name: Update the software package repository
      apt:
	update_cache: yes

    - name: Install PHP
      package:
	name: "{{ item }}"
	state: latest
      with_items:
	- php5
	- php5-mysql

We update the software repository and install PHP5. An execution output of the Ansible playbook is shown below:

$  ansible-playbook -i inventory/kvm/inventory playbooks/configuration/php.yml -K 
SUDO password: 

PLAY [Install PHP] *************************************************************

TASK [setup] *******************************************************************
ok: [ubuntu]

TASK [Update the software package repository] **********************************
changed: [ubuntu]

TASK [Install PHP] *************************************************************
changed: [ubuntu] => (item=[u'php5', u'php5-mysql'])

PLAY RECAP *********************************************************************
ubuntu                     : ok=3    changed=2    unreachable=0    failed=0  

Installing WordPress

As a final step, we will download, install and set up WordPress. The complete playbook is as follows:

---
- name: Setup Wordpress
  hosts: ubuntu
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [database]

  vars:
    wordpress_file: "/home/{{ ansible_user }}/Downloads/wordpress-latest.zip"
    wordpress_dest: "/var/www/html"

  tasks:
    - name: Update the software package repository
      apt:
	update_cache: yes

    - name: Create a database for wordpress
      mysql_db:
	name: wordpress
	state: present

    - name: Create downloads directory
      file:
	path: "/home/{{ ansible_user }}/Downloads"
	state: directory

    - name: Create target directory
      file:
	path: "{{ wordpress_dest }}/wordpress"
	state: directory

    - name: Download latest wordpress
      get_url:
	url: https://wordpress.org/latest.zip
	dest: "{{ wordpress_file }}"

    - name: Extract to /var/www/html
      unarchive:
	src: "{{ wordpress_file }}"
	dest: "{{ wordpress_dest}}"
	remote_src: True

    - name: Copy wp-config-sample.php to wp-config.php
      command: cp "{{ wordpress_dest }}/wordpress/wp-config-sample.php" "{{ wordpress_dest }}/wordpress/wp-config.php"

    - name: Update database credentials in the file
      replace:
	dest: "{{ wordpress_dest }}/wordpress/wp-config.php"
	regexp: "{{ item.regexp }}"
	replace: "{{ item.replace }}"
      with_items:
	- { regexp: 'database_name_here', replace: 'wordpress' }
	- { regexp: 'username_here', replace: 'guest' }
	- { regexp: 'password_here', replace: 'osfy'}

    - name: Restart apache2 server
      service:
	name: apache2
	state: restarted

We create variables to store the downloaded file for WordPress, and the target installation path. After updating the software repository, a database is created for the WordPress application. The download and target directories are created on the guest system, before actually downloading the latest WordPress sources. A configuration file is then created, and the database settings are updated. Although we explicitly specify the password here, the recommended practice is to store the encrypted passwords in an Ansible Vault file, and reference the same in the playbook. In future articles, I will demonstrate this use case. After completing the configuration, the web server is restarted. An execution run of the playbook is shown below:

 $  ansible-playbook -i inventory/kvm/inventory playbooks/configuration/wordpress.yml -K 
SUDO password: 

PLAY [Setup Wordpress] *********************************************************

TASK [setup] *******************************************************************
ok: [ubuntu]

TASK [Update the software package repository] **********************************
changed: [ubuntu]

TASK [Create a database for wordpress] *****************************************
changed: [ubuntu]

TASK [Create downloads directory] **********************************************
ok: [ubuntu]

TASK [Create target directory] *************************************************
changed: [ubuntu]

TASK [Download latest wordpress] ***********************************************
ok: [ubuntu]

TASK [Extract to /var/www/html] ************************************************
changed: [ubuntu]

TASK [Copy wp-config-sample.php to wp-config.php] ******************************
changed: [ubuntu]

TASK [Update database credentials in the file] *********************************
changed: [ubuntu] => (item={u'regexp': u'database_name_here', u'replace': u'wordpress'})
changed: [ubuntu] => (item={u'regexp': u'username_here', u'replace': u'guest'})
changed: [ubuntu] => (item={u'regexp': u'password_here', u'replace': u'osfy'})

TASK [Restart apache2 server] **************************************************
changed: [ubuntu]

PLAY RECAP *********************************************************************
ubuntu                     : ok=10   changed=7    unreachable=0    failed=0   

If you open the URL http://192.168.122.250/wordpress in a browser on the host system, you will see a screenshot as shown in Figure 2:

You can now proceed to complete the installation process from the browser. It is recommended that you follow the security best practices as recommended by the WordPress and PHP projects to harden this deployment.

Writing clean-up playbooks

It is essential to write clean-up playbooks to revert whatever changes you have made, so that you can roll back the system if things fail. Uninstalling should be done in the reverse order. For example, remove WordPress first, followed by PHP, MySQL and Apache.

The removal of WordPress could depend on your data retention policy. You might want to backup your PHP files, or you may decide to discard them. You might also want to retain the database. A complete removal of WordPress and the (G/)LAMP stack in the playbooks/admin folder is provided below for reference:

---
- name: Uninstall Wordpress
  hosts: ubuntu
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [web]

  vars:
    wordpress_dest: "/var/www/html"

  tasks:
    - name: Delete wordpress folder
      file:
	path: "{{ wordpress_dest }}/wordpress"
	state: absent

    - name: Drop database
      mysql_db:
	name: wordpress
	state: absent

---
- name: Uninstall PHP
  hosts: ubuntu
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [web]

  tasks:
    - name: Uninstall PHP packages
      package:
	name: "{{ item }}"
	state: absent
      with_items:
	- php5-mysql
	- php5

---
- name: Uninstall MySQL
  hosts: ubuntu
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [database]

  tasks:
    - name: Stop the database server
      service:
	name: mysql
	state: stopped

    - name: Uninstall MySQL packages
      package:
	name: "{{ item }}"
	state: absent
      with_items:
	- python-mysqldb
	- mysql-client
	- mysql-server

---
- name: Uninstall Apache web server
  hosts: ubuntu
  become: yes
  become_method: sudo
  gather_facts: true
  tags: [server]

  tasks:
    - name: Stop the web server
      service:
	name: apache2
	state: stopped

    - name: Uninstall apache2
      package:
	name: "{{ item }}"
	state: absent
      with_items:
	- apache2

The entire suite of playbooks is also available in my GitHub project ( https://github.com/shakthimaan/introduction-to-ansible ) for your reference.

With this article, we begin a new series on DevOps, starting out with Ansible, which helps you to build a strong foundation. As the Ansible website proclaims, proudly, “Deploy apps. Manage systems. Crush complexity.”

Introduction

Ansible is an IT automation tool that is used for provisioning, configuration, deployment and managing infrastructure. The project was first released in 2012, and is written in Python. The main objective of the tool is to be simple and easy to use. It is based on an agent-less (push-based) architecture, and the playbooks are written in plain English. Today, it also supports pull-based deployments. Ansible uses SSH to execute commands on remote machines. It is available under the GNU General Public License.

Installation

You can install Ansible using your GNU/Linux distribution package manager.

On Fedora, you can use Yum to install Ansible, as follows:

$ sudo yum install ansible

If you are using RHEL or CentOS, install the epel-release, and then use the Yum command to install Ansible.

On Ubuntu, you need to add the ppa repository before installing the tool, as shown below:

$ sudo apt-get install software-properties-common
$ sudo apt-add-repository ppa:ansible/ansible

$ sudo apt-get update
$ sudo apt-get install ansible

The Ansible documentation encourages Debian users to access the Ubuntu repository to obtain Ansible. You need to add the following line to /etc/apt/sources.list:

deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main

You can then install the software using the following commands:

$ sudo apt-get update
$ sudo apt-get install ansible

The Parabola GNU/Linux-libre distribution is a derivative of Arch Linux, without the binary blobs. You can install Ansible using the pacman utility:

$ pacman -S ansible

The latest Ansible version 2.2 (as of date) is what we will use in this article. Ansible is also available for BSD variants, Mac OS X, and Windows. You are encouraged to refer to the Ansible documentation for more information.

Virtualisation

Ansible can be used to provision new machines and also configure them. Instead of using bare metal machines, you can create multiple Virtual Machines (VMs) on your system. Lots of Free and Open Source Software (F/OSS) virtualization software is available.

QEMU is a machine emulator and virtualiser. It can also use host CPU support to run guest VMs for better performance. It is written by Fabrice Bellard, and released under the GNU General Public License (GPL). You can install it on Parabola GNU/Linux-libre, using the following command:

$ sudo pacman -S qemu

KVM or Kernel-based Virtual Machine (KVM) has direct support in the Linux kernel. It requires hardware support to be able to run guest operating systems. It is written in C, and is released under the GNU General Public License.

You need to check if your hardware first supports KVM. The ‘lscpu’ command will show an entry for ‘Virtualization’ if there is hardware support. For example:

$ lscpu

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 78
Model name:            Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
Stepping:              3
CPU MHz:               2275.341
CPU max MHz:           2800.0000
CPU min MHz:           400.0000
BogoMIPS:              4801.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              3072K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

You can also check the /proc/cpuinfo output as shown below:

$ grep -E "(vmx|svm)" --color=always /proc/cpuinfo

flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

The Libvirt project provides APIs to manage guest machines on KVM, QEMU and other virtualisation software. It is written in C, and is released under the GNU Lesser GPL. The Virtual Machine Manager (VMM) provides a graphical user interface for managing the guest VMs and is written in Python.

You can install all this software on Parabola GNU/Linux-libre using the following command:

$ sudo pacman -S libvirt virt-manager

A screenshot of Virtual Machine Manager is provided below:

Check your distribution documentation to install the appropriate virtualisation software packages.

You can use the VMM to create a new virtual machine, and install a GNU/Linux distribution using an .iso image. You can specify RAM, disk size and follow the installation steps for your particular distro. You can also import an existing .qcow2 disk image to use it as a virtual machine.

Ansible with libvirt-VM

The version of Ansible used for this article is given below:

$ ansible --version

ansible 2.2.1.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides

If you have the sshd daemon running on your local machine, you can use Ansible to test it. For example, a ping test on the localhost is shown below:

$ ansible localhost -m ping

localhost | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

You can also check how long the system has been up and running using the following command:

$ ansible localhost -a uptime

localhost | SUCCESS | rc=0 >>
 11:00:20 up  4:09,  0 users,  load average: 0.18, 0.14, 0.11

You can execute a shell command on the remote machine (localhost, in this case) as illustrated below:

$  ansible localhost -a "date"

localhost | SUCCESS | rc=0 >>
Sun Feb  5 11:24:53 IST 2017

The ‘setup’ command provides details of the remote target machine. A snippet output is provided below:

$  ansible localhost -m setup

localhost | SUCCESS => {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
            "192.168.10.1", 
            "192.168.5.6"
        ], 
        "ansible_all_ipv6_addresses": [
            "fe90::fc24:ff:feb9:cb61", 
            "ff80::5846:fac1:6afc:2e30"
        ], 
        "ansible_architecture": "x86_64", 
        "ansible_bios_date": "06/12/2016", 
        "ansible_bios_version": "R00ET45W (1.20 )", 
        "ansible_cmdline": {
            "BOOT_IMAGE": "/vmlinuz-linux-libre", 
            "cryptdevice": "/dev/sda1:cryptroot", 
            "quiet": true, 
            "root": "/dev/mapper/cryptroot", 
            "rw": true
        }, 
        ....

An Ubuntu 15.04 instance with VMM is used in the following examples with Ansible. The IP address of the instance is added to /etc/hosts:

192.168.122.250 ubuntu

The /etc/ansible/hosts file contains the following:

ubuntu

You can now do a ping test from the host to the Ubuntu VM using the following command sequence for the user ‘xetex’:

$  ansible ubuntu -m ping -u xetex --ask-pass
SSH password: 
ubuntu | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

To avoid prompting for the password, you can add the localhost public SSH key to the VM, as follows:

$  ssh-copy-id -i ~/.ssh/id_rsa.pub xetex@ubuntu

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/user/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
xetex@ubuntu's password: 

Number of key(s) added: 1

Now try logging into the machine, with ssh xetex@ubuntu and check to make sure that only the key you wanted was added.

You can now issue the following command to get the same result:

$ ansible ubuntu -m ping -u xetex

ubuntu | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

For the Ubuntu system, you can also add the defined user in the /etc/ansible/hosts file as follows:

ubuntu ansible_ssh_host=ubuntu ansible_ssh_user=xetex

The ping command is now simplified to:

$  ansible ubuntu -m ping

ubuntu | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

You can now try the earlier Ansible commands on the target Ubuntu VM as illustrated below:

$ ansible ubuntu -a uptime

ubuntu | SUCCESS | rc=0 >>
 12:32:14 up 25 min,  3 users,  load average: 0.02, 0.07, 0.06

$ ansible ubuntu -a date

ubuntu | SUCCESS | rc=0 >>
Sun Feb  5 12:32:45 IST 2017

$  ansible ubuntu -m setup
ubuntu | SUCCESS => {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
            "192.168.122.250"
        ], 
        "ansible_all_ipv6_addresses": [
            "ff20::5034:ff:fa9f:6123"
        ], 
        "ansible_architecture": "x86_64", 
        "ansible_bios_date": "04/01/2014", 
        "ansible_bios_version": "1.10.1-20151022_124906-anatol", 
        "ansible_cmdline": {
            "BOOT_IMAGE": "/boot/vmlinuz-3.19.0-15-generic", 
            "quiet": true, 
            "ro": true, 
            "root": "UUID=f43c2c72-5bc7-4a97-9a43-12e634ae232af", 
            "splash": true, 
            "vt.handoff": "7"
        }, 
        ...

Firstly, the Steel Bank Common Lisp (SBCL) compiler needs to be installed on the system using Pacman.

$ sudo pacman -S sbcl 
$ sudo pacman -S acpi xtrlock emacs mate-terminal iceweasel

The acpi utility is installed to show the battery status in the modeline. The xtrlock utility is used to lock the screen. When the screen is locked, you will need to enter your user password to unlock the same. GNU Emacs, Iceweasel and Mate-terminal software packages are also installed. Quicklisp is a package manager for Common Lisp and is used to install the StumpWM dependencies with SBCL.

An installation directory is created at ~/downloads/quicklisp, and the quicklisplisp file is fetched as follows:

$ mkdir ~/downloads/quicklisp
$ cd ~/downloads/quicklisp
$ curl -O https://beta.quicklisp.org/quicklisp.lisp

The quicklisp.lisp file is loaded using the SBCL interpreter. The installation steps and output are shown below:

~/downloads/quicklisp $ sbcl --load quicklisp.lisp 

    This is SBCL 1.3.17, an implementation of ANSI Common Lisp.
    More information about SBCL is available at <http://www.sbcl.org/>.

    SBCL is free software, provided as is, with absolutely no warranty.
    It is mostly in the public domain; some portions are provided under
    BSD-style licenses.  See the CREDITS and COPYING files in the
    distribution for more information.

      ==== quicklisp quickstart 2015-01-28 loaded ====

        To continue with installation, evaluate: (quicklisp-quickstart:install)

        For installation options, evaluate: (quicklisp-quickstart:help)

* (quicklisp-quickstart:install)

    ; Fetching #<URL "http://beta.quicklisp.org/client/quicklisp.sexp">
    ; 0.82KB
    ==================================================
    838 bytes in 0.00 seconds (0.00KB/sec)
    ; Fetching #<URL "http://beta.quicklisp.org/client/2017-03-06/quicklisp.tar">
    ; 250.00KB
    ==================================================
    256,000 bytes in 0.16 seconds (1602.56KB/sec)
    ; Fetching #<URL "http://beta.quicklisp.org/client/2015-09-24/setup.lisp">
    ; 4.94KB
    ==================================================
    5,054 bytes in 0.00 seconds (0.00KB/sec)
    ; Fetching #<URL "http://beta.quicklisp.org/asdf/2.26/asdf.lisp">
    ; 194.07KB
    ==================================================
    198,729 bytes in 0.09 seconds (2086.79KB/sec)
    ; Fetching #<URL "http://beta.quicklisp.org/dist/quicklisp.txt">
    ; 0.40KB
    ==================================================
    408 bytes in 0.00 seconds (0.00KB/sec)
    Installing dist "quicklisp" version "2017-07-25".
    ; Fetching #<URL "http://beta.quicklisp.org/dist/quicklisp/2017-07-25/releases.txt">
    ; 372.80KB
    ==================================================
    381,744 bytes in 0.15 seconds (2485.31KB/sec)
    ; Fetching #<URL "http://beta.quicklisp.org/dist/quicklisp/2017-07-25/systems.txt">
    ; 241.23KB
    ==================================================
    247,022 bytes in 0.10 seconds (2297.45KB/sec)

      ==== quicklisp installed ====

        To load a system, use: (ql:quickload "system-name")

        To find systems, use: (ql:system-apropos "term")

        To load Quicklisp every time you start Lisp, use: (ql:add-to-init-file)

        For more information, see http://www.quicklisp.org/beta/

    NIL

* (ql:add-to-init-file)

    I will append the following lines to #P"/home/guest/.sbclrc":

      ;;; The following lines added by ql:add-to-init-file:
      #-quicklisp
      (let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp"
                                             (user-homedir-pathname))))
        (when (probe-file quicklisp-init)
          (load quicklisp-init)))

    Press Enter to continue.

    #P"/home/guest/.sbclrc"

* (ql:quickload "clx")

    To load "clx":
      Install 1 Quicklisp release:
        clx
    ; Fetching #<URL "http://beta.quicklisp.org/archive/clx/2017-06-30/clx-20170630-git.tgz">
    ; 452.92KB
    ==================================================
    463,786 bytes in 0.12 seconds (3806.02KB/sec)
    ; Loading "clx"
    [package xlib]....................................
    ..................................................
    ..................................................
    ..................................................
    ..................................................
    ..................................................
    ..................................................
    ..................................................
    [package xlib/glx]................................
    [package xlib/gl].................................
    [package xlib/dpms]...............................
    [package xlib/xtest]..............................
    [package xlib/xinerama]...........................
    [package xlib-demo/clclock].......................
    [package xlib-demo/clipboard].....................
    [package xlib-demo/demos].........................
    [package xlib-demo/gl-test].......................
    [package xlib-demo/mandel].............
    ("clx")

* (ql:quickload "cl-ppcre")

    To load "cl-ppcre":
      Install 1 Quicklisp release:
        cl-ppcre
    ; Fetching #<URL "http://beta.quicklisp.org/archive/cl-ppcre/2015-09-23/cl-ppcre-2.0.11.tgz">
    ; 156.08KB
    ==================================================
    159,829 bytes in 0.08 seconds (1903.45KB/sec)
    ; Loading "cl-ppcre"
    [package cl-ppcre]................................
    ................
    ("cl-ppcre")

* (ql:quickload "alexandria")

    To load "alexandria":
      Install 1 Quicklisp release:
        alexandria
    ; Fetching #<URL "http://beta.quicklisp.org/archive/alexandria/2017-06-30/alexandria-20170630-git.tgz">
    ; 49.97KB
    ==================================================
    51,168 bytes in 0.04 seconds (1135.65KB/sec)
    ; Loading "alexandria"
    [package alexandria.0.dev].......................
    ("alexandria")

* (exit)

~/downloads/quicklisp $ 

The StumpWM v1.0.0 sources are fetched to ~/downloads/stumpwm directory, extracted and compiled using the following steps:

$ mkdir ~/downloads/stumpwm
$ cd ~/downloads/stumpwm
$ wget https://github.com/stumpwm/stumpwm/archive/1.0.0.tar.gz
$ tar xzvf 1.0.0.tar.gz
$ cd stumpwm-1.0.0/
$ ./autogen.sh 
$ ./configure 
$ make

You can then install the built stumpwm binary, which will get copied to /usr/local/bin/stumpwm as shown below:

$ sudo make install

$ which stumpwm
/usr/local/bin/stumpwm

I use the Lightweight X11 Display Manager (lxdm), and thus created a /usr/share/xsessions/stumpwm.desktop file to login to StumpWM from the login manager:

[Desktop Entry]
Encoding=UTF-8
Name=StumpWM
Comment=StumpWM
Exec=/usr/local/bin/stumpwm
Type=Application

A sample ~/.stumpwmrc configuration file is given below (should be self-explanatory):

;; -*-lisp-*-

(in-package :stumpwm)

;; Startup message: display the machine's name
(setf *startup-message* (machine-instance))

;; Turn on the modeline
(if (not (head-mode-line (current-head)))
    (toggle-mode-line (current-screen) (current-head)))

;; Lock screen
(define-key *root-map* (kbd "L") "exec xtrlock")

;; I like messages to be centered on the screen.
(setf *message-window-gravity* :center)
(setf *input-window-gravity* :center)

;; I thought that this mode-line was fabulous!
(defvar *battery-status-command*
  "acpi -b | awk -F '[ ,]' '{printf \"%s%s\", $3, $5}' | sed s/Discharging/\-/ | sed s/Unknown// | sed s/Full// | sed s/Charging/+/")

(defvar *vol-status-command*
  "amixer get Master | grep \"[[:digit:]]\\+%\" -o | tr -d \"\\n\"")

(setf *screen-mode-line-format*
      (list "%w [^B%n^b] ^>"
      '(:eval (run-shell-command *battery-status-command* t))
      " | Vol. "
      '(:eval (run-shell-command *vol-status-command* t))
      " | %d"))

;; urxvt
(define-key *root-map* (kbd "c") 
  "exec urxvt +sb -fn \"xft:Ubuntu Mono:pixelsize=15,style=regular\"")

;; Window information format
(setf *window-info-format* "%wx%h %n (%t - %c)")

;; Window format
(setf *window-format* "%m%n%s%10t")

;; Emacs
(defvar *emacs-command* nil
  "Start an emacs client frame. Starts an emacs daemon if necessary.")
(setf *emacs-command* "bash -c -i 'emacsclient -c -a \"\"'")

(define-key *root-map* (kbd "e") "run-emacs")

(defcommand run-emacs () ()
    (run-shell-command (concat "exec " *emacs-command*)))

;; iceweasel
(defcommand iceweasel-browser () ()
  "run iceweasel"
  (run-or-raise "iceweasel" '(:instance "iceweasel")))
(define-key *root-map* (kbd "b") "iceweasel-browser")

;; mate-terminal
(defcommand mate-terminal () ()
  (run-or-raise "mate-terminal --hide-menubar" '(:class "mate-terminal")))

(define-key *root-map* (kbd "C") "mate-terminal")

;; Clear rules
(clear-window-placement-rules)

(define-frame-preference "Default"
  ;; frame raise lock (lock AND raise == jumpto)
  (0 t   t :instance "emacs")
  (1 t   t :instance "iceweasel-browser"))

;; dvorak and தமிழ்
(stumpwm:run-shell-command "sh -c 'setxkbmap us,in dvorak,tam_unicode grp:ctrls_toggle'")

;; Start default applications
(run-emacs)
(iceweasel-browser)

On logging into StumpWM, GNU Emacs and Iceweasel browser are opened automatically. I also switch between English and Tamil keyboard layouts when required, and the two Control keys are used to toggle between them. A StumpWM screenshot is shown below:

You are encouraged to read the StumpWM manual to know more about its usage and configuration.

The objective of the workshop is to share the methodology and best practices on working with Free and Open Source Software projects, and also to guide the participants on career options. Although there is plenty of free documentation (including my own) available on the subject, some people prefer a formal, systematic approach to learning in a private coaching environment. Hence, this workshop is tailored for the same and is made a paid workshop, similar to personal tutoring.

The book has been in circulation for many years. So, I had to give two options - pay only for the workshop, or pay for both the workshop and the book (if they have not already bought the book). I have also kept additional copies of my book at the reserved-bit makerspace if people are interested. I had covered the following topics:

1. Careers with F/OSS
2. “i want 2 do project. tell me wat 2 do” best practices
3. Introduction to Git
4. Hands-on problem solving exercises
5. Real-world project example
6. Goal-driven development

The feedback has been positive and I am sharing the same below:

If you are interested in attending such a workshop, do write to me: author at shakthimaan dot com.

I attended Rootconf 2017 on May 11-12, 2017 at MLR Convention Centre, JP Nagar, Bengaluru.

The event had two parallel tracks, the Rootconf track at the main conference hall, and the DevConf.in track sponsored by Red Hat. A number of infrastructure hosting companies had sponsored the event. The selected talks had good diversity with topics ranging from monitoring, databases, networking, security etc.

Day I

The day’s proceedings began with the talk on the “State of open source monitoring landscape” by Bernd Erk. He gave a good overview of the available monitoring tools. He is involved in the Icinga2 project, which is re-written in C++ for performance. It is an alternative for the Nagios tool.

Aditya Patawari then provided an introduction to Kubernetes with a demo, as Spencer Krum could not make it to his talk on the “Anatomy of an alert”. After a short break, I attended “A little bot for big cause” session by Pooja Shah. The talk basically demonstrated how to create a bot by integrating various APIs.

I attended Manan Bharara’s talk on “Monitoring performance critical applications” where he had explained how they had used Clojure for monitoring at Otto in Germany. The presentation was very good, and the Oscillator Clojure code is available. He had provided good illustrations and code snippets in his presentation.

A number of stalls were available in the hallway, and I spent considerable time talking with the sponsors. At the Red Hat stall, I learnt some of the features and use cases for the Foreman provisioning tool. I was pictured with the Foreman helmet:

After lunch, I attended “SELinux: A deep dive” by Rejy M Cyriac. He gave an excellent introduction to SELinux, explaining the underlying concepts and rationale for the same. We also did basic hands-on exercises, but, I could not attend the last section as I had to move to the main conference hall for the flash talks. I sang the Nursery Rhymes version for DevOps. The video is available.

We then headed to the Rootconf party sponsored by Go-Jek!

Day II

The second day started with the talk by Kunal Grover on “Failure resilient architecture with microservice dependencies”. He presented the ways by which disruptions are performed in cloud infrastructure and how recovery mechanisms are tested. Ramya A then gave an interesting talk on “Asynchronous integration tests for microservices”. She had explained the pact.io family of frameworks to support Consumer Driven Contracts testing. Laxmi Nagarajan then gave a high level overview of “Capacity planning for AWS and configuring AWS autoscaling policies” sharing her Industry experience. After a short break, Colin Charles presented his story on “Capacity planning for your data stores” citing real world examples.

I then moved to the DevConf.in track to attend Ruchi Singh’s talk on “Migration of 300 microservices from AWS cloud to Snapdeal cloud”. The time was too short for such a talk, and it was an overview. I would have liked to see more details, given that they use Chef for provisioning and handle more than 16 TB of data as mentioned by Ruchi Singh.

After a quick lunch, I attended “Adventures in Postgres management” by Ramanan Balakrishnan. It was a good technical talk going over the failures and experiences learnt. After the talk I went to the hallway to listen to the Off-The-Record (OTR) session on mistakes to avoid when planning your infrastructure.

I returned to the main conference hall for doing some stretching exercises. In my opinion, all conferences should make this session mandatory, especially after you have been sitting for a long period of time. This session was then followed by the “Working with Secrets” talk by Shrey Agarwal, who gave an overview of using Hashicorp’s Vault for managing passwords. It was a very short introduction on the topic.

After a short beverage break, Toshaan Bharvani presented “Living with SELinux”, which was an excellent presentation on the subject. The initial slides had some material that Rejy M Cyriac had introduced the previous day, but, the content and presentation were good. With proper SELinux policies, he said that root in his production servers cannot do much. Rejy and Toshaan both asked people to use permissive mode instead of disabling SELinux altogether, so that you at least know what is being audited.

The last talk of the day was by Anurag Bhatia on “Understanding eyeball routing via RIPE Atlas”. He gave an overview of the RIPE Atlas project and how network metrics are measured and monitored wherever the RIPE device is installed. It is a TP-Link hardware whose firmware can be flashed using Libre software. Unfortunately, at this point the source of the firmware is not released as Free and Open Source Software. I was told that there is still an ongoing discussion on the same.

Conclusion

The talk by Manan Bharara and Rejy’s SELinux workshop were the highlights for day one for me. The content on the second day was much better and had greater depth in my opinion. Overall, it was a useful technical conference, and a good place to meet like-minded people.

I would like to thank Aerospike for sponsoring me to attend the conference.

We have covered a lot of ground in our quest to learn the intricacies of GNU Emacs and we trust that you have found your long journey fulfilling. This is the last article in the series.

Introduction

The GNU Emacs editor is written using Emacs Lisp and the C programming language. You can use GNU Emacs as an editor without knowing Emacs Lisp, but, being familiar with it will help you customize it or extend it to your needs. Emacs Lisp is a dialect of the Lisp programming language and is inspired by Maclisp and Common Lisp. The source files end with the filename extension .el and the byte-compiled files end with the .elc filename extension. You can also write scripts using Emacs Lisp and execute them as a batch operation. The code can thus be executed from the command line or from an executable file. Byte-compiling the source files can help you speed up the execution.

Comments

All comments begin with a semi-colon. An Emacs Lisp file usually has the following sections in the code, specified with three semi-colons. These always start in the left margin as shown below:

;;; Module --- Summary

;;; Commentary:

;;; Code:

;;; Module ends here

All comments outside functions begin with two semi-colons. The contents of the scratch buffer, shown below, are an example:

;; This buffer is for notes you don't want to save, and for Lisp evaluation.
;; If you want to create a file, visit that file with C-x C-f,
;; then enter the text in that file's own buffer.

Comments inside functions use a single semi-colon, and if they span multiple lines, they must be neatly aligned. For example:

...
(let* ((sexp (read (current-buffer)))    ; using `read' here
                                         ; easier than regexp
                                         ; matching, esp. with
                                         ; different forms of
                                         ; MONTH
 ...
 ))

Literals

The basic data types are available in Emacs Lisp. Numbers can be represented by integers or floats. Integers can have their sign representation before the digit (+1, -2). Floating point numbers can be represented using a decimal point (3.1415) or with an exponent (314.15e-2). A character (S) is represented by its integer code (83), while a string is a list of characters enclosed within double quotes (“A string”).

A symbol is an object with a name. A keyword symbol is one that starts with a colon (:). A vector is an array and can contain different types ([1 “two” :three]). The Boolean values for true and false are ’t’ and ‘nil’ respectively. A cons cell is an object with two slots. The first slot is called the CAR (Contents of the Address part of the Register number) and the second slot is called the CDR (Contents of the Decrement part of the Register number). A list is a series of linked cons cells. For example, in the list ‘(A B)’, the CAR is A and the CDR is B.

Sexp

Emacs Lisp uses prefix notation which consists of an operation followed by operands (arguments). All programs are represented as symbolic expressions (sexp). For example, the ‘+’ operation is applied to its arguments ‘1’ and ‘2’ in the following sexp:

(+ 1 2)

If you copy the above code in an Emacs buffer, you can evaluate the same by placing the cursor at the end of the expression and using the C-x C-e shortcut. The output ‘3’ will be displayed in the minibuffer.

You can also have nested sexps, wherein the innermost expressions are evaluated first.

(+ 1 (* 2 3))

Similar to the ‘+’ operation, you can use pre-defined keywords or functions in Emacs Lisp programs. The format function is similar to your printf statement in the C programming language.

(format "Hello, World!")

You can store a value in a variable using the setq operation as shown below:

(setq IST "Indian Standard Time")

IST     ; No enclosing parenthesis!
        ; produces "Indian Standard Time"

You can find the data type, using the type-of function. A few examples are shown below:

(type-of 1)                ; produces integer

(type-of 3.1415)           ; produces float

(type-of "A string")       ; produces string

(type-of :foo)             ; produces symbol

(type-of t)                ; produces symbol

(type-of '(1 2))           ; produces cons

(type-of [1 "two" :three]) ; produces vector

In the Bash shell, you can escape evaluation by using the backslash. Similarly, you can prevent the Emacs Lisp interpreter from evaluating an expression using quotes. For example:

(+ 1 2)     ; produces 3

'(+ 1 2)    ; produces the list (+ 1 2)

progn

The progn statement is used to execute several sexps. For example:

(progn
  (setq title "Introduction to Emacs Lisp")
  (setq author "Robert J. Chassell")
  (format "%s by %s" title author))  ; produces "Introduction to Emacs Lisp by Robert J. Chassell"

Functions

You can define your own function using the defun built-in keyword. The ‘say’ function simply prints the string “Hello, World!”" in the following example:

(defun say ()
  (format "Hello, World!"))

In order to execute the function, invoke it inside parenthesis as shown below:

(say)    ; produces "Hello, World!"

Arguments can be passed to a function. The ‘square’ function is demonstrated below:

(defun square (x)
  (* x x))

(square 3)  ; produces 9

You can also store a list of names and retrieve them using the car and cdr functions as illustrated below:

(setq teams '("GL" "DD" "KKR" "MI" "SRH" "RPS" "RCB" "KXIP"))

(car teams)       ; produces "GL"

(cdr teams)       ; produces "DD" "KKR" "MI" "SRH" "RPS" "RCB" "KXIP"

(car (cdr teams)) ; produces "DD"

let

The let statement allows you to bind a value to a variable in a local context. GNU Emacs 24 has lexical binding support. An example is given below:

(defun hello (name)
  (let ((new-name (concat "Hello " name)))
    new-name
      ))

(hello "Mark")   ; produces "Hello Mark"

Conditions

Conditions can be checked using the if and cond statements. The if statement takes a condition, and if the condition evaluates to true, the sexp immediately following the condition is executed. Otherwise, the else-form is evaluated. The syntax is

(if condition then-form else-form)

An example:

(if t
    (format "True")
  (format "False"))  ; produces "True"

The when condition statement is a variant of the if statement written as a macro. Macros are an important feature of Lisp programming that allow you to extend the language. They are primarily code generators. If the ‘when’ condition is true, then it evaluates the then-forms. The syntax and an example are given below:

(when condition then-forms)

(when t
  (print "Abracadabra!"))  ; produces "Abracadabra!"

Looping

You can use the while statement to perform looping in Emacs Lisp. The body of the ‘while’ loop is executed as long as the condition is true. An example that prints the numbers 0 to 4 is shown below:

(progn
  (setq i 0)
  (while (< i 5)
    (print i)
    (setq i (+ i 1))))

The dolist and dotimes macros can also be used for looping. The other approach is to use recursion. A literal definition in Emacs Lisp for computing the Fibonacci series is given below:

(defun fib (n)
  (cond ((= n 0) 0)
        ((= n 1) 1)
        (t (+ (fib (- n 1))
              (fib (- n 2))))))

The cond control structure takes a series of clauses, each has a condition and a body-form. If any condition evaluates to true, the body-form is executed. Otherwise, it proceeds to the next clause.

An iterative version of the Fibonacci series is given below:

(defun fib (n)
  (fib-helper n 0 1))

(defun fib-helper (n a b)
  (if (= n 0)
      a
    (fib-helper (- n 1) b (+ a b))))

(fib 10) ; produces 55

The Emacs Lisp Cookbook is very handy https://www.emacswiki.org/emacs/ElispCookbook.

You are encouraged to read ‘An Introduction to Programming in Emacs Lisp’ at https://www.gnu.org/software/emacs/manual/eintr.html and the Emacs Lisp Reference Manual at https://www.gnu.org/software/emacs/manual/elisp.html.

Gaurav (“dexter”) Pant started the day’s proceedings with a quick demo of Inkscape. He also provided the participants with an Inkscape quick reference sheet. The advantage of creating Scalable Vector Graphics (SVG) is that it can be used on a web page as well as enlarged for printing on posters and banners, without any distortion.

I took this time to create some illustrations. The first image was a logo with the letter “S”, whose shape was modified to fit into a boundary. The stick-figure-push SVG is already available in Openclipart. I modified it for a stick-figure-pull image. Few icons were then created for use in web pages. Finally, I completed an SVG of “The Common Man”.

All the images are available in Creative Commons Attribution-Share Alike 4.0 International license at Wikimedia ( https://commons.wikimedia.org/w/index.php?title=Special:ListFiles/Shakthimaan ).

The hackathon was a good initiative to encourage creative contributions to F/OSS. I was happy to see a number of excellent designers who were creating beautiful SVGs. Participants used both GIMP and Inkscape for their work. It was also good to have met the PLUG members.

PC: Siddharth Subramaniam

Such events help grow the activities in a region, and also showcases the diversity in a F/OSS group.

Keeping notes, maintaining to-do lists, planning projects and author documents with a fast and effective plain-text system - that’s what Emacs Org mode helps you do. There’s even a mobile version available as FOSS so that your favourite note-taking tool remains with you always.

In this next article in the GNU Emacs series, let’s learn about Org mode.

Introduction

Org-mode is an outline, major mode in GNU Emacs that can be used for taking notes, planning and documentation. It was written by Carsten Dominik in 2003. The Org file is written in plain text and markdown, and it can be exported into multiple output formats (TeX, HTML, PDF, etc.). Org-mode ships with GNU Emacs and this tutorial uses GNU Emacs 24.4.1 (M-x version) and Org mode version 8.2.10 (M-x org-version).

Structure

An Org file ends with the .org filename extension. The top-level headlines are marked using an asterisk (‘*’). For example, a college student may have the following high-level topics:

* College
* Home
* Recreation

You can create topics in the sub-section with more asterisks. For example:

* College
** Assignments
** Lab
** Exams
* Home
* Recreation

If you wish to hide the multiple asterisks at the sub-section level, you can add the following to your ~/.emacs:

(setq org-hide-leading-stars t)

The resultant Org file will look like what follows:

* College
 * Assignments
 * Lab
 * Exams
* Home
* Recreation

You can add unnumbered (use minus or plus symbols) or numbered lists as shown below:

* College
 * Assignments
 * Lab
   1. Compiler Design
   2. Programming Languages
 * Exams
   - Mathematics
   - Operating Systems
   - Databases
   - Compilers
* Home
* Recreation

You can cycle through the various levels in the Org file using the Tab and Shift-Tab keys.

A checkbox with a ‘/’ or a ‘%’ symbol can be used next to a topic name to indicate the completion status of the task. The lists under a topic can each have a checkbox too. Using C-c C-c will mark a checkbox for completion (‘X’) and will also update the statistics in the top-level checkbox. An example is shown below:

* College...
* Home [2/2]
  - [X] Read book
  - [X] Take print-out
* Recreation [50%]
  - [X] Read newspaper
  - [ ] Meditate

If a task is completed, you can move it out of the Org file using C-c C-x C-a. This will create an archive file with the completed items.

Planner

An Org mode file can be used as a planner. Each task can be marked with any of the following states - {TODO, DONE} using C-c C-t key combination. For example:

* TODO College...
* DONE Home [2/2]...
* Recreation [50%]...

You can also customize the TODO states depending on your workflow by setting org-todo-keywords in your Emacs startup file. For example:

(setq org-todo-keywords
  '((sequence "TODO(t)" "NEXT(n)" "STARTED(s)" "WAITING(w)" "|" "DONE(d)" "CANCELED(c)")))

The tasks can be scheduled using C-c C-s. A date is prompted for using the Calendar and is placed below the list entry. For example:

* TODO College
 * Assignments
 * Lab
   SCHEDULED: <2016-04-06 Wed>
   1. Compiler Design
   2. Programming Languages
 * Exams...

You can also add the time interval during which you intend to complete the task. The above example with a scheduled time is shown below:

* TODO College
 * Assignments
 * Lab
   SCHEDULED: <2016-04-06 Wed 14:00-16:00>
   1. Compiler Design
   2. Programming Languages
 * Exams...

A deadline can be added to a task using C-c C-d shortcut. An example is given below:

* TODO College
 * Assignments
 * Lab...
 * Exams
   DEADLINE: <2016-04-08 Fri>
   - Mathematics
   - Operating Systems
   - Databases
   - Compilers
* DONE Home [2/2]...
* Recreation [50%]...

You can have multiple Org files stored in your system, and you can instruct GNU Emacs where to find them by setting org-agenda-files in your Emacs start-up file as shown below:

(setq org-agenda-files (quote ("/tmp")))

Additionally, if you want an agenda view to see the scheduled items and deadlines, add the following to your GNU Emacs startup init file:

(define-key global-map "\C-ca" 'org-agenda)

In the Org file, when you press C-c a, the following agenda will show up in a new buffer:

Week-agenda (W14):
Monday      4 April 2016 W14
  test:       In   4 d.:  Exams
Tuesday     5 April 2016
Wednesday   6 April 2016
  test:       14:00-16:00 Scheduled:  Lab
Thursday    7 April 2016
Friday      8 April 2016
  test:       Deadline:   Exams
Saturday    9 April 2016
Sunday     10 April 2016

Tables

Org-mode has a built-in table editor which neatly aligns the column data. For example:

* TODO College...
* DONE Home [2/2]...
* Recreation [50%]
  - [X] Read newspaper
  - [ ] Meditate
  | Day       | Time | Status |
  |-----------+------+--------|
  | Monday    | 1.25 | Done   |
  | Tuesday   | 1.50 | Done   |
  | Wednesday |      |        |
  | Thursday  |      |        |
  | Friday    |      |        |
  | Saturday  |      |        |
  | Sunday    |      |        |

You can also use spreadsheet formula on these tables to perform calculations. For example:

* TODO College...
* DONE Home [2/2]...
* Recreation [50%]
  - [X] Read newspaper
  - [ ] Meditate
  | Day       | Time | Status |
  |-----------+------+--------|
  | Monday    | 1.25 | Done   |
  | Tuesday   | 1.50 | Done   |
  | Wednesday |      |        |
  | Thursday  |      |        |
  | Friday    |      |        |
  | Saturday  |      |        |
  | Sunday    |      |        |
  |-----------+------+--------|
  | Total     | 2.75 |        |
  #+TBLFM: @9$2=vsum(@2$2..@8$2)

Exporting

The Org file can be exported to multiple output formats (TeX, HTML, ASCII, PDF, etc.). Using C-c C-e will produce a buffer with the ‘Org Export Dispatcher’ menu to select an exporter. This is shown in the following figure:

You can also write your own backend customisations to suit your needs.

Literate programming

Donald Knuth coined the term ‘Literate Programming’ in 1984. To quote him:

"I believe that the time is ripe for significantly better documentation of programs, and that we can best achieve this by considering programs to be works of literature. Hence, my title: ‘Literate Programming’.

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do."

Org mode supports this style of programming using Babel. You need to activate the support for the programming languages in your GNU Emacs startup file. For example, the following code snippet helps to execute Bash shell scripts.

(org-babel-do-load-languages
 'org-babel-load-languages
 '((sh . t)
   ))

Consider a shell command to find the disk usage. You can create an Org file, and enclose the command in a Babel code block as shown below:

#+BEGIN_SRC sh
  df -h
#+END_SRC

When you press C-c C-c on this code block, you will be prompted with the string “Evaluate this sh code block on your system? (yes/no).” If you input “yes”, the output is produced in a Results section as shown below:

#+RESULTS:
| Filesystem | Size | Used | Avail | Use% | Mounted        | on |
| udev       | 1.9G | 0    | 1.9G  |   0% | /dev           |    |
| tmpfs      | 384M | 6.0M | 378M  |   2% | /run           |    |
| /dev/sda1  | 913G | 75G  | 792G  |   9% | /              |    |
| tmpfs      | 1.9G | 57M  | 1.9G  |   3% | /dev/shm       |    |
| tmpfs      | 5.0M | 4.0K | 5.0M  |   1% | /run/lock      |    |
| tmpfs      | 1.9G | 0    | 1.9G  |   0% | /sys/fs/cgroup |    |
| tmpfs      | 384M | 64K  | 384M  |   1% | /run/user/1000 |    |

You can learn more on Babel from their web page http://orgmode.org/worg/org-contrib/babel/.

A mobile version of Org mode is also available as Free and Open Source Software. You can use a third party service to sync your Org files between your mobile and system. Since the files are plain text, they can also be revision controlled using Git or any version control software.

Please refer to the Org reference manual at http://orgmode.org/#docs for more tips, customisation options and documentation.

Day I

I reached the venue early in the morning, to be of any help to the volunteers. The projector had to be checked, and I used my laptop to test the same. After changing couple of cables and adapters, the clarity on the screen was good.

This event had a single track where everyone sat in one big hall. I welcome this change!

Honza Král started the conference with his keynote, titled “(My) OSS Life”. He shared his experiences and learning in the Free and Open Source Software (F/OSS) world. At present, he maintains the Python drivers for Elasticsearch.

The keynote was followed by Anand Chitipotu’s talk on “Writing Beautiful Code”. He illustrated code examples on how to write simple, elegant, readable Python code. The use of meaningful variable names, comments were emphasized a lot. It was a short list of collated points on basic mistakes that newbie programmers make, and how to effectively write beautiful code.

Florian Fuchs then spoke on “Hacking Mailing Lists - The Mailman 3 API Ecosystem”. He explained the new architecture and API with code examples. He has been hacking on the new Mailman 3 web UI and the Python API bindings.

After attending these talks, I made a visit to the three booths at the conference - Red Hat, Python Software Foundation and reserved-bit. I also signed copies of my book that people had brought.

After lunch, I attended the “i18n-ise Django Apps” talk by Sundeep Anand, where he showed the internationalization processes for a Django application. All the relevant file modifications and commands involved were demonstrated.

John ‘warthog9’ Hawley is a Perl guy and gave an interesting keynote on building your own hardware, and why you should do that. He explained the various challenges he had faced, the process involved in the same. He had exclusively designed and produced a “Battle Bunny” embedded micro-Python kit for the conference and development sprints.

The “Building Trust in Releases” talk by Nigel Babu was very informative. He explained four important aspects in release management - cadence, documentation, testing, and empathy. This was also an experience report on DevOps practices, and was quite detailed and useful.

The last keynote of the day was by a Physics teacher, Praveen Patil. He shared his exploration on using Python to teach Physics to high school students. He is actively involved in ExpEYES Project, teacher training programs and also contributes content to National Repository of Open Educational Resources (NROER).

Day II

The morning keynote was by Katie Cunningham. She was initially testing the microphone and laptop by singing nursery rhymes. While the organizers decided to wait for people to arrive and settle down, there was time for lightning talks. So, I volunteered to start the day with the resounding “Nursery Rhymes”. After a couple of other lightning talks, Katie started her keynote on accessibility guidelines. It was quite an informative session.

“You can help develop Python - and you should!” talk by Stephen Turnbull was on the history of Python, PEP guidelines and the functioning of the community. I also had a chance to talk with him personally on the story of XEmacs.

Farhaan Bukhsh and Vivek Anand presented their Google Summer of Code (GSoC) work on the project Pagure, which is an alternative to GitHub and GitLab. They shared the past, present and future roadmap for the project. In the “Testing native binaries using CFFI” talk, Noufal Ibrahim demonstrated how to write Python bindings using CFFI.

After lunch, there was time for lightning talks. Different Python user group communities (PyDelhi, HydPy, PythonPune, PyLadies Pune) pitched about their work . I had prepared the sequel to “The Yet Another Lightning Talk” and requested the audience to sing on my behalf. The feedback was positive, as usual. The latest addition to the nursery rhyme is as follows:

Twinkle, Twinkle, unit tests,
How I wonder, where you exist!
I will write unit tests,
Until the project is laid to rest.

The afternoon keynote was by Nick Coghlan. He also shared his know-how on Free and Open Source Software and community best practices. “Django on Steroids - Lessons from Scale” by Sanket Saurav was a good technical, intermediate-level talk on Django customization, nginx settings, scaling and deployment.

The last keynote of the day and the conference was by Terri Oda on “Is OSS more secure?”. She presented the various dimensions on which one needs to answer the question. She concluded by saying that there is definitely more scope in F/OSS to be more secure given the number of people involved, and the transparency in the process.

Conclusion

The number of participants at the conference was more than five hundred, and they all came on a week day to attend! For a first-time event, that is quite impressive. This clearly shows that there is a demand for such events across India.

PC: Kushal Das

Initially, there was a lot of resistance (follow the thread) to this event, including the name of the event. Communities are meant for innovation, and stifling is futile. You can learn a lot in a community, and there are guidelines and best practices that are followed.

It was a four day event with development sprints, and hence it had to be a PyCon. Legally, the Python Software Foundation allows using the name “PyCon” for regional events too. Given the context and the necessity, I am happy that the Python Software Foundation (PSF) got the message right, and understood the need for the conference and supported it in a big way!

The development sprints had a limited seating capacity, and the registration got over early. I was informed that there were hundred more requests for the development sprints, which again emphasizes the need for such events! I also had a chance to meet some of the #dgplug (irc.freenode.net) folks with whom I have been interacting online on IRC.

It does take a lot of effort to organize a conference, and I congratulate the PyCon Pune team on their first event.

This article in the GNU Emacs series takes readers on how to use HTML mode, do indentation, and use the Magit interface.

HTML mode

You can use HTML mode to effectively edit HTML and CSS files using GNU Emacs. To start the mode, use M-x html-mode. You will see the string ‘HTML’ in the mode line.

Default template

A default HTML template can be started by opening a test.html file, and using C-c C-t html. It will produce the following content:

<html>
  <head>
<title>

You will then be prompted with the string ‘Title:’ to input the title of the HTML page. After you type ‘Hello World’, the default template is written to the buffer, as follows:

<html>
  <head>
<title>Hello World</title>
</head>
<body>
<h1>Hello World</h1>

<address>
<a href="mailto:user@hostname">shakthi</a>
</address>
</body>
</html>

Tags

You can enter HTML tags using C-c C-t. GNU Emacs will prompt you with the available list of tags. A screenshot of the available tags is shown in Figure 1:

The anchor tag can be inserted using ‘a’. You will then receive a message prompt: ‘Attribute:’. You can provide the value as ‘href’. It will then prompt you for a value, and you can enter a URL, say, ‘http://www.shakthimaan.com’. The anchor tag will be constructed in the buffer as you input values in the mini-buffer. You will be prompted for more attributes. If you want to finish, simply hit the Enter key, and the anchor tag will be completed. The final output is shown below:

<a href="http://www.shakthimaan.com"></a>

You can insert a h2 tag by specifying the same after C-c C-t. You can also add any attributes, as required. Otherwise, simply hitting the Enter key will complete the tag. The rendered text is as follows:

<h2></h2>

You can insert images using the alt tag. You can specify the src attribute and a value for the same. It is also a good practice to specify the alt attribute for the image tag. An example is shown below:

<img alt="image" src="http://shakthimaan.com/images/ShakthiK-workshop-A4-poster.png">

Unordered lists can be created using C-c C-t followed by ‘ul’. It will then prompt you for any attributes that you want included in the tag. You can hit the Enter key, which will prompt you with the string ‘List item:’ to key in list values. An example of the output is shown below:

<ul>
  <li>One
    <li>Two
      <li>Three
</ul>

You can neatly align the code by highlighting the above text and indenting the region using C-M-\. The resultant output is shown below:

<ul>
  <li>One
  <li>Two
  <li>Three
</ul>

If you wish to comment out text, you can select the region and type M-q. The text is enclosed using “<!--” and “-->”. For example, the commented address tags in the above example look like what follows:

<!-- <address> -->
<!-- <a href="mailto:shakthi@achilles">shakthi</a> -->
<!-- </address> -->

A number of major modes exist for different programming environments. You are encouraged to try them out and customize them to your needs.

Accents

In HTML mode, you can insert special characters, accents, symbols and punctuation marks. These characters are mapped to Emacs shortcuts. Some of them are listed in the following table:

Indentation

Consider the following paragraph:

“When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things.”

You can neatly fit the above text into 80 columns and 25 rows inside GNU Emacs using M-q. The result is shown below:

When we speak of free software, we are referring to freedom, not
price.  Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.

You can also neatly indent regions using the C-M-\ shortcut. For example, look at the following HTML snippet:

<table>
<tr>
<td>Tamil Nadu</td>
<td>Chennai</td> 
</tr>
<tr>
<td>Karnataka</td>
<td>Bengaluru</td> 
</tr>
<tr>
<td>Punjab</td>
<td>Chandigarh</td> 
</tr>
</table>

After indenting the region with C-M-\, the resultant output is shown below:

<table>
  <tr>
    <td>Tamil Nadu</td>
    <td>Chennai</td> 
  </tr>
  <tr>
    <td>Karnataka</td>
    <td>Bengaluru</td> 
  </tr>
  <tr>
    <td>Punjab</td>
    <td>Chandigarh</td> 
  </tr>
</table>

If you have a long line which you would like to split, you can use the C-M-o shortcut. Consider the quote:

“When you’re running a startup, your competitors decide how hard you work.” ~ Paul Graham

If you keep the cursor after the comma, and use C-M-o, the result is shown below:

"When you're running a startup, 
                                your competitors decide how hard you work." ~ Paul Graham

Magit

Magit is a fantastic interface to Git inside GNU Emacs. There are many ways in which you can install Magit. To install from the Melpa repository, add the following to your ~/.emacs:

(require 'package)
(add-to-list 'package-archives
             '("melpa" . "http://melpa.org/packages/") t)

When you do M-x list-packages, you will see ‘magit’ in the list. You can press ‘i’ to mark Magit for installation, followed by ‘x’ to actually install it. This will install Magit in ~/.emacs.d/elpa. The version installed on my system is magit-20160303.502.

When you open any file inside GNU Emacs that is version controlled using Git, you can start the Magit interface using M-x magit-status. I have bound this key to C-x g shortcut in ~/.emacs using the following:

(global-set-key (kbd "C-x g") 'magit-status)

The default magit screenshot for the GNU Emacs project README file is shown in Figure 2.

Pressing ‘l’ followed by ‘l’ will produce the history log in the magit buffer. A screenshot is provided in Figure 3.

You can make changes to the project sources and stage them to the index using the ’s’ shortcut. You can unstage the changes using the ‘u’ shortcut. After making changes to a file, you need to use M-x magit-status to update the Magit buffer status.

A sample screenshot of the modified files and staged changes is shown in Figure 4.

You can hit TAB and Shift-TAB to cycle through the different sections in the Magit buffer. To commit a message, press ‘c’ followed by ‘c’. It will pop up a buffer where you can enter the commit message.

You can create and checkout branches using the ‘b’ shortcut. A screenshot of the magit branch pop-up menu is shown in Figure 5.

All the basic Git commands are supported in Magit - diffing, tagging, resetting, stashing, push-pull, merging and rebasing. You are encourged to read the Magit manual ( https://magit.vc/ ) to learn more.

After reviewing many models and configurations, I purchased the Lenovo E460 model that comes with FreeDOS, and swapped the default HDD for SSD (< 500 GB).

Specification

• Intel(R) Core(TM) i5-6200 CPU @ 2.30 GHz (4 processors).
• 14" display
• 437 GB SSD disk
• 8 GB RAM
• Intel Corporation HD Graphics 520
• Intel Dual Band Wireless-AC 3165 Plus Bluetooth
• Intel I219-V Ethernet Controller
• 3 USB ports
• 1 HDMI port
• 4-in-1 Card Reader
• FreeDOS
• 1.81 kg

Parabola GNU/Linux Libre

I tried Trisquel GNU/Linux first on on this laptop. It is a derivative of Ubuntu without non-free software. I experimented with Qubes OS, but, its dom0 has proprietary blobs. GNU Guix is an interesting project, but, it does not have all the packages that I need (yet). I liked rolling distributions, and hence decided to try Parabola GNU/Linux Libre, a derivative of Arch, without the binary blobs.

There is no CD/DVD drive on this laptop, but, you can boot from USB. I first checked if all the software that I need are available in the Parabola GNU/Linux Libre repository, and then proceeded to install the same. I always encrypt the disk during installation. I have the Mate desktop environment with XMonad setup as a tiling window manager.

Audio works out of the box. I do not use the web cam. I had to use the package scripts to install Grisbi as it was not available in the base repository. Virtualization support exists on this hardware, and hence I use Virtual Machine Manager, QEMU and libvirt.

Command Output

All the hardware worked out of the box, except for the wireless which requires a binary blob. So, I purchased a ThinkPenguin Wireless N USB Adapter for GNU/Linux which uses the free ath9k Atheros wireless driver.

As mandatory, I am providing some command outputs.

$ lspci

00:00.0 Host bridge: Intel Corporation Skylake Host Bridge/DRAM Registers (rev 08)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 520 (rev 07)
00:14.0 USB controller: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller (rev 21)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-LP Thermal subsystem (rev 21)
00:16.0 Communication controller: Intel Corporation Sunrise Point-LP CSME HECI #1 (rev 21)
00:17.0 SATA controller: Intel Corporation Sunrise Point-LP SATA Controller [AHCI mode] (rev 21)
00:1c.0 PCI bridge: Intel Corporation Device 9d12 (rev f1)
00:1c.5 PCI bridge: Intel Corporation Sunrise Point-LP PCI Express Root Port #6 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-LP LPC Controller (rev 21)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-LP PMC (rev 21)
00:1f.3 Audio device: Intel Corporation Sunrise Point-LP HD Audio (rev 21)
00:1f.4 SMBus: Intel Corporation Sunrise Point-LP SMBus (rev 21)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection I219-V (rev 21)
01:00.0 Network controller: Intel Corporation Intel Dual Band Wireless-AC 3165 Plus Bluetooth (rev 99)
02:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS522A PCI Express Card Reader (rev 01)

$ uname -a

Linux aether 4.8.17-gnu-1 #1 SMP PREEMPT Wed Jan 18 05:04:13 UYT 2017 x86_64 GNU/Linux

$ df -h

Filesystem             Size  Used Avail Use% Mounted on
dev                    3.7G     0  3.7G   0% /dev
run                    3.7G  920K  3.7G   1% /run
/dev/mapper/cryptroot  437G   95G  321G  23% /
tmpfs                  3.7G   26M  3.7G   1% /dev/shm
tmpfs                  3.7G     0  3.7G   0% /sys/fs/cgroup
tmpfs                  3.7G  196K  3.7G   1% /tmp
/dev/sda1              976M   45M  865M   5% /boot
tmpfs                  745M   28K  745M   1% /run/user/1000

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.3G        2.4G        3.2G         83M        1.7G        4.6G
Swap:          2.1G          0B        2.1G

$ lscpu

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 78
Model name:            Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
Stepping:              3
CPU MHz:               499.951
CPU max MHz:           2800.0000
CPU min MHz:           400.0000
BogoMIPS:              4801.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              3072K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

Conclusion

I have been using the laptop for more than three months, and it has really been a smooth experience. It costed less than ₹ 55,000. The battery life is decent. I printed couple of Free Software stickers to identify my laptop. The “Inside GNU/Linux” sticker covers the web cam, and the “Free Software Foundation” sticker is pasted behind the screen. The folks at #parabola IRC channel on irc.freenode.net are quite helpful. The Parabola GNU/Linux Libre Wiki has excellent documentation for your reference.

In this next article in the GNU Emacs series, you will learn how to use its calendar, center text, macros and drawing tools.

Calendar

You can use and display a calendar inside GNU Emacs using the following command (also see figure below):

M-x calendar

Day

You can move forward by a day using the C-f shortcut, and move back a day using the C-b keys. You can move to the current date using the ’.’ key.

To start the week on a Monday, set the following in your ~/.emacs.

(setq calendar-week-start-day 1)

Week

If you wish to move forward by a week, you can use the C-n shortcut, and to move back by a week, use the C-p shortcut. The C-a shortcut can be used to move to the beginning of the week, while the C-e shortcut can be used to move to the end of the week.

Month

You can move to the beginning of a month using the M-a shortcut. To move to the end of the month, use M-e.

You can move forward and backward a month using the M-} and M-{ shortcuts, respectively.

If you wish to scroll forward three months, use the C-v shortcut. To scroll backward three months, use the M-v shortcut.

Year

In order to move forward a year, you can use the C-x ] shortcut, and to move back a year, you can use the C-x [ shortcut.

You can go to a specified date using the g d key combination. It will then prompt you with the messages “Year (>0):”, “Month name:” and “Day (1-31):”, and will take you to the specified date.

You can move to the beginning of the year using the M-< shortcut, and to the end of the year using the M-v shortcut.

You are encouraged to read the ‘Calendar’ section in the GNU Emacs manual at https://www.gnu.org/software/emacs/manual/html_node/emacs/Calendar_002fDiary.html#Calendar_002fDiary, to learn more.

Centering

Consider the following poem that I wrote in 2015:

Poem

Project issues in the way, 
In the way, in the way. 
Project issues in the way, 
My fair user. 

Fixing bugs right way, 
Right away, right away. 
Fixing bugs right way, 
My fair user. 

Merging pull requests as I say, 
As I say, as I say. 
Merging pull requests as I say, 
My fair user. 

All the tests are passing, hey! 
Passing, hey! Passing, hey! 
All the tests are passing, hey! 
My fair user. 

As a client, you should pay, 
You should pay, you should pay. 
As a client, you should pay, 
My fair user. 

Python really saved the day, 
Saved the day, saved the day. 
Python really saved the day, 
My fair user.

You can center the title “Poem” by placing the cursor on it, and typing “M-x set-justification-center”.

Marking and highlighting the poem, and using M-x center-region will center the poem. The output is shown below:

         Poem

Project issues in the way, 
 In the way, in the way. 
Project issues in the way, 
      My fair user. 

 Fixing bugs right way, 
Right away, right away. 
 Fixing bugs right way, 
     My fair user. 

Merging pull requests as I say, 
      As I say, as I say. 
Merging pull requests as I say, 
         My fair user. 

All the tests are passing, hey! 
  Passing, hey! Passing, hey! 
All the tests are passing, hey! 
         My fair user. 

  As a client, you should pay, 
You should pay, you should pay. 
  As a client, you should pay, 
         My fair user. 

 Python really saved the day, 
Saved the day, saved the day. 
 Python really saved the day, 
        My fair user. 

Macros

Macros are recorded key strokes that can be stored and replayed. You can start defining a keyboard macro using C-x ( command or the F3 key. You can then type a series of keys that constitute the macro. To finish defining the macro, you can use C-x ) or the F4 key. In order to execute the previous defined macro, you can use C-x e shortcut or F4.

Consider the following text in a buffer that contains a serial number, date and an examination subject list:

1,2015-03-02,English
2,2015-03-03,Physics
3,2015-03-05,Mathematics
4,2015-03-08,Biology
5,2015-03-10,Chemistry

Suppose you wish to add a space after each comma, you can define the following macro (exclude the semi-colon followed by the text) for the first line using the following key strokes:

F3    ; Start macro definition
C-s   ; Search for
,     ;   comma
Enter ;
Space ;
C-s   ; Search again for
,     ;   comma
Enter ;
Space ;
C-n   ; Move to next line
C-a   ; Move to beginning of line
F4    ; End macro definition

Using C-x e or F4 repeatedly will turn the above input CSV text into the following:

1, 2015-03-02, English
2, 2015-03-03, Physics
3, 2015-03-05, Mathematics
4, 2015-03-08, Biology
5, 2015-03-10, Chemistry

You can give a name (say, ‘comma’) to the previously defined macro using C-x C-k n. You can then execute the macro using M-x comma. You can also insert the named macro into a file using M-x insert-kbd-macro command. You can bind a macro to a key using C-x C-k b shortcut.

If you wish to apply the macro to each line in a region, you can use C-x C-k r keys. In order to cycle between the previous and next macros in the macro ring, you can use C-x C-k C-p and C-x C-k C-n shortcuts respectively. You can also delete a macro using C-x C-k C-d key combination.

Picture mode

You can draw diagrams inside Emacs using Picture mode. To start Picture mode, use M-x picture-mode command, and to exit use the C-c C-c shortcut.

The cursor movement keys in a buffer are also applicable in picture mode. To move the cursor right, you can use the C-f keys, and to move left by one character, you can use the C-b shortcut. To move the cursor up and down by one character, use the C-p and C-n shortcuts, respectively. The C-d shortcut is used to delete a character.

Before you move the cursor to draw in the buffer, you need to set the drawing direction. The following table summarizes the shortcut keys, and their associated drawing direction.

If you want to move the cursor forward in the drawing direction, you can use the C-c C-f shortcut. To move the cursor backward, use the C-c C-b key combination. If you want to delete a line, use the C-k command. You can insert a new line using the C-o shortcut. You can also draw a rectangle around a region using the C-c C-r shortcut. A drawing done using Picture mode is shown in Figure 2.

Artist mode

Artist mode can also be used to draw diagrams in GNU Emacs. You can enter this mode using M-x artist-mode, and exit the same using C-c C-c.

You can draw pictures using the keyboard alone or also use the mouse in Artist mode. In order to start and stop drawing, use the Enter key. This is equivalent to putting the pen down when drawing, and lifting it up when you want to perform a different action.

The buffer navigation commands to move right and left are the same as C-f and C-b shortcuts respectively. You can move up a column using the C-p shortcut, and move down a column using the C-n key.

You can draw geometric shapes using Artist mode. To select a shape or operation you can use C-c C-a C-o key combination. This will provide a list of shapes and actions you can perform. This list is shown in Figure 3:

The shortcuts listed in the following table are available for drawing specific shapes:

Figure 4 depicts an ellipse drawn using Artist mode:

Figure 5 is an example of polylines:

You can fill a shape using C-c C-a f key combination. The following Figure 6 shows a circular representation filled with dots.

You can also spray characters in the buffer using the C-c C-a S shortcut keys. An example is shown in Figure 7:

The character to be used for drawing can be changed using C-c C-a C-l shortcut. The character to fill shapes can be set using C-c C-a C-f key combination.

If you want to cut an area, you can draw a rectangle around it using C-c C-a C-k key combination. You can also copy the image area using the C-c C-a M-w keys, and paste the same using C-c C-a C-y or C-x r y shortcuts. To set the operation to erase text, you can use C-c C-a C-d key combination.

You can refer to the Emacs Wiki Artist mode for more documentation and help - http://www.emacswiki.org/emacs/ArtistMode.

After arriving early on the day of the workshop, I checked-in to my hotel accommodation. A view of the Kanteerva stadium from the hotel.

Pre-Conference Workshop

I had registered for the “Deep Dive into Erlang Ecosystem” workshop by Robert Virding, one of the creators of the Erlang programming language. He started the day’s proceedings with an introduction to Erlang basics and covered both sequential and concurrent programming. He also gave an overview of the Open Telecom Platform (OTP) and answered a number of questions from the participants. He, along with Joe Armstrong and Mike Williams, designed the Erlang programming language for telecommunication, keeping the system in mind and all the way from the ground-up.

He also mentioned how WhatsApp was able to handle two million concurrent connections on a single box, and they would peak at three million at times. As another Emacs and Lisp user, he wrote Lisp Flavoured Erlang (LFE). He did not have much time to talk about it during the workshop, but, he did share differences between Erlang, Elixir and other languages that are being built around the Erlang ecosystem.

Day I

The keynote of the day was from Robert Virding on “The Erlang Ecosystem”. He gave a good overview and history of the Erlang programming language, and the rationale for designing the same. He elaborated on the challenges they faced in the early days of computing, and the first principles that they had to adhere to. They did not intend the language to be functional, but, it turned out to be so, and greatly helped their use case. One of the beautiful expressions in Erlang to represent bit-level protocol formats in an expressive format is shown below:

<<?IP_VERSION:4, HLen:4, SrvcType:8, TotLen:16, 
      ID:16, Flgs:3, FragOff:13,
      TTL:8, Proto:8, HdrChkSum:16,
      SrcIP:32,
      DestIP:32, RestDgram/binary>>

Robert’s keynote was followed by another keynote by Brian McKenna on “No Silver Bullets in Functional Programming”. He gave the pros and cons of using Functional and other programming paradigms, and discussed the trade-offs. A number of code examples were shown to illustrate the concepts.

The next talk that I attended was by Aloïs Cochard on “Welcome to the Machines”. He gave an overview on the history of various Haskell libraries for data stream processing (pipes, conduit) and finally provided a tutorial on machines.

Abdulsattar Mohammed introduced the need for dependent types using Idris with simple examples in his “Dependently Typed Programming with Idris” talk. The concepts were well narrated with numerous code snippets.

The next talk by Debasish Ghosh on “An algebraic approach to functional domain modeling” was a modelling exercise on how to map business logic into functional algebra. He demonstrated a real world step-by-step process on the transformation from a problem domain to the solution domain consisting of algebraic data types, functions that operate on them, and business rules.

Ravi Mohan started his talk titled, “Equational Reasoning - From Code To Math and Back Again”, with his learning in the Functional Programming (FP) world, and an overview of how to go about reasoning from code to math. His laptop had ran out of battery power, and he did not have his laptop charger. Before his scheduled talk, he had re-created plain text notes of his slides and walked us through the content.

“Implementing Spark like system in Haskell” was an interesting session by Yogesh Sajanikar on his attempt to create a DSL for map-reduce jobs. He did cover much of the internals in his implementation and the challenges faced. The hspark code is available at https://github.com/yogeshsajanikar/hspark.

Day II

The second day began with the keynote by John Hughes on “Why Functional Programming Matters”. This was the best keynote of the conference, where John gave a very good historical perspective of FP and the experiences learnt in the process. His slide deck was excellent and covered all the necessary points that were part of his famous paper with the same title.

This was followed by a series of demos on cool features in Functional Programming languages - Erlang, Idris, APL, F# and Julia.

“Using F# in production: A retrospective” was a talk by Ankit Solanki on the lessons learned in using a functional language in implementing a tax e-filing application. They heavily use F# Type Providers to handle the variation in input CSV files.

“Real world functional programming in Ads serving” was a talk by Sathish Kumar from Flipkart on how they used functional programming in Java 8 for their product. They initially prototyped with Haskell, and used the constructs in Java.

I skipped the next talks, and spent time with Robert Virding in the Erlang booth.

Rethinking “State Management.” was presented by Tamizhvendan S. He narrated examples on state management for a cafe application using F#. He also gave a demo of Ionide text editor and its features.

Post-conference workshop

I attended John Hughes workshop on Property-based Testing. Initially, I thought he would be using Haskell QuickCheck, but, in the workshop he used the Erlang implementation. John mentioned that the Haskell and Erlang implementations are different, and their interests have diverged.

He started the workshop by taking an example of writing property tests for encoded SMS messages using Erlang. He also demonstrated on how a minimal test example is produced when a test fails. The choice of deciding on what properties to test is still an active research problem. He also demonstrated how to collect statistics from the test results to analyse and improve them.

The property-based testing has been used by his company, QuviQ, to test C protocols for the automobile industry. They were able to generate tests to detect bugs in the CAN bus implementation. Here is a summary of the statistics for a project:

3,000 pages of specification
20,000 lines of QuickCheck
1,000,000 LoC, 6 suppliers
200 problems
100 problems in the standard

He also shared his experience in generating tests for Klarna - an invoicing service web shop that uses Mnesia - the distributed Erlang database. He concluded by saying that we should not write tests, but, they shoud be generated.

Overall, the workshops were quite useful. It was good to have met both Robert Virding and John Hughes.

In this next article in the GNU Emacs series, we shall learn how to use GNU Emacs as a news reader.

Elfeed

Elfeed is an Emacs web feed reader that is extensible and supports both Atom and RSS feeds. It has written by Christopher Wellons.

Installation

We shall use Milkypostman’s Experimental Lisp Package Archive (MELPA) to install Elfeed. Create an initial GNU Emacs start-up file that contains the following:

(require 'package) ;; You might already have this line
(add-to-list 'package-archives
             '("melpa" . "https://melpa.org/packages/"))

(when (< emacs-major-version 24)
  ;; For important compatibility libraries like cl-lib
  (add-to-list 'package-archives '("gnu" . "http://elpa.gnu.org/packages/")))
(package-initialize) ;; You might already have this line

The above code snippet has been taken from the MELPA project documentation website http://melpa.org/#/getting-started, and has been tested on GNU Emacs 24.5.2.

You can now start GNU Emacs using the following command:

$ emacs -Q -l ~/elfeed-start.el

You can obtain the list of available packages using M-x list-packages, which will search the melpa.org and elpa.gnu.org repositories. You can search for ‘elfeed’ in this buffer, and select the same for installation by pressing the ‘i’ key. To actually install the package, press the ‘x’ (execute) key, and Elfeed will be installed in ~/.emacs.d/elpa directory.

Configuration

You can create a shortcut to start Elfeed using the following code snippet in your ~/elfeed-start.el file.

(global-set-key (kbd "C-x w") 'elfeed)

The list of feeds can be defined as shown below:

(setq elfeed-feeds
      '(("http://www.shakthimaan.com/news.xml" people)
        ("http://arduino.cc/blog/feed/" projects)
        ("http://planet-india.randomink.org/rss10.xml" people planet)
        ))

Tags can be added at the end of the feed. The above feeds include ‘people’, ‘projects’ and ‘planet’ tags.

Usage

You can use the C-x w shortcut to start Elfeed. If you press ‘G’, it will fetch the latest news feeds from the servers, starting with the message ‘3 feeds pending, 0 in process …’. A screenshot of Elfeed in GNU Emacs is shown below:

The RSS entries are stored in ~/.elfeed directory on your system.

You can read a blog entry by pressing the ‘Enter’ key. If you would like to open an entry in a browser, you can use the ‘b’ key. In order to copy the selected URL entry, you can use the ‘y’ key. To mark an entry as read, you can use the ‘r’ key, and to unmark an entry, press the ‘u’ key. You can add and remove tags for an entry using the ‘+’ and ’-’ keys, respectively.

You can also filter the feeds based on search critera. Pressing ’s’ will allow you to update the filter that you want to use. There are many filter options available. You can use ‘+’ to indicate that a tag must be present, and ’-’ to indicate that the tag must be absent. For example, “+projects -people”.

The filter text starting with ‘@’ represents a relative time. It can contain plain English text combined with dashes – for example, ‘@1-month-ago +unread’. The ’!’ notation can be used to negate a filter. To limit the number of entries to be displayed, you can use the ‘#’ pattern. For example, ‘+unread #5’ will list five unread blog articles. A screenshot of Elfeed with a filter applied is shown in the following figure:

You can also use regular expressions as part of your filter text. The default search filter can be changed by modifying the value of elfeed-search-filter. For example:

(setq-default elfeed-search-filter "@1-month-ago +unread")

The search format date can be customized as shown below:

(defun elfeed-search-format-date (date)
  (format-time-string "%Y-%m-%d %H:%M" (seconds-to-time date)))

Elfeed also has an export option to view the feeds in a browser. If you install the elfeed-web package from the packages list, you can then start it using M-x elfeed-web-start. You can then start a browser, and open http://localhost:8080/elfeed/ to view the feeds. A screenshot is shown below:

The entire contents of the elfeed-start.el configuration file are shown below:

(require 'package) ;; You might already have this line
(add-to-list 'package-archives
             '("melpa" . "https://melpa.org/packages/"))

(when (< emacs-major-version 24)
  ;; For important compatibility libraries like cl-lib
  (add-to-list 'package-archives '("gnu" . "http://elpa.gnu.org/packages/")))
(package-initialize) ;; You might already have this line

(global-set-key (kbd "C-x w") 'elfeed)

(defun elfeed-search-format-date (date)
  (format-time-string "%Y-%m-%d %H:%M" (seconds-to-time date)))

(setq elfeed-feeds
      '(("http://www.shakthimaan.com/news.xml" people)
        ("http://arduino.cc/blog/feed/" projects)
        ("http://planet-india.randomink.org/rss10.xml" people planet)
        ))

Gnus

Configuration

Gnus is an Emacs package for reading e-mail and Usenet news. The nnrss backend supports reading RSS feeds. Gnus is available by default in GNU Emacs. After launching emacs using emacs -Q in the terminal, you can start Gnus using M-x gnus. To add a new RSS entry, you can use ‘G R’. It will prompt you with the message ‘URL to Search for RSS:’. You can then provide the feed, for example, http://www.shakthimaan.com/news.xml. It will try to connect to the server and will provide you the message ‘Contacting host: www.shakthimaan.com:80’. After a successful connect, it will prompt for the title, ‘Title: Shakthimaan’s blog.’ You can simply hit Enter. You will then be prompted for a description, ‘Description: RSS feed for Shakthimaan’s blog.’ You can hit Enter to proceed. Now, the blog entry has been added to Gnus. In this fashion, you can add the other blog entries too. A screenshot of the main Gnus group buffer is shown below:

Usage

You can press ‘g’ to refresh the buffer and ask Gnus to check for latest blog entries. Using the ‘Enter’ key will open the feed, and the list of blogs for a feed. A screenshot is shown in Figure 5.

You can press ‘Enter’ on a blog entry, and it will open the contents in a new buffer. It will then be marked as read, indicated by ‘R’. A screenshot of a blog entry rendering text and image is shown in the following figure:

You can press ‘q’ to quit from any level inside Gnus. You are encouraged to read the Gnus tutorial ( http://www.emacswiki.org/emacs/GnusTutorial ) and manual ( http://www.gnus.org/manual/big-gnus.html ) to learn more, and to customize it for your needs.

In this next article in the GNU Emacs series, we shall learn how to execute shell commands, run shells, use spell-checkers and abbreviations, and print from GNU Emacs.

Shell mode commands

You can run a shell command from GNU Emacs using the M-! shortcut. For example, typing ‘M-!’ will prompt you with the message ‘Shell command:’ in the minibuffer. If you then type ‘date’, it will produce the output ‘Tue Dec 8 21:19:24 IST 2015’ in the minibuffer. But, if you want the output to be inserted in the current buffer you can use the C-u M-! command sequence.

Consider the following poem that I wrote:

"Water, water everywhere
Little power and aid to spare
Really tough for us to bear
We will face it with Chennaites' flair.

Lakes and rivers overflowing everywhere
No road, rail or plane to go anywhere
People really are in big despair
But we will ride it with Chennaites' flair.

More storms are forecast to be aware
Nothing like this in 100 years to compare
Stay indoors and please do take care
And we will take it with Chennaites' flair."

Suppose I want to know the number of words used, I can mark the poem as a region in the GNU Emacs buffer, and execute a shell command for the region using M-| shortcut. It then prompts with the string ‘Shell command on region’, and when I type in ‘wc -w’ it returns ‘90’.

Shells

There are three shells available in GNU Emacs - shell, ansi-term and eshell. Using M-x shell will invoke a shell for you. This starts a new buffer with the name ‘shell’ in it. You can use all the GNU Emacs buffer commands in this window. For example, C-p will move the cursor up, and C-n will move the cursor down. In order to interrupt a current job in the shell buffer, you can use C-c C-c. If you want to suspend the current job, you can use C-c C-z. You can also clear the output from the previous command using C-c C-o. For example, the output of ‘ifconfig’ command is shown below:

/tmp $ ifconfig docker0
docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
/tmp $

After executing ‘C-c C-o’, the previous command output gets cleared as shown below:

/tmp $ ifconfig docker0
 *** output flushed ***
/tmp $

You can move the cursor up to the previously executed commands using C-c C-p and move down using C-c C-n. You can also cycle backwards and forwards through the command history using the M-p and M-n shortcuts respectively. The C-c C-r shortcut moves to the first line of the output to the top of the window. You can move the cursor to the bottom of the window using C-c C-e. The C-r key binding allows you to search for a previously typed command from history.

The default shell is Bash, but, you can change it to use other shells like ksh, csh or zsh. You can open another shell buffer using C-u M-x shell. The new shell buffer will be called ’shell<2>". You can, of course, rename this buffer. You can thus open multiple shells to work inside GNU Emacs.

GNU Emacs also has a terminal emulator that you can invoke using M-x ansi-term. You can start a Bash session with this command to get the actual colours that you see in a terminal session. This is like a fallback shell if you do not want to use ‘M-x shell’. Eshell is the Emacs built-in shell written completely in Emacs Lisp (Elisp). You can start it using M-x eshell. The advantage of using this is that you can extend it, write your own customized Elisp functions and use it similar to shell scripting.

Screenshots of shell, ansi-term and eshells in GNU Emacs are shown in Figure 1, 2 and 3 respectively:

Spell check commands

You can check the spelling of a word by placing the cursor on it and using M-$ shortcut. For example, for the word ‘hte’, GNU Emacs provides the following spelling options:

(0) hate (1) HT (2) ht (3) GTE (4) the (5) He (6) Te (7) he (8) Hts
(9) hie (:) hoe (;) hue (<) Rte (=) Ste (>) Ute (@) ate (B) rte

It also lists some options in the minibuffer:

C-h or ? for more options; SPC to leave unchangedo Character to
replace word

On pressing the number 4, the word is replaced with the correct spelling. If the spelling is already correct, then GNU Emacs will tell you that the word is correct. You can also spell check a region using M-x ispell-region and a buffer using M-x ispell-buffer. If you would like to stop the spell checker while it is running, you can use M-x ispell-kill-ispell. There is a flyspell mode that can check your spelling as you type. You can enable it using M-x flyspell-mode. If you want this mode only in a buffer, you can use M-x flyspell-buffer.

Word abbreviation

GNU Emacs can complete words for you. If you type ‘ba’ and then hit M-/, then GNU Emacs will try to complete the word for you. If you continue to use ‘M-/’, it will cycle through the various options such as ‘backwards’, ‘ball’, ‘bash’, ‘bar’, ‘back’, ‘based’ etc. To enter into the abbreviation mode, you need to use M-x abbrev-mode followed by the Enter key.

The abbreviations can either be local or global. Suppose, you want to define a local abbreviation for ‘international’, you can type in ‘intl’ and use C-x a i l to define the expansion. It will prompt you with the message ‘Mode expansion for “intl”:’. You can then type the word ‘international’. The next time you enter ‘intl’ followed by the space key, GNU Emacs will automatically expand the same for you. In order to define a global expansion, you need to use C-x a i g command sequence. If you would like to remove all abbreviations for the current session, you can use M-x kill-all-abbrevs followed by the Enter key.

You can save the abbreviations to a file, say ~/.emacs.d/.abbrev_defs, using M-x write-abbrev-file for future use. You can also edit the stored abbreviations using M-x edit-abbrevs shortcut. If you would like to view all the defined abbreviations, you can use M-x list-abbrevs. The relevant contents of ~/.emacs.d/.abbrev_defs are shown below:

;;-*-coding: utf-8;-*-
(define-abbrev-table 'Buffer-menu-mode-abbrev-table '())

(define-abbrev-table 'completion-list-mode-abbrev-table '())

(define-abbrev-table 'edit-abbrevs-mode-abbrev-table '())

(define-abbrev-table 'emacs-lisp-mode-abbrev-table
  '(
    ("intl" "international" nil 2)
   ))

(define-abbrev-table 'fundamental-mode-abbrev-table '())

(define-abbrev-table 'global-abbrev-table '())

(define-abbrev-table 'lisp-mode-abbrev-table '())

(define-abbrev-table 'sh-mode-abbrev-table '())

(define-abbrev-table 'shell-mode-abbrev-table '())

Printing

You can print the buffer contents from GNU Emacs using M-x print-buffer command. If you would like to print a selected region, you can use M-x print-region command. These will be printed with page numbers and headers. You can also print a buffer without page numbers using M-x lpr-buffer. Similarly, to print a region without page numbers, use M-x lpr-region. If you are already using the Dired mode, you can select the files that you want to print using the P shortcut, and then execute in Dired mode to print them.

In this article in the GNU Emacs series, we shall learn how to play games and try out some software for fun.

Ten games that are available and can be played easily in GNU Emacs are listed and discussed in this article. We start with the Tetris game.

Tetris

Tetris is available in GNU Emacs and you can launch it using M-x tetris. The ‘Up’ arrow key can be used to spin the tile, while the space key can quickly drop the tile. You can pause the game by pressing the ‘p’ key. At the end of the game, a score is displayed in a buffer. For example:

112 shakthi shakthi <author@shakthimaan.com>  Mon Nov  2 21:12:04 2015
52 shakthi Shakthi Kannan <author@shakthimaan.com>  Mon Nov  2 15:24:43 2015
11 shakthi shakthi <author@shakthimaan.com>  Mon Nov  2 15:29:39 2015

A screenshot of Tetris in GNU Emacs:

Pong

You can play Ping Pong in GNU Emacs with two players using M-x pong. One player can use the ‘Up’ and ‘Down’ arrow keys, and the second player can use the number keys - ‘4’ and ‘6’. Each paddle is represented by three square tiles. If the ball hits the edge of the paddle, it changes its direction. You can use the ‘p’ key to pause the game. A live score is displayed during the game. A screenshot of Ping Pong:

Doctor

GNU Emacs comes with a psychotherapist with whom you can chat. Simply type M-x doctor and it will start a session with the therapist. After you type in your statement, you need to hit Enter twice, and the therapist will reply. It will usually ask you another question, and you can continue to chat with the doctor. An example of a session is given below:

I am the psychotherapist.  Please, describe your problems.  Each time
you are finished talking, type RET twice.

Hi

How do you do?  What brings you to see me?

I am trying to talk to the doctor.

Why do you say you are trying to talk to the doctor?

I want to share this transcript with my readers.

What makes you believe you might want to share this transcript your
readers?

I can, from GNU Emacs, copy this text and paste it in my article.

Emacs?  Hah!  I would appreciate it if you would continue.

I am looking forward to watching today's cricket match that India is playing.

Is it because india is playing that you came to me?

Yes, who do you think will win?

Earlier you said you would like to share this transcript your readers?

Yes, I am still going to do that.

Can you elaborate on that?

I am writing a GNU Emacs series for the OSFY magazine.

Are you sure?

Yes, I have been writing for the past few months.

Maybe your life has something to do with this.

I want people to learn and use GNU Emacs to improve their productivity.

Are you sure that that?

Absolutely!

Perhaps you could continue.

Bubbles

The objective of the Bubbles game is to remove a series of connected bubbles in the fewest number of moves. The game can be started using M-x bubbles. As you move through the grid of bubbles, the connected bubbles with the same colour get highlighted. You can press the space bar to remove them. A screenshot of bubbles:

At the end of the game, a summary of the score is shown.

Selected:    9
Score:    1932
Game Over!

GNU Emacs also keeps a record of the scores:

01959   Mon Nov  2 21:26:41 2015        shakthi <author@shakthimaan.com>
01932   Mon Nov  2 21:20:38 2015        shakthi <author@shakthimaan.com>

Conway’s life

The shortcut M-x life will start the Conway’s Game of Life in a new GNU Emacs buffer. The population increases in course of time, and the generation count is shown in the mode line. A screenshot of the cellular automation running inside GNU Emacs:

Morse and NATO conversions

You can convert text to Morse code and vice versa by selecting the text in a buffer and using M-x morse-region. For example, the text ‘morse code’ gets converted to the following:

--/---/.-./.../. -.-./---/-../.

You can get back the text by selecting the Morse text and applying M-x unmorse-region. Similarly, if you have a word that you would like to spell using the NATO phonetic alphabet, you can use M-x nato-region. To convert it back, you need to use M-x denato-region. For example, the text ‘abc’ gets converted to:

Alfa-Bravo-Charlie

Snake

The Snake game can be started using M-x snake. You can use the arrow keys to move the head. As you play, red boxes appear in the window. If you go over them, the length of the snake increases along with the score. At the end of the game, a summary of the scores is shown. A screenshot of the Snake game:

Dunnet

Dunnet is a text based adventure game that needs to be started in batch mode as shown below:

$ emacs -batch -l dunnet

Dead end
You are at a dead end of a dirt road.  The road goes to the east.
In the distance you can see that it will eventually fork off.  The
trees here are very tall royal palms, and they are spaced equidistant
from each other.
There is a shovel here.

The help command gives you the context of the game:

>help
Welcome to dunnet (2.01), by Ron Schnell (ronnie@driver-aces.com).
Here is some useful information (read carefully because there are one
or more clues in here):
- If you have a key that can open a door, you do not need to explicitly
  open it.  You may just use 'in' or walk in the direction of the door.

- If you have a lamp, it is always lit.

- You will not get any points until you manage to get treasures to a certain
  place.  Simply finding the treasures is not good enough.  There is more
  than one way to get a treasure to the special place.  It is also
  important that the objects get to the special place *unharmed* and
  *untarnished*.  You can tell if you have successfully transported the
  object by looking at your score, as it changes immediately.  Note that
  an object can become harmed even after you have received points for it.
  If this happens, your score will decrease, and in many cases you can never
  get credit for it again.

- You can save your game with the 'save' command, and use restore it
  with the 'restore' command.

- There are no limits on lengths of object names.

- Directions are: north,south,east,west,northeast,southeast,northwest,
                  southwest,up,down,in,out.

- These can be abbreviated: n,s,e,w,ne,se,nw,sw,u,d,in,out.

- If you go down a hole in the floor without an aid such as a ladder,
  you probably won't be able to get back up the way you came, if at all.

- To run this game in batch mode (no Emacs window), use:
     emacs -batch -l dunnet
NOTE: This game *should* be run in batch mode!

If you have questions or comments, please contact ronnie@driver-aces.com
My home page is http://www.driver-aces.com/ronnie.html

You can then give directions and proceed with the game. An example of a session is shown below:

>n
You can't go that way.
>s
You can't go that way.
>e
E/W Dirt road
You are on the continuation of a dirt road.  There are more trees on
both sides of you.  The road continues to the east and west.
There is a large boulder here.
>in
You can't go that way.
>w
Dead end
There is a shovel here.
>d
You can't go that way.
>e
E/W Dirt road
There is a large boulder here.
>n s e
You can't go that way.
>in
You can't go that way.
>e
Fork
You are at a fork of two passages, one to the northeast, and one to the
southeast.  The ground here seems very soft. You can also go back west.
>w
E/W Dirt road
There is a large boulder here.

You can exit the game by typing ‘quit’ at the prompt (>).

Gomoku

Gomoku is a strategy board game where you need to get five pieces in a row (any direction) to win. You and the computer will take turns to play the game. You need to use the Enter key to mark your cross in a position, and the computer will mark its position with a circle. The game ends when either player gets five continuous pieces. A screenshot of Gomoku is shown below:

Le Solitaire

Le Solitaire is a strategy game that consists of stones represented by ‘o’ and holes represented by ’.’ (a dot). The objective of the game is to remove all the stones except the last one. You can jump over another stone to create a hole. You can use the Shift key with the arrow keys to move a stone. A screenshot of the game in progress:

All the games and examples were tried on GNU Emacs 24.5.1.

In this next article in the GNU Emacs series, we shall learn how to perform text search in a buffer, and introduce the concept of windows and frames.

Search

You can copy the following poem, which I wrote in 2013, in the scratch buffer or a file inside GNU Emacs to try out the search commands:

Emacs is, an operating system 
Which unlike many others, is truly, a gem 
Its goodies can be installed, using RPM 
Or you can use ELPA, which has already packaged them 

You can customize it, to your needs 
You can also check EmacsWiki, for more leads 
Your changes work, as long as reload succeeds 
And helps you with, your daily deeds 

People say, it lacks a decent editor 
But after using its features, they might want to differ 
Using Magit’s shortcuts, you might infer 
That it is something, you definitely prefer 

Plan your life, with org-mode 
You don’t necessarily need, to write code 
TODO lists and agenda views, can easily be showed 
Reading the documentation, can help you come aboard 

Emacs is, a double-edged sword 
Its powerful features, can never be ignored 
Customization is possible, because of Free Software code 
And this is, my simple ode.

You can search for a word in a buffer using C-s shortcut. You will then be prompted with I-Search: in the minibuffer where you can type any text, and GNU Emacs will try to find words matching it, in the buffer. This is an incremental forward search and is case insensitive. Thus, if you search for the word ‘todo’ in the poem, it will match the string ‘TODO’. You can exit from the incremental search using the Enter key, or abort the search using the C-g key combination. If you want to do an incremental search in the reverse direction - from the cursor position to the top of the buffer – you can use the C-r shortcut.

If you place the cursor on the letter ‘E’ in ‘Emacs’ in the poem’s first line, and press C-s C-w, Emacs will try to find all occurrences of the word ‘Emacs’ in the text. Suppose, you have cut or copied text to the kill ring, you can search for this text by using C-s C-y shortcut. You can repeat the previous forward search using C-s C-s, and the previous backward search using C-r C-r shortcuts.

The first occurrence of a text can be looked up in the forward direction using C-s. This will prompt you in the minibuffer with a Search: string where you can type the text that you want to search for, and then press the Enter key. It will then search for the text and move the cursor to the matching word. This is a non-incremental forward search. Similarily, you can perform a non-incremental backward search using C-r. You can then input the search string to be searched for, followed by the Enter key.

Regular expression searches are very useful too. In order to search forward for an expression, you can use C-M-s followed by the Enter key, which will prompt you in the minibuffer with the string ‘Regexp search:’. You can then enter a regular expression. This will only match the first occurrence of the text and the search will then terminate. You can perform a one-time backward regular expression search using C-M-r shortcut. To perform an incremental forward search, you need to use C-M-s and you will be prompted with the string ‘Regexp I-search:’, where you can provide the pattern to match. For example, ‘[a-z]+-[a-z]+’ will match both the expressions ‘org-mode’ and ‘double-edged’ words in the poem. You can use C-M-r for an incremental backward regex search.

A common use case is to find and replace text in a buffer. The sequence to be used is M-x query-replace followed by the Enter key. You will then be prompted with the string ‘Query replace:’ where you will be asked which word or phrase is to be replaced. For example, if you mention ‘ode’, it will again prompt you with ‘Query replace ode with:’ and then you can enter the replacement string. You can also search and replace text by matching a regular expression with the C-M-% shortcut key combination.

Frames

The outermost user interface boundary of GNU Emacs is called a frame. In fact, when you split the GNU Emacs user interface, you are actually creating windows. So, in GNU Emacs, you have windows inside a frame. This is in contrast to today’s user applications, where the entire application is contained in a ‘window’. This is an important terminology to remember when using GNU Emacs.

You can create a new frame using C-x 5 2 key combination. You can move the cursor to the next frame using C-x 5 o (letter ‘o’), and delete the current frame using C-x 5 0 (zero) shortcut. This will not delete the existing buffers, but, only the view. In order to open a file in a new frame, you can use C-x 5 f. You can also open a file in a new frame in read-only mode using C-x 5 r. To switch to the buffer in a new frame, use C-x 5 b key combination.

Windows

You can split a frame vertically to create two windows using C-x 2 (Figure 1).

To split horizontally, you can use C-x 3 (Figure 2).

To move the cursor to the next window, use C-x o (the letter ‘o’). You can delete the current window using C-x 0 (zero). Note that this does not delete the buffer, but, just the view. If you have multiple windows and you want to retain the current window and remove the rest of the windows from the display, you can use C-x 1.

You can open a file in a new window using C-x 4 f. You can also select an existing buffer in another window using C-x 4 b. If you have multiple windows that you would like to be balanced equally, you can use C-x +. Figure 3 shows an Emacs screenshot with three windows that are balanced.

You can scroll the contents in the other window using C-M-v. You can scroll backwards using C-M-Shift-v.

You can also use the following shortcuts in your ~/.emacs to simplify the shortcuts used to split and remove windows.

(global-set-key (kbd "C-1") 'delete-other-windows)
(global-set-key (kbd "C-2") 'split-window-below)
(global-set-key (kbd "C-3") 'split-window-right)
(global-set-key (kbd "C-0") 'delete-window)

If you would like to make a window wider, you can use C-x } shortcut and to reduce it horizontally, you will need to use C-x {. You can use a prefix count to perform the operation ‘n’ times. For example, C-u 5 C-x { to shrink a window horizontally. To make a window taller, you can use C-x ^ shortcut; and to make it smaller, you have to use a negative prefix. For example, C-u -1 C-x ^. A screenshot of a custom Emacs frame with three windows is shown in Figure 4.

Anand Chandrasekaran, CTO, Mad Street Den began the day’s proceedings with his talk on “Deep learning: A convoluted overview with recurrent themes and beliefs”. He gave an overview and history of deep learning. He also discussed about LeNet, Deep Belief Network by Geoffrey Hinton, Backpropagation Algorithm (1974) by Paul Werbos, and Deep Convolutional Neural Networks (2012) by Alex Net, named after Alex Krizhevsky. Mad Street Den primarily work on computer vision problems. In one of their implementations, they extract 17,000 features from a dress, and provide recommendations to customers. They are one of the early users of NVIDIA GPUs. He also briefed on other deep learning tools like Amazon ML, Torch 7, and Google TensorFlow.

The second talk of the day was a sponsored talk on “Recent advancements in Deep Learning techniques using GPUs” by Sundara R Nagalingam from NVIDIA. He talked on the available GPU hardware and platforms for deep learning available from NVIDIA. It was a complete sales pitch. I did ask them if they have free and open source Linux device drivers for their hardware, but, at the moment they are all proprietary (binary blobs).

After a short tea break, Abhishek Thakur presented on “Applied Deep Learning”. This was one of two best presentations of the day. Abhishek illustrated binary classification and fine tuning. He also briefed on GoogleNet, DeepNet, and ImageNet Large Scale Visual Recognition Challenge (ILSVRC). Deep learning software such as Theano, Lasagne, and Keras were also discussed. A query can be of three types - navigational, transactional, or informational. Word2vec is a two-layer neural net that can convert text into vectors. You can find a large collection of images for input datasets at CIFAR.

The next two sessions were 20-minute each. The first talk was on “Residual Learning and Stochastic Depth in Deep Neural Networks” by Pradyumna Reddy, and the second was on “Expresso - A user-friendly tool for Deep Learning” by Jaley Dholakiya. The Expresso UI needs much work though. I headed early for lunch.

Food was arranged by Meal Diaries and it was delicious!

The post-lunch session began at 1410 IST with Arjun Jain talking on “Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation”. He gave a number of examples on how difficult it is to train models, especially the human body.

Vijay Gabale then spoke on “Deep Dive into building Chat-bots using Deep Learning”. This was the second best presentation of the day. He gave a good overview of chat-bots and the challenges involved in implementing them. There are four building blocks for chat bots - extract intent, show relevant results, contextual interaction and personalization. He also discussed on character-aware neural language models.

I then headed to the BoF session on “Getting Started with Deep Learning”. A panel of experts answered questions asked by the participants. It was suggested to start with toy data and move to big data. Andrew Ng’s Machine Learning course and Reinforcement Learning course were recommended. CS231n: Convolutional Neural Networks for Visual Recognition was also recommended for computer vision problems. Keras and Theano are useful tools to begin with. It is important to not just do a proof-of-concept, but, also see how things work in production. It is good to start to use and learn the tools, and subsequently delve into the math. Having references can help you go back and check them when you have the know-how. Data Nuggets and Kagil are two good sources for datasets. The Kaggle Facial Keypoints Detection (KFKD) tutorial was also recommended. Data science does involve both programming and math. We then headed for a short tea break.

Nishant Sinha, from MagicX, then presented his talk on “Slot-filling in Conversations with Deep Learning”. He gave an example of a semantic parser to fill slots using a simple mobile recharge example. He also discussed about CNN, Elman RNN and Jordan RNN. This was followed by the talk on “Challenges and Implications of Deep Learning in Healthcare” by Suthirth Vaidya from Predible Health. He spoke on the difficulties in dealing with medical data, especially biometric images. Their solution won the Multiple Sclerosis Segmentation Challenge in 2015.

The last talk of the day was on “Making Deep Neural Networks smaller and faster” by Suraj Srinivas from IISc, Bengaluru. He discussed how large model can be mapped to small models using model compression. This involves compressing matrices through four techniques - sparsify, shrink, break, and quantize. The objective is to scale down the solution to run on mobile and embedded platforms, and on CPUs. It was an interesting talk and a number of open research problems exist in this domain.

Overall, it was a very useful one day conference.

Internet Relay Chat (IRC) provides instant communication over the Web. It is generally used for one-to-one communication or group discussion forums, also known as channels. In this next article in the GNU Emacs series, we shall learn how to use GNU Emacs as an Internet Relay Chat (IRC) client.

What is IRC?

Internet Relay Chat (IRC) is a messaging protocol that was created by Jarkko Oikarinen in 1988. An IRC server has a number of channels (or rooms, as how they are called in other chat software) where both technical and non-technical discussions take place. Every user requires a nickname to chat. You will need to register your nickname with a password to identify yourself every time you log in to the IRC server. IRC can also be used for file sharing among users.

You can connect to any one of a number of IRC servers that are available worldwide, free. The most popular is Freenode (irc.freenode.net). The other IRC servers, to name a few, are IRCNet, OFTC, and EFNet. The Debian and Ubuntu projects have their own chat servers – irc.debian.org and irc.ubuntu.com. All IRC channel names begin with ‘#’. Some channels begin with ‘##’. Examples of channels are ##linux-india, #dgplug, #emacs, #ruby, #guile, #lisp, and #scheme. You can also create your own channels, or host your own IRC server. Some examples of free and open source IRC server software are IRCd, UnrealIRCd, and Synchronet. Each IRC channel has a topic that can include useful links relevant to the channel. Sometimes announcements, or news updates are also mentioned in the topic.

Basic commands

A number of commands can be given to the IRC server. A few of them are discussed below:

1. /list is used to provide a list of all the available channels in a server. For example: /list in irc.freenode.net returned the following:

   ...
   #linod           1       
   ##welding        3       Welcome to ##Welding, We're a little bare at the moment, but will help if we can. Tutorials: https://www.youtube.com/channel/UCJAFY2kKKb5sg79yld7T3hA
   #drupal-ph       1       "Drupalista! Welcome to Philippine Drupal Users Group. Have a good time chatting. If you have a question, please don't ask to ask but fire up your question in very specific and constructive way! Please join #drupal or #drupal-support if no one is around"
   #orx-project     4       Orx: Portable Game Engine
   #tinkerforge     5       
   #osi             10      The Open Source Initiative
   #xampp           1       
   #guitar          8       
   #bitcoin-ar      3       Comunidad Bitcoin Argentina
   #LargeHadrosaurCollider 19      Welcome to the LHC, est. 2001 | http://www.largehadrosaurcollider.net | August Birthdays: Digby 08/21, Josh 08/31 | At night it is pitch black, often for months. | http://tinyurl.com/psgdagl
   * End of /LIST

2. /msg NickServ REGISTER password e-mail is used to register your nickname to the IRC server. /msg NickServ IDENTIFY password is used to identify yourself to the server.

3. /me message displays the message for a user. For example:

   /me says "Hello, World!"
   
    * mbuf says "Hello, World!"

4. /whois nickname provides useful information for a user. For example:

    /whois mbuf
   
   * [mbuf] (~shakthi@123.123.123.123): Shakthi Kannan
   * [mbuf] #guile #scheme ##linux-india #stumpwm #guix #dgplug #lisp #emacs 
   * [mbuf] kornbluth.freenode.net :Frankfurt, Germany
   * [mbuf] is connecting from *@123.123.123.123 123.123.123.123
   * [mbuf] idle 00:41:52, signon: Thu Sep  3 20:36:52
   * [mbuf] is logged in as mbuf
   * [mbuf] End of WHOIS list.

5. /msg nickname is used to send a private message to a nickname and to start a private conversation.

6. /help provides useful help on basic IRC commands. These commands are:

     ADDBUTTON ALLCHAN   ALLCHANL  ALLSERV   AWAY
     BACK      BAN       CHANOPT   CHARSET   CLEAR
     CLOSE     COUNTRY   CTCP      CYCLE     DCC
     DEBUG     DEHOP     DELBUTTON DEOP      DEVOICE
     DISCON    DNS       ECHO      EXEC      EXECCONT
     EXECKILL  EXECSTOP  EXECWRITE FLUSHQ    GATE
     GETFILE   GETINT    GETSTR    GHOST     GUI
     HELP      HOP       ID        IGNORE    INVITE
     JOIN      KICK      KICKBAN   KILLALL   LAGCHECK
     LASTLOG   LIST      LOAD      MDEHOP    MDEOP
     ME        MENU      MKICK     MODE      MOP
     MSG       NAMES     NCTCP     NEWSERVER NICK
     NOTICE    NOTIFY    OP        PART      PING
     QUERY     QUIT      QUOTE     RECONNECT RECV
     SAY       SEND      SERVCHAN  SERVER    SET
     SETCURSOR SETTAB    SETTEXT   SPLAY     TOPIC
     TRAY      UNBAN     UNIGNORE  UNLOAD    URL
     USELECT   USERLIST  VOICE     WALLCHAN  WALLCHOP
   
   User defined commands:
   
     ACTION    AME       ANICK     AMSG      BANLIST
     CHAT      DIALOG    DMSG      EXIT      GREP
     J         KILL      LEAVE     M         ONOTICE
     RAW       SERVHELP  SPING     SQUERY    SSLSERVER
     SV        UMODE     UPTIME    VER       VERSION
     WALLOPS   WII       
   
   Plugin defined commands:
   
     UNLOAD    UNLOAD    LOAD      PY        LOAD
     RELOADALL SOURCE    TCL       RELOADALL UNLOADALL
     PL_RELOAD RELOAD    UNLOAD    LOAD      TIMER
   
   
   Type /HELP <command> for more information, or /HELP -l

7. /quit is used to disconnect and exit from IRC.

8. /join #channel - is used to join a channel. For example: /join #guix.

9. /nick newnickname changes your nick to newnickname. Suppose, you wish to move away from the computer, you can change your nick to nick|away, or nick|phone.

10. /part is used to leave a channel.

Using rcirc

If you are using a recent GNU/Linux distribution, you should already have rcirc as part of GNU Emacs. You can simply start it by typing M-x rcirc from inside Emacs. The ‘M’ key represents the Meta key, which is usually mapped to the ‘Alt’ key. After rcirc connects to the IRC server, you can use /nick to change your nickname, register (only the first time) your nick, identify yourself, join channels, and start chatting! Since everything is a buffer in GNU Emacs, each channel is a separate buffer. For example, #emacs@irc.freenode.net is the #emacs IRC channel. All your basic buffer navigation commands will work just like they would on a file!

Some basic rcirc commands

The rcirc commands for the above mentioned IRC commands are given in the following table:

~/.emacs

GNU Emacs is an extensible editor. There are a number of locations where Emacs checks for custom configurations before launching the editor. These are: ~/.emacs, _{/.emacs.el, and}/.emacs.d/init.el. The start-up files can be customised, and their locations can also be changed. There are a number of ways to organize, and manage your Emacs configuration. Until we get to learn Emacs lisp, and customize Emacs as a project, we shall use ~/.emacs for all our user-specific customizations.

rcirc customization

Create a ~/.emacs.d/etc folder in your $HOME directory and an Elisp file called init-rcirc.el in it. It should contain the following (change nick, user-name and full-name to suit your needs):

;; Default user.
(setq rcirc-default-nick "shaks")
(setq rcirc-default-user-name "shakthimaan")
(setq rcirc-default-full-name "Shakthi Kannan")

;; Channels to join at startup.
(setq rcirc-server-alist
      '(("irc.freenode.net" :channels ("##linux-india" "#dgplug" "#rcirc" "#emacs"))))

The above is an example of Emacs Lisp code. Comments begin with two semi-colons. The setq construct sets the second argument value to the first argument, which is a quoted symbol. For example, the symbol ’rcirc-default-nick is set to “shaks”. The rcirc-server-alist defines the initial list of channels to login at startup.

You can now start GNU Emacs from the GNOME terminal using the following command:

$ emacs -q -l ~/.emacs.d/etc/init-rcirc.el

You will then automatically connect to the four IRC channels.

How to use IRC

People join IRC channels to have their doubts regarding free and open source software clarified. Sometimes, off-topic discussions also happen. It is like live technical support, but has a social context to it. Whenever you are connected online, you must be logged in to IRC. You can have discussions in the channel, or in private, if the other party agrees. It is a good place to learn a lot about free and open source software, and you are bound to make a lot of friends. Since people from all over the world participate, which means they are online in different time zones, some channels log the discussions for future reference. As always, before asking a question, it is important for you to do your homework first. Take sufficient time and put in an effort to debug and identify the problem to the best of your ability.

Some users in the channel may ask you for more information before being able to provide you with any assistance. So, be prepared to provide all the information necessary about the bug or error when you seek help. Sometimes, people might be logged in the channel, but, they might be away from the computer. So, even if you don’t get a response, be patient; come back later and ask again.

You should not paste more than four continuous lines of text in the channel, as it will ‘flood’ the screen for everyone else. Instead, use an external paste service like gist.github.com, or fpaste.org. These services will provide a shortened URL that you can pass around in the channel. Whoever is interested in helping you will view the contents from the link. If you enter text in the channel, it means that it is addressed to everyone in the channel. If you wish to say something to a specific user, mention their nickname first, and then type in the text.

Most IRC client software provide you with panels that list the channels that you are logged in, and show the list of users. If someone mentions your nickname in a channel, then the corresponding channel will change colour or representation to indicate that there is a message for you. A few users are channel operators (or moderators) and they have special privileges. They are similar to ‘root’ users in a system, and they exist to keep the signal-to-noise ratio to a minimum, and keep a vigil on the channel.

An IRC bot is a client software that connects to the IRC server as a user, but can respond to commands. It can thus be programmed to provide many services in a channel. You can customize existing bots, or write your own. Examples of IRC bots are Cerberus, Gambot and irccd. Cinch is an example of an IRC bot building framework written in Ruby. Bots can be used during an IRC meeting session to keep track of user questions. They can evaluate programming language constructs and return meaningful errors to newbies in the channel. They can be used to send a notification to the channel if a project test build fails, or when a new bug has been filed. The possibilities are endless.

IRC meeting protocol

A free and open source software project will have a dedicated IRC channel where the project members will meet to have discussions. Meetings can be scheduled, and can happen in different time zones depending on where the users are located. There is a protocol and etiquette to be followed during such meetings. The speaker or moderator should not be interrupted during the session.

If you wish to ask a question, type ’?’ and wait. When the speaker has finished and feels that you can type in your text, you will be asked to do so. After you have finished typing your content, end with ‘EOF’. Similarly, if you need to speak during the session, type ’!’, and wait. You can give your consent or dissent to statements made in the channel using +1 or -1, respectively.

You are encouraged to read the rcirc manual and customize rcirc to your needs. If you have made it this far, do connect to irc.freenode.net, and feel free to say ‘Hi’ to me. I am ‘mbuf’ on irc.freenode.net. A screenshot of an rcirc session is shown below:

19:35 <mbuf>    http://ecb.sourceforge.net/screenshots/index.html
19:35 ***       Arpita QUIT Client Quit
19:36 <rtnpro>  !
19:36 <mbuf>    rtnpro, shoot!
19:37 <rtnpro>  How do we get the emacs code browser?
19:37 <rtnpro>  <EOF>
19:37 <mbuf>    rtnpro, 1. Need to install "ecb" from your distro package manager
                2. you could have searched this on the Internet :)
19:38 <rtnpro>  It is not in my distro
19:38 <sumitc>  !
19:38 <mbuf>    rtnpro, and which distro are you using?
19:38 <rtnpro>  Its Fedora 9
19:39 <rtnpro>  I have got emacs but not emacs code browser
19:39 <mbuf>    rtnpro, you can always install from source
19:39 <techno_freak>    rtnpro, http://ecb.sourceforge.net/downloads.html
19:39 ***       khushbu QUIT Ping timeout: 244 seconds
19:39 <rtnpro>  ok
19:39 <mbuf>    sumitc, shoot!
19:39 <sumitc>  what is a tag-file?
19:40 <rtnpro>  What factors should decide the choice of our editor?
19:40 <mbuf>    rtnpro, wait!
19:40 ***       pushkal JOIN
19:40 <mbuf>    sumitc, the TAGS file contains the details of the reference count,
                and locations of variables/functions et. al.
19:41 <sumitc>  So, a tag file is always associated with a specific file?
19:41 <mbuf>    sumitc, no, it can have information of files in a directory
19:41 <sumitc>  ok
19:41 <sumitc>  <eof>
19:42 <mbuf>    sumitc, think of it as a database that answers all your queries
               regarding code references

References

1. Freenode. https://freenode.net/

2. rcirc manual. https://www.gnu.org/software/emacs/manual/html_mono/rcirc.html

3. IRC Help. http://www.irchelp.org/

4. Fedora project - How to use IRC. https://fedoraproject.org/wiki/How_to_use_IRC

Continuing with our series on Emacs, in this article, let’s explore basic buffer navigation, formatting, and help commands.

Buffers

You can view a list of buffers in GNU Emacs using the C-x C-b command, and can switch to a buffer using the C-x b shortcut. The default list of buffers as seen in GNU Emacs 24.3.1 on Ubuntu 14.10 is shown below:

CRM Buffer                  Size Mode            File
.   *scratch*               191 Lisp Interaction 
*   *Messages*              485 Fundamental
%   *Completions*           264 Completion List
%   *Help*                 1855 Help

In order to kill (close) a buffer, you can use C-x k key combination. If you have multiple buffers that you would like to close, you can use M-x kill-some-buffers command, which will prompt you for a confirmation for every buffer. You can also rename a buffer using the M-x rename-buffer command. If you would like to save the buffer contents to a file, you can use C-x s, and it will prompt you with many options before proceeding to save the file. You can also save a modified buffer using C-x C-s.

Navigation

It is important to learn how to use the Emacs navigation commands instead of the arrow keys. When you learn to use macros, you will find it useful to automate steps. A summary of the basic navigation shortcuts are given in the following table:

You can scroll down a buffer using C-v, and scroll up using M-v shortcut. If you would like to move to the beginning of a file, use M-< command. You can move to the end of a file using M-> shortcut.

In order to move the cursor to a specific line, use M-x goto-line or M-g g. If you would like to position the contents where the cursor is located to the center of the screen, you can use C-l. If you want to repeat a command ‘n’ (‘n’ is an integer) number of times, you can use M-n following by the command. You can also use C-u n, but, if you don’t specify ‘n’ here, the next command, by default, will be executed four times.

Clipboard

You can delete a character where the cursor is located using C-d. To remove a word, you can use M-d. You can remove text from the cursor position till the end of line using C-k. The cut text goes into a kill ring buffer. You can paste whatever you have removed using C-y. You can cycle through the text that has been removed by repeatedly using M-y after C-y. Consider the text:

When you have a question about Emacs, the Emacs manual is often the
best starting point.

If you place the cursor before “about” in the above sentence, and press ‘M-d’, it will remove " about“. You can now see the value of the kill-ring using C-h v (describe variable), and it will prompt you with the message”Describe variable:“, where you can enter kill-ring. You will see an entry for” about“. Now, place the cursor after”question" and press ‘M-d’ again, and “Emacs” will be removed. If you again check the kill-ring value, you will see an entry for " Emacs".

You can now move the cursor to any location in the buffer, and press ‘C-y’, and it will paste “Emacs”. If you then press M-y, it will paste “about” which was the text that was cut before “Emacs”. This way, you can cycle through the kill ring buffer.

To select a region of text, you first need to mark the starting position with the cursor using C-space. You can then use any of the navigation commands to highlight the text that you want. You can either cut the marked region using C-w, or copy it using M-w. This will get copied to the kill-ring. You can then paste the text to any buffer. You can also select the entire buffer using C-x h.

Formatting text

It is good to have text within 80 columns and 25 rows (80x25), not just for readability, but, to avoid making any assumptions about the users’ screen size. If you have a really long sentence or paragraph, you can reformat it using the M-q shortcut from anywhere in the sentence or paragraph.

You can transpose two characters using C-t shortcut. For example, if you have misspelt “emacs” as “emcas”, you can place the cursor on ‘a’ and type C-t to swap the letters ‘a’ and ‘c’ to get “emacs”. You can transpose two words using the M-t command. For example, if you have the phrase “morning good” in a buffer, and you type M-t anywhere on the word “morning”, it will swap the words to give you “good morning”. You can also swap lines and sentences using C-x C-t and M-x transpose-sentences commands respectively.

If you would like to capitalize the first character of a word in upper case, you need to place the cursor on it, and key in M-c to make the first letter capital. You can turn all those letters from the cursor location to the rest of the word into capitals using M-u command. For example, if you have the word “sentence”, and you place the cursor on the letter ’t’, and you key in M-u the result will be “senTENCE”. To make all the characters lower case, you can use M-l. You can turn the entire previous word into capital letters using the M - M-c shortcut. Similarily, to make the previous words either upper case and lower case, you can use M - M-u and M - M-l commands, respectively.

Info

You can read manuals written in Info format inside GNU Emacs. When you type M-x info, it opens up a menu with the built-in documentation available from ‘Info’ files. The Info mode is mostly used to read hypertext manuals. You can use Tab and Shift-tab to navigate through the available links. Since the contents are in a buffer, all the basic navigation commands are applicable. To exit from Info, you can use q (quit).

You can press the return key on a link to open its contents. For example, when you click on “Emacs FAQ” it opens “The GNU Emacs FAQ” contents. The following are the sections listed in its buffer:

* Menu
* FAQ notation
* General questions
* Getting help
* Status of Emacs
* Common requests
* Bugs and problems
* Compiling and installing Emacs
* Finding Emacs and related packages
* Key bindings
* Alternate character sets
* Mail and news
* Concept index

Suppose, you hit the return key on “FAQ notation”, it will open its contents with the title “1 FAQ notation”. If you want to cycle through the headings at this level, you can press n (next topic) and p (previous topic). If you want to move one level up, to the “The GNU Emacs FAQ”, you can use the u shortcut.

If you simply want to move to the topmost node (“The GNU Emacs FAQ”), you can use the t shortcut. For example, if you were in the section “3.1 I’m just starting Emacs; how do I do basic editing?” and you pressed ’t’, it will take you to the topmost section - “The GNU Emacs FAQ”.

You can move to the previous node that you visited using the l shortcut. You can also explicitly specify the node you want to move to using the g shortcut. For example, if you are in the “1 FAQ notation” page, and you pressed g, it will then prompt you with the message “Go to node:”. You can then type “Extended” and hit TAB to auto-complete to “Extended commands”, which is one of the links in the page.

You are encouraged to try the built-in info tutorial available by pressing h in the “info” buffer.

Help

You can read manual pages inside GNU Emacs using M-x man command followed by the return key. Suppose, you try this command from the scratch buffer, it will then prompt you with the message “Manual entry:”. You can then input ls (list directory contents), and it will open a new buffer and window titled “Man ls” with the contents of the manual entry for the ls command.

From the scratch buffer, you can scroll forward the other window (‘Man ls’) using C-M-v (Ctrl-Alt-v), and scroll back using C-M-shift-v (Ctrl-Alt-Shift-v).

You can list all the available, active key-bindings using the C-h b command.

The C-h f shortcut is used to provide documentation for a function available in your installed version of GNU Emacs. For example, if you use C-h f, it will prompt you with the message “Describe function:”. If you input scroll-other-window, it will open a new buffer and window titled “Help” with the description and usage of the function. The output as seen in GNU Emacs 24.3.1 is shown below:

scroll-other-window is an interactive built-in function in `C source
code'.

It is bound to <M-next>, C-M-v, ESC <next>.

(scroll-other-window &optional ARG)

Scroll next window upward ARG lines; or near full screen if no ARG.
A near full screen is `next-screen-context-lines' less than a full screen.
The next window is the one below the current one; or the one at the top
if the current one is at the bottom.  Negative ARG means scroll downward.
If ARG is the atom `-', scroll downward by nearly full screen.
When calling from a program, supply as argument a number, nil, or `-'.

If `other-window-scroll-buffer' is non-nil, scroll the window
showing that buffer, popping the buffer up if necessary.
If in the minibuffer, `minibuffer-scroll-window' if non-nil
specifies the window to scroll.  This takes precedence over
`other-window-scroll-buffer'.

[back]

On the other hand, if you know the command, and would like to get help related to it, you can use the C-h k command. This will prompt you with the message “Describe key (or click or menu item):”. You must key in the actual shortcut, for example, C-h b, and it will update the buffer “Help”, and open it in a new window, showing the description for the shortcut as shown below:

C-h b runs the command describe-bindings, which is an interactive
compiled Lisp function.

It is bound to C-h b, <f1> b, <help> b, <menu-bar> <help-menu>
<describe> <list-keybindings>.

(describe-bindings &optional PREFIX BUFFER)

Show a list of all defined keys, and their definitions.
We put that list in a buffer, and display the buffer.

The optional argument PREFIX, if non-nil, should be a key sequence;
then we display only bindings that start with that prefix.
The optional argument BUFFER specifies which buffer's bindings
to display (default, the current buffer).  BUFFER can be a buffer
or a buffer name.

[back]

You are encouraged to try out all the commands so that you understand what actions they perform. When you use GNU Emacs as your main text editor, you will be able to regularly practice these commands, and master them.

Introduction

Dired is a directory editor in GNU Emacs. It opens a buffer containing a list of directories and files to operate on. It is a ‘read-only’ mode and hence you cannot input any text. This article explores some of the basic commands that can be used in Dired mode.

Let us first create a sample directory, sub-directories and files that we can use for our demonstration. Open a terminal and execute the following commands:

$ mkdir /tmp/test/network -p
$ mkdir /tmp/test/kernel -p

$ date > /tmp/test/date.txt
$ locale > /tmp/test/locale.txt

$ cat /etc/resolv.conf > /tmp/test/network/resolv.conf
$ ifconfig > /tmp/test/network/ifconfig.output

$ dmesg > /tmp/test/kernel/dmesg.txt

In general, filename extensions don’t have any meaning on *nix systems. Some applications do check the filename extension before using them. The extension is only for the benefit of the user, and hence you can have a filename without any extension on *nix.

Invocation

After opening GNU Emacs, you can enter Dired mode using M-x dired. It will prompt you for the directory to be opened in the buffer where you can input ‘/tmp/test’. You will see a buffer with the following contents:

/tmp/test:
total used in directory 24 available 146721468
drwxrwxr-x  4 shakthi shakthi 4096 Jul  3 11:36 .
drwxrwxrwt 12 root    root    4096 Jul  3 11:38 ..
-rw-rw-r--  1 shakthi shakthi   29 Jul  3 11:36 date.txt
drwxrwxr-x  2 shakthi shakthi 4096 Jul  3 11:36 kernel
-rw-rw-r--  1 shakthi shakthi  270 Jul  3 11:36 locale.txt
drwxrwxr-x  2 shakthi shakthi 4096 Jul  3 11:36 network

The contents of the buffer are similar to the ls -al output as observed in the terminal. The files were created by user ‘shakthi’, and this is indicated in the owner and group fields. The ’.’ entry represents the current ‘/tmp/test’ directory. The ’..’ listing represents the parent directory, which is ‘/tmp’ and owned by the ‘root’ user.

Exit

You can exit from Dired mode by simply pressing q (quit-window) in the Dired buffer.

Navigation

You can move to a previous line or the next line in the Dired buffer using the p and n keys, respectively. If you wish to move the cursor to the previous and next directories, you can use the ‘<’ and ‘>’ keys. If the cursor is at ‘kernel’ directory as shown below …

/tmp/test:
total used in directory 24 available 146721468
drwxrwxr-x  4 shakthi shakthi 4096 Jul  3 11:36 .
drwxrwxrwt 12 root    root    4096 Jul  3 11:38 ..
-rw-rw-r--  1 shakthi shakthi   29 Jul  3 11:36 date.txt
drwxrwxr-x  2 shakthi shakthi 4096 Jul  3 11:36 kernel      <-- CURSOR
-rw-rw-r--  1 shakthi shakthi  270 Jul  3 11:36 locale.txt
drwxrwxr-x  2 shakthi shakthi 4096 Jul  3 11:36 network

… then, when you press ‘>’, the cursor will move to ‘network’, which is the next directory in the buffer.

/tmp/test:
total used in directory 24 available 146721468
drwxrwxr-x  4 shakthi shakthi 4096 Jul  3 11:36 .
drwxrwxrwt 12 root    root    4096 Jul  3 11:38 ..
-rw-rw-r--  1 shakthi shakthi   29 Jul  3 11:36 date.txt
drwxrwxr-x  2 shakthi shakthi 4096 Jul  3 11:36 kernel      
-rw-rw-r--  1 shakthi shakthi  270 Jul  3 11:36 locale.txt
drwxrwxr-x  2 shakthi shakthi 4096 Jul  3 11:36 network     <-- CURSOR

You can move to the parent directory using the ‘^’ key. To enter into a directory, move to its listing in the Dired buffer, and simply hit the return key.

Viewing files

To view a file, you can place the cursor on its entry and use the f or v key, or simply hit the return key. This will open the file in a new buffer (view). To return to the Dired buffer, you can type C-x b and the minibuffer will prompt you with the message “Switch to buffer (…)”. If you then press TAB, it will open a new Completions buffer that will list the available buffers. You can also type the entire ‘test’ buffer name, or type it partially and hit TAB for auto-completion, and hit the return key to get to the ‘test’ buffer. If you wish to close a buffer, you can use C-k to kill it. This will only close the buffer, but, the file will still exist! You can use the ‘+’ key to create a new sub-directory.

Marking and unmarking files

Dired mode allows you to operate on multiple files and directories. In order to run commands on them, you need to first select the files. The m key can be used to mark a file or directory for subsequent operations. For example, pressing ’m’ on the date.txt entry will mark it, and this is indicated by an asterisk in front of the line, as shown below:

  /tmp/test:
  total used in directory 24 available 146933380
  drwxrwxr-x  4 shakthi shakthi 4096 Jul  5 09:53 .
  drwxrwxrwt 10 root    root    4096 Jul  5 09:46 ..
* -rw-rw-r--  1 shakthi shakthi   29 Jul  5 09:46 date.txt
  drwxrwxr-x  2 shakthi shakthi 4096 Jul  5 09:46 kernel
  -rw-rw-r--  1 shakthi shakthi  270 Jul  5 09:46 locale.txt
  drwxrwxr-x  2 shakthi shakthi 4096 Jul  5 09:46 network

You can unmark the above selection using the u key. This is applicable for both files and directories. To undo a selection, you can also use M-del (M and the Delete key). Suppose, you wish to mark all the directories in the Dired buffer, you can use ‘* /’ key combination. The resultant Dired buffer is shown below:

  /tmp/test:
  total used in directory 24 available 146933432
  drwxrwxr-x  4 shakthi shakthi 4096 Jul  5 09:53 .
  drwxrwxrwt 10 root    root    4096 Jul  5 09:46 ..
  -rw-rw-r--  1 shakthi shakthi   29 Jul  5 09:46 date.txt
* drwxrwxr-x  2 shakthi shakthi 4096 Jul  5 09:46 kernel
  -rw-rw-r--  1 shakthi shakthi  270 Jul  5 09:46 locale.txt
* drwxrwxr-x  2 shakthi shakthi 4096 Jul  5 09:46 network

If you want to invert the selection, use t. The buffer will look like the following:

  /tmp/test:
  total used in directory 24 available 146933432
  drwxrwxr-x  4 shakthi shakthi 4096 Jul  5 09:53 .
  drwxrwxrwt 10 root    root    4096 Jul  5 09:46 ..
* -rw-rw-r--  1 shakthi shakthi   29 Jul  5 09:46 date.txt
  drwxrwxr-x  2 shakthi shakthi 4096 Jul  5 09:46 kernel
* -rw-rw-r--  1 shakthi shakthi  270 Jul  5 09:46 locale.txt
  drwxrwxr-x  2 shakthi shakthi 4096 Jul  5 09:46 network

To mark all the files, you can use ‘* s’ key combination.

Modifying the buffer

After you have selected files, you can remove them from the listing by pressing the k key. This does not delete the files! You can always press g to refresh the Dired buffer contents. You can change the ‘ls’ command line options used in the Dired listing using the C-u s key combination.

Actions

You can perform a number of operations on the marked files and directories. To copy a file, you can press C on a file, and it will prompt you in the minibuffer regarding where you would like to create the new copy. You can use R to rename a file. If you would like to delete a file, you can press D. You can change the mode of a file with the M command. You can compress or uncompress a file with the Z command.

A regular expression search can be done on selected files with the A command. You can create a symbolic link file with the S key. You can run a shell command on marked files using the ’!’ command. For example, if you want to perform a word count (wc) on the date.txt file, you can press ’!’ on the date.txt entry listing, and it will prompt you in the minibuffer with “! on date.txt:” message. If you input ‘wc’, the results will again be shown in the minibuffer. On my system, it returned ‘1 6 29 date.txt’. You can also run asynchronous shell commands on files using the ‘&’ command, and a new buffer will be opened that will contain the results.

Deletion

You can mark files for deletion using the d command. After selecting the files or directories, if you press x, the respective actions will be performed on the files. To demonstrate an example, let us use ‘d’ on the ‘kernel’ directory to mark it for deletion. The letter ‘D’ will be added to its entry as shown below:

  /tmp/test:
  total used in directory 24 available 146933432
  drwxrwxr-x  4 shakthi shakthi 4096 Jul  5 09:53 .
  drwxrwxrwt 10 root    root    4096 Jul  5 09:46 ..
  -rw-rw-r--  1 shakthi shakthi   29 Jul  5 09:46 date.txt
D drwxrwxr-x  2 shakthi shakthi 4096 Jul  5 09:46 kernel
  -rw-rw-r--  1 shakthi shakthi  270 Jul  5 09:46 locale.txt
  drwxrwxr-x  2 shakthi shakthi 4096 Jul  5 09:46 network

When you press ‘x’, you will be prompted to confirm deletion with the message “Delete kernel (yes or no)”. When you input “yes” and hit the return key, the directory contents will be removed. GNU Emacs will perform auto-save on files, and also create backup files ending with ‘~’. You can mark such files using the ‘~’ command.

Regex

You can use ‘% d’ to select files based on a regular expression (regex) to mark for deletion. To simply mark files based on regex, you can use ‘% m’ shortcut. To rename the marked files, you can use ‘% R’.

Find

You can search for files matching a pattern using the M-x find-name-dired command. Suppose you wish to find all the .txt files in our original /tmp/test directory, you can use ‘M-x find-name-dired’ in the Dired buffer. It will prompt you in the minibuffer with the following message “Find-name (directory): /tmp/test”. After you press the return key, it will ask for the regex with the message “Find-name (filename wildcard):”. If you input ‘*.txt’, it will return the following contents:

/tmp/test/:
find . \( -name \*.txt \) -ls
6826601    4 -rw-rw-r--   1 shakthi  shakthi        29 Jul  5 09:46 date.txt
6826616   64 -rw-rw-r--   1 shakthi  shakthi     63050 Jul  5 09:46 kernel/dmesg.txt
6826602    4 -rw-rw-r--   1 shakthi  shakthi       270 Jul  5 09:46 locale.txt

find finished at Sun Jul  5 10:27:02

You can also find files based on regex patterns that exist in the file contents using the M-x find-grep-dired. Suppose, you wish to find the files that have the text ‘eth’ in them, you can use ‘M-x find-grep-dired’, which will prompt you in the minibuffer with the message “Find-grep (directory): /tmp/test”. After you press the return key, it will prompt you for the regex with the message “Find-grep (grep regexp):”, where you can input ‘eth’ and this will return the following results:

/tmp/test/:
find . \( -type f -exec grep -q -e eth \{\} \; \) -ls
6826605    4 -rw-rw-r--   1 shakthi  shakthi      1620 Jul  5 09:46 network/ifconfig.output
6826616   64 -rw-rw-r--   1 shakthi  shakthi     63050 Jul  5 09:46 kernel/dmesg.txt

find finished at Sun Jul  5 10:31:27

You can also execute a find operation on marked files using M-x find-dired.

Help

To open the help menu for Dired, you can use the h shortcut key. To see the Dired log messages, you can press the ? key.

You may refer to a quick reference card at https://www.gnu.org/software/emacs/refcards/pdf/dired-ref.pdf, and try out more Dired commands.

Introduction

GNU Emacs is a very popular text editor written in C and Emacs Lisp. It can be run on many platforms, and can be easily customised and extended for user needs. It was created by Richard Stallman, the founder of the GNU project. This article is the first in a series on how to use it. There are a number of tutorials available for GNU Emacs; this series of articles provide one approach to understanding and learning the software. You are encouraged to refer to the official GNU Emacs reference manual for more information, and it supersedes everything. Here are a few interesting quotes on Emacs.

“I use Emacs, which might be thought of as a thermonuclear word processor” – Neal Stephenson (In the Beginning… Was the Command Line)

“Emacs is undoubtedly the most powerful programmer’s editor in existence. It’s a big, feature-laden program with a great deal of flexibility and customizability. … Emacs has an entire programming language inside it that can be used to write arbitrarily powerful editor functions.” – Eric S. Raymond (from ‘The Art of UNIX Programming’)

“Personally, I feel inspired whenever I open Emacs. Like a craftsman entering his workshop, I feel a realm of possibility open before me. I feel the comfort of an environment that has evolved over time to fit me perfectly - an assortment of packages and keybindings which help me bring ideas to life day after day.” – Daniel Higginbotham (in ‘Clojure for the Brave and True’)

“EMACS could not have been reached by a process of careful design, because such processes arrive only at goals which are visible at the outset, and whose desirability is established on the bottom line at the outset. Neither I nor anyone else visualized an extensible editor until I had made one, nor appreciated its value until he had experienced it. EMACS exists because I felt free to make individually useful small improvements on a path whose end was not in sight.” – Richard Stallman

Installation

You can use your favourite GNU/Linux distribution package manager to install GNU Emacs. On Debian/Ubuntu, you can install with the following command:

$ sudo apt-get install emacs

On Fedora, you can use the Yum package manager as shown below:

$ sudo yum install emacs

The emerge tool can install GNU Emacs on Gentoo, as follows:

# emerge --ask app-editors/emacs

On Arch, the Pacman software can help you in installing GNU Emacs:

$ sudo pacman -S emacs

Use the Zypper package manager in SUSE as shown below:

$ sudo zypper install emacs

Startup

If you use the Unity interface, you can search for emacs in the Dash, and it will show the Emacs icon that you can click to open the editor. This is illustrated in the following figure.

On the Metacity interface or any other desktop environment with a panel, you can open GNU Emacs from “Applications” -> “Accessories” -> “Emacs” as shown below:

You can also open the editor from the terminal by simply typing ‘emacs’ and hitting the return key.

$ emacs

The version that I have used for this article is GNU Emacs 24.3.1.

Exit

To exit from GNU Emacs, you need to use C-c C-q, where ‘C’ stands for the Control key. You can also use your mouse and close the editor by clicking on the ‘x’, but, GNU Emacs was designed to be completely usable with a keyboard, and I am going to encourage you to only use the keyboard shortcuts.

Concepts

While you can work using GUI editors, I am going to teach you to work entirely on the keyboard to experience the power of shortcuts in GNU Emacs. You can disconnect the mouse and your touchpad when working on GNU Emacs. Just as an exercise, I’d encourage you to remove your mouse completely and see how you can work with a computer for one day. You will realise that a lot of user interfaces are heavily dependent and designed for mouse interactions! By only using the keyboard, you can be blazingly fast and productive.

The keyboard shortcuts in GNU Emacs may seem to involve many keys. But, please bear with me on this, because the way the shortcuts are designed, you will be able to remember them easily. As you practice, you will gain insight into how consistently they have been defined.

GNU Emacs is a ‘stateless’ editor for the most part. By ‘stateless’, I mean that there are no specific state transitions that need to happen before you can use the commands. There does exist the concept of modes. When you open GNU Emacs, you will see menus, buffer and a mode line as illustrated in the following figure:

As mentioned earlier, we will not be clicking on the menus or icons with a mouse, but only use keyboard shortcuts. Everything is a buffer in GNU Emacs. Each buffer can have one major mode and one or more minor modes. The mode determines the keyboard shortcuts that are applicable primarily on the buffer. Examples of major modes are given below:

The mode line exists below the buffer and it gives you a lot of information on the status of the buffer, such as what modes are active and other useful information. Below the mode line is the mini-buffer, where any commands you issue are indicated and prompts for user input are shown. This is the overall view of the default GNU Emacs interface.

In today’s user interface applications, an application is treated as a window on a desktop. But, when you open GNU Emacs, you are actually opening a frame. A frame can be split it into many windows. Everything is a buffer in GNU Emacs. So, you can have many frames of GNU Emacs, and inside each you can have one or more windows containing buffers (or files).

Features

Although GNU Emacs was designed primarily for text editing, it can do a lot more. ‘A lot’ is probably a highly simplified term. It has got support for syntax highlighting for a large number of programming languages. You can also generate code snippets from templates using the yasnippet package. You can also enter markup or markdown text. There is support for indentation of text and programs depending on the programming languages and modes. Internationalization support is available, and you can use it to even enter text in Indian languages.

A number of configurations are available for setting it up for your development work, including automating tasks for compiling, executing, testing and deployment. When you become familiar with Emacs Lisp, you can implement your own modules. Since, it is Lisp, it is also easily extensible. You can write your own macros to perform repeated tasks. You can also query an inbuilt help system for information, shortcuts, tutorials and other documentation. You are not at all dependent on the Internet for information, and thus you can work offline too. Version control support is available for many centralized (cvs, svn) and decentralized systems (Git, Hg).

org-mode is a very popular mode for managing your notes. You can use it for planning your day-to-day activities. GNU Emacs can be used as publishing software. You can create wikis, blogs and publish books using it. This article is written using org-mode. Spell-checking modules are also available for your documentation needs. It is also possible to export plain text into a number of formats (PDF, HTML etc.).

A number of Emacs lisp packages are available for networking. You can use Gnus for checking your e-mails, and reading and writing to newsgroups. Emacs Relay Chat (ERC) can be used for connecting to Internet Relay Chat (IRC) channels. There are modules that support the Jabber protocol for communicating with chat servers. There is also support for viewing Web pages inside Emacs. There are a large number of Emacs modules available through package repositories such as MELPA (melpa.org), and Marmalada (marmalade-repo.org).

History

The first version of Emacs (macros) was written by Richard Stallman and Guy L. Steele, Jr. in 1976 for the TECO editor in MIT. It was written in a low-level language for the PDP-10 assembler. People were able to freely hack on the code, make improvements and share their changes. This was the original hacker culture that existed in MIT. Unfortunately, business entities started to make software proprietary and this hacker culture ceased to exist, especially in the MIT AI labs. Richard Stallman wanted to revive the hacker culture and started the GNU project. He wrote the second implementation of the editor entirely in C in 1984, and released it as the first program of the GNU project. Today, it is a very popular editor that is widely used, and has more than 570 contributors. The official web site for GNU Emacs is at http://www.gnu.org/software/emacs/.

In this final article in the Haskell series, we shall explore how to use it for web programming.

Scotty is a web framework written in Haskell, which is similar to Ruby’s Sinatra. You can install it on Ubuntu using the following commands:

$ sudo apt-get install cabal-install
$ cabal update
$ cabal install scotty

Let us write a simple `Hello, World!’ program using the Scotty framework:

-- hello-world.hs

{-# LANGUAGE OverloadedStrings #-}

import Web.Scotty

main :: IO ()
main = scotty 3000 $ do
  get "/" $ do
    html "Hello, World!"

You can compile and start the server from the terminal using the following command:

$ runghc hello-world.hs 
Setting phasers to stun... (port 3000) (ctrl-c to quit)

The service will run on port 3000, and you can open localhost:3000 in a browser to see the `Hello, World!’ text. You can then stop the service by pressing Control-c in the terminal. You can also use Curl to make a query to the server. Install and test it on Ubuntu as shown below:

$ sudo apt-get install curl

$ curl localhost:3000
Hello, World!

You can identify the user client that made the HTTP request to the server by returning the “User-Agent” header value as illustrated in the following example:

-- request-header.hs

{-# LANGUAGE OverloadedStrings #-}

import Web.Scotty

main :: IO ()
main = scotty 3000 $ do
  get "/agent" $ do
    agent <- header "User-Agent"
    maybe (raise "User-Agent header not found!") text agent

You can execute the above code in a terminal using the following command:

$ runghc request-header.hs  
Setting phasers to stun... (port 3000) (ctrl-c to quit)

If you open the URL localhost:3000/agent in the browser, it returns the following User-Agent information on Ubuntu 14.10 Mozilla/5.0 (X11; Linux x86₆₄) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/41.0.2272.76 Chrome/41.0.2272.76 Safari/537.36. The Curl version is returned for the same URL request as shown below:

$ curl localhost:3000/agent -v

 * Hostname was NOT found in DNS cache
 *   Trying 127.0.0.1...
 * Connected to localhost (127.0.0.1) port 3000 (#0)
 > GET /agent HTTP/1.1
 > User-Agent: curl/7.37.1
 > Host: localhost:3000
 > Accept: */*
 > 
 < HTTP/1.1 200 OK
 < Transfer-Encoding: chunked
 < Date: Wed, 29 Apr 2015 07:46:21 GMT
 * Server Warp/3.0.12.1 is not blacklisted
 < Server: Warp/3.0.12.1
 < Content-Type: text/plain; charset=utf-8
 < 
 * Connection #0 to host localhost left intact

 curl/7.37.1

You can also return different content types (HTML, text, JSON) based on the request. For example:

-- content-type.hs

{-# LANGUAGE OverloadedStrings #-}

import Web.Scotty as W
import Data.Monoid
import Data.Text
import Data.Aeson

main :: IO ()
main = scotty 3000 $ do
  get "/hello" $ do
    html $ mconcat ["<h1>", "Hello, World!", "</h1>"]

  get "/hello.txt" $ do
    text "Hello, World!"

  get "/hello.json" $ do
    W.json $ object ["text" .= ("Hello, World!" :: Text)]

You can start the above server in a terminal as follows:

$ runghc content-type.hs 
Setting phasers to stun... (port 3000) (ctrl-c to quit)

You can then open the three URLs listed above in a browser to see the different output. The respective outputs when used with Curl are shown below:

$ curl localhost:3000/hello
<h1>Hello, World!</h1>

$ curl localhost:3000/hello.txt
Hello, World!

$ curl localhost:3000/hello.json
{"text":"Hello, World!"}

You can also pass parameters in the URL when you make a request. The param function can be used to retrieve the parameters as indicated below:

-- params.hs

{-# LANGUAGE OverloadedStrings #-}

import Web.Scotty
import Data.Monoid

main :: IO ()
main = scotty 3000 $ do
  get "/user" $ do
    name <- param "name"
    html $ mconcat ["<h1>Hello ", name, "</h1>"]

You can start the above server using the runghc command:

$ runghc params.hs 
Setting phasers to stun... (port 3000) (ctrl-c to quit)

You can now try the URL requests with and without parameters. The observed outputs are shown below:

$ curl localhost:3000/user
<h1>500 Internal Server Error</h1>Param: name not found!

$ curl localhost:3000/user?name=Shakthi
<h1>Hello Shakthi</h1>

The Hspec testing framework can be used for integration testing the web application. Install the required dependencies as shown below:

$ cabal install happy hspec hspec-wai hspec-wai-json

The content type example has been updated to use Hspec, as illustrated below:

-- content-type-spec.hs

{-# LANGUAGE OverloadedStrings, QuasiQuotes #-}
module Main (main) where

import Data.Monoid
import Data.Text

import           Network.Wai (Application)
import qualified Web.Scotty as W
import           Data.Aeson (object, (.=))

import           Test.Hspec
import           Test.Hspec.Wai
import           Test.Hspec.Wai.JSON

main :: IO ()
main = hspec spec

app :: IO Application
app = W.scottyApp $ do
  W.get "/hello.txt" $ do
    W.text "Hello, World!"

  W.get "/hello" $ do
    W.html $ mconcat ["<h1>", "Hello, World!", "</h1>"]

  W.get "/hello.json" $ do
    W.json $ object ["text" .= ("Hello, World!" :: Text)]

spec :: Spec
spec = with app $ do
  describe "GET /" $ do
    it "responds with text" $ do
      get "/hello.txt" `shouldRespondWith` "Hello, World!"

    it "responds with HTML" $ do
      get "/hello" `shouldRespondWith` "<h1>Hello, World!</h1>"

    it "responds with JSON" $ do
      get "/hello.json" `shouldRespondWith` [json|{text: "Hello, World!"}|]

You can compile the above code as shown below:

$ ghc --make content-type-spec.hs     
Linking content-type-spec ...

The following output is observed when you run the above built test executable:

$ ./content-type-spec 

GET /
  responds with text
  responds with HTML
  responds with JSON

Finished in 0.0010 seconds
3 examples, 0 failures

Please refer to the hspec-wai webpage at https://github.com/hspec/hspec-wai for more information.

Template support is available through many Haskell packages. The use of the blaze-html package is demonstrated below. Install the package first using the following command:

$ cabal install blaze-html

Consider a simple web page with a header and three unordered lists. Using blaze-html, the template can be written in Haskell DSL as follows:

-- template.hs

{-# LANGUAGE OverloadedStrings #-}

import Web.Scotty as W
import Text.Blaze.Html5
import Text.Blaze.Html.Renderer.Text

main :: IO ()
main = scotty 3000 $ do
  get "/" $ do
    W.html . renderHtml $ do
      h1 "Haskell list"
      ul $ do
        li "http://haskell.org"
        li "http://learnyouahaskell.com/"
        li "http://book.realworldhaskell.org/"

You can compile the above code using GHC:

$ ghc --make template.hs 
Linking template ...

You can then execute the built executable, which starts the server as shown below:

$ ./template 
Setting phasers to stun... (port 3000) (ctrl-c to quit)

Opening a browser with URL localhost:3000 will render the expected HTML file. You can also verify the resultant HTML output using the Curl command as shown below:

$ curl localhost:3000
<h1>Haskell list</h1><ul><li>http://haskell.org</li><li>http://learnyouahaskell.com/</li><li>http://book.realworldhaskell.org/</li></ul>

It is good to separate the views from the actual application code. You can move the template content to a separate file as shown below:

-- Haskell.hs

{-# LANGUAGE OverloadedStrings #-}

module Haskell where

import Text.Blaze.Html5

render :: Html
render = do
  html $ do
    body $ do
      h1 "Haskell list"
      ul $ do
        li "http://haskell.org"
        li "http://learnyouahaskell.com/"
        li "http://book.realworldhaskell.org/"

The main application code is now simplified as shown below:

-- template-file.hs

{-# LANGUAGE OverloadedStrings #-}

import qualified Haskell
import Web.Scotty as W
import Text.Blaze.Html
import Text.Blaze.Html.Renderer.Text

blaze :: Text.Blaze.Html.Html -> ActionM ()
blaze = W.html . renderHtml

main :: IO ()
main = scotty 3000 $ do
  get "/" $ do
    blaze Haskell.render

You need to place both the source files (Haskell.hs and template-file.hs) in the same top-level directory, and you can then compile the template-file.hs file that will also compile the dependency Haskell.hs source file as shown below:

$ ghc --make template-file.hs 

[1 of 2] Compiling Haskell          ( Haskell.hs, Haskell.o )
[2 of 2] Compiling Main             ( template-file.hs, template-file.o )
Linking template-file ...

You can now run the server as follows:

$ ./template-file 
Setting phasers to stun... (port 3000) (ctrl-c to quit)

Executing template-file produces the same output as in the case of the template.hs example.

$ curl localhost:3000
<html><body><h1>Haskell list</h1><ul><li>http://haskell.org</li><li>http://learnyouahaskell.com/</li><li>http://book.realworldhaskell.org/</li></ul></body></html>

You can refer the Scotty wiki page at https://github.com/scotty-web/scotty/wiki for more information.

The clay package is a CSS preprocessor similar to LESS and Sass. You can install it using the following Cabal command:

    $ cabal install clay

Let us consider a simple CSS example to generate a list of fonts to be used in the body section of a HTML page. The corresponding Clay Haskell embedded DSL looks like the following:

-- clay-simple.hs

{-# LANGUAGE OverloadedStrings #-}

import Clay

main :: IO ()
main = putCss exampleStylesheet

exampleStylesheet :: Css
exampleStylesheet = body ? fontFamily ["Baskerville", "Georgia", "Garamond", "Times"] [serif]

You can compile the above code as follows:

$ ghc --make clay-simple.hs 
[1 of 1] Compiling Main             ( clay-simple.hs, clay-simple.o )
Linking clay-simple ...

You can then execute clay-simple to generate the required CSS output as shown below:

$ ./clay-simple

body
{
  font-family : "Baskerville","Georgia","Garamond","Times", serif;
}

/* Generated with Clay, http://fvisser.nl/clay */

A more comprehensive example is shown below for the HTML pre tag:

-- clay-pre.hs

{-# LANGUAGE OverloadedStrings #-}

import Clay

main :: IO ()
main = putCss $
  pre ?
    do border dotted (pt 1) black
       whiteSpace (other "pre")
       fontSize (other "8pt")
       overflow (other "auto")
       padding (em 20) (em 0) (em 20) (em 0)

You can compile the above clay-pre.hs file as shown below:

$ ghc --make clay-pre.hs 
[1 of 1] Compiling Main             ( clay-pre.hs, clay-pre.o )
Linking clay-pre ...

Executing the above complied clay-pre binary produces the following output:

$ ./clay-pre 

pre
{
  border      : dotted 1pt rgb(0,0,0);
  white-space : pre;
  font-size   : 8pt;
  overflow    : auto;
  padding     : 20em 0em 20em 0em;
}

/* Generated with Clay, http://fvisser.nl/clay */

You can also add custom values using the Other type class or the fallback operator `-:’ to explicitly specify values. For example:

-- clay-custom.hs

{-# LANGUAGE OverloadedStrings #-}

import Clay

main :: IO ()
main = putCss $
  body ?
       do fontSize (other "11pt !important")
          "border" -: "0"

Compiling and executing the above code produces the following output:

$ ghc --make clay-custom.hs 
[1 of 1] Compiling Main             ( clay-custom.hs, clay-custom.o )
Linking clay-custom ...

$ ./clay-custom 

body
{
  font-size : 11pt !important;
  border    : 0;
}

/* Generated with Clay, http://fvisser.nl/clay */

You can explore more of clay from the official project homepage http://fvisser.nl/clay/.

A number of good books are available for further learning. I recommend the following books available online and in print:

1. Bryan O’Sullivan, Don Stewart, and John Goerzen. (December 1, 2008). Real World Haskell (http://book.realworldhaskell.org/). O’Reilly.

2. Miran Lipovaca. (April 21, 2011). Learn You a Haskell for Great Good! A Beginner’s Guide (http://learnyouahaskell.com/). No Starch Press.

The https://www.haskell.org website also has plenty of useful resources. You can also join the haskell-cafe@haskell.org and beginners@haskell.org mailing lists ( https://wiki.haskell.org/Mailing_lists ) for discussions. The folks in the #haskell channel on irc.freenode.net are also very helpful.

I hope you enjoyed learning Haskell through this series, as much as I did creating them. Please feel free to write to me (author at shakthimaan dot com) with any feedback or suggestions.

Happy Hacking!

In this article we shall explore network programming in Haskell.

Let us begin with a simple TCP (Transmission Control Protocol) client and server example. The network package provides a high-level interface for communication. You can install the same in Fedora, for example, using the following command:

$ sudo yum install ghc-network

Consider the following simple TCP client code:

-- tcp-client.hs

import Network
import System.IO

main :: IO ()
main = withSocketsDo $ do
         handle <- connectTo "localhost" (PortNumber 3001)
         hPutStr handle "Hello, world!"
         hClose handle

After importing the required libraries, the main function connects to a localhost server running on port 3001, sends a string “Hello, world!”, and closes the connection.

The connectTo function defined in the Network module accepts a hostname, port number and returns a handle that can be used to transfer or receive data.

The type signatures of the withSocketsdo and connectTo functions are as under:

ghci> :t withSocketsDo
withSocketsDo :: IO a -> IO a

ghci> :t connectTo
connectTo :: HostName -> PortID -> IO GHC.IO.Handle.Types.Handle

The simple TCP server code is illustrated below:

-- tcp-server.hs

import Network
import System.IO

main :: IO ()
main = withSocketsDo $ do
         sock <- listenOn $ PortNumber 3001
         putStrLn "Starting server ..."
         handleConnections sock

handleConnections :: Socket -> IO ()
handleConnections sock = do
  (handle, host, port) <- accept sock
  output <- hGetLine handle
  putStrLn output
  handleConnections sock

The main function starts a server on port 3001 and transfers the socket handler to a handleConnections function. It accepts any connection requests, reads the data, prints it to the server log, and waits for more clients.

Firstly, you need to compile the tcp-server.hs and tcp-client.hs files using GHC:

$ ghc --make tcp-server.hs
[1 of 1] Compiling Main             ( tcp-server.hs, tcp-server.o )
Linking tcp-server ...

$ ghc --make tcp-client.hs 
[1 of 1] Compiling Main             ( tcp-client.hs, tcp-client.o )
Linking tcp-client ...

You can now start the TCP server in a terminal:

$ ./tcp-server 
Starting server ...

You can then run the TCP client in another terminal:

$ ./tcp-client

You will now observe the “Hello, world” message printed in the terminal where the server is running:

$ ./tcp-server 
Starting server ...
Hello, world!

The Network.Socket package exposes more low-level socket functionality for Haskell and can be used if you need finer access and control. For example, consider the following UDP (User Datagram Protocol) client code:

-- udp-client.hs

import Network.Socket

main :: IO ()
main = withSocketsDo $ do
         (server:_) <- getAddrInfo Nothing (Just "localhost") (Just "3000")
         s <- socket (addrFamily server) Datagram defaultProtocol
         connect s (addrAddress server)
         send s "Hello, world!"
         sClose s

The getAddrInfo function resolves a host or service name to a network address. A UDP client connection is then requested for the server address, a message is sent, and the connection is closed. The type signatures of getAddrInfo, addrFamily, and addrAddress are given below:

ghci> :t getAddrInfo
getAddrInfo
  :: Maybe AddrInfo
     -> Maybe HostName -> Maybe ServiceName -> IO [AddrInfo]

ghci> :t addrFamily
addrFamily :: AddrInfo -> Family

ghci> :t addrAddress
addrAddress :: AddrInfo -> SockAddr

The corresponding UDP server code is as follows:

-- udp-server.hs

import Network.Socket

main :: IO ()
main = withSocketsDo $ do
         (server:_) <- getAddrInfo Nothing (Just "localhost") (Just "3000")
         s <- socket (addrFamily server) Datagram defaultProtocol
         bindSocket s (addrAddress server) >> return s
         putStrLn "Server started ..."
         handleConnections s

handleConnections :: Socket -> IO ()
handleConnections conn = do
  (text, _, _) <- recvFrom conn 1024
  putStrLn text
  handleConnections conn

The UDP server binds to localhost and starts to listen on port 3000. When a client connects, it reads a maximum of 1024 bytes of data, prints it to stdout, and waits to accept more connections. You can compile the udp-server.hs and udp-client.hs files using the following commands:

$ ghc --make udp-server.hs 
[1 of 1] Compiling Main             ( udp-server.hs, udp-server.o )
Linking udp-server ...

$ ghc --make udp-client.hs 
[1 of 1] Compiling Main             ( udp-client.hs, udp-client.o )
Linking udp-client ...

You can start the UDP server in one terminal:

$ ./udp-server 
Server started ...

You can then run the UDP client in another terminal:

$ ./tcp-client

You will now see the “Hello, world!” message printed in the terminal where the server is running:

$ ./udp-server 
Server started ...
Hello, world!

The network-uri module has many useful URI (Uniform Resource Identifier) parsing and test functions. You can install the same on Fedora using the following command:

$ cabal install network-uri

The parseURI function takes a string and attempts to convert it into a URI. It returns ‘Nothing’ if the input is not a valid URI, and returns the URI, otherwise. For example:

ghci> :m + Network.URI

ghci> parseURI "http://www.shakthimaan.com"
Just http://www.shakthimaan.com

ghci> parseURI "shakthimaan.com"
Nothing

The type signature of the parseURI function is given below:

ghci> :t parseURI
parseURI :: String -> Maybe URI

A number of functions are available for testing the input URI as illustrated in the following examples:

ghci> isURI "shakthimaan.com"
False

ghci> isURI "http://www.shakthimaan.com"
True

ghci> isRelativeReference "http://shakthimaan.com"
False

ghci> isRelativeReference "../about.html"
True

ghci> isAbsoluteURI "http://www.shakthimaan.com"
True

ghci> isAbsoluteURI "shakthimaan.com"
False

ghci> isIPv4address "192.168.100.2"
True

ghci> isIPv6address "2001:0db8:0a0b:12f0:0000:0000:0000:0001"
True

ghci> isIPv6address "192.168.100.2"
False

ghci> isIPv4address "2001:0db8:0a0b:12f0:0000:0000:0000:0001"
False

The type signatures of the above functions are as follows:

ghci> :t isURI
isURI :: String -> Bool

ghci> :t isRelativeReference
isRelativeReference :: String -> Bool

ghci> :t isAbsoluteURI
isAbsoluteURI :: String -> Bool

ghci> :t isIPv4address
isIPv4address :: String -> Bool

ghci> :t isIPv6address
isIPv6address :: String -> Bool

You can make a GET request for a URL and retrieve its contents. For example:

import Network
import System.IO

main = withSocketsDo $ do
    h <- connectTo "www.shakthimaan.com" (PortNumber 80)
    hSetBuffering h LineBuffering
    hPutStr h "GET / HTTP/1.1\nhost: www.shakthimaan.com\n\n"
    contents <- hGetContents h
    putStrLn contents
    hClose h

You can now compile and execute the above code, and it returns the index.html contents as shown below:

$ ghc --make get-network-uri.hs
[1 of 1] Compiling Main             ( get-network-uri.hs, get-network-uri.o )
Linking get-network-uri ...

$ ./get-network-uri 
HTTP/1.1 200 OK
Date: Sun, 05 Apr 2015 01:37:19 GMT
Server: Apache
Last-Modified: Tue, 08 Jul 2014 04:01:16 GMT
Accept-Ranges: bytes
Content-Length: 4604
Content-Type: text/html
...

You can refer to the network-uri package documentation at https://hackage.haskell.org/package/network-uri-2.6.0.1/docs/Network-URI.html for more detailed information.

The whois Haskell package allows you to query for information about hosting servers and domain names. You can install the package on Ubuntu, for example, using:

$ cabal install whois

The serverFor function returns a whois server that can be queried for obtaining more information regarding an IP or domain name. For example:

ghci> :m + Network.Whois

ghci> serverFor "shakthimaan.com"
Loading package array-0.4.0.1 ... linking ... done.
Loading package deepseq-1.3.0.1 ... linking ... done.
Loading package bytestring-0.10.0.2 ... linking ... done.
Loading package old-locale-1.0.0.5 ... linking ... done.
Loading package time-1.4.0.1 ... linking ... done.
Loading package unix-2.6.0.1 ... linking ... done.
Loading package network-2.6.0.2 ... linking ... done.
Loading package transformers-0.4.3.0 ... linking ... done.
Loading package mtl-2.2.1 ... linking ... done.
Loading package text-1.2.0.4 ... linking ... done.
Loading package parsec-3.1.9 ... linking ... done.
Loading package network-uri-2.6.0.1 ... linking ... done.
Loading package split-0.2.2 ... linking ... done.
Loading package whois-1.2.2 ... linking ... done.

Just (WhoisServer {hostname = "com.whois-servers.net", port = 43, query = "domain "})

You can use the above specific information with the whois1 function to make a DNS (Domain Name System) query:

ghci> whois1 "shakthimaan.com" WhoisServer {hostname = "com.whois-servers.net", port = 43, query = "domain "}
Just "\nWhois Server Version 2.0\n\nDomain names in the .com and .net domains can now be registered\n
...

You can also use the whois function to return information on the server as shown below:

ghci> whois "shakthimaan.com"
Just "\nWhois Server Version 2.0\n\nDomain names in the .com and .net domains can now be registered\n
...

The type signatures of serverFor, whois1 and whois functions are as follows:

ghc> :t serverFor
serverFor :: String -> Maybe WhoisServer

ghci> :t whois1
whois1 :: String -> WhoisServer -> IO (Maybe String)

ghci> :t whois
whois :: String -> IO (Maybe String, Maybe String)

The dns package provides a number of useful functions to make Domain Name System queries, and handle the responses. You can install the same on Ubuntu, for example, using the following commands:

$ sudo apt-get install zlib1g-dev
$ cabal install dns

A simple example of finding the IP addresses for the haskell.org domain is shown below:

ghci> import Network.DNS.Lookup
ghci> import Network.DNS.Resolver

ghci> let hostname = Data.ByteString.Char8.pack "www.haskell.org"

ghci> rs <- makeResolvSeed defaultResolvConf

ghci> withResolver rs $ \resolver -> lookupA resolver hostname
Right [108.162.203.60,108.162.204.60]

The defaultResolvConf is of type ResolvConf and consists of the following default values:

--     * 'resolvInfo' is 'RCFilePath' \"\/etc\/resolv.conf\".
--
--     * 'resolvTimeout' is 3,000,000 micro seconds.
--
--     * 'resolvRetry' is 3.

The makeResolvSeed, and withResolver functions assist in making the actual DNS resolution. The lookupA function obtains all the A records for the DNS entry. Their type signatures are shown below:

ghci> :t makeResolvSeed
makeResolvSeed :: ResolvConf -> IO ResolvSeed

ghci> :t withResolver
withResolver :: ResolvSeed -> (Resolver -> IO a) -> IO a

ghci> :t lookupA
lookupA
  :: Resolver
     -> dns-1.4.5:Network.DNS.Internal.Domain
     -> IO
          (Either
             dns-1.4.5:Network.DNS.Internal.DNSError
             [iproute-1.4.0:Data.IP.Addr.IPv4])

The lookupAAAA function returns all the IPv6 ‘AAAA’ records for the domain. For example:

ghci> withResolver rs $ \resolver -> lookupAAAA resolver hostname
Right [2400:cb00:2048:1::6ca2:cc3c,2400:cb00:2048:1::6ca2:cb3c]

Its type signature is shown below:

lookupAAAA
  :: Resolver
     -> dns-1.4.5:Network.DNS.Internal.Domain
     -> IO
          (Either
             dns-1.4.5:Network.DNS.Internal.DNSError
             [iproute-1.4.0:Data.IP.Addr.IPv6])

The MX records for the hostname can be returned using the lookupMX function. An example for the shakthimaan.com website is as follows:

ghci> import Network.DNS.Lookup
ghci> import Network.DNS.Resolver

ghci> let hostname = Data.ByteString.Char8.pack "www.shakthimaan.com"

ghci> rs <- makeResolvSeed defaultResolvConf

ghci> withResolver rs $ \resolver -> lookupMX resolver hostname
Right [("shakthimaan.com.",0)]

The type signature of the lookupMX function is as under:

ghci> :t lookupMX
lookupMX
  :: Resolver
     -> dns-1.4.5:Network.DNS.Internal.Domain
     -> IO
          (Either
             dns-1.4.5:Network.DNS.Internal.DNSError
             [(dns-1.4.5:Network.DNS.Internal.Domain, Int)])

The nameservers for the domain can be returned using the lookupNS function. For example:

ghci> withResolver rs $ \resolver -> lookupNS resolver hostname
Right ["ns22.webhostfreaks.com.","ns21.webhostfreaks.com."]

The type signature of the lookupNS function is shown below:

ghci> :t lookupNS
lookupNS
  :: Resolver
     -> dns-1.4.5:Network.DNS.Internal.Domain
     -> IO
          (Either
             dns-1.4.5:Network.DNS.Internal.DNSError
             [dns-1.4.5:Network.DNS.Internal.Domain])

You can also return the entire DNS response using the lookupRaw function as illustrated below:

ghci> :m + Network.DNS.Types

ghci> let hostname = Data.ByteString.Char8.pack "www.ubuntu.com"

ghci> rs <- makeResolvSeed defaultResolvConf

ghci> withResolver rs $ \resolver -> lookupRaw resolver hostname A
Right (DNSFormat 
  {header = DNSHeader 
    {identifier = 29504, 
     flags = DNSFlags 
       {qOrR = QR_Response, 
        opcode = OP_STD, 
        authAnswer = False, 
        trunCation = False, 
        recDesired = True, 
        recAvailable = True, 
        rcode = NoErr}, 
     qdCount = 1, 
     anCount = 1, 
     nsCount = 3, 
     arCount = 3}, 
   question = [
     Question 
       {qname = "www.ubuntu.com.", 
        qtype = A}], 
     answer = [
       ResourceRecord 
         {rrname = "www.ubuntu.com.", 
          rrtype = A, 
          rrttl = 61, 
          rdlen = 4, 
          rdata = 91.189.89.103}], 
     authority = [
       ResourceRecord 
         {rrname = "ubuntu.com.", 
          rrtype = NS, 
          rrttl = 141593, 
          rdlen = 16, 
          rdata = ns2.canonical.com.},
       ResourceRecord 
         {rrname = "ubuntu.com.", 
          rrtype = NS, 
          rrttl = 141593, 
          rdlen = 6, 
          rdata = ns1.canonical.com.},
       ResourceRecord
         {rrname = "ubuntu.com.", 
          rrtype = NS, 
          rrttl = 141593, 
          rdlen = 6, 
          rdata = ns3.canonical.com.}], 
    additional = [
      ResourceRecord 
        {rrname = "ns2.canonical.com.", 
         rrtype = A, 
         rrttl = 88683, 
         rdlen = 4, 
         rdata = 91.189.95.3},
      ResourceRecord 
        {rrname = "ns3.canonical.com.", 
         rrtype = A, 
         rrttl = 88683, 
         rdlen = 4, 
         rdata = 91.189.91.139},
      ResourceRecord 
        {rrname = "ns1.canonical.com.", 
         rrtype = A, 
         rrttl = 88683, 
         rdlen = 4, 
         rdata = 91.189.94.173}]})

Please refer the Network.DNS hackage web page https://hackage.haskell.org/package/dns for more information.

In this article we shall explore access to Redis and PostgreSQL databases using Haskell modules. The hackage website at https://hackage.haskell.org/packages/#cat:Database provides a vast number of database packages that you can use, a couple of which will be covered here..

You will need to install the cabal-install tool on Fedora, for example, using the following command:

$ sudo yum install cabal-install

• Connecting to the Redis database

Let’s use the hedis package to connect to the Redis server. Install the Fedora dependency package alex, and the Redis server as shown below:

$ sudo yum install alex redis

You can then install the hedis package using the following commands:

$ cabal update
$ cabal install hedis

This installs the latest hedis version 0.6.5. You can now start the Redis server on Fedora using the service command:

$ sudo service redis start

You can then test connectivity to the Redis server using the redis-cli command by issuing the PING command as follows:

$ redis-cli

127.0.0.1:6379> PING
PONG

You can also test the same using the hedis package inside the GHCi interpreter as illustrated below:

$ ghci
GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.

ghci> :m Database.Redis 

ghci> conn <- connect defaultConnectInfo

Loading package array-0.4.0.1 ... linking ... done.
Loading package base-unicode-symbols-0.2.2.4 ... linking ... done.
Loading package deepseq-1.3.0.1 ... linking ... done.
Loading package old-locale-1.0.0.5 ... linking ... done.
Loading package time-1.4.0.1 ... linking ... done.
Loading package transformers-0.3.0.0 ... linking ... done.
Loading package bytestring-0.10.0.2 ... linking ... done.
Loading package text-0.11.3.1 ... linking ... done.
Loading package stm-2.4.2 ... linking ... done.
Loading package primitive-0.5.0.1 ... linking ... done.
Loading package vector-0.10.0.1 ... linking ... done.
Loading package hashable-1.1.2.5 ... linking ... done.
Loading package transformers-base-0.4.1 ... linking ... done.
Loading package monad-control-0.3.2.1 ... linking ... done.
Loading package containers-0.5.0.0 ... linking ... done.
Loading package attoparsec-0.10.4.0 ... linking ... done.
Loading package mtl-2.1.2 ... linking ... done.
Loading package BoundedChan-1.0.3.0 ... linking ... done.
Loading package bytestring-lexing-0.4.3.2 ... linking ... done.
Loading package unix-2.6.0.1 ... linking ... done.
Loading package network-2.6.0.2 ... linking ... done.
Loading package resource-pool-0.2.3.2 ... linking ... done.
Loading package hedis-0.6.5 ... linking ... done.

ghci> runRedis conn ping
Right Pong

It is recommended that you use defaultConnectInfo to connect to the database, and its type is ConnectInfo:

ghci> :t defaultConnectInfo
defaultConnectInfo :: ConnectInfo

The different options that can be used in defaultConnectInfo are as follows:

connectHost           = "localhost"
connectPort           = PortNumber 6379 -- Redis port
connectAuth           = Nothing         -- No authentication
connectDatabase       = 0               -- SELECT database 0
connectMaxConnections = 10              -- Up to 10 connections
connectMaxIdleTime    = 20              -- Keep connection open for 20 seconds

The types of conn, connect, runRedis and ping are given below:

ghci> :t conn
conn :: Connection

ghci> :t connect
connect :: ConnectInfo -> IO Connection

ghci> :t runRedis
runRedis :: Connection -> Redis a -> IO a

ghci> :t ping
ping :: RedisCtx m f => m (f Status)

If the Redis server was not started, and you tried to issue the ping command, the following exception will be automatically thrown by the package:

ghci> runRedis conn ping
*** Exception: connect: does not exist (No route to host)

You can automate the above code snippets into Haskell code with a main function as demonstrated below:

{-# LANGUAGE OverloadedStrings #-}
import Database.Redis

main :: IO (Either Reply Status)
main = do
  conn <- connect defaultConnectInfo
  runRedis conn ping

The OverloadedStrings extension allows string literals to be polymorphic for the IsString class. You can compile and run the above code inside GHCi, as follows:

$ ghci ping.hs

GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( ping.hs, interpreted )
Ok, modules loaded: Main.

ghci> main
...
Right Pong

The echo Redis command is used to print a message that is passed as an argument to it. The equivalent hedis echo command expects the message to be of type ByteString. For example:

{-# LANGUAGE OverloadedStrings #-}
import Database.Redis
import qualified Data.ByteString as B

bytes :: B.ByteString
bytes = "Hello, World" :: B.ByteString

main :: IO (Either Reply B.ByteString)
main = do
  conn <- connect defaultConnectInfo
  runRedis conn $ echo bytes

Loading the above code in GHCi produces the following output:

ghci> main
Right "Hello, World"

The type signature of the echo function is as follows:

echo
  :: RedisCtx m f =>
     Data.ByteString.Internal.ByteString
     -> m (f Data.ByteString.Internal.ByteString)

You can set a value to a key using the set function in hedis. An example is shown below:

{-# LANGUAGE OverloadedStrings #-}
import Database.Redis

main :: IO (Either Reply Status)
main = do
  conn <- connect defaultConnectInfo
  runRedis conn $ set "a" "apple"

Loading the above set.hs code in GHCi and testing the same produces the following output:

ghci> :l set.hs
[1 of 1] Compiling Main             ( set.hs, interpreted )
Ok, modules loaded: Main.

ghci> main
Right Ok

The type signature of the set function is shown below:

ghci> :t set
set
  :: RedisCtx m f =>
     Data.ByteString.Internal.ByteString
     -> Data.ByteString.Internal.ByteString -> m (f Status)

You can verify the value of the key `a’ from the redis-cli command, and it must return the value “apple”:

127.0.0.1:6379> get a
"apple"

You can also retrieve the value of a key using the get function. For example:

{-# LANGUAGE OverloadedStrings #-}
import Database.Redis
import Control.Monad.IO.Class

main :: IO ()
main = do
  conn <- connect defaultConnectInfo
  runRedis conn $ do
         result <- get "a"
         liftIO $ print result

Executing the above code in GHCi gives the expected result:

ghci> :l get.hs
[1 of 1] Compiling Main             ( get.hs, interpreted )
Ok, modules loaded: Main.

ghci> main
Right (Just "apple")

The liftIO function transforms an IO action into a Monad. Its type signature is shown below:

ghci> :t liftIO
liftIO :: MonadIO m => IO a -> m a

The type signature of the get function is as follows:

ghci> :t get
get
  :: RedisCtx m f =>
     Data.ByteString.Internal.ByteString
     -> m (f (Maybe Data.ByteString.Internal.ByteString))

You are encouraged to read the Database.Redis documentation page that contains a comprehensive list of commands and their usage at https://hackage.haskell.org/package/hedis-0.6.5/docs/Database-Redis.html.

• Accessing the PostgreSQL database

We shall now explore accessing a PostgreSQL database using the postgresql-simple (0.4.10.0) package. You will need to install and configure PostgreSQL for your GNU/Linux distribution. Please follow your distribution documentation to do so. On Fedora, for example, you can install the database server using the following command:

$ sudo yum install postgresql-server postgresql-contrib

You can then start the database server using the following service command:

$ sudo service postgresql start

You can now install the postgresql-simple package using the cabal command:

$ cabal install postgresql-simple

Let us first create a database and a schema using the Postgresql command-line utility psql:

$ psql -U postgres
Password for user postgres: 
psql (9.3.5)
Type "help" for help.

postgres=# \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   
-----------+----------+----------+-------------+-------------+-----------------------
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
(3 rows)

postgres=# CREATE DATABASE test;
CREATE DATABASE

postgres-# \c test
You are now connected to database "test" as user "postgres".

test=# create schema social;
CREATE SCHEMA

test=# \dn
 public | postgres
 social | postgres

We can then create a users’ table with an id, first name and last name using the postgresql-simple package.

{-# LANGUAGE OverloadedStrings #-}

import Database.PostgreSQL.Simple

main :: IO ()
main = do
  conn <- connect defaultConnectInfo
    { connectUser = "postgres"
    , connectPassword = "postgres123"
    , connectDatabase = "test"
    }

  execute conn "create table social.users (id INT, fname VARCHAR(80), lname VARCHAR(80))" ()

  close conn

Loading the above code in GHCi creates the table social.users as shown below:

$ ghci create.hs 

GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( create.hs, interpreted )
Ok, modules loaded: Main.

ghci> main

Loading package array-0.4.0.1 ... linking ... done.
Loading package deepseq-1.3.0.1 ... linking ... done.
Loading package bytestring-0.10.0.2 ... linking ... done.
Loading package containers-0.5.0.0 ... linking ... done.
Loading package text-0.11.3.1 ... linking ... done.
Loading package attoparsec-0.10.4.0 ... linking ... done.
Loading package blaze-builder-0.3.1.1 ... linking ... done.
Loading package dlist-0.5 ... linking ... done.
Loading package hashable-1.1.2.5 ... linking ... done.
Loading package transformers-0.3.0.0 ... linking ... done.
Loading package mtl-2.1.2 ... linking ... done.
Loading package old-locale-1.0.0.5 ... linking ... done.
Loading package syb-0.4.0 ... linking ... done.
Loading package pretty-1.1.1.0 ... linking ... done.
Loading package template-haskell ... linking ... done.
Loading package time-1.4.0.1 ... linking ... done.
Loading package unordered-containers-0.2.3.0 ... linking ... done.
Loading package primitive-0.5.0.1 ... linking ... done.
Loading package vector-0.10.0.1 ... linking ... done.
Loading package aeson-0.6.2.1 ... linking ... done.
Loading package random-1.0.1.1 ... linking ... done.
Loading package scientific-0.2.0.2 ... linking ... done.
Loading package case-insensitive-1.0.0.1 ... linking ... done.
Loading package blaze-textual-0.2.0.8 ... linking ... done.
Loading package postgresql-libpq-0.9.0.2 ... linking ... done.
Loading package binary-0.7.4.0 ... linking ... done.
Loading package cereal-0.3.5.2 ... linking ... done.
Loading package entropy-0.2.2.1 ... linking ... done.
Loading package tagged-0.6 ... linking ... done.
Loading package crypto-api-0.11 ... linking ... done.
Loading package cryptohash-0.9.0 ... linking ... done.
Loading package network-info-0.2.0.5 ... linking ... done.
Loading package uuid-1.3.8 ... linking ... done.
Loading package postgresql-simple-0.4.10.0 ... linking ... done.

You can verify the created table from the psql prompt:

test=# \d social.users
 id     | integer               | 
 fname  | character varying(80) | 
 lname  | character varying(80) | 

You can also list the databases in the PostgreSQL server using the query_ function as illustrated below:

{-# LANGUAGE OverloadedStrings #-}

import Database.PostgreSQL.Simple

main :: IO ()
main = do
  conn <- connect defaultConnectInfo
    { connectUser = "postgres"
    , connectPassword = "postgres123"
    , connectDatabase = "test"
    }
  databases <- query_ conn "SELECT datname FROM pg_database"
  print (databases :: [Only String])

  close conn

Executing the above code in GHCi produces the following output:

$ ghci show.hs 
GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( show.hs, interpreted )
Ok, modules loaded: Main.

ghci> main
[Only {fromOnly = "template1"},Only {fromOnly = "template0"},Only {fromOnly = "postgres"},Only {fromOnly = "test"}]

You can now insert a record into the databaes using the execute function:

execute conn "insert into social.users (id, fname, lname) values (?, ?, ?)" ["1" :: String, "Edwin" :: String, "Brady" :: String]

After executing the above code, you can verify the database entry from the psql prompt:

test=# select * from social.users;
 id | fname | lname 
----+-------+-------
  1 | Edwin | Brady
(1 row)

You can also do batch inserts using the executeMany function. For example:

executeMany conn "insert into social.users (id, fname, lname) values (?, ?, ?)" [("2" :: String, "Simon" :: String, "Marlow" :: String), ("3" :: String, "Ulf" :: String, "Norell" :: String)]

After running the above code, you can check the newly added rows in the database from the psql command-line tool:

test=# select * from social.users;
 id | fname | lname  
----+-------+--------
  1 | Edwin | Brady
  2 | Simon | Marlow
  3 | Ulf   | Norell
(3 rows)

You can also change a record entry using the UPDATE statement as shown below:

execute conn "update social.users SET lname = 'Peyton Jones' where fname = 'Simon'" ()

The corresponding entry is updated as seen from the psql prompt:

test=# select * from social.users;
 id | fname |    lname     
----+-------+--------------
  1 | Edwin | Brady
  3 | Ulf   | Norell
  2 | Simon | Peyton Jones
(3 rows)

It is recommended to catch exceptions when running database commands. Consider the following example, where the number of arguments passed does not match with the expected:

{-# LANGUAGE OverloadedStrings #-}

import Database.PostgreSQL.Simple
import Control.Exception
import GHC.Int

main :: IO ()
main = do
  conn <- connect defaultConnectInfo
    { connectUser = "postgres"
    , connectPassword = "postgres123"
    , connectDatabase = "test"
    }
  result <- try (execute conn "insert into social.users (id, fname, lname) values (?, ?, ?)" ["4" :: String, "Laurel" :: String]) :: IO (Either SomeException Int64)
  case result of
      Left ex  -> putStrLn $ "Caught exception: " ++ show ex
      Right val -> putStrLn $ "The answer was: " ++ show val
  close conn

The error is observed when the main function is executed as shown below:

ghci> main
Caught exception: FormatError {fmtMessage = "3 '?' characters, but 2 parameters", fmtQuery = "insert into social.users (id, fname, lname) values (?, ?, ?)", fmtParams = ["4","Laurel"]}

You can also retrieve multiple records from the database and use the results using a map function. An example is illustrated below:

{-# LANGUAGE OverloadedStrings #-}

import Database.PostgreSQL.Simple
import Control.Monad
import Data.Text as Text

main :: IO ()
main = do
  conn <- connect defaultConnectInfo
    { connectUser = "postgres"
    , connectPassword = "postgres123"
    , connectDatabase = "test"
    }
  users <- query_ conn "SELECT fname, lname FROM social.users"
  forM_ users $ \(fname, lname) ->
      putStrLn $ Text.unpack fname ++ " " ++ Text.unpack lname
  close conn

The output after executing the above code in GHCi returns the actual data:

ghci> main
Edwin Brady
Ulf Norell
Simon Peyton Jones

Please refer to the Database.PostgreSQL.Simple documentation for more examples and usage at https://hackage.haskell.org/package/postgresql-simple-0.4.10.0/docs/Database-PostgreSQL-Simple.html.

In this ninth article on Haskell, I shall cover access to Sqlite, and MySQL databases using Haskell modules. A number of packages are available from the https://hackage.haskell.org/packages/#cat:Database website, but, I will illustrate a few of them with examples.

You first need to install the cabal-install tool on Fedora, for example, using the following command:

$ sudo yum install cabal-install

You can then install HDBC.Sqlite3 using the following command:

$ cabal install HDBC-sqlite3

This installs the latest 2.3.3.0 version from https://hackage.haskell.org/package/HDBC-sqlite3. You can also install the Sqlite3 package on Fedora for testing, as follows:

$ sudo yum install sqlite

To initiate a connection to a database, you can test it out in the GHCi prompt using the connectSqlite3 function, as shown below:

$ ghci

GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.

ghci> :module Database.HDBC Database.HDBC.Sqlite3

ghci> conn <- connectSqlite3 "students.db"

Loading package array-0.4.0.1 ... linking ... done.
Loading package deepseq-1.3.0.1 ... linking ... done.
Loading package old-locale-1.0.0.5 ... linking ... done.
Loading package time-1.4.0.1 ... linking ... done.
Loading package transformers-0.3.0.0 ... linking ... done.
Loading package bytestring-0.10.0.2 ... linking ... done.
Loading package text-0.11.3.1 ... linking ... done.
Loading package old-time-1.1.0.1 ... linking ... done.
Loading package containers-0.5.0.0 ... linking ... done.
Loading package utf8-string-0.3.7 ... linking ... done.
Loading package mtl-2.1.2 ... linking ... done.
Loading package convertible-1.1.0.0 ... linking ... done.
Loading package HDBC-2.4.0.0 ... linking ... done.
Loading package HDBC-sqlite3-2.3.3.0 ... linking ... done.

The signature of the connectSqlite3 function is as follows:

ghci> :t connectSqlite3
connectSqlite3 :: FilePath -> IO Connection

The type of conn is a Connection.

ghci> :t conn
conn :: Connection

If you already have an existing Sqlite3 database, you can give the full path to the database and connect to it, or else you can now create a table using the Sqlite CREATE TABLE syntax as shown below:

ghci> run conn "CREATE TABLE names (id INTEGER NOT NULL, fname VARCHAR(80), lname VARCHAR(80))" []
0

The type signature of run is as follows:

ghci> :t run
run
  :: IConnection conn => conn -> String -> [SqlValue] -> IO Integer

It takes three arguments as input and performs an IO computation that returns an integer indicating the status of the execution. The first argument to run is the connection, the second argument is the Sqlite command to be executed, and finally is the array of SqlValues that provide a mapping between Haskell values and SQL databases.

Both Haskell and SQL databases have types, and different databases may have different representations of the types. In order to provide a consistent mapping between the two, each HDBC driver implements the relation using SqlValue.

You can now insert a record into the database using the following command:

ghci> run conn "INSERT INTO names (id, fname, lname) VALUES(1, 'Edwin', 'Brady')" []
1

ghci> commit conn

The type signature of commit is given here:

ghci> :t commit
commit :: IConnection conn => conn -> IO ()

It takes a connection and completes the pending IO actions. To read the result from Haskell you can use the quickQuery function from the GHCi prompt, as follows:

ghci> quickQuery conn "SELECT * from names" []
[[SqlByteString "1",SqlByteString "Edwin",SqlByteString "Brady"]]

The type signature of the quickQuery function is as follows:

quickQuery
  :: IConnection conn =>
     conn -> String -> [SqlValue] -> IO [[SqlValue]]

You can also verify the result of the above actions using the sqlite3 executable in the command prompt as illustrated below:

$ sqlite3 students.db 

SQLite version 3.8.4.3 2014-04-03 16:53:12
Enter ".help" for usage hints.

sqlite> .schema
CREATE TABLE names (id INTEGER NOT NULL, fname VARCHAR(80), lname VARCHAR(80));

sqlite> select * from names;
1|Edwin|Brady

You can also do batch processing for inserts by preparing the statements and executing them:

ghci> batch <- prepare conn "INSERT INTO names VALUES (?, ?, ?)"

ghci> execute batch [toSql (2 :: Int), toSql "Simon", toSql "Marlow"]
1

ghci> execute batch [toSql (3 :: Int), toSql "Ulf", toSql "Norell"]
1

ghci> commit conn

The type signatures of the prepare and execute functions are given below:

ghci> :t prepare
prepare :: IConnection conn => conn -> String -> IO Statement

ghci> :t execute
execute :: Statement -> [SqlValue] -> IO Integer

You can once again check the records in the database using the quickQuery function:

ghci> quickQuery' conn "SELECT * from names" []

[[SqlByteString "1",SqlByteString "Edwin",SqlByteString "Brady"],[SqlByteString "2",SqlByteString "Simon",SqlByteString "Marlow"],[SqlByteString "3",SqlByteString "Ulf",SqlByteString "Norell"]]

You can also run an update query to the database. For example:

ghci> run conn "UPDATE names set lname = 'Peyton Jones' WHERE fname = 'Simon'" []
1

ghci> commit conn

Verifying the output from the Sqlite3 command prompt, you get:

sqlite> select * from names;

1|Edwin|Brady
2|Simon|Peyton Jones
3|Ulf|Norell

The HDBC driver provides many functions to retrieve information regarding the database and the drivers. A few examples are illustrated below:

ghci> hdbcDriverName conn
"sqlite3"

ghci> hdbcClientVer conn
"3.8.4.3"

ghci> dbTransactionSupport conn
True

ghci> getTables conn
["names"]

ghci> describeTable conn "names"

[("id",SqlColDesc {colType = SqlIntegerT, colSize = Nothing, colOctetLength = Nothing, colDecDigits = Nothing, colNullable = Nothing}),("fname",SqlColDesc {colType = SqlUnknownT "varchar(80)", colSize = Nothing, colOctetLength = Nothing, colDecDigits = Nothing, colNullable = Nothing}),("lname",SqlColDesc {colType = SqlUnknownT "varchar(80)", colSize = Nothing, colOctetLength = Nothing, colDecDigits = Nothing, colNullable = Nothing})]

It is considered good practice to use handleSqlError before running any HDBC commands to catch errors that may arise during the database transactions. For example:

ghci> handleSqlError $ quickQuery conn "SELECT * from namesaaa" []

*** Exception: user error (SQL error: SqlError {seState = "", seNativeError = 1, seErrorMsg = "prepare 23: SELECT * from namesaaa: no such table: namesaaa"})

To disconnect from the database, you can use the disconnect function provided by HDBC as shown below:

ghci> disconnect conn

Let’s now look at how to access a MySQL database using the mysql-simple package. You will also need to create or grant privileges for an existing user to use the MySQL database server. Please follow your GNU/Linux distribution manual on how to install and configure a MySQL server. On Fedora, for example, you must have mysql and mysql-server installed:

$ sudo yum install community-mysql community-mysql-server

You can install the mysql-simple Haskell package using:

$ cabal install mysql-simple

Create a test database using the mysql command line tool as shown below:

$ mysql -u user -p

Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.5.38-log MySQL Community Server (GPL)

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database test;
Query OK, 1 row affected (0.03 sec)

Let us create a users’ table that has an id as well as first name and last name fields, using the mysql-simple package, as follows:

{-# LANGUAGE OverloadedStrings #-}

import Database.MySQL.Simple

main :: IO ()
main = do
  conn <- connect defaultConnectInfo
    { connectUser = "user"
    , connectPassword = "password"
    , connectDatabase = "test"
    }

  execute conn "create table users (id INT, fname VARCHAR(80), lname VARCHAR(80))" ()

  close conn

The OverloadedStrings extension allows string literals to be polymorphic for the IsString class. The defaultConnectInfo is of type ConnectInfo:

ghci> :t defaultConnectInfo
defaultConnectInfo :: ConnectInfo

ConnectInfo can take many parameters to describe the connectivity to the MySQL server. For example:

connectInfo :: ConnectInfo
connectInfo = ConnectInfo { connectHost = "localhost",
                            connectPort = 3306,
                            connectUser = "user",
                        connectPassword = "password",
                        connectDatabase = "test",
                         connectOptions = [],
                            connectPath = "",
                             connectSSL = Nothing }

The above code to create a table can be compiled directly in GHCi and the main function can be executed as given below:

$ ghci create.hs

GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( create.hs, interpreted )
Ok, modules loaded: Main.

ghci> main

Loading package array-0.4.0.1 ... linking ... done.
Loading package deepseq-1.3.0.1 ... linking ... done.
Loading package old-locale-1.0.0.5 ... linking ... done.
Loading package time-1.4.0.1 ... linking ... done.
Loading package bytestring-0.10.0.2 ... linking ... done.
Loading package text-0.11.3.1 ... linking ... done.
Loading package blaze-builder-0.3.1.1 ... linking ... done.
Loading package primitive-0.5.0.1 ... linking ... done.
Loading package vector-0.10.0.1 ... linking ... done.
Loading package pcre-light-0.4 ... linking ... done.
Loading package containers-0.5.0.0 ... linking ... done.
Loading package attoparsec-0.10.4.0 ... linking ... done.
Loading package blaze-textual-0.2.0.8 ... linking ... done.
Loading package base16-bytestring-0.1.1.6 ... linking ... done.
Loading package mysql-0.1.1.7 ... linking ... done.
Loading package mysql-simple-0.2.2.4 ... linking ... done.

You can check with the mysql command line utility for the created table:

mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> desc users;
+-------+-------------+------+-----+---------+-------+
| Field | Type        | Null | Key | Default | Extra |
+-------+-------------+------+-----+---------+-------+
| id    | int(11)     | YES  |     | NULL    |       |
| fname | varchar(80) | YES  |     | NULL    |       |
| lname | varchar(80) | YES  |     | NULL    |       |
+-------+-------------+------+-----+---------+-------+
3 rows in set (0.00 sec)

You can now list the databases available in the MySQL server using the query_ function as illustrated below:

{-# LANGUAGE OverloadedStrings #-}

import Database.MySQL.Simple

main :: IO ()
main = do
  conn <- connect defaultConnectInfo
    { connectUser = "user"
    , connectPassword = "password"
    , connectDatabase = "test"
    }
  databases <- query_ conn "SHOW databases"
  print (databases :: [Only String])

  close conn

You can compile the above code directly with GHCi and execute the main function, as follows:

 $ ghci show-databases.hs

GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( show-databases.hs, interpreted )
Ok, modules loaded: Main.

ghci> main

Loading package array-0.4.0.1 ... linking ... done.
Loading package deepseq-1.3.0.1 ... linking ... done.
...
[Only {fromOnly = "information_schema"},Only {fromOnly = "mysql"},Only {fromOnly = "performance_schema"},Only {fromOnly = "test"}]

You can try inserting a record into the database using the execute function:

execute conn "insert into users (id, fname, lname) values (?, ?, ?)" ["1" :: String, "Edwin" :: String, "Brady" :: String]

After running the code, you can check the database entry using the mysql client program as shown below:

mysql> select * from users;

+------+-------+-------+
| id   | fname | lname |
+------+-------+-------+
|    1 | Edwin | Brady |
+------+-------+-------+
1 row in set (0.00 sec)

You can also do batch inserts using the executeMany function. For example:

executeMany conn "insert into users (id, fname, lname) values (?, ?, ?)" [("2" :: String, "Simon" :: String, "Marlow" :: String), ("3" :: String, "Ulf" :: String, "Norell" :: String)]

You can verify the execution of the code from the mysql utility:

mysql> select * from users;
+------+-------+--------+
| id   | fname | lname  |
+------+-------+--------+
|    1 | Edwin | Brady  |
|    2 | Simon | Marlow |
|    3 | Ulf   | Norell |
+------+-------+--------+
3 rows in set (0.01 sec)

You can change a record entry using the UPDATE MySQL command:

execute conn "update users SET lname = 'Peyton Jones' where fname = 'Simon'" ()

Executing the code in GHCi, and checking the results with the mysql prompt gives the following changed output:

mysql> select * from users;
+------+-------+--------------+
| id   | fname | lname        |
+------+-------+--------------+
|    1 | Edwin | Brady        |
|    2 | Simon | Peyton Jones |
|    3 | Ulf   | Norell       |
+------+-------+--------------+
3 rows in set (0.00 sec)

It is important to catch any exceptions that may arise on executing the database commands. Consider the following example, where the number of arguments passed does not match with the expected:

{-# LANGUAGE OverloadedStrings #-}

import Database.MySQL.Simple
import Control.Exception
import GHC.Int

main :: IO ()
main = do
  conn <- connect defaultConnectInfo
    { connectUser = "user"
    , connectPassword = "password"
    , connectDatabase = "test"
    }
  result <- try (execute conn "insert into users (id, fname, lname) values (?, ?, ?)" ["4" :: String, "Laurel" :: String]) :: IO (Either SomeException Int64)
  case result of
      Left ex  -> putStrLn $ "Caught exception: " ++ show ex
      Right val -> putStrLn $ "The answer was: " ++ show val
  close conn

The error is caught when the main function is executed inside GHCi:

ghci> main
Caught exception: FormatError {fmtMessage = "3 '?' characters, but 2 parameters", fmtQuery = "insert into users (id, fname, lname) values (?, ?, ?)", fmtParams = ["4","Laurel"]}

You can also map through the results returned from the database and use them for your needs. The following is an illustration of the same:

{-# LANGUAGE OverloadedStrings #-}

import Database.MySQL.Simple
import Control.Monad
import Data.Text as Text

main :: IO ()
main = do
  conn <- connect defaultConnectInfo
    { connectUser = "user"
    , connectPassword = "password"
    , connectDatabase = "test"
    }
  users <- query_ conn "SELECT fname, lname FROM users"
  forM_ users $ \(fname, lname) ->
      putStrLn $ Text.unpack fname ++ " " ++ Text.unpack lname
  close conn

The resultant output when executing the main function in GHCi is given below:

ghci> main

Edwin Brady
Simon Peyton Jones
Ulf Norell

You are encouraged to read Database.MySQL.Simple documentation from https://hackage.haskell.org/package/mysql-simple-0.2.2.4/docs/Database-MySQL-Simple.html for more information.

Let’s take a look at the property-based testing of Haskell programs and at the Cabal tool, which is used to build and manage Haskell packages and applications.

One of the main features of testing in Haskell is property-based testing. The type system allows you to infer and derive types, and also helps in auto-generating test cases. QuickCheck is a popular property-based testing library for Haskell. If your program is pure, you can write tests to ascertain the properties and invariants of your programs, and the tests can be auto-generated and executed.

You can install QuickCheck on Fedora, for example, by using the following command:

$ sudo yum install ghc-QuickCheck-devel

Consider a simple function to add two integers:

mySum :: Int -> Int -> Int
mySum a b = a + b

We can ascertain the property of the function that ‘a + b’ is the same as ‘b + a’ using the QuickCheck library. You must first define the invariant in a function as shown below:

prop_mySum a b = mySum a b == mySum b a

You can test the code directly in the GHCi prompt, using the following command:

$ ghci sum.hs 
GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( sum.hs, interpreted )
Ok, modules loaded: Main.

ghci> prop_mySum 2 3
Loading package array-0.4.0.1 ... linking ... done.
Loading package deepseq-1.3.0.1 ... linking ... done.
Loading package old-locale-1.0.0.5 ... linking ... done.
Loading package time-1.4.0.1 ... linking ... done.
Loading package random-1.0.1.1 ... linking ... done.
Loading package containers-0.5.0.0 ... linking ... done.
Loading package pretty-1.1.1.0 ... linking ... done.
Loading package template-haskell ... linking ... done.
Loading package QuickCheck-2.6 ... linking ... done.
True

You can also invoke the quickCheck function in a main function, as shown below:

import Test.QuickCheck

mySum :: Int -> Int -> Int
mySum a b = a + b

prop_mySum :: Int -> Int -> Bool
prop_mySum a b = mySum a b == mySum b a

main :: IO ()
main = quickCheck prop_mySum

Compiling and executing the above code produces the following output:

$ ghc --make sum.hs
[1 of 1] Compiling Main             ( sum.hs, sum.o )
Linking sum ...

$ ./sum 
+++ OK, passed 100 tests.

You can also dump the input that was generated for the various test cases using the verboseCheck function, as shown below:

main :: IO ()
main = verboseCheck prop_mySum

Executing the above code with the updated main function will yield 100 input test cases that were generated in runtime.

ghci> main
Passed:
0
0
Passed:
-1
1
Passed:
64
-44
Passed:
-2159
2134
Passed:
-927480859
61832343
...

The head function in Haskell expects to receive a non-empty list. You can write a headExists function to check if the head exists for a list of integers, as shown below:

headExists :: [Int] -> Bool
headExists list
    | null list    = False
    | otherwise = True

You can load the above code in GHCi and test it out, as follows:

ghci> headExists []
False

ghci> headExists [1, 2, 3]
True

Let’s assume that, by mistake, you wrote an incorrect property-based test where the headExists function will always return ‘False’, ignoring the ‘otherwise’ case.

import Test.QuickCheck

headExists :: [Int] -> Bool
headExists list
    | null list    = False
    | otherwise = True

prop_headExists :: [Int] -> Bool
prop_headExists emptyList = headExists emptyList == False

main :: IO ()
main = quickCheck prop_headExists

Testing the code produces the following output:

$ ghc --make head.hs 
[1 of 1] Compiling Main             ( head.hs, head.o )
Linking head ...

$ ./head 
*** Failed! Falsifiable (after 3 tests):             
[0]

The QuickCheck library generated test cases for different [Int] types and it returned a failure after the third test, for which the input was [0]. Clearly, the ‘headExists [0]’ computation will return ‘True’ and not ‘False’.

The way we defined the property is incorrect. We know that if the list is empty, then its length is zero. We can write a helper function lengthZero for the above, as follows:

lengthZero :: [Int] -> Bool
lengthZero list
    | length list == 0 = True
    | otherwise = False

We can then use this function to assert that for any Integer list, if headExists returns ‘False’ then the lengthZero function must return ‘True’. The complete code is shown below:

import Data.List
import Test.QuickCheck

headExists :: [Int] -> Bool
headExists list
    | null list    = False
    | otherwise = True

lengthZero :: [Int] -> Bool
lengthZero list
    | length list == 0 = True
    | otherwise = False

prop_headExists :: [Int] -> Bool
prop_headExists list = headExists list == not (lengthZero list)

main :: IO ()
main = quickCheck prop_headExists

Executing the code produces the required output:

$ ghc --make head.hs 
[1 of 1] Compiling Main             ( head.hs, head.o )
Linking head ...

$ ./head 
+++ OK, passed 100 tests.

We can also re-write the above code based on conditional properties. The property that the headExists function will return ‘True’ only for non-empty lists can be defined as a constraint. The notation syntax is condition ==> property. In our example, if the condition that the list is non-empty is ‘True’, then the property that the headExists function for the list must return is ‘True’. Also, when the list is empty, the headExists function must return ‘False’. These two conditions can be written as follows:

import Data.List
import Test.QuickCheck

headExists :: [Int] -> Bool
headExists list
    | null list    = False
    | otherwise = True

prop_headExists :: [Int] -> Property
prop_headExists list = length list > 0 ==> headExists list == True

prop_emptyList :: [Int] -> Property
prop_emptyList list = length list == 0 ==> headExists list == False

main :: IO ()
main = do
     quickCheck prop_headExists
     quickCheck prop_emptyList

Testing the code produces the following output:

$ ghc --make cond.hs 
[1 of 1] Compiling Main             ( cond.hs, cond.o )
Linking cond ...

 $ ./cond 
+++ OK, passed 100 tests.
*** Gave up! Passed only 38 tests.

These tests can be integrated with Hspec or HUnit for a more verbose output.

Cabal

Cabal is a software tool that is used to describe a Haskell application, list its dependencies, and provide a manifestation to distribute the source and binaries. It is not to be confused with a distribution package manager like RPM or the Debian package management system. You can install Cabal using your distribution package manager. On Fedora, for example, you can use the following command:

$ sudo yum install cabal-install

Haskell software programs are available in hackage.haskell.org, and each project has a .cabal file. Let us take an example of the HSH-2.1.2 package at http://hackage.haskell.org/package/HSH which allows you to use shell commands and expressions within Haskell programs. You can download HSH-2.1.2.tar.gz and extract it using:

$ tar xzvf HSH-2.1.2.tar.gz 

HSH-2.1.2/
HSH-2.1.2/COPYING
HSH-2.1.2/HSH.cabal
HSH-2.1.2/testsrc/
HSH-2.1.2/testsrc/runtests.hs
HSH-2.1.2/HSH.hs
HSH-2.1.2/HSH/
HSH-2.1.2/HSH/Command.hs
HSH-2.1.2/HSH/ShellEquivs.hs
HSH-2.1.2/HSH/Channel.hs
HSH-2.1.2/COPYRIGHT
HSH-2.1.2/Setup.lhs

The .cabal file has various fields that describe the Haskell application. The contents of the HSH.cabal for version 2.1.2 are given below:

Name: HSH
Version: 2.1.3
License: LGPL
Maintainer: John Goerzen <jgoerzen@complete.org>
Author: John Goerzen
Stability: Beta
Copyright: Copyright (c) 2006-2014 John Goerzen
Category: system
license-file: COPYRIGHT
extra-source-files: COPYING
homepage: http://software.complete.org/hsh
Synopsis: Library to mix shell scripting with Haskell programs
Description: HSH is designed to let you mix and match shell expressions with
 Haskell programs. With HSH, it is possible to easily run shell
 commands, capture their output or provide their input, and pipe them
 to and from other shell commands and arbitrary Haskell functions at will.
  Category: System

Cabal-Version: >=1.2.3
Build-type: Simple

flag buildtests
  description: Build the executable to run unit tests
  default: False

library
  Exposed-Modules: HSH, HSH.Command, HSH.ShellEquivs, HSH.Channel
  Extensions: ExistentialQuantification, OverlappingInstances,
    UndecidableInstances, FlexibleContexts, CPP
  Build-Depends: base >= 4 && < 5, mtl, process, regex-compat, MissingH>=1.0.0,
    hslogger, filepath, regex-base, regex-posix, directory,
    bytestring
  if !os(windows)
    Build-Depends: unix
  GHC-Options: -O2 -threaded -Wall

Executable runtests
  if flag(buildtests)
    Buildable: True
    Build-Depends: base >= 4 && < 5, mtl, process, regex-compat,
      MissingH>=1.0.0,
      hslogger, filepath, regex-base, regex-posix, directory,
      bytestring, HUnit, testpack
    if !os(windows)
      Build-Depends: unix
  else
    Buildable: False
  Main-Is: runtests.hs
  HS-Source-Dirs: testsrc, .
  Extensions: ExistentialQuantification, OverlappingInstances,
    UndecidableInstances, FlexibleContexts, CPP
  GHC-Options: -O2 -threaded

Enter the HSH-2.1.2 directory and configure the project using the cabal configure command as shown below:

$ cd HSH-2.1.2

$ cabal configure
Resolving dependencies...
Configuring HSH-2.1.2...

You can then compile the project sources using the cabal build step:

$ cabal build

Building HSH-2.1.2...
Preprocessing library HSH-2.1.2...
[1 of 4] Compiling HSH.Channel      ( HSH/Channel.hs, dist/build/HSH/Channel.o )
...
[2 of 4] Compiling HSH.Command      ( HSH/Command.hs, dist/build/HSH/Command.o )
...
[3 of 4] Compiling HSH.ShellEquivs  ( HSH/ShellEquivs.hs, dist/build/HSH/ShellEquivs.o )
...
[4 of 4] Compiling HSH              ( HSH.hs, dist/build/HSH.o )
In-place registering HSH-2.1.2...

You can install the built library files using the cabal install command. By default, it installs to ~/.cabal folder as shown below:

$ cabal install

Resolving dependencies...
Configuring HSH-2.1.2...
Building HSH-2.1.2...
Preprocessing library HSH-2.1.2...
In-place registering HSH-2.1.2...
Installing library in /home/guest/.cabal/lib/HSH-2.1.2/ghc-7.6.3
Registering HSH-2.1.2...
Installed HSH-2.1.2

You can also generate HTML documentation for the source code using the cabal haddock option. The HTML files can also be made available at hackage.haskell.org.

$ cabal haddock

Running Haddock for HSH-2.1.2...
Preprocessing library HSH-2.1.2...
Warning: The documentation for the following packages are not installed. No
links will be generated to these packages: MissingH-1.3.0.1, rts-1.0,
hslogger-1.2.6, network-2.6.0.2
Haddock coverage:
...
Documentation created: dist/doc/html/HSH/index.html

If you make changes to the sources and wish to generate a new release, you can update the Version field in the HSH.cabal file.

Version: 2.1.3

In order to make a new tarball, use the cabal sdist command:

$ cabal sdist

Distribution quality warnings:
...
Building source dist for HSH-2.1.3...
Preprocessing library HSH-2.1.3...
Source tarball created: dist/HSH-2.1.3.tar.gz

To test the installed application, you can run GHCi from a directory other than the HSH-2.1.2 sources directory. For example:

$ ghci
GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.

ghci> :m + HSH

ghci HSH> runIO "date"
Sun Jan  4 14:22:37 IST 2015

You should not run GHCi from the sources directory, since it will find the module in it and try to use it instead of the installed modules in ~/.cabal folder.

You can also test the installation by writing a program:

import HSH.Command

main :: IO ()
main = do
  runIO "date"

You can compile and execute the above as shown below:

$ ghc --make test.hs 
[1 of 1] Compiling Main             ( test.hs, test.o )
Linking test ...

$ ./test 
Sun Jan  4 14:25:19 IST 2015

You are encouraged to read the Cabal guide at https://www.haskell.org/cabal/ for information on specific fields and their options.

Day 1

I took an early train from Chennai to reach Bengaluru, and headed straight to the venue. The workshops were scheduled on the first day. After completing the registration formalities, I went to observe the DevSprint. I met a number of dgplug folks with whom I have been interacting online on #dgplug (irc.freenode.net).

After lunch, I attended Prof. Ronojoy Adhikari’s workshop on “Reasoning under uncertainity with Python”. Dorai Thodla began the workshop with an introduction on “Data Science” and its use and applications in the industry. Prof. Ronojoy then spoke on “Data Science: Theory” and “Probability Theory for Data Science”. We already use Boolean logic in our programming. We also have probability concepts implemented in Python. He then proceeded to demonstrate “Probabilistic Programming in Python” using Lea. He had presented a short preview of this talk at the Chennaipy August 2015 meet-up. Prof. Ronojoy showed very interesting Lea one-liners with IPython Notebook. A sample is provided below:

In [1]: from lea import *

In [2]: faircoin = Lea.fromVals('Head', 'Tail')

In [3]: faircoin.random(10)

Out[3]: ('Tail',
         'Tail',
         'Tail',
         'Head',
         'Head',
         'Tail',
         'Tail',
         'Head',
         'Tail',
         'Head')

In [4]: die = Lea.fromVals(1, 2, 3, 4, 5, 6)

In [5]: die1 = die.clone()
        die2 = die.clone()

In [6]: dice = die1 + die2

In [7]: dice

OUt[10]: 2 : 1/36
         3 : 2/36
         4 : 3/36
         5 : 4/36
         6 : 5/36
         7 : 6/36
         8 : 5/36
         9 : 4/36
        10 : 3/36
        11 : 2/36
        12 : 1/36

I had installed the Anaconda distribution and Lea for the workshop, but, couldn’t get Pomegranate to compile. Later, I worked with Jacob Schreiber to get it built on Ubuntu 14.10.

Prof. Ronojoy then explained Bayesian networks, and how the probabilities can be fed to compute the probability for a causal-effect relationship. He demonstrated numerous examples with live coding. His work is part of the Value from Data initiative. Prof. Ronojoy and Dorai’s “Bayesian Inference and Data Analytics” video lectures are available in YouTube.

Day 2

After collecting my speaker kit, I attended the “KG award” presentation, which was given to Vijay Kumar, followed by the key note by Dr. Ajit Kumar. Vijay Kumar is doing an excellent job in organizing the monthly Chennaipy meet-ups. It was also good to meet Dr. Ajit Kumar after many years, having known his work and involvement with the Phoenix project.

During the tea break, I took the time to test my laptop with the projector in auditorium-1, where I was scheduled to speak. I had few demos in my talk, and I needed to set everything up before the talk. After the tea break, I attended the session by Pratap Vardhan on “Consuming Government Data with Python and D3”. The insights from the data and the analysis by Gramener were quite interesting.

I then moved to the “dgplug” 1130 IST staircase meeting, where I met more #dgplug students, and was able to connect IRC nicknames to people’s faces. My suggestion to the students was to start to identify their interests, and to specialize on them. Around noon, I moved back to auditorium-1 to setup everything that I needed for my talk - “Pretty Printing in Python”. There is one slot which I never wanted - the one before lunch because I know that I will be the one standing between the audience and their lunch, and that is exactly what I got. Nevertheless, the slot had to be taken by someone, and I was prepared for it. I had requested the organizers for a whiteboard and markers for my talk, and they had arranged for the same. I had lot of material to cover in 45 minutes. I did rush through, but, paused at times to explain the concepts. The audience were quite interested in 3D printing, and they asked some really interesting questions at the end of the talk. The video is available in YouTube:

After a quick lunch, I headed for the lightning talks to present “Nursery Rhymes”. We are all familiar with nursery rhymes, and the idea is to teach programming languages and software development using them. The audience needed something to wake them after a heavy lunch, and they thoroughly enjoyed the singing! The video link is provided below:

I then attended the “Machine learning techniques for building a large scale production ready prediction system using Python” by Arthi Venkataraman. After a short tea break, I attended the talk by Pooja Salpekar on “Test Driven Development with Ansible”. She gave an overview of using Serverspec. The subject to test is still determined by the developer, and testing with Vagrant is not the same as testing in production deployments. For the last session of the day, I attended the “Python and Riak DB, a perfect couple for huge scale distributed computing” by Naren Arya, but, it was yet another generic talk.

In the evening, I participated in the discussion on building and running communities organized by Satyaakam Goswami on how Python communities are organizing themselves in India, and the challenges faced by them. I shared my experience with the Chennaipy meet-ups, and insights from my “i want 2 do project. tell me wat 2 do” presentation and book. A video recording of the discussion is available:

Day 3

Nicholas H. Tollervey began the day’s proceedings with his keynote. He shared his experience on Python and education. The Micro Python and BBC micro:bit projects are interesting initiatives. The audience did have questions on why they cannot use Android or any other mobile phone for Python development. But, I think they missed the point that Open Hardware initiatives are important and the openness is not the same with present day mobile devices. Nicholas also covered important points on how to organize Python Dojo meet-ups, and as a teacher, the experiences he shared were very useful.

After the tea break, I attended the session by Sumith Kulal on “SymEngine: The future fast core of computer algebra systems”. It was an introductory session on SymEngine, and he shared the roadmap and plan that they have for the software. I was then networking with people before attending “How to build microservices using ZeroMQ and WSGI” by Srinath G. S. It was yet another generic talk, and I headed for early lunch.

The previous night I prepared for another lightning talk titled “Yet Another Lightning Talk”, to once again, wake the audience after their heavy lunch. The talk went as I had planned, and the audience loved it! The video recording is available in YouTube:

A number of people approached me after the lightning talk, appreciating the creativity, and I did make a lot of new friends. The last talk of the day that I attended was on “Python load balancer; 0 to 1 million requests per second”, and yet again it turned out to be a a very generic talk, without going into specifics. I left early to catch a flight.

It will be helpful if organizers can identify talks where the speaker has contributed to the project. When you hire people for free and open source software (F/OSS) work, you would like to see what their contribution to the project is, and how they have added value to the project. I am interested to hear on what the speaker has done. I can always read the project web site, and find out more information about the project. I wish all conferences add this criteria in their talk selection process.

The workshops have been useful. The content of the talks need to be improved though. I was able to meet fellow F/OSS developers during the conference, and also made new friends. I am pleased that I was able to deliver a talk, and two lightning talks.

I would like to thank Manufacturing System Insights for sponsoring my travel, and allowing me to 3D print during office hours!

August 2015 meetup

The first talk was titled “Paho Python client for MQTT” presented on Saturday, August 22, 2015. The slides and notes for the presentation are given below:

MQTT Architecture

Source: Device to Cloud: MQTT and the power of topic notation

Topics

$SYS/broker/load/connections
$SYS/broker/load/messages/received
$SYS/broker/load/messages/sent

home/living-room/temperature
home/kitchen/humidity
home/store-room/brightness

MQTT Protocol

Source: MQTT Essentials Part 3: Client, Broker and Connection Establishment

Source: MQTT Essentials Part 4: MQTT Publish, Subscribe & Unsubscribe

Mosquitto MQTT Installation

$ sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa

  You are about to add the following PPA to your system:
 
  More info: https://launchpad.net/~mosquitto-dev/+archive/ubuntu/mosquitto-ppa
  Press [ENTER] to continue or ctrl-c to cancel adding it

  gpg: keyring `/tmp/tmpd67wf6/secring.gpg' created
  gpg: keyring `/tmp/tmpd67wf6/pubring.gpg' created
  gpg: requesting key 262C4500 from hkp server keyserver.ubuntu.com
  gpg: /tmp/tmpd67wf6/trustdb.gpg: trustdb created
  gpg: key 262C4500: public key "Launchpad mosquitto" imported
  gpg: Total number processed: 1
  gpg:               imported: 1  (RSA: 1)
  OK

$ apt-get update
$ apt-get install python-pip mosquitto mosquitto-clients
$ pip install paho-mqtt

Connectivity

Start mosquitto server

$ sudo /etc/init.d/mosquitto status
mosquitto stop/waiting

$ sudo /etc/init.d/mosquitto start
mosquitto start/running, process 11346

Verification

$ netstat -na | grep :1883
tcp        0      0 0.0.0.0:1883            0.0.0.0:*               LISTEN     
tcp6       0      0 :::1883                 :::*                    LISTEN 

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.10
Release:	14.10
Codename:	utopic

Pub/Sub

Subscribe

$ mosquitto_sub -d -t hello/world

Client mosqsub/11678-achilles sending CONNECT
Client mosqsub/11678-achilles received CONNACK
Client mosqsub/11678-achilles sending SUBSCRIBE (Mid: 1, Topic: hello/world, QoS: 0)
Client mosqsub/11678-achilles received SUBACK
Subscribed (mid: 1): 0

Publish

$ mosquitto_pub -d -t hello/world -m "Vanakkam"
Client mosqpub/11712-achilles sending CONNECT
Client mosqpub/11712-achilles received CONNACK
Client mosqpub/11712-achilles sending PUBLISH (d0, q0, r0, m1, 'hello/world', \
  ... (8 bytes))
Client mosqpub/11712-achilles sending DISCONNECT

Verification

Client mosqsub/11678-achilles received PUBLISH (d0, q0, r0, m0, 'hello/world', \
  ... (8 bytes))
Vanakkam

Paho Python connectivity

  import paho.mqtt.client as mqtt

  def on_connect(client, userdata, flags, rc):
      print("Connected with result code "+str(rc))
      client.subscribe("hello/world")

  def on_message(client, userdata, msg):
      print(msg.topic+" "+str(msg.payload))

  client = mqtt.Client()
  client.on_connect = on_connect
  client.on_message = on_message

  client.connect("localhost", 1883, 60)

  client.loop_forever()

$ python client.py

$ mosquitto_pub -d -t hello/world -m "Hello, World"

iot.eclipse.org

  import paho.mqtt.client as mqtt

  def on_connect(client, userdata, flags, rc):
      print("Connected with result code "+str(rc))
      client.subscribe("$SYS/#")

  def on_message(client, userdata, msg):
      print(msg.topic+" "+str(msg.payload))

  client = mqtt.Client()
  client.on_connect = on_connect
  client.on_message = on_message

  client.connect("iot.eclipse.org", 1883, 60)

  client.loop_forever()

Source Code

  paho-mqtt-1.1:
  -rw-rw-r-- 1 shakthi shakthi  2031 Feb  1  2015 about.html
  -rw-rw-r-- 1 shakthi shakthi  3736 Feb  1  2015 CONTRIBUTING.md
  -rw-rw-r-- 1 shakthi shakthi  1569 Feb  1  2015 edl-v10
  -rw-rw-r-- 1 shakthi shakthi 11695 Feb  1  2015 epl-v10
  drwxrwxr-x 2 shakthi shakthi  4096 Feb  1  2015 examples
  -rw-rw-r-- 1 shakthi shakthi   156 Feb  1  2015 LICENSE.txt
  -rw-rw-r-- 1 shakthi shakthi  9230 Feb  1  2015 notice.html
  -rw-rw-r-- 1 shakthi shakthi   814 Feb  1  2015 PKG-INFO
  -rw-rw-r-- 1 shakthi shakthi 30871 Feb  1  2015 README.rst
  -rw-rw-r-- 1 shakthi shakthi   972 Feb  1  2015 setup.py
  drwxrwxr-x 3 shakthi shakthi  4096 Feb  1  2015 src

Embedded

Source: Raspberry Pi to Arduino

References

• mqtt.org
• hivemq.com
• mosquitto.org
• www.arduino.cc
• www.raspberrypi.org
• cooking-hacks.com
• pypi.python.org/pypi/paho-mqtt
• iot.eclipse.org
• github.com/eschulte/epresent

September 2015 meetup

The September 2015 talk was scheduled on Saturday, September 26, 2015 and I gave a talk on “G-code Visualization Tools”. The talk began with an introduction and history of G-codes. A Maya pyramid STL file was then used with Pronterface to generate the following G-codes:

G90         ; Set to Absolute Positioning
G21         ; Set Units to Millimeters
M103        ; Turn off extruder
M105        ; Get extruder temperature
M106        ; Fan (cooling) on
M140 S60.0  ; Set Bed temperature
M141 S30.0  ; Set Chamber temperature
M142 S0.0   ; Holding pressure
M113 S1.0   ; Set Extruder PWM
M108 S210.0 ; Set Extruder Speed
M104 S200.0 ; Set Extruder Temperature
G1 X-54.72 Y-53.352 Z0.72 F60.0 ; G1 Move, F Feedrate, mm/rev
M101        ; Turn extruder 1 on
G1 X-54.72 Y53.352 Z0.72 F240.0
G1 X-51.84 Y53.352 Z0.72 F240.0
...

I then gave a demo of “Yet Another GCode Viewer” written by Jonathan Winterflood that uses the Pyglet library to render the G-code paths. You can also zoom, rotate and scale the image after it has loaded the input data. The brim that is printed for 3D objects is clearly visible in the rendition. A screenshot is shown below:

Blender is a free and open source software for computer graphics, 3D modelling and animations. It is written in C, C++ and Python. A Blender gcode reader plugin written by Simon Kirkby exists to visualize G-codes. This plugin generates the image from the G-codes, and converts the change in paths into arcs for a smoother image. The advantage of this plugin is that you can use all the goodies from Blender and Python. A screenshot of the pyramid using the Blender plugin is shown below:

The Visualization Toolkit (VTK) is yet another free and open source software for image processing and visualization. It requires its input file in a VTK file format. After using the gcode2vtk utility to convert the Maya pyramid G-code file to a VTK file format, I was able to demonstrate the G-code path simulation using ParaView - a data analysis and visualization application. An illustration of the Maya pyramid in ParaView is shown below:

ParaView is written in C, C++, Fortran and Python. Both Blender and ParaView allow you to run simulations on the G-codes.

All the visualization tools discussed are free and open source software, and you can install them on your favourite *nix system.

In this article we shall cover testing of Haskell programs.

HUnit is a unit testing framework available for Haskell. It is similar to JUnit, which is used for the Java programming language. You can install HUnit on Fedora using the following command:

$ sudo yum install ghc-HUnit-devel

Consider a simple example that follows:

import Test.HUnit

test1 = TestCase $ assertEqual "Test equality" 3 (2 + 1)

The TestCase is a constructor defined in the Test data type. The definition is as follows:

-- Test Definition
-- ===============

-- | The basic structure used to create an annotated tree of test cases.
data Test
    -- | A single, independent test case composed.
    = TestCase Assertion
    -- | A set of @Test@s sharing the same level in the hierarchy. 
    | TestList [Test]
    -- | A name or description for a subtree of the @Test@s.
    | TestLabel String Test

On executing the above code with GHCi, you get the following output:

$ ghci test.hs 

GHCi, version 7.6.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
[1 of 1] Compiling Main             ( one.hs, interpreted )
Ok, modules loaded: Main.

ghci> runTestTT test1

Loading package array-0.4.0.1 ... linking ... done.
Loading package deepseq-1.3.0.1 ... linking ... done.
Loading package HUnit-1.2.5.2 ... linking ... done.
Cases: 1  Tried: 1  Errors: 0  Failures: 0
Counts {cases = 1, tried = 1, errors = 0, failures = 0}

You can build a test suite of tests with TestList as shown below:

import Test.HUnit

test1 = TestList [ "Test addition" ~: 3 ~=? (2 + 1)
                 , "Test subtraction" ~: 3 ~=? (4 - 1)
                 ]

The ‘~=?’ operation is shorthand to assert equality. Its definition is as follows:

-- | Shorthand for a test case that asserts equality (with the expected 
--   value on the left-hand side, and the actual value on the right-hand
--   side).
(~=?) :: (Eq a, Show a) => a     -- ^ The expected value 
                        -> a     -- ^ The actual value
                        -> Test

The ‘~:’ operation is shorthand to attach a label to a test. Its definition is shown below:

(~:) :: (Testable t) => String -> t -> Test
label ~: t = TestLabel label (test t)

By compiling and executing the above code with GHCi, you get the following result:

ghci> runTestTT test1

Cases: 2  Tried: 2  Errors: 0  Failures: 0
Counts {cases = 2, tried = 2, errors = 0, failures = 0}