rmed

Connecting things in C++

2024-07-14T11:18:00+02:00

Recently I've had to tackle the task of making two different software architectures, a modern one and a legacy one (as it happens), work together in C++. These two architectures were based on two completely different event-based frameworks, each one with their own event loop, and it was clear that they were not intended to work toegether as one, but rather assume that they are completely isolated components that handle the entire application lifecycle.

While it might seem like a rather complicated conundrum at first glance, I've come to realize that the solution is, in my humble opinion, quite simple in modern C++.

Overview of the architecture

Let's set the scene. For this post I'm going to assume that I have access to (at least) C++11 and my two components are based on the following frameworks:

A graphical application written in Qt 6 (this is the modern component)
A set of servers and remote communication handlers based on (Boost) Asio (this is the legacy component)

Now, before people start suggesting a complete rewrite of the legacy in Qt, we are going to assume that there are some constraints in place that require both elements in the software, modifying the legacy as little as possible.

These two components have their own event loop to handle their specific events (obviously). In the case of Qt, the event loop might be started like this:

#include <QGuiApplication>

int main(int argc, char* argv[])
{
    QGuiApplication app{argc, argv};

    // Event loop starts here
    app.exec();

    return 0;
}

Where the QGuiApplication::exec() method has the following information in Qt's documentation:

Enters the main event loop and waits until exit() is called, and then returns the value that was set to exit() (which is 0 if exit() is called via quit()).

For Asio, the event loop might be started like this:

#include <boost/asio.hpp>

int main(int argc, char* argv[])
{
    boost::asio::io_context context{};

    // Create a work guard to have the event loop run forever
    auto work_guard{boost::asio::make_work_guard(context)};

    // Event loop starts here
    context.run();

    return 0;
}

The work_guard is there only to prevent the event loop from stopping if there are no more pending tasks. As with Qt, the io_context::run() method states the following in its documentation:

The run() function blocks until all work has finished and there are no more handlers to be dispatched, or until the io_context has been stopped.

Now, regarding how to run both event loops at the same time, the solution is rather easy: simply run each of them in a different thread. For example, Qt might be launched in the main thread, and Asio in a secondary thread that will be stopped when the program terminates. Easy, right?

But, then again... These two are supposed to communicate and exchange information among themselves. How shall we do that?

Information exchange

This has suddenly become one of those classic synchronisation problems that universities love so much: "need to access something from a different thread? Get a mutex/atomic/semaphore/other-method in there and be done with it!"

While that is certainly a possible approach, what happens if we need to run some code in a specific event loop? I'm pretty sure Qt will complain (and that's putting it lightly) if we attempt to modify some graphical element from a different thread. Or perhaps the software needs to call a method in the Asio thread to notify a remote server about something? A lock-based solution is a possibility, but I believe it would really increase the complexity of the application (even more).

Instead, why not use some sort of message exchange-like mechanism so that their processing mechanisms do not interfere with each other?

Just of a moment, I'm going to forget about the fact that I'm talking about software components. Instead, let's imagine there are two co-workers, Alice and Bob, working side by side, each with their own tasks. Occasionally they need to interact with each other to do their job, like in the following conversation:

Bob: "Hey, Alice. I just finished writing that report our manager asked for. Would you mind giving it a look and forwarding it to her if you think it's okay as it is? I left it in our shared drive."

Alice: "Sure, just let me finish writing this email first."

Bob: "Of course, no rush. Thanks!"

A few minutes later

Alice: "Alright, I forwarded the report. By the way, I received a document from Carl, from our overseas branch, regarding the new project you are working on. I think he sent it to me by error because we tend to work together in most projects. I'll send it to you."

Bob: "Fantastic, thank you Alice."

See that? That is exactly the kind of interaction that we are looking for. Mr. Bob Qt has processed something and asks his colleague Ms. Alice Asio to review the information and send it to a remote location. Furthermore, Ms. Alice Asio receives a message and passes it on to Mr. Bob Qt. This is purely an event-based architecture, where both of them react to actions and offer a reaction independently.

But the question remains: how are we going to translate that to C++?

Everything is a function

One of my favourite things about the C++11 standard has to be std::function: it's like a good old (and unsafe) void*, but treated as an object and type safe. Pretty cool, if you ask me.

With std::function I should be able to store a pointer to a bound method (or free function, if I so desired) and call it from wherever I want, which solves half of the problem: Qt side would be able to invoke methods from the Asio side and vice versa. For example, we could define a Callback class like:

// callback.hpp
#pragma once

#include <functional>

template<class... TArgs>
class Callback
{
public:
    // The methods we are going to call must not return anything
    using Callable = std::function<void(TArgs...)>;

    // Don't want to allow building an empty callback
    Callback() = delete;

    // Copyable
    Callback(const Callback&) = default;
    Callback& operator=(const Callback&) = default;

    // Movable
    Callback(Callback&&) = default;
    Callback& operator=(Callback&&) = default;

    virtual ~Callback() = default;

    /// @brief Callback constructor from a bound method.
    ///
    /// @details
    /// This constructor initializes the internal callable with a pointer to the specified bound @p method.
    /// The developer must guarantee the lifetime of the @p object to prevent invalid memory access.
    ///
    /// @tparam TClass              Object type.
    ///
    /// @param[in] object           Pointer to the object whose method is going to be called.
    /// @param[in] method           Pointer to the method that is going to be called. The arguments of the Callback
    ///                             must match those of this method.
    template<class TClass>
    Callback(TClass* object, void (TClass::*method)(TArgs...))
        // In C++20 I would use std::bind_front()
        : m_callable
        {
            // Object and method pointers are copied into the lambda's context
            [object, method](TArgs args...)
            {
                // In C++17 I would use std::invoke()
                (object->*method)(args...);
            }
        }
    {
    }

    /// @brief Invoke the underlying callable.
    ///
    /// @param[in] args             Arguments to provide to the invokation.
    void operator()(TArgs... args) const
    {
        m_callable(args...);
    }

private:
    Callable m_callable;
};

And, here is how it could be used:

// main.cpp
#include <iostream>
#include <string>

#include "callback.hpp"

class Test
{
public:
    void do_stuff(int a, const std::string& b)
    {
        std::cout << "Doing stuff with " << a << " and " << b << std::endl;
    }
};

int main()
{
    // Create dummy object to cal its method
    Test test{};

    // Create a callback and invoke it
    Callback<int, const std::string&> cb{&test, &Test::do_stuff};
    cb(42, "a message");

    return 0;
}

The output would be:

Doing stuff with 42 and a message

Now, let's break it down. The Callback object is a simple wrapper around std::function that forces it to point to a method that returns and only method arguments are provided as part of the template. A Callback object has to be constructed by providing an object pointer and a method pointer belonging to that class (TClass) and matching the arguments provided to the Callback (TArgs...). The invocation of the stored class method itself is done via a lambda function, which comes in very handy to store the pointers for deferred execution.

Whenever a Callback object is invoked, the operator() is called, which in turn calls the underlying std::function, which in turn calls the lambda, which in turn calls Test::do_stuff(int a, const std::string& b) of object test.

Even though this is pretty cool (again, my opinion), it has an obvious limitation: invocation of the Callback happens in the same thread from where it was called, meaning that it does not solve the problem at hand entirely.

Making it dance

The interesting thing about event loops is that most, if not all, have a way to post an event to the event loop. Redundant, I know. But we can use such capabilities to our advantage. For Qt, I'm going to focus on QMetaObject::invokeMethod(), and for Asio, on boost::asio::post().

Both of these will allow us to execute something in the corresponding event loop, by enqueueing the task and notifying the event loop to handle it (eventually) in its own thread.

With this information, I'm going to rewrite the previous Callback a little, while still retaining the same API in terms of invocation (the operator() operator).

Qt version

// qt-callback.hpp
#pragma once

#include <functional>

#include <QObject>

template<class... TArgs>
class QtCallback
{
public:
    // The methods we are going to call must not return anything
    using Callable = std::function<void(TArgs...)>;

    // Don't want to allow building an empty callback
    QtCallback() = delete;

    // Copyable
    QtCallback(const QtCallback&) = default;
    QtCallback& operator=(const QtCallback&) = default;

    // Movable
    QtCallback(QtCallback&&) = default;
    QtCallback& operator=(QtCallback&&) = default;

    virtual ~QtCallback() = default;

    /// @brief Qt Callback constructor from a bound method.
    ///
    /// @details
    /// This constructor initializes the internal callable with a pointer to the specified bound @p method.
    /// The developer must guarantee the lifetime of the @p object to prevent invalid memory access.
    ///
    /// @tparam TClass              Object type. Must derive from QObject.
    ///
    /// @param[in] object           Pointer to the QObject-based object whose method is going to be called.
    /// @param[in] method           Pointer to the method that is going to be called. The arguments of the Callback
    ///                             must match those of this method.
    template<typename TClass, std::enable_if_t<std::is_base_of_v<QObject, TClass>, bool> = true>
    QtCallback(TClass* object, void (TClass::*method)(TArgs...))
        : m_callable
        {
            // Object and method pointers are copied into the lambda's context
            [object, method](TArgs... args)
            {
                QMetaObject::invokeMethod(
                    // Event loop/thread is obtained from the object itself
                    object,
                    // Yes, a lambda within a lambda to copy the arguments for the other thread
                    [object, method, args...]()
                    {
                        // In C++17 I would use std::invoke()
                        //
                        // Try to be efficient about memory usage, because copied arguments are
                        // discarded after the lambda returns.
                        (object->*method)(std::move(args)...);
                    },
                    // Make sure that the event is enqueued so that it gets executed in Qt's thread
                    Qt::QueuedConnection
                );
            }
        }
    {
    }

    /// @brief Invoke the underlying callable.
    ///
    /// @param[in] args             Arguments to provide to the invokation.
    void operator()(TArgs... args) const
    {
        m_callable(args...);
    }

private:
    Callable m_callable;
};

This QtCallback works exactly the same way as the original Callback, with one difference: the object pointer provided in the constructor must derive from QObject as this allows Qt to determine the thread in which to execute the function. This is done via the QMetaObject::invokeMethod() call inside the first lambda:

QMetaObject::invokeMethod(
    // Event loop/thread is obtained from the object itself
    object,
    // Yes, a lambda within a lambda to copy the arguments for the other thread
    [object, method, args...]()
    {
        // In C++17 I would use std::invoke()
        //
        // Try to be efficient about memory usage, because copied arguments are
        // discarded after the lambda returns.
        (object->*method)(std::move(args)...);
    },
    // Make sure that the event is enqueued so that it gets executed in Qt's thread
    Qt::QueuedConnection
);

There is some Qt magic at work here, but the summary is:

Arguments (args...) are copied from the parent lambda
The arguments are std::move()d when calling the method to try to prevent further copies
This lambda is executed by Qt considering a Qt::QueuedConnection, which guarantees that it will be enqueued in the event loop and executed from its thread

This is completely thread safe, as the invocation is deferred to Qt's own thread.

Asio version

// asio-callback.hpp
#pragma once

#include <functional>

#include <boost/asio.hpp>

template<class... TArgs>
class AsioCallback
{
public:
    // The methods we are going to call must not return anything
    using AsioCallback = std::function<void(TArgs...)>;

    // Don't want to allow building an empty callback
    AsioCallback() = delete;

    // Copyable
    AsioCallback(const AsioCallback&) = default;
    AsioCallback& operator=(const AsioCallback&) = default;

    // Movable
    AsioCallback(AsioCallback&&) = default;
    AsioCallback& operator=(AsioCallback&&) = default;

    virtual ~AsioCallback() = default;

    /// @brief Callback constructor from a bound method.
    ///
    /// @details
    /// This constructor initializes the internal callable with a pointer to the specified bound @p method.
    /// The developer must guarantee the lifetime of the @p object to prevent invalid memory access.
    ///
    /// @tparam TClass              Object type.
    ///
    /// @param[in] context          Pointer to the Asio context (event loop).
    /// @param[in] object           Pointer to the object whose method is going to be called.
    /// @param[in] method           Pointer to the method that is going to be called. The arguments of the Callback
    ///                             must match those of this method.
    template<class TClass>
    AsioCallback(boost::asio::io_context* context, TClass* object, void (TClass::*method)(TArgs...))
        : m_callable
        {
            // Object and method pointers are copied into the lambda's context
            [context, object, method](TArgs... args)
            {
                boost::asio::post(
                    // Event loop is accessed via the context
                    context->get_executor(),
                    // Yes, a lambda within a lambda to copy the arguments for the other thread
                    [object, method, args...]() {
                        // In C++17 I would use std::invoke()
                        //
                        // Try to be efficient about memory usage, because copied arguments are
                        // discarded after the lambda returns.
                        (object->*method)(std::move(args)...)
                    }
                )
            }
        }
    {
    }

    /// @brief Invoke the underlying callable.
    ///
    /// @param[in] args             Arguments to provide to the invokation.
    void operator()(TArgs... args) const
    {
        m_callable(args...);
    }

private:
    Callable m_callable;
};

Like QtCallback, the AsioCallback remains mostly unchanged, except for an additional context parameter in the constructor which allows us to tell Asio the event loop in which to enqueue the invocation of the function/lambda. As before, we pass Asio a second lambda to execute as part of boost::asio::post():

boost::asio::post(
    // Event loop is accessed via the context
    context->get_executor(),
    // Yes, a lambda within a lambda to copy the arguments for the other thread
    [object, method, args...]() {
        // In C++17 I would use std::invoke()
        //
        // Try to be efficient about memory usage, because copied arguments are
        // discarded after the lambda returns.
        (object->*method)(std::move(args)...)
    }
)

Again, here is a small summary of what this is doing:

Arguments (args...) are copied from the parent lambda
The arguments are std::move()'d when calling the method to try to prevent further copies
This lambda is enqueued (posted) by Asio to the executor of the provided context so that it is deferred and executed from its thread

Again, this is completely thread safe, as the invocation is deferred to Asio's own thread.

Conclusions

This was just a small introduction to how std::function() can be (ab)used to connet different event loops in a thread-safe way with as little overhead as possible.

Of course, it is possible to improve this solution way further, such as using std::shared_ptr in for the callback arguments to prevent any copy while passing the information to the other thread. Or even having a generic Callback object that other framework-dependent specialized callbacks can derive from to allow our code to be completely agnostic of who is listening on the other side of the callback, which is whay I did in my Asio-based framework Kouta.

Goodbye cookies, hello logs

2022-07-20T18:00:00+02:00

Currently (as of 2022-07-19) I'm using Matomo to have some self-hosted web analytics. In particular, I'm intrested in knowing which posts people find interesting over time (monthly). However, I'm not too fond of having to enable cookies in my website, so I have been looking for some way to process server logs from Nginx to get that information without having to bother visitors with cookies. That's where GoAccess comes in.

What is GoAccess?

As stated in the official GoAccess website:

GoAccess is an open source real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.

It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly.

That is exactly what I'm looking for. In addition, it can be used as a TUI to browse the logs or generate a report in HTML, which I could then send to my email weekly or monthly.

Install GoAccess

Initially, I tried to install GoAccess from the Debian package repository, but the version in the repository is not the latest at the time of writing (1.6.2). Therefore, I followed the instructions in the GoAccess documentation to use GoAccess' official Debian/Ubuntu repository, giving me the latest stable version of the software.

After installation I was eager to try out the parsing of the access logs of my site:

goaccess /var/log/nginx/website_access.log

The terminal UI shows a screen asking me to specify the log format my files use. Because I'm using the default Nginx configuration, I chose the NCSA Combined Log Format option and lo and behold, I get a terminal dashboard with my statistics. Initially I chose the Common Log Format (CLF), but GoAccess was not correctly identifying clients and crawlers.

Now, my intent, as I'll show later, is to automate this, so I went ahead and uncommented the log-format COMBINED entry in /etc/goaccess/goaccess.conf to default to this format whenever goaccess is executed.

Parsing all the logs!

The previous command parsed the latest access log from Nginx, but depending on the log rotation configured it may not be too helpful (e.g. it could just contain the entries from the previous day). Luckily, my log rotation is configured to keep a certain number of old logs compressed with filenames such as website_access.log.10.gz, meaning that I can ingest them as well to get a more complete dataset.

In order to do so, and again, following the documentation, it is enough to execute the following:

zcat /var/log/nginx/website_access.log.*.gz | goaccess /var/log/nginx/website_access.log -

Now I get a complete overview from all the access logs of the site, with some interesting sections for web analytics:

Unique visitors per day - Including spiders: as its name implies, registers all the clients that have accessed a URL. This includes crawlers, but we'll take care of that later
Requested Files (URLS): in my case, pages accessed by visitors, ordered from most visited to least visited
Time distribution: when the visitors are most active during the day
Geo location: summary of countries that visitors are from

But there is also other data that I may find interesting for other reasons:

Static Requests: static assets downloaded
Not Found URLs (404s): Come on, people, stop trying to acces a non-existent /wp-login.php

The navigation needs a bit of getting used to, but once you get the hang of it it feels quite Vim-like (arrows to go up/down in the dashboard, numbers to jump to a specific section, return to open the section, j/k to go up/down in the section).

Incremental log processing

For sites that have a lot of traffic, the log files can grow pretty quickly making them harder to parse and process quickly. For that reason, I find the persistence capabilities of GoAccess really cool. It allows you to save an initial snapshot of the logs in an internal database and then only add new entries when parsing logs.

To do this, GoAccess keeps track of the timestamp of the latest entries and will only add log lines to the database if their timestamp is more recent.

On to business! I created the snapshot using only the compressed logs:

zcat /var/log/nginx/website_access.log.*.gz | goaccess --persist --db-path logs/www.rmedgar.com -

Note that --db-path points to a directory that I created beforehand.

Afterwards, I append data from not compressed logs:

goaccess --restore --persist --db-path logs/www.rmedgar.com /var/log/nginx/website_access.log.1 /var/log/nginx/website_access.log -

Note that in order to append data to an existing database, both --restore and --persist must be used.

Finally, the dataset can be read directly from the database using the following command:

goaccess --restore --db-path logs/www.rmedgar.com

If I wanted to periodically add logs to this database, I could set up a cronjob running the following command:

goaccess --restore --persist --db-path logs/www.rmedgar.com --process-and-exit

With --process-and-exit the TUI will not be displayed and goaccess will terminate after adding the new entries.

Filtering and reporting

Current version of GoAccess (1.6.2) has no way to filter a dataset after it has been stored in the internal database. You can only browse it as-is. The drawback with this is that I'd have to ingest the data once per report I wanted to generate for myself. For example, I'm interested in the following:

Report of visits of humans per month: for some statistics regarding visitors
Report of visits of crawlers per month: again, it's always funny to see crawlers trying to access a non-existent Wordpress instance

That means I would have to follow this process each month:

At the beginning of the month, ingest all logs into two different databases (with --ignore-crawlers and with --crawlers-only respectively), using the --persist option and filtering with grep or similar to get only entries from the current month. These databases should not overwrite the ones from the previous month
Hourly (for example) read latest log files applying the same filters and using the --restore --persist options to update the respective database

Although a bit inconvenient in terms of having to process the same dataset twice, it can be easily scripted, so it's not much of an issue. Still, there is an open issue in the GoAccess repository that covers this sort of use case, so I imagine the feature may arrive at some point.

Periodic ingest

Taking into account the previous information, I devised the following script to be executed as part of a cronjob:

#!/usr/bin/env bash

# Directory in which the databases will be stored
STORAGE_PATH="/srv/web-analytics"

# Main
while getopts f:p:h: flag
do
    case "${flag}" in
        f) ADDITIONAL_FILTERS=${OPTARG};;
        p) ADDITIONAL_PARAMS=${OPTARG};;
        h)
            show_help
            exit;;
    esac
done

# Name of the site being analyzed
SITENAME=${@:$OPTIND:1}

# Site variant
VARIANT=${@:$OPTIND+1:1}

# Absolute path to the base log file.
#
# For example:
#
# /var/log/nginx/mysite.log
LOG_PATH=${@:$OPTIND+2:1}

if [ -z $SITENAME ] || [ -z $VARIANT ] || [ -z $LOG_PATH ]
then
    show_help
    exit
fi


# Ingest the data, creating the initial database if needed
ingest() {
    full_date=$(date '+%Y-%m-%d %H:%M:%S')
    dir_name=$(date '+%Y-%m' -d "$full_date")
    date_filter="/$(date +'%b/%Y'):"

    # Database directory
    db_path="$STORAGE_PATH/$SITENAME/$dir_name/$VARIANT"

    # Create directory and database if it does not exist already
    if [ ! -d $db_path ] 
    then
        mkdir -p "$db_path"

        # Begin with compressed logs
        if [ -z $ADDITIONAL_FILTERS ]
        then
            zcat $LOG_PATH.*.gz | grep "$date_filter" | goaccess --persist --db-path $db_path --process-and-exit $ADDITIONAL_PARAMS -
        else
            zcat $LOG_PATH.*.gz | grep "$date_filter" | grep "$ADDITIONAL_FILTERS" | goaccess --persist --db-path $db_path --process-and-exit $ADDITIONAL_PARAMS -
        fi
    fi

    # Incremental logs (parse those with a number)
    for LOG in $(find `dirname $LOG_PATH` -name "`basename $LOG_PATH`*" -not -name "*.gz" -type f | xargs echo)
    do
        echo "Ingesting $LOG"

        if [ -z $ADDITIONAL_FILTERS ]
        then
            cat $LOG | grep "$date_filter" | goaccess --restore --persist --db-path $db_path --process-and-exit $ADDITIONAL_PARAMS -
        else
            cat $LOG | grep "$date_filter" | grep "$ADDITIONAL_FILTERS" | goaccess --restore --persist --db-path $db_path --process-and-exit $ADDITIONAL_PARAMS -
        fi
    done
}


# Show help message
show_help() {
    echo "Ingest log files for web analytics"
    echo ""
    echo "Syntax: nginx-analytics.sh [-f <filter>] [-p <params>] [-h] <sitename> <variant> <log_path>"
    echo ""
    echo "Options:"
    echo ""
    echo "  -f <filter>     Additional filter to use before ingesting the logs (grep)"
    echo "  -p <params>     Additional parameters to provide to goaccess"
    echo "  -h              Show this help message and exit"
    echo ""
    echo "Argsuments:"
    echo ""
    echo "  sitename        Name of the site being analyzed"
    echo "  variant         Name of the report variant for this site"
    echo "  log_path        Absolute path to the base log file"
}


ingest

The usage is really simple. Supposing I wanted to have the following reports for my site:

Complete report: logs parsed as is, nginx-analytics.sh mysite complete /var/log/nginx/mysite_access.log
Crawler report: only extract information regarding crawler visits, nginx-analytics.sh -p "--crawlers-only" mysite crawlers /var/log/nginx/mysite_access.log
Client report: only extract information regarding normal client visits, nginx-analytics.sh -p "--ignore-crawlers" mysite clients /var/log/nginx/mysite_access.log

This will generate the appropriate databases:

srv/web-analytics/mysite/YY-mm/complete
srv/web-analytics/mysite/YY-mm/crawkers
srv/web-analytics/mysite/YY-mm/clients

Next executions of the script with the same parameters will just append new data as long as we are in the same month. Otherwise, the databases corresponding to the next month will be created.

Moreover, if I wanted to add more filtering (e.g. to get only hits on the blog pages of the site), I could use the -f <filter> parameter to have an additional grep filter before ingesting the logs.

Conclusion

Even with the aforemenioned drawback, I think I will set up my web analytics using my Nginx logs and shut down the Matomo instance, removing cookies from my website in the process. What's more, I think GoAccess coud prove very useful in the future in the eventuallity that I have to analyze other web logs in bulk.

One thing I didn't really explore is the option to generate an HTML report, nor the real-time server for checking the statistics in real-time, but those seem to be really powerful options to gain further insight.

SWIG, C, Python, and other tales

2021-05-09T18:45:00+02:00

Python has lots of libraries available for almost anything imaginable. However, there are times when, for one reason or another, you need to use native code written in C/C++ (e.g. legacy code, hard requirement for a specific project, etc.). Although there are several options to overcome this need, we are going to have a look at SWIG.

Contents

Enter Swig
Look at that tree!
Python wrapper
The quirks
Conclusions
References

Enter SWIG

As described in the official site of SWIG:

SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages. SWIG is used with different types of target languages including common scripting languages such as Javascript, Perl, PHP, Python, Tcl and Ruby.

What SWIG does is create a wrapper that allows our favorite programming language to interact with the native code almost transparently. I say "almost" because there are some quirks that should be taken into account depending on the language you are trying to write a wrapper for. Take Python for instance: how would C pointers and arrays be translated into the Python world, where there is garbage collection and pointers are not really something you see every day?

In this post we will see some of these special cases and how to deal with them.

Look at that tree!

Let's have an example C code defining a tree based on the following points:

The tree has a specific age measured in years
The tree has a specific height measured in meters
The tree is not a cilinder! Therefore, its trunk can have different diameters depending on the height
The tree has multiple branches
- Each branch has a specific length
- Each branch has a given number of leaves
  - Each leaf can be a specific color: green, orange, red, brown

This is how our tree might look like in C:

typedef enum Color {
    GREEN,
    ORANGE,
    RED,
    BROWN
} Color;

typedef struct Leaf {
    Color color;
} Leaf;

typedef struct Branch {
    double length;
    size_t num_leaves;
    Leaf *leaves;
} Branch;

typedef struct Tree {
    unsigned int age;
    unsigned int height;
    double *diameters;
    size_t num_branches;
    Branch *branches;
} Tree;

For the sake of simplicity, the age and height of the tree will be considered as integer values, and we won't care about where the branches or leaves are in the tree.

Next, we will want to have a function to initialize our tree with some default values:

Tree *new_tree(int age, int height, size_t num_branches) {
    Tree *my_tree = malloc(sizeof(Tree));

    // Trunk
    my_tree->age = age;
    my_tree->height = height;

    int i, j;

    // Trunk diameters
    double *diam = malloc(sizeof(double) * height);

    for (i = 0; i < height; i++) {
        diam[i] = 1.2;
    }

    my_tree->diameters = diam;

    // Branches
    Branch *branches = malloc(sizeof(Branch) * num_branches);

    for (i = 0; i < num_branches; i++) {
        branches[i].length = 1.5;
        branches[i].num_leaves = 42;

        Leaf *leaves = malloc(sizeof(Leaf) * branches[i].num_leaves);

        // Happy little tree
        for (j = 0; j < branches[i].num_leaves; j++) {
            leaves[j].color = GREEN;
        }

        branches[i].leaves = leaves;
    }

    my_tree->num_branches = num_branches;
    my_tree->branches = branches;

    return my_tree;
}

As well as another function that simply prints the tree structure:

void show_tree(Tree *t) {
    int i, j;
    char *leaf_color;

    // Basic info
    printf("My tree is %d years old and %d meters high\n", t->age, t->height);
    printf("These are the diameters of my tree:\n");

    for (i = 0; i < t->height; i++) {
        printf("%f ", t->diameters[i]);
    }

    printf("\n");

    // Branch info
    printf("Don't forget about all the branches in my tree:\n");

    for (i = 0; i < t->num_branches; i++) {
        printf("- Branch %d is %f meters long and has %zu leaves with following colors:\n", i, t->branches[i].length, t->branches[i].num_leaves);

        for (j = 0; j < t->branches[i].num_leaves; j++) {
            switch (t->branches[i].leaves[j].color) {
                case GREEN:
                    leaf_color = "green";
                    break;

                case ORANGE:
                    leaf_color = "orange";
                    break;

                case RED:
                    leaf_color = "red";
                    break;

                case BROWN:
                    leaf_color = "brown";
                    break;

                default:
                    break;
            }
            printf("%s ", leaf_color);
        }

        printf("\n");
    }
}

Our final supertree.c file would look like this:

#include "supertree.h"

Tree *new_tree(int age, int height, size_t num_branches) {
    Tree *my_tree = malloc(sizeof(Tree));

    // Trunk
    my_tree->age = age;
    my_tree->height = height;

    int i, j;

    // Trunk diameters
    double *diam = malloc(sizeof(double) * height);

    for (i = 0; i < height; i++) {
        diam[i] = 1.2;
    }

    my_tree->diameters = diam;

    // Branches
    Branch *branches = malloc(sizeof(Branch) * num_branches);

    for (i = 0; i < num_branches; i++) {
        branches[i].length = 1.5;
        branches[i].num_leaves = 42;

        Leaf *leaves = malloc(sizeof(Leaf) * branches[i].num_leaves);

        // Happy little tree
        for (j = 0; j < branches[i].num_leaves; j++) {
            leaves[j].color = GREEN;
        }

        branches[i].leaves = leaves;
    }

    my_tree->num_branches = num_branches;
    my_tree->branches = branches;

    return my_tree;
}

void show_tree(Tree *t) {
    int i, j;
    char *leaf_color;

    // Basic info
    printf("My tree is %d years old and %d meters high\n", t->age, t->height);
    printf("These are the diameters of my tree:\n");

    for (i = 0; i < t->height; i++) {
        printf("%f ", t->diameters[i]);
    }

    printf("\n");

    // Branch info
    printf("Don't forget about all the branches in my tree:\n");

    for (i = 0; i < t->num_branches; i++) {
        printf("- Branch %d is %f meters long and has %zu leaves with following colors:\n", i, t->branches[i].length, t->branches[i].num_leaves);

        for (j = 0; j < t->branches[i].num_leaves; j++) {
            switch (t->branches[i].leaves[j].color) {
                case GREEN:
                    leaf_color = "green";
                    break;

                case ORANGE:
                    leaf_color = "orange";
                    break;

                case RED:
                    leaf_color = "red";
                    break;

                case BROWN:
                    leaf_color = "brown";
                    break;

                default:
                    break;
            }
            printf("%s ", leaf_color);
        }

        printf("\n");
    }
}


int main() {
    Tree *tree = new_tree(1, 1, 3);

    show_tree(tree);

    free(tree);
    return 0;
}

While supertree.h would look like this:

#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>

typedef enum Color {
    GREEN,
    ORANGE,
    RED,
    BROWN
} Color;

typedef struct Leaf {
    Color color;
} Leaf;

typedef struct Branch {
    double length;
    size_t num_leaves;
    Leaf *leaves;
} Branch;

typedef struct Tree {
    unsigned int age;
    unsigned int height;
    double *diameters;
    size_t num_branches;
    Branch *branches;
} Tree;

Tree *new_tree(int age, int height, size_t num_branches);
void show_tree(Tree *t);

Compiling and running our code results in the following:

$ gcc supertree.c -o supertree

$ ./supertree
My tree is 1 years old and 1 meters high
These are the diameters of my tree:
1.200000
Don't forget about all the branches in my tree:
- Branch 0 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green
- Branch 1 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green
- Branch 2 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green

Python wrapper

In order to create the Python wrapper as a module we will first create a SWIG input file supertree.i with the following content:

%module supertree

%{
#include "supertree.h"
%}

%include "supertree.h"

Now for the creation of the wrapper itself:

$ swig -python -py3 supertree.i

This will result in the following two files:

supertree_wrap.c: generated C code for the interface
supertree.py: python module that will serve as the entrypoint for the interface

Although the SWIG-generated code might be a bit overwhelming to read, let's check what our main Tree struct would look like:

class Tree(object):
    thisown = property(lambda x: x.this.own(), lambda x, v: x.this.own(v), doc="The membership flag")
    __repr__ = _swig_repr
    age = property(_supertree.Tree_age_get, _supertree.Tree_age_set)
    height = property(_supertree.Tree_height_get, _supertree.Tree_height_set)
    diameters = property(_supertree.Tree_diameters_get, _supertree.Tree_diameters_set)
    num_branches = property(_supertree.Tree_num_branches_get, _supertree.Tree_num_branches_set)
    branches = property(_supertree.Tree_branches_get, _supertree.Tree_branches_set)

    def __init__(self):
        _supertree.Tree_swiginit(self, _supertree.new_Tree())
    __swig_destroy__ = _supertree.delete_Tree

# Register Tree in _supertree:
_supertree.Tree_swigregister(Tree)


def new_tree(age: "int", height: "int", num_branches: "size_t") -> "Tree *":
    return _supertree.new_tree(age, height, num_branches)

SWIG translates C structs into Python classes, so managing their attributes should be pretty straight-forward when working with an object of this class. Note as well that everything comes from the _supertree_ module, which will be the native module resulting from compiling the wrapper.

The SWIG documentation contains an example on compiling everything using distutils, but here we are going to do it manually. As we are compiling for Python 3, it will be necessary to bave the Python development libraries installed. On Debian/Ubuntu systems:

# apt install python3-dev

Now, for the compilation itself, it can be done in three steps:

$ gcc -O2 -fPIC -c supertree.c
$ gcc -O2 -fPIC -c supertree_wrap.c $(python3-config --includes)
$ gcc -shared supertree.o supertree_wrap.o -o _supertree.so

We should now have a _supertree.so file with our compiled Python module, let's try it out!

>>> import supertree
>>> mytree = supertree.new_tree(1, 1, 3)
>>> supertree.show_tree(mytree)
My tree is 1 years old and 1 meters high
These are the diameters of my tree:
1.200000
Don't forget about all the branches in my tree:
- Branch 0 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green
- Branch 1 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green
- Branch 2 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green

Great, we can now effectively call the underlying C code from Python!

The quirks

As I was mentioning earlier, there are some teeny-tiny things to take into account when using the SWIG wrapper. Let's start by trying to modify some attributes of the Tree object from Python:

>>> import supertree
>>> mytree = supertree.new_tree(1, 1, 3)
>>> mytree.age
1
>>> mytree.age = 12
>>> mytree.age
12

No issues so far, let's try modifying the only diameter value we have (remember, our tree is only 1 meter long):

>>> mytree.diameters
<Swig Object of type 'double *' at 0x7f48ab15eae0>
>>> mytree.diameters[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'SwigPyObject' object is not subscriptable

Huh, apparently we can't access the double pointer to read/modify the data. Let's see what happens with the branches:

>>> mytree.branches
<supertree.Branch; proxy of <Swig Object of type 'Branch *' at 0x7f48ab0a5720> >
>>> mytree.branches[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'Branch' object is not subscriptable
>>> mytree.branches.length
1.5
>>> mytree.branches.num_leaves
42

Interesting, we have the same problem as before, but because the Branch C struct is mapped to the Python class, it seems we can only access the first element.

These are the sort of problems I was talking about: there are certain things that simply do not have a direct translation from C to the target language. That being said, SWIG is flexible enough and offers several solutions for these situations. Let's have a look at some of them!

Arrays/pointers of core data types

Previously we saw that we cannot access the pointer of double that contains the diameters of our tree trunk. This is not entirely accurate, as we cannot access it directly. The main problem here is that we are getting a SwigPyObject instance that wraps said pointer, so there is no straightforward way to access the memory from the Python side.

One possible solution comes from the ctypes Python library: ideally, we should be able to create a pointer to the memory address containing the diameters and read/modify the values from Python. Now, you might be wondering "where do we get the memory address from?", well the answer to that is from the SwigPyObject itself!

Let's have a look at the internals of the object:

>>> dir(mytree.diameters)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__int__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'acquire', 'append', 'disown', 'next', 'own']

Most of these are SWIG-specific, but we can also see that the object implements the special __int__ method, which allows the usage of int(obj). Let's see what it returns for this instance:

>>> int(mytree.diameters)
27381776

Although it may seem an incredibly random value at first, that is indeed the memory address of our diameters!. Don't believe me? Check it out:

>>> import ctypes
>>> mem = (ctypes.c_double * mytree.height).from_address(int(mytree.diameters))
>>> pointer = ctypes.pointer(mem)
>>> pointer.contents
<__main__.c_double_Array_1 object at 0x7fc49c1275c0>
>>> pointer.contents[0]
1.2
>>> pointer.contents[0] = 4.5
>>> pointer.contents[0]
4.5

Of course, if we now print the tree structure we should be able to see the changes at the C level:

>>> supertree.show_tree(mytree)
My tree is 1 years old and 1 meters high
These are the diameters of my tree:
4.500000
Don't forget about all the branches in my tree:
- Branch 0 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green
- Branch 1 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green
- Branch 2 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green

Warning

Be careful with this way of accessing memory, as there are no bounds whatsoever and you have to take care of the indexing yourself.

Arrays/pointers of C structures: access by index

The previous way of accessing pointers works wonders for core data types, however it is not viable when dealing with more complex types such as the Branch and Leaf C structures. As we saw previously whenever we try to access these elements we only get the first object in the array, making it a bit hard to work with.

One of the charms of SWIG is that it provides a way of extending the generated code with custom methods. This means that we can modify our supertree.i file to include something like the following:

%extend Branch {
    Leaf *__getitem__(size_t i) {
        return &$self->leaves[i];
    }
}

What we are doing here is implement the special __getitem__ method from Python, which should allow us to access leaves in our branch instances by index. What's more, given that the function returns a pointer, we can also modify the element in-place!

After regenerating the wrapper, and recompiling, we see that the method has been added to our Python class in supertree.py:

class Branch(object):
    thisown = property(lambda x: x.this.own(), lambda x, v: x.this.own(v), doc="The membership flag")
    __repr__ = _swig_repr
    length = property(_supertree.Branch_length_get, _supertree.Branch_length_set)
    num_leaves = property(_supertree.Branch_num_leaves_get, _supertree.Branch_num_leaves_set)
    leaves = property(_supertree.Branch_leaves_get, _supertree.Branch_leaves_set)

    def __getitem__(self, i: "size_t") -> "Leaf *":
        return _supertree.Branch___getitem__(self, i)

    def __init__(self):
        _supertree.Branch_swiginit(self, _supertree.new_Branch())
    __swig_destroy__ = _supertree.delete_Branch

Let's try it out:

>>> import supertree
>>> mytree = supertree.new_tree(1,1,3)
>>> mytree.branches[4]
<supertree.Leaf; proxy of <Swig Object of type 'Leaf *' at 0x7f22157be840> >
>>> mytree.branches[4].color
0

>>> # Try changing the color
>>> for i in range(4,36):
...     mytree.branches[i].color = supertree.RED

>>> supertree.show_tree(mytree)
My tree is 1 years old and 1 meters high
These are the diameters of my tree:
1.200000
Don't forget about all the branches in my tree:
- Branch 0 is 1.500000 meters long and has 42 leaves with following colors:
green green green green red red red red red red red red red red red red red red red red red red red red red red red red red red red red red red red red green green green green green green
- Branch 1 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green
- Branch 2 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green

As you can see, we have effectively changed the color of several leaves of the final object at the same time.

Warning

Be careful with this way of accessing memory, as there are no bounds whatsoever and you have to take care of the indexing yourself.

Arrays/pointers of C structures: access by array

What if we had several arrays in a single structure? Obviously the __getitem__ method would only allow us to access the elements of a single array, so we might need something a bit more powerful than that.

The following Stack Overflow answer provides a very interesting solution. Basically, what we will want to do is tell SWIG to generate a special array class for a specific data type by means of the arrays.i SWIG file and then implement a custom method to return that array. For example:

%extend Tree {
    %pythonappend branch_list() {
        newval = _BranchArray.frompointer(val)
        newval.ptr_retain = val
        val = newval
    }

    Branch *branch_list() {
        return $self->branches;
    }
}

%include <carrays.h>
%array_class(Branch, _BranchArray)

Breaking it down:

We add the arrays.i and array class definitions at the end of the file
We extend the Tree class with a native branch_list() function that returns a pointer to the branches
Because we want a proper list from Python, we use the pythonappend directive to further extend the native method in order to create the array. This has to appear before the definition of the native method itself

Note that pythonappend, as its name implies, will add code after the function body of Branch *branch_list() and just before returning the value:

class Tree(object):
    thisown = property(lambda x: x.this.own(), lambda x, v: x.this.own(v), doc="The membership flag")
    __repr__ = _swig_repr
    age = property(_supertree.Tree_age_get, _supertree.Tree_age_set)
    height = property(_supertree.Tree_height_get, _supertree.Tree_height_set)
    diameters = property(_supertree.Tree_diameters_get, _supertree.Tree_diameters_set)
    num_branches = property(_supertree.Tree_num_branches_get, _supertree.Tree_num_branches_set)
    branches = property(_supertree.Tree_branches_get, _supertree.Tree_branches_set)

    def branch_list(self) -> "Branch *":
        val = _supertree.Tree_branch_list(self)

        newval = _BranchArray.frompointer(val)
        newval.ptr_retain = val
        val = newval


        return val


    def __init__(self):
        _supertree.Tree_swiginit(self, _supertree.new_Tree())
    __swig_destroy__ = _supertree.delete_Tree

If we access the data with this:

>>> import supertree
>>> mytree = supertree.new_tree(1,1,3)
>>> branch_list = mytree.branch_list()
>>> branch_list[0].length
1.5

>>> # Modify an element
>>> modified_branch = branch_list[1]
>>> modified_branch.length = 5
>>> branch_list[1] = modified_branch
>>> supertree.show_tree(mytree)
My tree is 1 years old and 1 meters high
These are the diameters of my tree:
1.200000
Don't forget about all the branches in my tree:
- Branch 0 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green
- Branch 1 is 5.000000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green
- Branch 2 is 1.500000 meters long and has 42 leaves with following colors:
green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green green

With this we now have the tools to read and modify all the information in our tree!

Warning

Be careful with this way of accessing memory, as there are no bounds whatsoever and you have to take care of the indexing yourself.

Conclusion

The examples we saw here might have been a bit rudimentary but I hope they serve as a small introduction into the world of SWIG.

As we saw, SWIG is a very powerful tool that offers many possibilities to allow us to interact with lower-level C code from an external library or program. While it is mostly transparent and most (simple) cases will require no modification to the wrapper, it might still be a good idea to make some adaptations to make the wrapper easier to work with.

Finally, I do recommend checking the SWIG documentation to have a deeper understanding of how more complex constructs would work. Although the learning curve might look a bit steep at first, I believe it will be worth it in the end.

References

SWIG documentation
Python ctypes library
SWIG Python Structure Array (Stack Overflow)

Migrate Isso comments to Remark42

2021-01-12T13:53:00+01:00

I recently changed the comment system of the website from Isso to Remark42. Although Remark42 can directly import comments from Disqus and Wordpress, there is no direct approach to migrating comments from an Isso instance, and it would be a shame to lose all of the existing comments. That is why I decided to dig into the documentation of Remark42 and its source code to figure out a way to perform the migration myself.

Before starting, a couple of clarifications:

Firstly, there is nothing wrong with Isso. It is a very nice piece of software and I have been using it for a long time without any problems. However, lately I had been noticing that some comments were not being rendered correctly (or at least not as I would expect them to render) and thought that it may be positive to try out a different comment engine. After looking at several options I opted for Remark42 because it offers several functionalities that I was already using in Isso:

Self-hosted
Notifications on new comments
Anonymous comments
Markdown support
Code highlighting
Reply subscription

Secondly, I'm not using Remark42 inside a Docker container. Instead, I have downloaded the binary and run it as a service, loading the configuration from a file. That means that if I write any remark42.linux-amd64 command in the post, I assume that the configuration has already been loaded beforehand, for simplicity.

Native migration options

As I mentioned previously, Remark42 natively supports migrating from Disqus and Wordpress. Taking this into account, my first thought was to generate an XML document with the comment data as Disqus would generate it by following the schema. I wrote a small script that extracted the comments from the Isso SQLite database and rendered the XML document using Jinja2.

I then tried to import the XML with the following command:

$ remark42.linux-amd64 import -f isso-import.xml -s mysite

Although this worked perfectly and Remark42 was correctly populated, I noticed a couple of issues:

After importing the data, Remark42 wouldn't retain the relation between comments (original comment, reply, etc.)
Remark42 considered all users to be Disqus users regardless of me identifying them as anonymous users

After playing around with Remark42 in my testing environment, I found out that users are identified using the following pattern: <PREFIX>_<HASH>, where <PREFIX> corresponds to the authentication method and <HASH> is a SHA-1 of the user identifier (which varies depending on the authentication method). In this case, users were identified as disqus_<HASH>, which was far from ideal.

Browsing the source code, I found the module in charge of importing Disqus comments, and the particular line that handled creation of users:

User: store.User{
    ID:   "disqus_" + store.EncodeID(comment.AuthorUserName),
    Name: comment.AuthorName,
    IP:   comment.IP,
},

Apparently, the same thing happens when importing comments from Wordpress, although with the wordpress_ prefix, so it wouldn't be possible for me to import comments using the Wordpress format either.

Time for plan B.

Solution: fake backup

One nice feature offered by Remark42 is the fact that it performs periodic backups of the comments. According to the README:

Backup file is a text file with all exported comments separated by EOL. Each backup record is a valid json with all key/value unmarshaled from Comment struct

If that is the case, it should be possible to convert the data from Isso to this backup format and import it normally using the restore command. I tried to perform a manual backup of the testing site and the resulting file (after decompressing) was as follows:

{"version":1,"users":[],"posts":[]}
{"id":"1","pid":"","text":"TEST","user":{"name":"test","id":"anonymous_4e1243bd22c66e76c2ba9eddc1f91394e57f9f83","picture":"","ip":"","admin":false},"locator":{"site":"mysite",url":"http://127.0.0.1"},"score":0,"vote":0,"time":"2021-01-12T12:00:35Z","title":"Test"}

That should be very easy to write using a script! Just need to understand the Comment struct definition:

type Comment struct {
    ID        string          `json:"id"`      // comment ID, read only
    ParentID  string          `json:"pid"`     // parent ID
    Text      string          `json:"text"`    // comment text, after md processing
    Orig      string          `json:"orig"`    // original comment text
    User      User            `json:"user"`    // user info, read only
    Locator   Locator         `json:"locator"` // post locator
    Score     int             `json:"score"`   // comment score, read only
    Vote      int             `json:"vote"`    // vote for the current user, -1/1/0.
    Controversy float64       `json:"controversy,omitempty"` // comment controversy, read only
    Timestamp time.Time       `json:"time"`    // time stamp, read only
    Edit      *Edit           `json:"edit,omitempty" bson:"edit,omitempty"` // pointer to have empty default in json response
    Pin       bool            `json:"pin"`     // pinned status, read only
    Delete    bool            `json:"delete"`  // delete status, read only
    PostTitle string          `json:"title"`   // post title
}

type Locator struct {
    SiteID string `json:"site"`     // site id
    URL    string `json:"url"`      // post url
}

type Edit struct {
  Timestamp time.Time `json:"time" bson:"time"`
  Summary   string    `json:"summary"`
}

The data structures for Isso, on the other hand, are as follows in the SQLite database:

threads table:
- id
- uri
- title
comments table:
- tid: Thread ID
- id
- parent: Comment ID when replying
- created
- modified
- mode
- remote_addr
- text: Plaintext (without markdown rendering)
- author
- email
- website
- likes
- dislikes
- voters
- notification

The script

Now, I'm not going to attempt a 1-to-1 migration of the data. Instead, I'm only going to export the data I consider most relevant. For comments, that means:

tid
id
parent
created
text
author
email

The following script will parse the data from the isso_comments.db file in the same directory and write the backup file for Remark42. As I prefer pre-rendering the markdown here, the following requirement must be installed before executing the script (Python 3):

$ pip3 install markdown

Below you can find the complete script:

# -*- coding: utf-8 -*-

import hashlib
import json
import sqlite3

from datetime import datetime
from markdown import markdown

# Change to admin email
ADMIN_USER = 'set_to_email'


def main():
    conn = sqlite3.connect('isso_comments.db')

    threads = {}
    comments = []

    cursor = conn.cursor()

    # Threads
    cursor.execute('SELECT * FROM threads')

    for tid, uri, title in cursor:
        threads[tid] = {
            'uri': uri,
            'title': title
        }

    # Extract comments
    cursor.execute('SELECT tid, id, parent, created, text, author, email FROM comments')

    for tid, id, parent, created, text, author, email in cursor:
        comments.append({
            'thread': tid,
            'id': id,
            'parent': parent,
            'created': datetime.fromtimestamp(created).strftime('%Y-%m-%dT%H:%M:%SZ%z'),
            'text': text,
            'author': author,
            'email': email
        })

    # Convert threads
    with open('output', 'w') as f:
        f.write('{"version":1,"users":[],"posts":[]}\n')

        for comment in comments:
            # Prepare user
            author = comment['author'] or 'Anonymous'

            user = {
                'name': author,
                'picture': '',
                'ip': '',
                'admin': True if ADMIN_USER == comment['email'] else False
            }

            if comment['email']:
                user['id'] = 'email_{}'.format(hashlib.sha1(comment['email'].encode()).hexdigest()),

            else:
                user['id'] = 'anonymous_{}'.format(hashlib.sha1(author.encode()).hexdigest()),

            item = {
                'id': str(comment['id']),
                'pid': str(comment['parent']) if comment['parent'] else '',
                'text': markdown(comment['text'], extensions=['nl2br', 'extra', 'codehilite']),
                'user': user,
                'locator': {
                    'site': 'mysite',
                    'url': threads[comment['thread']]['uri'],
                },
                'score': 0,
                'vote': 0,
                'time': comment['created'],
                'title': threads[comment['thread']]['title']
            }

            f.write(json.dumps(item)+'\n')


if __name__ == '__main__':
    main()

Note that the ADMIN_USER variable can be set to the original admin email in Isso to also set the admin flag in Remark42.

After execution, the output file should have been created. Given that Remark42 expects the backup to be gzipped simply run:

$ gzip output

And then restore the backup (note that this will overwrite any existing data):

$ remark42.linux-amd64 restore -s mysite -p BACKUP_DIR -f output.gz

Nvidia Optimus, Clevo, and power management

2019-08-21T13:11:23+02:00

I've had a PC Specialist Optimus IX (Clevo N850EP6) laptop for a while, dual booting Debian testing (with kernel 4.19) and Windows 10. This laptop has a hybrid graphics system with an integrated Intel UHD 630 and a dedicated NVIDIA GTX 1060 and, while it works pretty well for the most part, there have been several issues regarding power management, freezes (looking at you lspci!) and other malfunctions.

"Luckily" I am not the only one who has had these issues, so I finally could find a good solution for my case.

Enter Optimus and Bumblebee

Optimus is a technology developed by NVIDIA which allows switching between the integrated and the dedicated GPUs as needed. In practice, you run the desktop and most common applications, which do not usually require many resources, using the integrated GPU, while other applications such as games would use the dedicated GPU for best performance.

This achieves a good balance between battery life and performance, but while this switch is automatic in Windows systems, GNU/Linux systems require some additional work.

This is where Bumblebee comes in. The Bumblebee daemon allows performing this switch by rendering graphics using the NVIDIA GPU and displaying them using the integrated GPU. In order to do so, applications need to be executed as either with primusrun or optirun as follows:

primusrun APPLICATION_PATH

This approach has worked really well for me in the past, so I expected it to work as well in this new laptop, so I followed the Debian documentation for Bumblebee and installed everything as I did last time. This was not the case.

How about some power management with that?

Bumblebee includes a power management module called bbswitch, which is in charge of disabling the discrete GPU when not in use, which in turn improves the battery life greatly. It is installed as a kernel module and last time I used it, it worked out of the box.

However, after a while I noticed that the fans were usually at top speed and that the laptop would completely freeze when performing certain actions such as suspending (which was a gamble, as it would work less than 50% of the time), shutting down, and even running lspci.

After some search, it turns out that many people had this problem. The Arch Linux wiki suggested adding the acpi_osi=!"Windows 2015" or acpi_osi=! acpi_osi="Windows 2009" to the kernel boot parameters, but neither worked out for me.

I also checked the workarounds in the Bumblebee issue referenced in the Arch wiki, but these didn't work either. Another user suggested disabling the GPU on boot and then manually enabling it when needed. Didn't work either.

At this point I was starting to think that I wouldn't be able to have a completely working Debian installation in my machine, but among all the comments in the GitHub issue I found one that looked really promising.

Solution

This message by user phkb01 was hidden by GitHub (with more than 300 others), so I couldn't find it the first time around:

It seems that using the linux power management instead of bbswitch works.

I used the patch from PR 983, configured bumblebee with PMMethod=none and AlwaysUnloadKernelDriver=true. I enabled power management for the Nvidia card with powertop.

Now, the Device stats tab of powertop reports 0% usage for the Nvidia card when not using it, and 100% when optirun is running, so I guess it works.

"Might as well try it out" - I thought.

First, I made a backup of the original Bumblebee configuration in /etc/bumblebee/bumblebee.conf:

sudo cp /etc/bumblebee/bumblebee.conf /etc/bumblebee/bumblebee.conf.bak

And then I modified the [driver-nvidia] section (I use the NVIDIA driver with Bumblebee) as follows:

[driver-nvidia]
#....
PMMethod=none # original value was bbswitch
#....
AlwaysUnloadKernelDriver=true # original value was false

The following means that Bumblebee will not try to manage the power of the discrete GPU through bbswitch, therefore the card should stay on until it is told otherwise through another mechanism.

After rebooting, nothing really seemed to have changed: fans were still running and I could still run primusrun glxgears, so the GPU was working. I checked the output of powertop and saw a power consumption of 25 W idle. So I tried the second part of phkb01's message and enabled power management for the NVIDIA GPU in powertop and then, silence. Fans turned off and power consumption went down to around 10 W.

Mark me impressed.

Turns out that bbswitch was the issue all along and the kernel is quite capable of managing the power of the GPU by itself. Just to be sure, I started executing lspci multiple times and suspended the laptop. Everything worked as expected.

When toggling the power management setting in powertop I noted that it executed the following command:

echo auto > /sys/bus/pci/devices/0000:01:00.0/power/control

This change is not permanent, and will be reset on next boot. Therefore, I wrote a simple systemd service to enable power management on boot:

# /etc/systemd/system/nvidia-pm.service
[Unit]
Description=Nvidia GPU power management enabler
After=sysvinit.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/sh -c "echo auto > /sys/bus/pci/devices/0000:01:00.0/power/control"
ExecStop=/usr/bin/sh -c "echo on > /sys/bus/pci/devices/0000:01:00.0/power/control"

[Install]
WantedBy=sysvinit.target

Note that 0000:01:00.0 is the ID of my NVIDIA GPU, so it may change in other systems. Also, you can chech the status of the GPU by running:

cat /sys/bus/pci/devices/0000:01:00.0/power/runtime_status

This will return active when the GPU is being used and suspended otherwise.

Finally, enable and start the service:

sudo systemctl enable nvidia-pm.service
sudo systemctl start nvidia-pm.service

Bonus: HDMI output

I don't really use a second HDMI screen, so I didn't notice this until I had to present something and Debian wouldn't recognize the second screen.

The HDMI port is wired to the discrete GPU, so just plugging the cable will not make the screen show up in the settings manager. Apparently, Intel has a tool just for these cases called intel-virtual-output, and it is available in Debian package xserver-xorg-video-intel.

After installing it and running the program, I was greeted with the following message: no Virtual outputs on ":0"

After some searching, I found this Arch wiki page with the solution. Applied to Debian:

First, create /usr/share/X11/xorg.conf.d/20-intel.conf with the following content:

Section "Device"
    Identifier "intelgpu0"
    Driver "intel"
EndSection

Then, edit /etc/bumblebee/xorg.conf.nvidia (after making a backup):

Section "ServerLayout"
    Identifier  "Layout0"
    Option      "AutoAddDevices" "true"     # Bumblebee defaults to false
    Option      "AutoAddGPU" "false"
EndSection

Section "Device"
    Identifier  "DiscreteNvidia"
    Driver      "nvidia"
    VendorName  "NVIDIA Corporation"
    Option "ProbeAllGpus" "false"
    Option "NoLogo" "true"
    Option "UseEDID" "true"                 # Bumblebee defaults to false
    Option "AllowEmptyInitialConfiguration" # Added line
    #Option "UseDisplayDevice" "none"       # Commented this line
EndSection

Section "Screen"
    Identifier "Screen0"
    Device "DiscreteNVidia"
EndSection

After rebooting and running (the -f flag is used to keep it in the foreground):

primusrun intel-virtual-output -f

My settings manager was able to detect the screen correctly.

Dynamic fields with Flask-WTF

2019-03-02T13:53:00+01:00

Lately I've been working on a side project that requires dynamically adding or removing fields to a form: ideally, I should be able to add any number of fields and the server should receive the data correctly.

While a combination of FieldList and FormField works great when adding fields in the backend, I wanted to dynamically add and remove fields in the client (through Javascript) without having to send requests to the server until finally submitting the form with all the dynamic fields. Here's how that went.

TL;DR: Example code is available at https://gist.github.com/rmed/def5069419134e9da0713797ccc2cb29.

Edited July 21, 2019: Added persistence to SQLite database and endpoint to display race details.

Edited November 23, 2020: Seeing how several people have asked about this, I have included additional field types to lap subforms (<select>and <textarea>). I also revisited the way the form template is rendered and handled, which translates to easier implementation without having to manually write the HTML. Kudos to all the people in the comments that identified issues and edge cases that I hadn't considered!

Scenario

Let's assume that we have a simple web application used to record the lap times of one or more runners: every time a runner completes a lap, we want to record the amount of time (in minutes) it took them to complete that lap. Once all runners have completed all laps, we will submit the results to the server and store them in a SQLite database.

For this scenario, we could use the following two forms:

MainForm: this would be the form submitted at the end of all laps and contains entries for each of the laps completed
LapForm: this sub-form would contain the name of the runner and the time it took them to complete the lap, as well as other miscellaneous fields. These fields will be added and removed dynamically from the frontend

In order to store the data in the database, we will also implement the following models:

Race: will act as container for the laps
Lap: stores name and time of each runner, as well as other details such as category and notes on the specific lap

Implementation

This example application will have four files: app.py, templates/macros.html (for separate form rendering macro), templates/index.html (for displaying the form), and templates/show.html (for showing details of a single race).

`app.py`

The app.py file will contain form definitions, model definitions, and the endpoints used to manage form submission and displaying race details. Starting with the forms:

from flask_wtf import FlaskForm
from wtforms import Form, FieldList, FormField, IntegerField, SelectField, \
        StringField, TextAreaField, SubmitField
from wtforms import validators

class LapForm(Form):
    """Subform.

    CSRF is disabled for this subform (using `Form` as parent class) because
    it is never used by itself.
    """
    runner_name = StringField(
        'Runner name',
        validators=[validators.InputRequired(), validators.Length(max=100)]
    )
    lap_time = IntegerField(
        'Lap time',
        validators=[validators.InputRequired(), validators.NumberRange(min=1)]
    )
    category = SelectField(
        'Category',
        choices=[('cat1', 'Category 1'), ('cat2', 'Category 2')]
    )
    notes = TextAreaField(
        'Notes',
        validators=[validators.Length(max=255)]
    )


class MainForm(FlaskForm):
    """Parent form."""
    laps = FieldList(
        FormField(LapForm),
        min_entries=1,
        max_entries=20
    )

The previous lines implement the forms we mentioned earlier. Note that there will be at least 1 LapForm and a maximum of 20. This means that when accessing the page for the first time, we are guaranteed at least one lap subform and that we can only add a maximum of 20 lap subforms.

Continuing with the database models:

from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy()

class Race(db.Model):
    """Stores races."""
    __tablename__ = 'races'

    id = db.Column(db.Integer, primary_key=True)


class Lap(db.Model):
    """Stores laps of a race."""
    __tablename__ = 'laps'

    id = db.Column(db.Integer, primary_key=True)
    race_id = db.Column(db.Integer, db.ForeignKey('races.id'))

    runner_name = db.Column(db.String(100))
    lap_time = db.Column(db.Integer)
    category = db.Column(db.String(4))
    notes = db.Column(db.String(255))

    # Relationship
    race = db.relationship(
        'Race',
        backref=db.backref('laps', lazy='dynamic', collection_class=list)
    )

Note that there is a backreference defined in the Lap.race relationship, which would allow us to simply add new laps to an already existing Race object (more on that later).

Now, let's implement the application and the main endpoints:

from flask import Flask, render_template

app = Flask(__name__)
app.config['SECRET_KEY'] = 'sosecret'
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///test.db'
db.init_app(app)
# Create all database tables
db.create_all(app=app)

@app.route('/', methods=['GET', 'POST'])
def index():
    form = MainForm()
    template_form = LapForm(prefix='laps-_-')

    if form.validate_on_submit():
        # Create race
        new_race = Race()

        db.session.add(new_race)

        for lap in form.laps.data:
            new_lap = Lap(**lap)

            # Add to race
            new_race.laps.append(new_lap)

        db.session.commit()


    races = Race.query

    return render_template(
        'index.html',
        form=form,
        races=races,
        _template=template_form
    )

@app.route('/<race_id>', methods=['GET'])
def show_race(race_id):
    """Show the details of a race."""
    race = Race.query.filter_by(id=race_id).first()

    return render_template(
        'show.html',
        race=race
    )

if __name__ == '__main__':
    app.run()

As you can see, when submitting the form in the index endpoint, and after it has been validated, a new_race is created and added to the database session. After that, we create as many laps as received in the request (remember, a maximum of 20!). Given that this data is submitted as a dictionary, it is possible to populate the new lap object using the dictionary as keyword arguments. Finally, since the Race.laps relationship is a list, we can simply append new objects to it.

Note that we are also manually creating a LapForm as follows:

template_form = LapForm(prefix='laps-_-')

And then passing it to index.html as the variable _template. We will talk more about this later on, but this form is going to be used to render the template form used to add new laps directly from the browser.

The prefix argument guarantees that all the fields in template_form are going to be named laps-_-<FIELD>, which matches the naming rules required for dynamic fields to work.

Putting it all together:

# -*- coding: utf-8 -*-
# app.py

from flask import Flask, render_template
from flask_sqlalchemy import SQLAlchemy
from flask_wtf import FlaskForm
from wtforms import Form, FieldList, FormField, IntegerField, SelectField, \
        StringField, TextAreaField, SubmitField
from wtforms import validators


class LapForm(Form):
    """Subform.

    CSRF is disabled for this subform (using `Form` as parent class) because
    it is never used by itself.
    """
    runner_name = StringField(
        'Runner name',
        validators=[validators.InputRequired(), validators.Length(max=100)]
    )
    lap_time = IntegerField(
        'Lap time',
        validators=[validators.InputRequired(), validators.NumberRange(min=1)]
    )
    category = SelectField(
        'Category',
        choices=[('cat1', 'Category 1'), ('cat2', 'Category 2')]
    )
    notes = TextAreaField(
        'Notes',
        validators=[validators.Length(max=255)]
    )


class MainForm(FlaskForm):
    """Parent form."""
    laps = FieldList(
        FormField(LapForm),
        min_entries=1,
        max_entries=20
    )


# Create models
db = SQLAlchemy()


class Race(db.Model):
    """Stores races."""
    __tablename__ = 'races'

    id = db.Column(db.Integer, primary_key=True)


class Lap(db.Model):
    """Stores laps of a race."""
    __tablename__ = 'laps'

    id = db.Column(db.Integer, primary_key=True)
    race_id = db.Column(db.Integer, db.ForeignKey('races.id'))

    runner_name = db.Column(db.String(100))
    lap_time = db.Column(db.Integer)
    category = db.Column(db.String(4))
    notes = db.Column(db.String(255))

    # Relationship
    race = db.relationship(
        'Race',
        backref=db.backref('laps', lazy='dynamic', collection_class=list)
    )



# Initialize app
app = Flask(__name__)
app.config['SECRET_KEY'] = 'sosecret'
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///test.db'
db.init_app(app)
db.create_all(app=app)


@app.route('/', methods=['GET', 'POST'])
def index():
    form = MainForm()
    template_form = LapForm(prefix='laps-_-')

    if form.validate_on_submit():
        # Create race
        new_race = Race()

        db.session.add(new_race)

        for lap in form.laps.data:
            new_lap = Lap(**lap)

            # Add to race
            new_race.laps.append(new_lap)

        db.session.commit()


    races = Race.query

    return render_template(
        'index.html',
        form=form,
        races=races,
        _template=template_form
    )


@app.route('/<race_id>', methods=['GET'])
def show_race(race_id):
    """Show the details of a race."""
    race = Race.query.filter_by(id=race_id).first()

    return render_template(
        'show.html',
        race=race
    )


if __name__ == '__main__':
    app.run()

`templates/macros.html`

This file contains the following macro:

{# Render lap form.

This macro is intended to render both regular lap subforms (received from the
server) and the template form used to dynamically add more forms.

Arguments:
    - subform: Form object to render
    - index: Index of the form. For proper subforms rendered in the form loop,
             this should match `loop.index0`, and for the template it should be
             '_'
#}
{%- macro render_lap_form(subform, index) %}
    <div id="lap-{{ index }}-form" class="{% if index != '_' %}subform{% else %}is-hidden{% endif %}" data-index="{{ index }}">
        <div>
            {{ subform.runner_name.label }}
            {{ subform.runner_name }}
        </div>
        <div>
            {{ subform.lap_time.label }}
            {{ subform.lap_time}}
        </div>
        <div>
            {{ subform.category.label }}
            {{ subform.category }}
        </div>
        <div>
            {{ subform.notes.label }}
            {{ subform.notes }}
        </div>

        <a class="remove" href="#">Remove</a>
        <hr/>
    </div>
{%- endmacro %}

This macro receives the following arguments:

subform: LapForm to render in HTML, it can be either a subform included in MainForm.laps or an independent form
index: The index of the form being rendered

Although the usual way to use this macro is to simply call it while iterating the laps attribute of the main form, it will also be used to render the invisible template form used to add new forms with Javascript. To do so, it is only necessary to specify the value of index as '_'.

The main benefit of rendering the template this way is that we don't have to manually change the HTML each time we change the form definition code in Python, including validators.

`templates/index.html`

This file has the following main purposes:

To display current laps in the form (when submitting data)
To allow the user to add/remove laps as needed
To serve as index for the races in the database

Let's take a look at the first point:

{% import "macros.html" as macros %}

<html>
    <head>
        <title>Lap logging</title>

        <style>
            .is-hidden {
                display: none;
            }
        </style>
    </head>

    <body>
        <a id="add" href="#">Add Lap</a>
        <hr/>

        {# Show all subforms #}
        <form id="lap-form" action="" method="POST" role="form">
            {{ form.hidden_tag() }}

            <div id="subforms-container">
                {% for subform in form.laps %}
                    {{ macros.render_lap_form(subform, loop.index0) }}
                {% endfor %}
            </div>

            <button type="submit">Send</button>
        </form>

        {% if form.errors %}
            {{ form.errors }}
        {% endif %}

        {# Form template #}
        {{ macros.render_lap_form(_template, '_') }}


        {# Show races #}
        {% for race in races %}
            <p><a href="{{ url_for('show_race', race_id=race.id) }}">Race {{ race.id }}</a></p>
        {% endfor %}
    </body>
</html>

We end up with something like this:

Note that we are calling the render_lap_form macro introduced before twice. First when rendering the subforms contained within the main form:

<div id="subforms-container">
    {% for subform in form.laps %}
        {{ macros.render_lap_form(subform, loop.index0) }}
    {% endfor %}
</div>

And then when rendering the form template:

{# Form template #}
{{ macros.render_lap_form(_template, '_') }}

If we tried to submit the form we would be sending a single lap subform (the only one present) and we would see the data received by the server.

However, how do we add new forms so that WTForms recognizes the new data? If we take a look at the HTML generated by Flask when rendering the template, we will see that the fields looke like the following:

<div id="lap-0-form" class="subform" data-index="0">
    <div>
        <label for="laps-0-runner_name">Runner name</label>
        <input id="laps-0-runner_name" name="laps-0-runner_name" required="" type="text" value="">
    </div>
    <div>
        <label for="laps-0-lap_time">Lap time</label>
        <input id="laps-0-lap_time" name="laps-0-lap_time" required="" type="text" value="">
    </div>
    <div>
        <label for="laps-0-category">Category</label>
        <select id="laps-0-category" name="laps-0-category">
            <option value="cat1">Category 1</option>
            <option value="cat2">Category 2</option>
        </select>
    </div>
    <div>
        <label for="laps-0-notes">Notes</label>
        <textarea id="laps-0-notes" name="laps-0-notes"></textarea>
    </div>

    <a class="remove" href="#">Remove</a>
    <hr>
</div>

Apparently, WTForms recognizes inputs laps-0-runner_name and laps-0-lap_time as the first LapForm so if we submitted a form which also included laps-1-runner_name and laps-1-lap_time, then WTForms would get both LapForm objects in order.

Remember the macros.render_lap_form(_template, '_') call? That translates to the following HTML:

<div id="lap-_-form" class="is-hidden" data-index="_">
    <div>
        <label for="laps-_-runner_name">Runner name</label>
        <input id="laps-_-runner_name" name="laps-_-runner_name" required="" type="text" value="">
    </div>
    <div>
        <label for="laps-_-lap_time">Lap time</label>
        <input id="laps-_-lap_time" name="laps-_-lap_time" required="" type="text" value="">
    </div>
    <div>
        <label for="laps-_-category">Category</label>
        <select id="laps-_-category" name="laps-_-category">
            <option value="cat1">Category 1</option>
            <option value="cat2">Category 2</option
        ></select>
    </div>
    <div>
        <label for="laps-_-notes">Notes</label>
        <textarea id="laps-_-notes" name="laps-_-notes"></textarea>
    </div>

    <a class="remove" href="#">Remove</a>
    <hr>
</div>

Can you spot the difference? Instead of laps-0- appearing in the fields (and labels), it has been replaced with laps-_-, which is the prefix we set before for the template form. This way, we have a unique way of identifying the template and then adjusting the index when inserting a new subform.

Now that we have the template, we need to have the capability of adding and removing fields using the Add and Remove links defined in the template. For this, I will use the following Javascript functions (assuming JQuery is imported, but could easily be adapted for vanilla Javascript):

const ID_RE = /(-)_(-)/;

/**
 * Replace the template index of an element (-_-) with the
 * given index.
 */
function replaceTemplateIndex(value, index) {
    return value.replace(ID_RE, '$1'+index+'$2');
}

/**
 * Adjust the indices of form fields when removing items.
 */
function adjustIndices(removedIndex) {
    var $forms = $('.subform');

    $forms.each(function(i) {
        var $form = $(this);
        var index = parseInt($form.data('index'));
        var newIndex = index - 1;

        if (index < removedIndex) {
            // Skip
            return true;
        }

        // This will replace the original index with the new one
        // only if it is found in the format -num-, preventing
        // accidental replacing of fields that may have numbers
        // intheir names.
        var regex = new RegExp('(-)'+index+'(-)');
        var repVal = '$1'+newIndex+'$2';

        // Change ID in form itself
        $form.attr('id', $form.attr('id').replace(index, newIndex));
        $form.data('index', newIndex);

        // Change IDs in form fields
        $form.find('label, input, select, textarea').each(function(j) {
            var $item = $(this);

            if ($item.is('label')) {
                // Update labels
                $item.attr('for', $item.attr('for').replace(regex, repVal));
                return;
            }

            // Update other fields
            $item.attr('id', $item.attr('id').replace(regex, repVal));
            $item.attr('name', $item.attr('name').replace(regex, repVal));
        });
    });
}

/**
 * Remove a form.
 */
function removeForm() {
    var $removedForm = $(this).closest('.subform');
    var removedIndex = parseInt($removedForm.data('index'));

    $removedForm.remove();

    // Update indices
    adjustIndices(removedIndex);
}

/**
 * Add a new form.
 */
function addForm() {
    var $templateForm = $('#lap-_-form');

    if ($templateForm.length === 0) {
        console.log('[ERROR] Cannot find template');
        return;
    }

    // Get Last index
    var $lastForm = $('.subform').last();

    var newIndex = 0;

    if ($lastForm.length > 0) {
        newIndex = parseInt($lastForm.data('index')) + 1;
    }

    // Maximum of 20 subforms
    if (newIndex >= 20) {
        console.log('[WARNING] Reached maximum number of elements');
        return;
    }

    // Add elements
    var $newForm = $templateForm.clone();

    $newForm.attr('id', replaceTemplateIndex($newForm.attr('id'), newIndex));
    $newForm.data('index', newIndex);

    $newForm.find('label, input, select, textarea').each(function(idx) {
        var $item = $(this);

        if ($item.is('label')) {
            // Update labels
            $item.attr('for', replaceTemplateIndex($item.attr('for'), newIndex));
            return;
        }

        // Update other fields
        $item.attr('id', replaceTemplateIndex($item.attr('id'), newIndex));
        $item.attr('name', replaceTemplateIndex($item.attr('name'), newIndex));
    });

    // Append
    $('#subforms-container').append($newForm);
    $newForm.addClass('subform');
    $newForm.removeClass('is-hidden');

    $newForm.find('.remove').click(removeForm);
}


$(document).ready(function() {
    $('#add').click(addForm);
    $('.remove').click(removeForm);
});

The functions work as follows:

replaceTemplateIndex: Used to replace -_- in newly added subforms with their appropriate index
adjustIndices: When removing a subform, adjusts the indices of the next subforms so that there is no gap in the numbering
removeForm: Removes a subform
addForm: Copies the template and assigns it the index corresponding to the last subform + 1

The complete templates/index.html file, including a table with all the races in the database, looks as follows:

{# templates/index.html #}
{% import "macros.html" as macros %}

<html>
    <head>
        <title>Lap logging</title>

        {# Import JQuery #}
        <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>

        <script>
            const ID_RE = /(-)_(-)/;

            /**
             * Replace the template index of an element (-_-) with the
             * given index.
             */
            function replaceTemplateIndex(value, index) {
                return value.replace(ID_RE, '$1'+index+'$2');
            }

            /**
             * Adjust the indices of form fields when removing items.
             */
            function adjustIndices(removedIndex) {
                var $forms = $('.subform');

                $forms.each(function(i) {
                    var $form = $(this);
                    var index = parseInt($form.data('index'));
                    var newIndex = index - 1;

                    if (index < removedIndex) {
                        // Skip
                        return true;
                    }

                    // This will replace the original index with the new one
                    // only if it is found in the format -num-, preventing
                    // accidental replacing of fields that may have numbers
                    // intheir names.
                    var regex = new RegExp('(-)'+index+'(-)');
                    var repVal = '$1'+newIndex+'$2';

                    // Change ID in form itself
                    $form.attr('id', $form.attr('id').replace(index, newIndex));
                    $form.data('index', newIndex);

                    // Change IDs in form fields
                    $form.find('label, input, select, textarea').each(function(j) {
                        var $item = $(this);

                        if ($item.is('label')) {
                            // Update labels
                            $item.attr('for', $item.attr('for').replace(regex, repVal));
                            return;
                        }

                        // Update other fields
                        $item.attr('id', $item.attr('id').replace(regex, repVal));
                        $item.attr('name', $item.attr('name').replace(regex, repVal));
                    });
                });
            }

            /**
             * Remove a form.
             */
            function removeForm() {
                var $removedForm = $(this).closest('.subform');
                var removedIndex = parseInt($removedForm.data('index'));

                $removedForm.remove();

                // Update indices
                adjustIndices(removedIndex);
            }

            /**
             * Add a new form.
             */
            function addForm() {
                var $templateForm = $('#lap-_-form');

                if ($templateForm.length === 0) {
                    console.log('[ERROR] Cannot find template');
                    return;
                }

                // Get Last index
                var $lastForm = $('.subform').last();

                var newIndex = 0;

                if ($lastForm.length > 0) {
                    newIndex = parseInt($lastForm.data('index')) + 1;
                }

                // Maximum of 20 subforms
                if (newIndex >= 20) {
                    console.log('[WARNING] Reached maximum number of elements');
                    return;
                }

                // Add elements
                var $newForm = $templateForm.clone();

                $newForm.attr('id', replaceTemplateIndex($newForm.attr('id'), newIndex));
                $newForm.data('index', newIndex);

                $newForm.find('label, input, select, textarea').each(function(idx) {
                    var $item = $(this);

                    if ($item.is('label')) {
                        // Update labels
                        $item.attr('for', replaceTemplateIndex($item.attr('for'), newIndex));
                        return;
                    }

                    // Update other fields
                    $item.attr('id', replaceTemplateIndex($item.attr('id'), newIndex));
                    $item.attr('name', replaceTemplateIndex($item.attr('name'), newIndex));
                });

                // Append
                $('#subforms-container').append($newForm);
                $newForm.addClass('subform');
                $newForm.removeClass('is-hidden');

                $newForm.find('.remove').click(removeForm);
            }


            $(document).ready(function() {
                $('#add').click(addForm);
                $('.remove').click(removeForm);
            });
        </script>

        <style>
            .is-hidden {
                display: none;
            }
        </style>
    </head>

    <body>
        <a id="add" href="#">Add Lap</a>
        <hr/>

        {# Show all subforms #}
        <form id="lap-form" action="" method="POST" role="form">
            {{ form.hidden_tag() }}

            <div id="subforms-container">
                {% for subform in form.laps %}
                    {{ macros.render_lap_form(subform, loop.index0) }}
                {% endfor %}
            </div>

            <button type="submit">Send</button>
        </form>

        {% if form.errors %}
            {{ form.errors }}
        {% endif %}

        {# Form template #}
        {{ macros.render_lap_form(_template, '_') }}


        {# Show races #}
        {% for race in races %}
            <p><a href="{{ url_for('show_race', race_id=race.id) }}">Race {{ race.id }}</a></p>
        {% endfor %}
    </body>
</html>

`templates/show.html`

For this template we simply want to display the results of a specific race (through the /<race_id> URL). Therefore, a simple table should fit this purpose perfectly:

{# templates/show.html #}
<html>
    <head>
        <title>Race details</title>

    </head>
    <body>
        <a href="{{ url_for('index') }}">Back to index</a>

        {% if not race %}
            <p>Could not find race details</p>
        {% else %}
            <table>
                <thead>
                    <tr>
                        <th>Runner name</th>
                        <th>Lap time</th>
                        <th>Category</th>
                        <th>Notes</th>
                    </tr>
                </thead>
                <tbody>
                    {% for lap in race.laps %}
                        <tr>
                            <td>{{ lap.runner_name }}</td>
                            <td>{{ lap.lap_time }}</td>
                            <td>
                                {%- if lap.category == 'cat1' %}
                                    Category 1
                                {%- elif lap.category == 'cat2' %}
                                    Category 2
                                {%- else %}
                                    Unknown
                                {%- endif %}
                            </td>
                            <td>{{ lap.notes }}</td>
                        </tr>
                    {% endfor%}
                </tbody>
            </table>
        {% endif %}
    </body>
</html>

Testing

In order to test the application, simply run python3 app.py and try adding and removing fields. After submitting the form, check the Race link added at the bottom of the page, which should look like this:

Using RPi Zero as a Keyboard Part 3: Sending and receiving reports

2017-07-13T11:40:14+02:00

In this third and last part of the series we will use the new keyboard to send keystrokes to the connected computer. To do this, I will provide a couple of examples using Bash and Python, although they can be easily translated to other languages.

Edited July 8, 2018: Tested on Raspbian Stretch (lite) 2018-06-27. Added null character in between the ll in Hello.

Index

Setup and device definition
Report descriptor
Sending and receiving reports (this post)

Input reports

Input reports are those sent from the keyboard to the computer. As we saw in the previous post, input reports for keyboards are 8 bytes long:

1 byte: modifier keys (Control, Shift, Alt, etc.), where each bit corresponds to a key
1 byte: unused/reserved for OEM
6 bytes: pressed key codes

In order to press a regular key (e.g. A or B), its code has to be included inside the 6 byte segment. Remember that this segment represents the pressed keys, so until a key stops appearing there the host will keep pressing that key constantly.

The modifier keys, however, are 1 bit each. The modifier byte has the following structure (bit 0 is on the rightmost part):

[Right Meta | Right Alt | Right Shift | Right Control | Left Meta | Left Alt | Left Shift | Left Control]

If a given bit is set to 1, then the modifier key in question is pressed.

Now, as an example, let's write Hello World! with the virtual keyboard. Note that the device, once activated as shown in the first post (and having connected the Pi to a computer through the data port), will be accessible from the path /dev/hidg0 and can be interacted with like a normal file.

For example, in bash:

#!/bin/bash

function write_report {
    echo -ne $1 > /dev/hidg0
}

# H (press shift and H)
write_report "\x20\0\xb\0\0\0\0\0"

# e
write_report "\0\0\x8\0\0\0\0\0"

# ll
write_report "\0\0\xf\0\0\0\0\0"
write_report "\0\0\0\0\0\0\0\0"
write_report "\0\0\xf\0\0\0\0\0"

# o
write_report "\0\0\x12\0\0\0\0\0"

# SPACE
write_report "\0\0\x2c\0\0\0\0\0"

# W (press shift and W)
write_report "\x20\0\x1a\0\0\0\0\0"

# o
write_report "\0\0\x12\0\0\0\0\0"

# r
write_report "\0\0\x21\0\0\0\0\0"

# l
write_report "\0\0\xf\0\0\0\0\0"

# d
write_report "\0\0\x7\0\0\0\0\0"

# ! (press shift and 1)
write_report "\x20\0\x1e\0\0\0\0\0"

# Release al keys
write_report "\0\0\0\0\0\0\0\0"

And for Python (3):

#!/usr/bin/env python3

NULL_CHAR = chr(0)

def write_report(report):
    with open('/dev/hidg0', 'rb+') as fd:
        fd.write(report.encode())


# H (press shift and H)
write_report(chr(32)+NULL_CHAR+chr(11)+NULL_CHAR*5)

# e
write_report(NULL_CHAR*2+chr(8)+NULL_CHAR*5)

# ll
write_report(NULL_CHAR*2+chr(15)+NULL_CHAR*5)
write_report(NULL_CHAR*8)
write_report(NULL_CHAR*2+chr(15)+NULL_CHAR*5)

# o
write_report(NULL_CHAR*2+chr(18)+NULL_CHAR*5)

# SPACE
write_report(NULL_CHAR*2+chr(44)+NULL_CHAR*5)

# W (press shift and W)
write_report(chr(32)+NULL_CHAR+chr(26)+NULL_CHAR*5)

# o
write_report(NULL_CHAR*2+chr(18)+NULL_CHAR*5)

# r
write_report(NULL_CHAR*2+chr(21)+NULL_CHAR*5)

# l
write_report(NULL_CHAR*2+chr(15)+NULL_CHAR*5)

# d
write_report(NULL_CHAR*2+chr(7)+NULL_CHAR*5)

# ! (press shift and 1)
write_report(chr(32)+NULL_CHAR+chr(30)+NULL_CHAR*5)

# Release all keys
write_report(NULL_CHAR*8)

Note that both examples may require superuser permissions in order to access the device file, and that the key codes must be sent in binary format. Also note that the only report that explicitly release the keys is the last one, as that is when we end writing and no more keys are going to be pressed.

Output reports

Output reports are sent from the host to the keyboard in order to set LED states. These reports are 1 byte long and each bit represents the state of a single LED. Their structure is as follows:

[CONSTANT | CONSTANT | CONSTANT | Kana | Compose | Scroll Lock | Caps Lock | Num Lock]

Take into account that these may not only be sent when pressing keys such as Num Lock. For instance, when dealing with virtual machines, the host will change the state of the LEDs depending on the state in the main operating system or the operating system inside the virtual machine.

A very simple example of reading an output report in Python (3) is as follows:

#!/usr/bin/env python3

with open('/dev/hidg0', 'rb+') as fd:
    # Read only 1 byte
    output = fd.read(1)

Final note

I hope this short series of posts was somewhat useful. Having the Raspberry Pi Zero act as a USB device, in this case a HID such as a keyboard, offers many different possibilities such as automated testing, fast shortcuts, or device prototyping. That, together with the fact that it is such a low-cost board and the multiple periferials and components available, make it a great base for many different types of projects.

Here I've only covered the basis of a keyboard (HID) gadget, but there are many types available that seem as interesting and worth looking at. This is only the beginning!

Post series index

Setup and device definition
Report descriptor
Sending and receiving reports (this post)

Using RPi Zero as a Keyboard Part 2: Report descriptor

2017-07-05T12:40:00+02:00

In this second part of the series we'll see the structure of HID report descriptors and how to apply them to the virtual keyboard. When done, the gadget will be completely configured and ready to use.

Edited August 19, 2018: Added binary report descriptor for those that do not have the hidrd tool.

Edited July 8, 2018: Tested on Raspbian Stretch (lite) 2018-06-27. Copy report descriptor from existing keyboard.

Edited August 13, 2019: Updated link to USB HID document.

Edited August 19, 2020: Fixed incorrect write destination for the report length and added not regarding xxd (thanks Bob)

Index

What is it?

As the name suggests, a report descriptor specifies the structure of the reports sent from/to a USB device. In practice, any generic descriptor would be enough for our gadget considering that input reports are always going to be 8 bytes long and output reports are going to be 1 byte long.

Input reports (sent from keyboard to computer) have the following structure for a total of 8 bytes:

1 byte: modifier keys (Control, Shift, Alt, etc.), where each bit corresponds to a key
1 byte: unused/reserved for OEM
6 bytes: pressed key codes

While output reports (sent from computer to keyboard) have the following structure:

1 byte: LED states (Num lock, Caps lock, etc.), where each bit corresponds to a LED

However, this may not be the case for other human interface devices. For instance, a mouse may have more than three buttons and treat the (X, Y) coordinates very differently from another one.

Understanding the descriptor

The following is a descriptor given as an example in the HID specification, [1] page 59:

Usage Page (Generic Desktop),
Usage (Keyboard),
Collection (Application),
   Report Size (1),
   Report Count (8),
   Usage Page (Key Codes),
   Usage Minimum (224), 
   Usage Maximum (231),
   Logical Minimum (0),
   Logical Maximum (1),
   Input (Data, Variable, Absolute),            ;Modifier byte
   Report Count (1),
   Report Size (8),
   Input (Constant),                            ;Reserved byte
   Report Count (5),
   Report Size (1),
   Usage Page (LEDs),
   Usage Minimum (1),
   Usage Maximum (5),
   Output (Data, Variable, Absolute),           ;LED report
   Report Count (1),
   Report Size (3),
   Output (Constant),                           ;LED report padding
   Report Count (6),
   Report Size (8),
   Logical Minimum (0),
   Logical Maximum(255),
   Usage Page (Key Codes),
   Usage Minimum (0),
   Usage Maximum (255),
   Input (Data, Array),
End Collection

Let's go over it bit by bit.

Usage Page (Generic Desktop),
Usage (Keyboard),

Here we specify that the report is intended for a generic USB device, in this case a keyboard.

Collection (Application),

From the specification [1] : "A group of Main items that might be familiar to applications. It could also be used to identify item groups serving different purposes in a single device". Generally speaking, this is the type of collection to use for keyboards or mice.

   Report Size (1),
   Report Count (8),
   Usage Page (Key Codes),
   Usage Minimum (224), 
   Usage Maximum (231),
   Logical Minimum (0),
   Logical Maximum (1),
   Input (Data, Variable, Absolute),            ;Modifier byte

The previous is used to specify the modifier keys in the input report. This defines 8 fields which are 1 bit long each, resulting in the modifier byte mentioned previously. Also note that the values stored in these fields must be in the range [224, 231], as these are the key codes defined for the modifier keys:

Left Control: 224 (0x00e0)
Left Shift: 225 (0x00e1)
Left Alt: 226 (0x00e2)
Left Meta (Windows key): 227 (0x00e3)
Right Control: 228 (0x00e4)
Right Shift: 229 (0x00e5)
Right Alt: 230 (0x00e6)
Right Meta (Windows key): 231 (0x00e7)

Finally, as these are represented as bits, the final values in the fields will be either 1 or 0 (key is pressed or not).

   Report Count (1),
   Report Size (8),
   Input (Constant),                            ;Reserved byte

The previous defines the constant/reserved byte in the input report, which is again formed by 8 fields which are 1 bit long each.

   Report Count (5),
   Report Size (1),
   Usage Page (LEDs),
   Usage Minimum (1),
   Usage Maximum (5),
   Output (Data, Variable, Absolute),           ;LED report

This defines LEDs that can be set in the output report, formed by 5 fields which are 1 bit long each. LEDs are specified in the following order:

Num Lock
Caps Lock
Scroll Lock
Compose
Kana

In this case, the allowed values are in the range [1, 5] given that the value specified is converted to binary to set each specific bit. For example, a value of 3 (011) will set LEDs Num Lock and Caps Lock.

   Report Count (1),
   Report Size (3),
   Output (Constant),                           ;LED report padding

As I mentioned previously, output reports are 1 byte long. The block above set 5 bits, so it is necessary to add 3 constant fields which are 1 bit long each in order to complete the byte.

   Report Count (6),
   Report Size (8),
   Logical Minimum (0),
   Logical Maximum(255),
   Usage Page (Key Codes),
   Usage Minimum (0),
   Usage Maximum (255),
   Input (Data, Array),
End Collection

Finally, this specifies the key codes pressed in the keyboard in the input report. As shown, the block specifies 6 fields which are 8 bits (1 byte) long each. The value of each field corresponds to the HID code of the key pressed, in the range [0, 255].

Given that it is the last part of the definition of the report, the collection is closed.

Setting the descriptor

Once the report descriptor has been defined, we need to give it to the gadget. This is done through the /sys/kernel/config/usb_gadget/mykeyboard/functions/hid.usb0/report_desc file we skipped in the last post. However, we cannot simply write the previous text into the file. It is necessary to convert it into a binary format.

While hidrd is a great tool for reading and writing such reports, it might be easier to simply copy a report from an already existing keyboard. Such reports can be found in the /sys/class/input/eventXX/device/device/report_descriptor file, where XX is the corresponding event number. Keyboards are usually assigned event0.

If you cannot simply copy the file, especially when using a startup script to set up the keyboard, you can copy its hexadecimal representation to your script and then write the report descriptor manually.

For the following commands you will need the xxd program. In Raspbian systems it is available as a standalone package and can be installed with the following:

apt install xxd

Other systems may include the tool as part of the vim package, so you may need to install vim:

apt install vim

Now, on to the report. First, get a one-liner hex representation of the report bytes:

xxd -p /sys/class/input/event0/device/device/report_descriptor | tr -d '\n' > /tmp/mydesc

Then you can copy this file to the Raspberry Pi and use it to set the report descriptor. The function section of startup script shown in the previous post would look like this:

echo 1 > functions/hid.usb0/protocol
echo 8 > functions/hid.usb0/report_length
echo 1 > functions/hid.usb0/subclass
xxd -r -ps /tmp/mydesc functions/hid.usb0/report_desc

If you don't want to use the additional file, you may simply include the contents in the script directly. For this example, I'm writing the report descriptor used to test this:

echo 1 > functions/hid.usb0/protocol
echo 8 > functions/hid.usb0/report_length
echo 1 > functions/hid.usb0/subclass
echo "05010906a101050719e029e71500250175019508810275089501810175019503050819012903910275019505910175089506150026ff00050719002aff008100c0" | xxd -r -ps > functions/hid.usb0/report_desc

With this, the configuration of the gadget should be complete.

References

Post series index

Using RPi Zero as a Keyboard Part 1: Setup and device definition

2017-06-26T14:40:00+02:00

The Raspberry Pi Zero is a cool little piece of hardware with many possibilities. One of them is that it can work as a USB host OR as a USB gadget, meaning that it is possible to implement different types of devices such as ethernet, HID (keyboard, mouse, gamepad, etc.), audio, mass storage, etc. In this 3-part series of post we'll see how to configure and use a simple and generic keyboard gadget to send keys to the connected host.

In this part I'll go over the process of defining the gadget, breaking down what each different configuration files is used for and giving example values.

Edited October 12, 2017: Corrected the bmAttributes value used and added explanation for the values.

Edited July 8, 2018: Tested on Raspbian Stretch (lite) 2018-06-27.

Edited August 1, 2018: Added link to gist with gadget creation and tester scripts at the end of the post.

Edited August 13, 2019: Updated link to USB HID document.

Post series index

ConfigFS

First, let's talk about ConfigFS. For this part, I'm going to reference a brilliant presentation by Matt Porter [1], which greatly summarizes the key points of ConfigFS.

In essence, ConfigFS is a virtual filesystem which exposes a userspace API for the creation of USB devices. This was introduced in Linux 3.11, and recent versions of the Raspbian operating system already include the module.

Furthermore, this allows creating several devices at the same time, meaning that the Pi can act as a composite USB device with different functionalities.

Usage in Raspberry Pi Zero

First, let's assume that we have a microSD card with the following two partitions created when writing the official Raspbian image to the card:

BOOT: contains the configuration applied during boot time
DATA: contains the filesystem of the OS

Before using ConfigFS, it is necessary to load the module. Although it should be possible to use modprobe for this, it's just easier to add the following line at the end of BOOT/config.txt:

dtoverlay=dwc2

And the following two lines at the end of DATA/etc/modules:

dwc2
libcomposite

Once done, ConfigFS will be loaded in /sys/kernel/config/usb_gadget when the Pi is started.

Defining a USB device

The way to define a new device in ConfigFS is to create a directory inside the virtual filesystem. Note that these commands are only for illustration purposes, as they are expected to be executed during runtime. For this example, the device will be called mykeyboard:

mkdir /sys/kernel/config/usb_gadget/mykeyboard
cd /sys/kernel/config/usb_gadget/mykeyboard

Listing the contents of this new directory shows that several files and directories have been created automatically. The meaning of each individual file can be obtained from the official USB specification [2]. Regarding the files:

bcdDevice: device release number, assigned by manufacturer (format: 0x0000)
bcdUSB: USB specification number that the device implements (format: 0x0000)
bDeviceClass: USB class code, assigned by USB organization (format: 0x00)
bDeviceProtocol: USB protocol code, assigned by USB organization (format: 0x00)
bDeviceSubClass: USB subclass code, assigned by USB organization (format: 0x00)
bMaxPacketSize0: maximum packet size for the device, only possible values are 8, 16, 32, and 64 (format: 0x00)
idProduct: product ID, assigned by manufacturer (format: 0x0000)
idVendor: vendor ID, assigned by USB organization (format: 0x0000)
UDC: USB Device Controller, used to attach the gadget to the UDC driver in the machine

Note the format comment at the end of each file. This means that the file should contain the given number of hex characters. For example, if 0x0000 is shown, then the file must contain a 4-digit hexadecimal number, such as 0x12AB.

Now, let's start writing content to these files. Vendor and product IDs can be obtained from the Linux USB project [3]:

echo 0x0100 > bcdDevice # Version 1.0.0
echo 0x0200 > bcdUSB # USB 2.0
echo 0x00 > bDeviceClass
echo 0x00 > bDeviceProtocol
echo 0x00 > bDeviceSubClass
echo 0x08 > bMaxPacketSize0
echo 0x0104 > idProduct # Multifunction Composite Gadget
echo 0x1d6b > idVendor # Linux Foundation

In addition, note that the following directories are also present:

configs/: contains specific configurations for the device
functions/: defines the capabilities of the virtual device
os_desc/: OS Descriptors (ignore it for now)
strings/: localized description of the device (e.g. in English)

Let's begin with the localization. For English, the hexadecimal code would be 0x409, which must be created as a directory inside strings/. Once present, it is populated with three files:

manufacturer : name of the manufacturer of the device
product: product name/description
serialnumber: complete serial number of the product

Therefore:

mkdir strings/0x409

echo "My manufacturer" > strings/0x409/manufacturer
echo "My virtual keyboard" > strings/0x409/product
echo "0123456789" > strings/0x409/serialnumber

Now let's define the functions of the device. ConfigFS supports many different types of devices, but in this case we are interested in a USB keyboard. In order to have the device behave like a HID, a directory named hid.usb0 has to be created inside functions/. It contains the following files:

protocol: the protocol for the device
report_desc: binary descriptor for the reports sent by the keyboard
report_length: length of the reports sent by the keyboard
subclass: type of HID

For now we are going to ignore the report_desc, as I will cover that in the next post, but we can write the values for the other files, as these are common for all keyboards:

mkdir functions/hid.usb0

echo 1 > functions/hid.usb0/protocol
echo 8 > functions/hid.usb0/report_length # 8-byte reports
echo 1 > functions/hid.usb0/subclass
# echo "..." > functions/hid.usb0/report_desc # Check second post

Lastly, for the configuration, creating a new directory inside configs/ will include the following:

MaxPower: maximum power for the device (in mA)
bmAttributes: configuration characteristics bitmask (in hex format: 0x00), the following bits can be set depending on the power characteristics:
- bit 7: bus powered (e.g. 10000000)
- bit 6: self powered (e.g. 01000000)
- bit 5: remote wakeup (e.g. 00100000)
- bit 4 to bit 0: reserved
strings/: directory for localization of the configuration description (we will use 0x409 for English, as before)

Note that it is possible to set several bits in the bmAttributes bitmask at the same time. For instance, to indicate that the device is bus powered and has remote wakeup capabilities, the bitmask should be 10100000, which in hex translates to 0xa0.

Therefore:

mkdir configs/c.1
mkdir configs/c.1/strings/0x409

echo 0x80 > configs/c.1/bmAttributes # Only bus powered
echo 100 > configs/c.1/MaxPower # 100 mA
echo "Example configuration" > configs/c.1/strings/0x409/configuration

Once that is set up, the HID function defined previously must be linked to the configuration like so:

ln -s functions/hid.usb0 configs/c.1

Activating the device

Even though we are still missing the report descriptor, the device that was just configured can be activated by attaching the gadget to a UDC driver:

ls /sys/class/udc > UDC

After this (and after adding the report descriptor), the host to which the Pi is connected should recognize it as a USB keyboard.

TL;DR

The following script defines a USB keyboard as a gadget in ConfigFS. It should be executed by the Pi during runtime (e.g. on startup):

#!/bin/bash

# Create gadget
mkdir /sys/kernel/config/usb_gadget/mykeyboard
cd /sys/kernel/config/usb_gadget/mykeyboard

# Add basic information
echo 0x0100 > bcdDevice # Version 1.0.0
echo 0x0200 > bcdUSB # USB 2.0
echo 0x00 > bDeviceClass
echo 0x00 > bDeviceProtocol
echo 0x00 > bDeviceSubClass
echo 0x08 > bMaxPacketSize0
echo 0x0104 > idProduct # Multifunction Composite Gadget
echo 0x1d6b > idVendor # Linux Foundation

# Create English locale
mkdir strings/0x409

echo "My manufacturer" > strings/0x409/manufacturer
echo "My virtual keyboard" > strings/0x409/product
echo "0123456789" > strings/0x409/serialnumber

# Create HID function
mkdir functions/hid.usb0

echo 1 > functions/hid.usb0/protocol
echo 8 > functions/hid.usb0/report_length # 8-byte reports
echo 1 > functions/hid.usb0/subclass
# echo "..." > functions/hid.usb0/report_desc # Check second post

# Create configuration
mkdir configs/c.1
mkdir configs/c.1/strings/0x409

echo 0x80 > configs/c.1/bmAttributes
echo 200 > configs/c.1/MaxPower # 200 mA
echo "Example configuration" > configs/c.1/strings/0x409/configuration

# Link HID function to configuration
ln -s functions/hid.usb0 configs/c.1

# Enable gadget
ls /sys/class/udc > UDC

Bonus: The previous snippet and the test scripts used in the next posts are available at https://gist.github.com/rmed/0d11b7225b3b772bb0dd89108ee93df0

References

Post series index

Bridge? wat-bridge?

2016-10-13T17:27:00+02:00

Instant messaging services are very common nowadays. For instance, I use Telegram, while many people I know use WhatsApp, and that's cool. This, however, leads to a question I am asked often: "why don't you install WhatsApp?". The answer to that can be summarized in two main points: "because I don't want to" and "because I cannot install WhatsApp on my phone". But then again, it is interesting to see the user fragmentation between different services, so the Telegram bot API got me thinking about creating a "bridge" between WhatsApp and Telegram, just for fun.

TL;DR: source code available at https://github.com/rmed/wat-bridge

Architecture

The architecture that wat-bridge follows is actually very simple. It consists of two main components, the WhatsApp listener and the Telegram listener/bot (in the figure, wabot and tgbot respectively, that are executed in separate threads and communicate through signals, relaying messages from one service to the other.

This architecture is designed to work with one-to-one conversations and has the following flow:

Friend-A sends message from WhatsApp
wabot receives message and sends it to tgbot
tgbot receives the signal and sends the message to the owner of the bot
The owner of the bot reads the message and replies through the bot using defined commands
tgbot receives message and sends it to wabot
wabot receives the signal and sends the message to the specified contact/phone

Note that among those steps some checks are performed, for instance whether the phone that sent the message from WhatsApp is a known contact or is blacklisted.

The Telegram bot uses the pyTelegramBotAPI library, while the WhatsApp listener uses the Yowsup library. In addition, I have included a simple database using TinyDB in order to keep contacts and blacklisted phone numbers.

Configuration

In order to keep things simple, wat-bridge uses a configuration file with the following structure:

[tg]
owner = OWNER_ID
token = TOKEN

[wa]
phone = PHONE_NUMBER
password = PASSWORD

[db]
path = PATH_TO_DB

The Telegram token is obtained by talking to the BotFather through Telegram and creating a bot, while the owner ID can be obtained by using the /me command.

The WhatsApp phone must include the country code (only two digits) followed by the number, for instance 49xxxxxxxxx, and the password can be obtained through the Yowsup cli interface.

Lastly, the database path is the full path to the file that will contain blacklist and contacts. Note that this path should be readable/writable by the user that executes the application.

Commands

The commands available in the Telegram bot are very straightforward:

/help: shows a help message (incredible, right?)
/add <name> <phone>: adds a new phone to the database, assigning a name to it
/blacklist: shows blacklisted Whatsapp phones
/blacklist <phone>: blacklists a WhatsApp phone number
/contacts: list stored contacts
/me: obtain owner Telegram ID
/rm <name>: remove a contact from database
/send <name> <message>: send message to Whatsapp contact
/unblacklist <phone>: unblacklist a phone number

In case anyone is wondering why contacts are used/needed to send messages to WhatsApp... I'm just too lazy to write phone numbers every time :) Furthermore, the database is configured to use query cache, so looking up phone numbers or contact names should not be much of a bottleneck.

Messages from blacklisted phones are directly ignored upon arrival. Surely this can come in handy, right?

Conclusions

The resulting code is not beautiful whatsoever, but it works, which I actually didn't expect, to be honest. It is a fun little toy project to play around with, but I don't think it should be used in a production environment or for serious business.

Also, as a reminder, this requires a real WhatsApp phone number. I registered my second SIM card to play, but I don't think I will use it that much.

Source code is released under MIT license and available at https://github.com/rmed/wat-bridge.