Categories
Miscellaneous

Long Time No See

A long time has passed since the last post on this blog. Not because I was lazy. It was merely because there were more important things to do than writing blog posts about things that most people can look-up in the internet anyway.

However, the Open Source software projects were still going on. Not so frequent updates but once in a while. The current Corona pandemic now gives me some possibilities to finish things that were long time on my list. First and most importantly is to gain independance of hosting all my software on my own and maintaining the infrastructure for it. Still, some main parts will be on me. Such as build tools, issue tracking and automation.

However, I managed to host all my code now at GitHub. This task alone cost me about two weeks until each and every Subversion repository was migrated. I have been writing code now for more than 20 years. That’s why about 110 software projects piled up at my previous Subversion repository. Most of them are not public, only 26 can be accessed by everyone. But migrating all 110 physically took me 3 days. Another 10 days I was busy to update the CI/CD pipelines for the still active projects (around 50). And the last week passed with upgrading the Open Source projects to new software versions, documenting them, changing the workflows, upgrading build tools and writing CI/CD tools for these changes. Finally, I managed to bump up the versions of the major OSS projects – after 3 weeks of work. Most of them were API breaking. That’s why the major versions increased (Check Maven Central for an overview).

You will find updates on them here in this blog – and you will see more updates coming soon. The main changes are:

  • Upgrading to Java 9: My Java projects will not support any older runtime environment.
  • Documentation moves to GitHub along withe code and the respective version. It is still going on. So this blog will become less important for documentation and the respective sections will be removed from the menu (but still be available).
  • Development workflow will follow the Gitflow workflow model now.

Feel free to contact me for any of the projects, the new or the old ones. For the moment, I wish you all the best and stay healthy!

Ralph

PS: Of course, I will try to blog more IT stuff and more frequently than before πŸ™‚

Categories
Kubernetes

IPv6 with Kubernetes

Awwww – so much work I had put into setting up a Kubernetes cluster (this blog will run there in a few days). I set up the pods and containers, cron jobs, services, and, and, and. Then I started renewing my SSL certificates from LetsEncrypt. This renewal failed hilariously, but with a weird error message:

1
Timeout

What? I can reach my websites. Did I miss something? I checked connectivity. The IP addresses were right, the ACME challenge directory was available as required by LetsEncrypt, the DNS was working properly. Why couldn’t LetsEncrypt servers not reach my cluster? I soon found out that they prefer IPv6 over IPv4 which I had both enabled. But the IPv6 connection failed. From everywhere. Ping6 though succeeded.

Further analysis revealed that Kubernetes is not able to expose IPv6 services at all (or at least at now, so I researched). What shall I do now? All my work was based on the assumption that IPv4 and IPv6 will be there. But it’s not with Kubernetes. Of course I could move my reverse proxy out of Kubernetes and put it in front of it. But that would require more work as all the automation scripts for LetsEncrypt would need to be rebased. Testing again and again. Let aside the disadvantage of not having it all self-contained in containers anymore. Another solution must be there.

Luckily there was an easy solution: socat. It’s a small Linux tool that can copy network traffic from one socket to another. So that was setup easily with a systemd script (sock_80.service):

1
2
3
4
5
6
7
8
9
10
11
12
[Unit]
 Description=socat Service 80
 After=network.target
 
[Service]
 Type=simple
 User=root
 ExecStart=/usr/bin/socat -lf /var/log/socat80.log TCP6-LISTEN:80,reuseaddr,fork,bind=[ip6-address-goes-here] TCP4:ip4-address-goes-here:80
 Restart=on-abort
 
[Install]
 WantedBy=multi-user.target

That’s it. Enabled it (systemctl enable sock_80.service), reloaded systemd (systemctl daemon-reload), and started the service (systemctl start sock_80). VoilΓ‘! Here we go. IPv6 traffic is now routed to IPv4. I repeated it with port 443 and the setup is done. And LetsEncrypt servers are happy too πŸ™‚

Categories
Apache Linux Perl

How to automate LetsEncrypt

A new service is born: Let’s Encrypt. It offers free SSL certificates that you can use for web servers, email servers or whatever service you want to secure with TLS. This blog post presents my strategy to automate certificate creation and renewal. Please, install Let’s Encrypt on your web server box before you start to follow the presented strategy.

The key to success is to have Let’s Encrypt running without any further interaction. I use webroot authentication – which allows me to leave the productive web service up and running while the certificates are being issued or renewed. Therefore, I created a file named “myserver.ini” in folder /etc/letsencrypt. This configuration file contains all details that are required for the certification process;

1
2
3
4
5
6
7
8
rsa-key-size = 4096
authenticator = webroot
webroot-path = /path/to/webroot/
server = https://acme-v01.api.letsencrypt.org/directory
renew-by-default = True
agree-tos
email = <my-email-address>
domains = domain1.com, domain2.com

The second component of my strategy is the central piece: a script called “renewCertificates.pl”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
#!/usr/bin/perl
 
my $DOMAINS = {
    'myserver' => {
        'configFile' => '/etc/letsencrypt/myserver.ini',
        'leSubDir'   => 'domain1.com',
        'certDir'    => '/var/www/domain1.com/certs',
    },
};
 
my $domain;
my $renewed = 0;
chdir ('/usr/local/letsencrypt');
foreach $domain (keys(%{$DOMAINS})) {
    print "INFO  - $domain - START\n";
    my $cmd = '/usr/local/scripts/checkCertExpiry.sh 30 '.$DOMAINS->{$domain}->{'certDir'}.'/cert.pem >/dev/null';
    my $rc = system($cmd);
    if ($rc) {
        $cmd = './letsencrypt-auto certonly --config '.$DOMAINS->{$domain}->{'configFile'}.' --renew-by-default';
        $rc = system($cmd);
        if (!$rc) {
            $cmd = 'cp /etc/letsencrypt/live/'.$DOMAINS->{$domain}->{'leSubDir'}.'/* '.$DOMAINS->{$domain}->{'certDir'}.'/';
            $rc = system($cmd);
            if ($rc) {
                print "ERROR - $domain - Cannot deploy\n";
            } else {
                print "INFO  - $domain - Deployed\n";
                $renewed = 1;
            }
        } else {
            print "ERROR - $domain - Cannot generate certificates\n";
        }
    } else {
        print "INFO  - $domain - Certificate does not expire within 30 days\n";
    }
    print "INFO  - $domain - END");
}
 
if ($renewed) {
   system("/etc/init.d/apache2 reload");
}
 
exit 0;

This scripts allows renewal of multiple certificates by supporting multiple configurations. Lines 3-9 describe these configurations. leSubDir (line 6) is the sub directory that Let’s Encrypt creates in the certification process. It is the name of the first domain specified in the configuration file, here: domain1.com. certDir (line 7) is the target path where the certificates will be deployed to.

A second script supports this procedure by telling whether a certificate will expire within a certain number of days (see line 16 above):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/bash
 
# First parameter specifies if certificate expire in the next X days
DAYS=$1
 
target=$2
if [ ! -f "$target" ]; then
    echo "Certificate does not exist (RC=2)"
    exit 2;
fi
 
openssl x509 -checkend $(( 86400 * $DAYS )) -enddate -in "$target" >/dev/null 2>&1
expiry=$?
if [ $expiry -eq 0 ]; then
    echo "Certificate will not expire (RC=0)"
    exit 0
else
    echo "Certificate will expire (RC=1)"
    exit 1
fi

This script returns 0 when the given certificate will not expire, otherwise it returns a non-0 value. The Perl script above will renew certificates 30 days before expiration only.

The last piece is the Apache configuration to be used on these domains:

1
2
3
4
    SSLEngine on
    SSLCertificateFile /var/www/domain1.com/certs/cert.pem
    SSLCertificateKeyFile /var/www/domain1.com/certs/privkey.pem
    SSLCertificateChainFile /var/www/domain1.com/certs/fullchain.pem

I run the central Perl script above daily and do not need to worry about certificates anymore πŸ™‚

Categories
Bugzilla for Java

Bugzilla 4 Java V2.0.3 released

A new version 2.0.3 of B4J has been released. It ensures compatibility with latest Bugzilla releases and introduces a workaround for “Untrusted Authentication Request” errors on some installations. This is the complete list of changes:

Bug

  • [BFJ-86] – JiraRpcSession does not allow inheritance
  • [BFJ-87] – BugzillaHttpSession does not allow inheritance
  • [BFJ-88] – BugzillaRpcSession does not allow inheritance
  • [BFJ-89] – AbstractLazyRetriever uses Long.getLong() instead of LangUtils.getLong()
  • [BFJ-90] – Untrusted Authentication Request

Improvement

  • [BFJ-85] – Overcome Jira's search result limitation

New Feature

  • [BFJ-91] – Add login token usage for RPC usage

You can download the new version here or visit the Homepage of the utility where you will find more documentation.

The Maven Site is available as well. The Maven coordinates are:

<dependency>
      <groupId>eu.ralph-schuster</groupId>
      <artifactId>b4j</artifactId>
      <version>2.0.3</version>
</dependency>
Categories
Neolog Watchface

Neolog Watchface for Pebble

neolog1Yes! I published my first watchface for the Pebble watch. I am a proud owner of a Neolog and a Pebble Time Steel. I wanted to join both, the wonderful Neolog design and the smart display of Pebble. So I simply wrote this watchface. Enjoy it! All details are available here…

Categories
Linux Perl

Nonblocking sockets and Perl’s Net::Daemon

I was writing a Perl-based proxy for line-based SMTP protocol. The main reason doing this is because I receive unwanted e-mail bounces that are not filtered out by my SpamAssassin. The idea was to hook into the mail delivery chain and to collect e-mail addresses that I use. The proxy can later then filter out any bounce message that was not originated by myself.

I decided to use the Net::Daemon module which has a quite fancy interface. One just writes a single function which handles the client connection. As I didn’t want to learn every detail of SMTP protocol, I simply use non-blocking sockets. So whoever of the two parties wants to talk, it can do so and my proxy will just listen to the chat. The IO::Select documentation tells you to do this when you have multiple sockets to react on:

1
2
3
4
5
6
7
8
9
10
11
12
# Prepare selecting
$select = new IO::Select();
$select->add($socket1);
$select->add($socket2);
 
# Run until timed out / select socket
while (@CANREAD = $select->can_read(30)) {
    foreach $s (@CANREAD) {
        #... read the socket and do your stuff
    }
}
# Whenever there is a problem (like talk ended) the loop exits here

However, this code doesn’t work as expected. The can_read() function will not return an empty list when the sockets closed. It still returns any socket and the loop goes on forever. In fact, as we are in non-blocking mode, the script now eats up CPU time. πŸ™

There are two solutions to it. The first is to check whether the given socket is still connected and then exit the loop:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Prepare selecting
$select = new IO::Select();
$select->add($socket1);
$select->add($socket2);
 
# Run until timed out / select socket
while (@CANREAD = $select->can_read(30)) {
    foreach $s (@CANREAD) {
        if (!$s->connected()) {
            return;
        }
        #... It's safe now to read the socket and do your stuff
    }
}

The second and more clean method is just to remove the closed socket from the IO::Select object:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Prepare selecting
$select = new IO::Select();
$select->add($socket1);
$select->add($socket2);
 
# Run until timed out / select socket
while (@CANREAD = $select->can_read(30)) {
    foreach $s (@CANREAD) {
        if (!$s->connected()) {
            $select->remove($s);
            next;
        }
        #... It's safe now to read the socket and do your stuff
    }
}

Then the selector runs empty and will exit the loop as well.

Categories
Miscellaneous

Neolog Watchface for Pebble Time

This is the homepage of the Neolog Watchface for the Pebble Time watch. Actually it is my first watchface or even Pebble development. But as I am a proud owner of the Neolog watch and a Pebble Time, I didn’t want to miss the extraordinary design of my Neolog at the Pebble. So I simply wrote this watchface. Enjoy it!

Download the watchface

You can download the latest version 1.0 at Pebble’s AppStore.

Download the source code

The source code is available at my Subversion repository

Request a Change/Report a Bug

My Jira installation can be used to report bugs or request enhancements.

Categories
Java

Multi-threading: Pitfalls when initializing static variables

As a developer you often have to initialize static variables in a multi-threaded environment. The basic solution that most programmers apply is:

1
2
3
4
5
6
7
8
9
private static Object staticVar = null;
 
public static synchronized Object getStaticVar() {
    if (staticVar == null) {
        // initialize
        staticVar = ...
    }
    return staticVar;
}

This is a simple but expensive method. Each thread that needs the variable must synchronize with each other although the variable has long been initialized. So, usually the next step is to synchronize less:

1
2
3
4
5
6
7
8
9
10
11
12
private static Object SYNCHRONIZER = new Object();
private static Object staticVar = null;
 
public static Object getStaticVar() {
    if (staticVar == null) {
        synchronized (SYNCHRONIZER) {
            // initialize
            staticVar = ...
        }
    }
    return staticVar;
}

Yep. That does it, doesn’t it? The answer is: half! Imagine two simultaneous threads entering the method at the same time. If both will see staticVar being null then both will try to enter the synchronized block. And of course, both will initialize the variable nevertheless another thread did it before. So, we add another evaluation to ensure that only one thread will initialize:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
private static Object SYNCHRONIZER = new Object();
private static Object staticVar = null;
 
public static Object getStaticVar() {
    if (staticVar == null) {
        synchronized (SYNCHRONIZER) {
            if (staticVar == null) {
               // initialize
               staticVar = ...
            }
        }
    }
    return staticVar;
}

Most books end here but omit a very crucial part. The code works perfect as long as the initialization is a simple operation only. Let’s make the initialization a bit more sophisticated:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
private static Object SYNCHRONIZER = new Object();
private static List<String> staticVar = null;
 
public static List<String> getStaticVar() {
    if (staticVar == null) {
        synchronized (SYNCHRONIZER) {
            if (staticVar == null) {
               // initialize
               staticVar = new ArrayList<String>();
               staticVar.add("value 1");
               staticVar.add("value 2");
               staticVar.add("value 3");
            }
        }
    }
    return staticVar;
}

The first glance doesn’t reveal anything. But it happened several times in one of my own applications that two threads were not correctly synchronized. Occasionally, one thread found the list to be empty. What happened?

The magic is that staticVar is being set at the very first beginning of the synchronized block. Meanwhile another thread was entering the method and saw the variable not being null. It immediately started using the list although it was not yet initialized completely. The correct solution is therefore:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
private static Object SYNCHRONIZER = new Object();
private static List<String> staticVar = null;
 
public static List<String> getStaticVar() {
    if (staticVar == null) {
        synchronized (SYNCHRONIZER) {
            if (staticVar == null) {
               // initialize
               List<String> tmp = new ArrayList<String>();
               tmp.add("value 1");
               tmp.add("value 2");
               tmp.add("value 3");
               staticVar = tmp;
            }
        }
    }
    return staticVar;
}

For readability, we could refactor the method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
private static Object SYNCHRONIZER = new Object();
private static List<String> staticVar = null;
 
public static List<String> getStaticVar() {
    if (staticVar == null) {
        synchronized (SYNCHRONIZER) {
            if (staticVar == null) {
               staticVar = createList();
            }
        }
    }
    return staticVar;
}
 
private static List<String> createList() {
   // initialize
   List<String> tmp = new ArrayList<String>();
   tmp.add("value 1");
   tmp.add("value 2");
   tmp.add("value 3");
   return tmp;
}
Categories
Miscellaneous

DIY Calendars 2015

Here they are, in time πŸ™‚ The DIY calendars for 2015. Good planning…

Categories
Eclipse Java RsBudget

RsBudget 2.0 released

logoIt’s done. My first official Eclipse/RCP application is out. RsBudget is an Expense Tracker for everyone. I’ve been developing it now for three years while constantly using it for private purposes. That’s how it grew to its functionality as it is today. I simply used these previous versions in order to feel and learn what’s been missing. Now it’s up to you to tell me what there is to be done next (a few tasks are already waiting ;)).

The application still misses some features, e.g. nice graphical statistics. But I don’t regard them as a must-have so far. They will be added with next versions, some will be available as commercial add-ons later.

The main features are:

  • General Expense Planning
  • Monthly Expense Planning, Tracking and Control
  • Categorization of expenses
  • Comparison of planned and actual values
  • Free text field for personal notes for each month
  • Forecasting of balances and profit/loss
  • Statistics and History
  • Export of transaction records to Excel and CSV
  • Multi-language support (English and German)
  • Online Help
  • Online Update

RsBudget runs on all major desktop platforms (Windows, MacOS, Linux) with Java 7 installed. Just download your version here!