It turns out that I can also run local private registries on the same host as my mirror as its own container, and what's more, I can put these all together in a simple Docker Compose system.
]]> A simple Registry Mirror containerUsing a Docker Compose v3 configuration, I can set up a service like this:
registry_mirror:
container_name: registry-mirror
restart: always
image: registry:2
environment:
REGISTRY_STORAGE_DELETE_ENABLED: "true"
REGISTRY_PROXY_REMOTEURL: "https://registry-1.docker.io"
ports:
- "50000:5000"
volumes:
- mirror:/var/lib/registry
Note that I give a proper name to the container instead of letting Docker Compose name it for me, so I can easily check on its status or logs via e.g docker logs -f registry-mirror.
I have a pretty simple configuration here, but if I need a more sophisticated one, I can just update the environment hash with new settings.
For building and sharing private images, I have this other service:
registry_private:
container_name: registry-private
restart: always
image: registry:2
ports:
- "50001:5000"
volumes:
- private:/var/lib/registry
Nearly zero customization on the registry config, other than a different exposed port.
For the meantime, I keep the image storage of both registry services in simple Docker Compose named volumes:
volumes:
mirror:
private:
I have this all down in a GitHub repo, such that starting the Compose system is as easy as a docker-compose up -d.
The only other thing to do next is to make your Docker host (or Docker Machine VMs) use these registries. For the registry mirror, we need to run dockerd with --registry-mirror, while for the private registry, we use --insecure-registry. On systems with a /etc/docker/daemon.json, the configuration would look like this:
{
"insecure-registries": ["127.0.0.1:50001"],
"registry-mirrors": ["http://127.0.0.1:50000"]
}
Perhaps the best use case for setting this all up is when you want to have several Docker Machines using these registries. In my case, all I have to do is to deploy these like this:
docker-machine create --driver=kvm --engine-registry-mirror=http://192.168.122.1:50000 --engine-insecure-registry http://192.168.122.1:50001 my-machine
In my reference libvirt/KVM setup, my Docker host running this Registry Compose system is listening on 192.168.122.1 as well a localhost. This means that on my-machine, I can pull private images like e.g. docker pull 192.168.122.1:50001/my-private-image:4.2 while still being able to fetch Docker Hub hosted images like normal.
Here are a few little bits of Things I've Learned...
]]>.DEFENITEness (whether if it is a type object, or an object instance)Int/Real .rand method do emit a result between 0 inclusive and n exclusive.make/made in grammar Match objects access stores for data in grammar action classesEven after the month's squashathon ended, there's still a few things I think I can do in the meantime:
Thanks to Zoffix, moritz, AlexDaniel, and others for guiding me, so looking forward to helping out further soon!
How about you, dear readers? Maybe you can help out too?
]]>buildpack-deps more than once, in different machines; as a result, I used a lot of network bandwidth for image pulling. Aside from this, I also run several VMs via libvirt/KVM, via Docker Machine to start up VMs for Docker Swarm and Minikube.
I wanted to save my bandwidth usage, and it turns out there is a way to do this by building on some Docker features...
]]> Using "remote" Docker daemon environment for the Docker clientOne approach is to not use your local (bare-metal) Docker context at all, and use a Docker Machine's environment instead:
$ docker-machine create my-remote-machine
$ eval $(docker-machine env my-remote-machine)
$ docker info
Minikube (which builds on docker-machine) also encourages this approach:
$ minikube start
$ eval $(minikube docker-env)
Either setup will change your shell's environment to use a "remote" Docker server, which means docker pulls will put the images in the remote's own storage. This is good in cases where you retain a single machine or minikube instance, but should you delete the instance, you'd have to re-pull again.
Another approach, especially for Minikube, is to use an SSH pipe to send images from local to remote, like this:
$ docker pull myimage:4.2
$ docker save myimage:4.2 | minikube ssh docker load
Again, this borrows from docker-machine:
$ docker save myimage:4.2 | docker-machine ssh my-remote-machine docker load
It is good enough for testing local private images in a swarm, but can get tedious real fast.
Fortunately, Docker's basic architecture already provides a way to reduce the tedium by means of the Docker Registry. One of the Registry's features is that it can run as a proxy (pull-through cache) of the Docker Index. One can thus start a persistent container like this:
$ docker run -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io -e REGISTRY_STORAGE_DELETE_ENABLED=true -d -p 50000:5000 --restart=always --name registry registry:2
This lets me run a Registry proxy on localhost:50000, which I can then point Docker daemons to via the registry-mirrors configuration, e.g. in /etc/docker/daemon.json:
{
"registry-mirrors: ["http://127.0.0.1:50000"]
}
There are a few caveats with it though:
docker pull myimage, translating them into requests for docker pull myimage:latest; thus, it is preferable to use named tags on images so that the proxy Registry will only update its local image copy when there are updates from the remote Docker Index. The SSH pipe trick would be preferable in this case.Making this work on docker-machine and minikube is a matter of passing in certain flags. In my case, the registry container also listens on my libvirt/KVM default network bridge (192.168.122.1) thus I have to do
$ minikube start --registry-mirror=http://192.168.122.1:50000
for minikube. For docker-machine, it looks like this:
$ docker-machine create --engine-registry-mirror=http://192.168.122.1:5000 my-docker-machine]]>
I have a few things going on under subdomains that I wanted to move off to my main domain as subpaths instead. As such I have a /etc/nginx/sites-enabled/foo configuration like this:
server {
root /srv/somewhere/over/the/rainbow/www;
index index.php index.html index.htm;
server_name foo.mydomain.net;
access_log /srv/somewhere/over/the/rainbow/log/access.log;
error_log /srv/somewhere/over/the/rainbow/log/error.log info;
location / {
try_files $uri $uri/ =404;
}
location ~ \.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/var/run/php5-fpm.sock;
}
}
I wanted to move things off to /etc/nginx/sites-enabled/main, which has configurations for location / {...} and whatnot. Naturally, I thought I could get away with doing this:
location /foo/ {
alias /srv/somewhere/over/the/rainbow/www/;
index index.php;
location ~ *\.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/var/run/php5-fpm.sock;
}
}
But alas, requests for http://mydomain.net/foo/blah.php returned 404 instead of my expected app.
Reading about nginx location again, I noticed this:
To find location matching a given request, nginx first checks locations defined using the prefix strings (prefix locations). Among them, the location with the longest matching prefix is selected and remembered. Then regular expressions are checked, in the order of their appearance in the configuration file. The search of regular expressions terminates on the first match, and the corresponding configuration is used. If no match with a regular expression is found then the configuration of the prefix location remembered earlier is used.
I was under the impression that the prefix-based locations will also nest their last match with other location blocks within them, but apparently they do not. Hence the configuration becomes
location /foo/ {
alias /srv/somewhere/over/the/rainbow/www/;
index index.php;
location ~ /foo/*\.php$ {
include snippets/fastcgi-php.conf;
fastcgi_pass unix:/var/run/php5-fpm.sock;
}
}
Taking out the location ~ /foo/*\.php$ { ... } block from the nest will not work, which is consistent with the statement above.
This is further validated in this post with
location /app1/ {
# something special for app1 here, e.g.
# access control
auth_basic ...
access ...
location = /app1/login {
# something special for /app1/login,
# eveything from /app1/ is inherited
proxy_pass ...
}
location ~ \.jpg$ {
expires 1m;
}
}
After getting schooled and confirming it all works, now I just have to change my old subdomain configs to just use a return:
return 301 https://mydomain.net/foo$request_uri;
And I'm all set.
]]>I've been using Slackware Linux since around 2010-2011, moving from Debian mainly because I got tired of the package churn and following the internal politics of DDs. Initially I tried it as a way of getting a close to BSD experience on Linux, since I was already using OpenBSD at the time (more on that on a later post.) Turns out that having a prior BSD experience helps a lot, especially as Slackware is as "upstream" as upstream gets (very minimal, if any, distro-specific changes, as a matter of policy.)
Since it was Slackware's birthday just sometime ago, let me share some tips and tricks I've learned over the years.
]]> Useslacktrack as a next choice after SlackBuilds
Both SlackBuilds.org and alienBOB's repo have a large cache of additional software packages to get from, but there are few cases where sometimes you'd need to install software without a SlackBuild. In those cases, one might just go straight to doing ./configure && make && make install for those software that claim to be well-behaved and install only into /usr/local and nowhere else. For more complex software, something like GNU stow may be required to track the file changes and try to make sense out of the install.
On Slackware though, there's already a GNU stow alike in the form of slacktrack, which not only provides a way to track file changes, but can optionally make a full Slackware package that can be installed using pkgtools. This is a nice alternative when you don't want to write out a full SlackBuild setup for that package (yet); instead, you write a .build file that is a very thin wrapper for your software's build/install system.
As an example, I use the acpi_call Linux kernel module and tpacpi-bat software on my ThinkPad X220, which are not on those two repos above, but on GitHub. For my case, all I need to do is to fork those GitHub repos, add a .build and slack-desc file for each (and a doinst.sh for the kernel module so I can run depmod upon install/upgrade,) then invoke slacktrack, like so for tpacpi-bat:
# cd tpacpi-bat
# slacktrack -Qp tpacpi-bat_3.0-x86_64-1zakame.tgz '/bin/sh -x tpacpi-bat.build'
This produces a Slackware package in /tmp, so while tpacpi-bat is already installed and tracked by a dummy record in /var/log/packages, I can still do something like
# installpkg /tmp/tpacpi-bat_3.0-x86_64-1zakame.tgz
To install the package with a proper slack-desc (and in the case of acpi_call, an actual package post-install script.)
slackpkg upgrade-allThis one is something I picked up quite recently. What I used to do is that after a slackpkg upgrade-all with a kernel update, I'd fire up a separate root shell to run /usr/share/mkinitrd/mkinitrd_command_generator.sh --run /boot/vmlinuz-generic | sh so I can have a new /boot/initrd.gz once slackpkg prompts me to run LILO.
It turns out that slackpkg has a simple mechanism of customizing package post-installation work, since slackpkg is after all a wrapper around the standard pkgtools. In this case, what I needed to customize is the lookkernel() function in /usr/libexec/slackpkg/functions.d/post-functions.sh. Copy that function over to a new file (say, /usr/libexec/slackpkg/functions.d/z-lookkernel.sh) and edit so it becomes something like this:
lookkernel() {
NEWKERNELMD5=$(md5sum /boot/vmlinuz 2>/dev/null)
if [ "$KERNELMD5" != "$NEWKERNELMD5" ]; then
if [ -x /usr/share/mkinitrd/mkinitrd_command_generator.sh ]; then
echo -e "\n
Your kernel image was updated.
Do you want slackpkg to run mkinitrd now? (Y/n)"
answer
if [ "$ANSWER" != "n" ] && [ "$ANSWER" != "N" ]; then
( /usr/share/mkinitrd/mkinitrd_command_generator.sh --run /boot/vmlinuz-generic | sh )
fi
fi
if [ -x /sbin/lilo ]; then
echo -e "\n
Your kernel image was updated. We highly recommend you run: lilo
Do you want slackpkg to run lilo now? (Y/n)"
answer
if [ "$ANSWER" != "n" ] && [ "$ANSWER" != "N" ]; then
/sbin/lilo
fi
else
echo -e "\n
Your kernel image was updated and lilo is not found on your system.
You may need to adjust your boot manager(like GRUB) to boot appropriate
kernel."
fi
fi
}
This customization will now let slackpkg upgrade-all prompt me about updating my initrd upon a kernel upgrade, along with a subsequent prompt to update LILO.
startx over display managersFor some, this might very obvious, but this one is something I learned early on, even when on Debian or OpenBSD. I can't count how many times I've read questions from people asking about what to do next after upgrading their display driver and/or kernel, rebooting, and getting a blank screen instead, with some of them not even able to switch virtual consoles back to tty1. Or, it boots up fine to presenting the display manager, only to break upon logging in because of somesuch.
On a default install, Slackware would drop you to runlevel 3, which will not run a display manager. This is a good default, since this means you have startx as an option to run X11, and even more, you have some option to check if your configuration is ok, by looking at your /var/log/dmesg or /var/log/Xorg.0.log. Moreover, logging in to console shell lets you have a chance to set up your environment the way you want it, especially if you need some X11 tweaks. Which leads to next item...
I know most people have a simple login shell configuration, e.g. on bash, just having a .bashrc and maybe even a .profile. But there's more to that than just those two files. For example, if you're on a Slackware64 with multilib enabled, you can source /etc/profile.d/32dev.sh to get additional environment tweaks for compiling software under 32-bit mode.
For bash, I recommend reading up the INVOCATION section of the manpage. For zsh, there's a corresponding STARTUP/SHUTDOWN FILES section in its manpage as well. Other shells definitely have their own style of setup, so be sure to check out their docs.
If you're going to use X11 via startx, it is often helpful to look around /etc/X11/xinit and peek at the various xinitrc* files. Reading startx's manpage should also give a clearer idea about configuring it so you can run your preferred apps and window manager/desktop environment.
These last couple of tips are pretty much something that can be applied to just about any Linux/BSD system, not just Slackware. However, these are often overlooked so I think it best to tell here, and moreover, I think this shows how well and generic Slackware is.
Kudos once more to Patrick Volkerding and and the Slackware team!
]]>For the longest time I haven't enabled SSL on my blog or other HTTP sites; I used to use Comodo, then StartCom SSL. However, I got wind that StartCom (along with WoSign) will soon be distrusted by Google Chrome, et al. after some reviews of their operation. The good thing though, is that there's now a Let's Encrypt initiative which provides free SSL certificates, so I figure it is high time for me to re-integrate SSL once more.
]]> The Let's Encrypt site recommends the CertBot script as a starting point for integration, but upon grabbing the software, it wanted to install Python stuff into my system. I don't really mind, but would rather opt for something that already uses already-installed libraries, so I looked for alternatives. Being a Perl guy, I found Crypt::LE (via ZeroSSL) as an acceptable alternative, as it can leverage my existing Perl 5.26 environment.On the other hand, my system Perl is of a different version (5.20,) so I need to do a bit of work to integrate Crypt::LE. First off, I'd use PAR::Packer and make a binary:
$ cpanm PAR::Packer
$ cd Crypt-LE
$ pp -M Crypt::OpenSSL::Bignum -M Cpanel::JSON::XS -M IO::Socket::SSL -Ilib script/le.pl -o crypt-le
This lets me have a self-sufficient client that I can put to /usr/local, that doesn't even use the system Perl. If I had Docker on my server, I would've just pulled the ZeroSSL client, but right now I don't, so I make do :)
I can then make my cert:
# crypt-le --key account.key --email REDACTED --csr domain.csr --csr-key domain.key --crt domain.crt --generate-missing --domains www.domain.tld,domain.tld --unlink --path /my/path/to/.well-known/acme-challenge --live
And renewing my cert would just be same command above, just with a --renew <days> added. Wrap that up in a script that can be put in crontab, and I'm all set.
After installing the cert at the usual places, I can then verify their correct installation with
$ openssl s_client -showcerts -connect zakame.net:443
as well as with a proper browser, or two.
After this, I'm mulling whether to continue using Movable Type for this blog, or maybe move to other engines. Hugo seems like a good candidate.
]]>Furthermore, I'm coming back to using Emacs, so, I should post some bits about it as well, maybe some pieces off my ~/.emacs.d/init.el configuration. Much of it is on GitHub though, and is constantly more updated there than elsewhere, so I'll probably concentrate on large bits of customization such as Helm and programming/IDE modes for Perl/Ruby. And probably, some bits off my other dotfiles as well... So, looking forward to writing again!
]]>Key commands:
update-service from daemontools-run Ubuntu package is used here to add the service directory in /etc/sshd. This is really just a fancy way in Ubuntu to do cd /etc/service && ln -sf /etc/ssh/service sshd which would be done otherwise on other systems.docker ps -l -q is a very useful Docker idiom, so much that I've made and alias of it in my .bashrc as docker last.docker inspect can be used to retrieve external information about the running container, such as its IP address, that can then be used on the host to connect to (like in this example, via ssh.)You can get this image from the Docker index under zakame/base, default root password is docker. Additionally, I set sshd to not use PAM authentication here as I'm on a Slackware host, and I needed to make /var/run/sshd manually this directory is normally made at startup during init.
awk and sed, then became great in the late 90's to 2000's as the CGI programming language. Now, with the advent of web applications programming, she's doing it better with a lot of frameworks held together by better glue. And let's not forget the objects and events.
Looking forward to another great year building great stuff with Perl!
]]>init inside containers. Here are some notes regarding this kind of usage:
]]>
Docker service under daemontools
Docker has a simple server mode called using docker -d, which simply listens to a local socket (typically /var/run/docker.sock) and emits logs to STDOUT. This mode is naturally suitable for running docker under daemontools as a supervise service, so much so that it is almost a no-brainer to configure:
$ cat /service/docker/run
#!/bin/sh
exec 2>&1
exec docker -d
Notice that the docker output is redirected, which can then be collected in a logger subprocess like multilog:
$ cat /service/docker/log/run
#!/bin/sh
exec multilog t ./main
Docker containers can be thought of as lightweight virtual machines; some even view it as a better chroot environment with its own networking and namespaces separate from the host system. Thus, one can run init like supervisor services inside these containers, and daemontools is a good choice for such a supervisor:
# docker run -i -t ubuntu /bin/bash
# sed -e 's/main$/main universe' /etc/apt/sources.list > /etc/apt/sources.list.new
# mv /etc/apt/sources.list{.new,}
# apt-get update
# apt-get install daemontools-run
# sh -c 'exec /usr/bin/svscanboot &'
# ps axf
... add daemons, scripts and install them in /etc/service
# exit
Once you've created an image, you can use docker commit with the -run option to add a Cmd that will be run be default when creating container with docker run -d:
# $myservice_id=`docker ps -l -q`
# docker commit -run '{"Cmd": ["/usr/bin/svscanboot"]}' $myservice_id myservice
If you have services listening on ports, be sure to add a PortSpecs key in your docker commit -run invocation with the appropriate ports to be exposed on the container, like this:
# docker commit -run '{"Cmd": ["/usr/bin/svscanboot"], "PortSpecs": ["22", "80"]}' $myservice_id my_ssh_web_container
Once created, you can now run your container with docker run -d:
# docker run -d my_ssh_web_container
# docker inspect `docker ps -l -q` | grep IPAddress
... ssh or browse the IP address found above...
cpanfile listing down Plack, XMLRPC, and Starman dependencies, so install those manually according to the wiki doc. In addition, install some graphics library like Imager and set it in mt-config.cgi or MT will complain about a missing ImageDriver.PIDFilePath that need to be added:
CGIPath should point to the URL going to the MT installation, normally this can be some path in $url/cgi-bin/mt but one can just shorten it to $url/mtStaticPath must not be your website's document root, but rather the path where MT's static files will be (e.g. /var/www/html/mt-static if the website root is in /var/www/html)StaticFilePath should be set to correspond to the URL in StaticPath/path/to/mt/html would be ok.