An odd issue I ran into the other day: I had a Varnish 3 instance that had logic hinging on req.backend.healthy to show a special error page if all the backends were down. That logic inexpliciably triggered even though all my backends were up! After much head-scratching I identified the issue: one of my historical VCLs was still loaded that no longer had any healthy backends (due to repeated autoscaling up / down), and although the current definition of that director had healthy backends, the historical one did not. Varnish has a habit of not letting go of old VCLs even if you specify vcl.discard on them. So, req.backend.healthy will show the director as down if any prior definition of that director is down. Since the only way to definitively remove VCLs from memory is a restart (which flushes the memory cache), this makes req.backend.healthy fairly unreliable.

This is in v3 and may not apply to v4 anymore.

Two Factor Authentication (2FA) is an additional layer of protection you can add to your Minecraft server. You should already be relying on SSH keys to access your server, but those keys can be lost or leaked if you use them on untrusted machines. 2FA protects you from hacks resulting from someone gaining access to your password or SSH key.

DuoSecurity is a company that provides a 2FA service that is free for personal use. A Mobile Push notification is sent via their Android / iOS app whenever a login is attempted and requires an affirmative response before login can proceed. There is also SMS and computer-generated voice calls, but those consume credits that have a cost to refill. There is also a cost for going over 10 users, which should not be an issue for adminstation of a Minecraft server.

I performed a source install to get the latest version, there are also packages for RHEL/CentOS/Debian/Ubuntu on their website.

They offer a SSH-specific installation and a PAM installation that covers all auth on the machine. PAM is likely the more comprehensive solution but is a more involved process, plus requires a SELinux policy update (and the server resulting from this series has SELinux enabled). Their website tells you how to install the relevant SELinux policy but the necessary objects are missing from their download. I’ve send them a mail and will update if I get clarification. For the record, the error I got was:

1
2
3
4
5
6
7
8
9
10
11
12

make: Entering directory `/home/centos/duo_unix-1.9.14/pam_duo'
checkmodule -M -m -o authlogin_duo.mod authlogin_duo.te
checkmodule:  loading policy configuration from authlogin_duo.te
checkmodule:  unable to open authlogin_duo.te
make: [semodule] Error 1 (ignored)
semodule_package -o authlogin_duo.pp -m authlogin_duo.mod
semodule_package:  Could not open file No such file or directory:  authlogin_duo.mod
make: [semodule] Error 1 (ignored)
semodule -i authlogin_duo.pp
semodule:  Failed on authlogin_duo.pp!
make: [semodule] Error 1 (ignored)
make: Leaving directory `/home/centos/duo_unix-1.9.14/pam_duo'

SSH setup is straightforward. Install the DuoSecurity app on your mobile device. Create an account on their website. Navigate their portal and select Applications, then +New Application. Give it name of Minecraft SSH. This will result in a window showing a Integration key, Secret key (requiring a click to show), and an API hostname. Collect those and store them to the side (securely!)

https://www.duosecurity.com/docs/duounix#instructions describes the compile process:

1
2
3
4

wget https://dl.duosecurity.com/duo_unix-latest.tar.gz
tar zxf duo_unix-latest.tar.gz
cd duo_unix-1.9.14
./configure --prefix=/usr && make && sudo make install

After installing the binaries, modify /etc/duo/login_duo.conf and fill in the ikey/skey/host values from above. Uncomment pushinfo as well so that more details about the login are sent to you.

https://www.duosecurity.com/docs/duounix#centos describes modifying SSH to work with the new system. Keep an alternate root-logged in session open in an alternate window while attempting this so you can back out if you make a mistake. Do not log out of the ‘backup’ window until you’ve thoroughly exercised the system. I will not be responsible if you lock yourself out.

Add ForceCommand /usr/sbin/login_duo to the end of /etc/sshd/sshd_config, then restart sshd (which does not kick out existing sessions) with systemctl restart sshd.service

Now attempt to log in to your account. You will be prompted to enroll in 2FA. Upon logging in again you will be prompted for your 2FA method, which will look like:

1
2
3
4
5
6
7
8

$  ssh minecraft.example.com -i ~/.ssh/minecraft.pem -l centos
Duo two-factor login for centos

Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-1234
 2. Phone call to XXX-XXX-1234
 3. SMS passcodes to XXX-XXX-1234

I always choose #1 as I do not want to expend credits, though if push failed I could resort to the other two. When I choose #1 I get a popup on my phone asking me to accept / deny the login, and after choosing Accept I am able to log in.

Now I do not have to worry someone will get ahold of my minecraft.pem and gain access to the server. Further, if I get a login-request on my phone and haven’t attemped to log in, I know my key has been compromised.

I had cause to call CenturyLink support recently. As one of the troubleshooting steps they seriously claimed that a power strip is not capeable of providing sufficient power to a DSL modem, and that it must be connected directly to the wall to receive sufficient power.

I suspect this was to force me to unplug and powercycle the modem, but I had done that several times already. The idea that a DSL modem draws more power than will flow through a power strip is ridiculous.

DataDog is a nifty monitoring / statistics gathering system. It is something like a akin to a combination of Graphite / Grafana, but with a social aspect so that your team can attach discussions to a given point in time. They have a free tier that retains data for a day, which is handy for visualizing the state of the Minecraft server.

Java applications normally expose their statistics via JMX. I did not see anything Minecraft-specific in my stock instance, but Java itself exposes several counters that are informative.

I created my Datadog account, procured my API key, and installed the agent with:

1

DD_API_KEY=MyAPIKey  bash -c "$(curl -L https://raw.githubusercontent.com/DataDog/dd-agent/master/packaging/datadog-agent/source/install_agent.sh)"

JMX is not enabled by default for Java processes, so I updated my systemd unit file in /etc/systemd/system/minecraft.service to include the JMX configuration:

NewRelic is an exceptionally useful tool for monitoring java applications, or at least those that deal with web or other transational workloads. I tried hooking it up to MineCraft and it doesn’t report anything. The free version doesn’t let me look at the JVM stats (threads and such) so it appears to be a waste. However, they also provide a general unix server agent that does provide some nifty dashboards. The procedure to install it is:

1
2
3
4

rpm -Uvh https://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-repo-5-3.noarch.rpm
yum install newrelic-sysmond
nrsysmond-config --set license_key=YourLicenseKey
/etc/init.d/

The RPM is EL5-era so doesn’t understand SystemD, so I created a unit file:

In the first installment I launched a basic Minecraft service on CentOS7. However, a proper service should not be run from the command line, instead it should be controlled by the system service daemon. In years past this would be by writing a ‘sysV init script’ which would try to determine if the process was running, if not launch it and capture its PID for future reference, and capture its output to a file. CentOS-7 has switched away from that model to one called SystemD which makes much of that easier. There is some controversy over the SystemD model (is it UNIX-y? Too monolithic? Taking over everything?) but seems pretty handy for what it needs to do, plus has some nice security features.

So I will create a Minecraft service definition for SystemD. I used http://0pointer.de/blog/projects/systemd-for-admins-3.html to help me with this. The unit file will look like:

A new Server

This series will go through how to host a MineCraft server, and go totally overboard on the configuration/management of it. I’ll be integrating a variety of management/monitoring tools that go far beyond the needs of the average ‘friends & family’ server, because it is fun. I’ll starting the basics and build up from there.

So, first we need a machine. I’m going to use a Amazon AWS machine for this. I’ll be using some AWS-specific features, but I don’t think any of them will be critical. In some cases I’ll show the non-AWS alternative.

—assiclink-and-elbs.markdown layout: post title: “EC2 ClassicLink and ELBs” date: 2015-01-09 16:08:02 -0800 comments: true

categories: AWS VPC

I was looking at EC2 ClassicLink today. It is going to make VPC migrations much easier. Some things I noticed:

• Connectivity to external-facing ELBs only works if the ELBs have ‘world’ incoming access, or specifically mention the public IPs of the source. That is because the ELB IP resolves to its public IP.
• Connectivity to internal-facing ELBs work fine, assuming the source security group has access.
• My default-VPC IP space overlapped with my EC2 Classic node’s DNS resolver in /etc/resolv.conf, so attaching it to the VPC broke DNS. Update 1/25/15: I retested this and it is working now, so I’m not sure that this is a problem anymore.
• The ReInvent session says that instance-DNS won’t resolve, so you won’t be able to use ec2-* domain names to reach specific instances — it’ll have to be IPs.

PaperTrailApp (Referral link) is a SaaS RSyslog host that currently offers 100MB/mo of capacity free. In setting up my home network and servers I wanted to have my RouterBOARD 750GL send its logs to PaperTrail. The RouterBoard supports syslog, so configuring it is pretty straightforward — with one exception. My ISP utilizes PPPoE and so I have an additional interface configured in my RouterBOARD for it. It seems to get confused as to what source IP to use when talking to PaperTrail and the syslog packets go into the bitbucket. I resolved this by hardcoding my source IP into the src-address for the syslog. The resulting configuation is:

1
2
3
4
5
6
7
8

# jan/07/2015 20:12:24 by RouterOS 6.22
# software id = 2JL8-DKGT
#
/system logging action
set 3 remote=<IP by resolving your log*.papertailapp.com address> remote-port=<port defined in the console>
add bsd-syslog=yes name=papertrail remote==<IP by resolving your log*.papertailapp.com address> remote-port=<port defined in papertrail console> src-address=<pppoe address> target=remote
/system logging
add action=papertrail topics=!async,!packet,!ddns,!debug

I was playing with Mesos/Marathon + Docker recently and tried to integrate Consul. Mesos sets the Docker container ‘name’ to a internal value that messes with the Consul Registrator, something like mesos-aa998abcabab. Registrator can accept an environmental variable telling it what service name to register as instead. The format for that is:

Registrator will register the instance with a service name of nginx-xyz