Stuff I'm Up To

Technical Ramblings

VMWare Horizon Load Balancing — November 21, 2018

VMWare Horizon Load Balancing

We’re in the process of installing a new Horizon 7 infrastructure  and as part of the process the vendor added load balancers all over the place. I asked with question of why not use an Open Source solution for that?

My go to web server, proxy, load balancer is Nginx and as we already have a HA pair setup I thought we’d try to use that – even if it meant putting in a new one dedicated to the task in the longer term.


As the plan is to use a load balancer in front of the connection servers and the only tunnelling that will take place will be for external systems, our requirement will be to LB the https traffic (TCP 443) for the authentication. The PCoIP/Blast traffic will be directed straight to the ESX Host/client.

The previous document on load balancing with Nginx means I only need to add in the config needed for horizon. By using the same syncing of config it immediately becomes available on the secondary load balancer.

I created a new config file /etc/nginx/sites-available/horizon and then as standard, symbolic link it to sites-enabled to make it live.

upstream connectionservers {
ip_hash;
server 192.168.0.236:443;
server 192.168.0.237:443;
}
server {
listen 443 ssl;
server_name horizon.domain.tld;
location ~ / {
proxy_pass https://connectionservers;
}
}

This adds our two connection servers into an upstream group called connectionservers which I then point the  proxy_pass  directive to.

The ip_hash directive ensures we have session stickiness based on the clients IP address. When a client connects they’ll stay directed to the connection server they were given until and unless the connection server becomes unavailable.

nginx.conf

Within the nginx.conf ensure you have the reverse proxy options set in the http {} section:

enable reverse proxy
proxy_redirect off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwared-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
client_header_buffer_size 64k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 16k;
proxy_buffers 32 16k;
proxy_busy_buffers_size 64k;

The SSL configuration on the HA pair is standard throughout all of our servers that it “proxies” for. We have a wildcard certificate and the HA proxies only services under *.domain.tld – our horizon.domain.tld fits this pattern so no changes necessary.

All the standard Nginx SSL related security settings for certificate, stapling, ciphers, HSTS are located in our /etc/nginx/snippets/ssl.conf file and is included in the nginx.conf using:

include snippets/ssl.conf

snippets/ssl.conf

ssl_certificate /etc/ssl/certs/wildcard.pem;
ssl_certificate_key /etc/ssl/private/wildcard_key.cer;
ssl_dhparam /etc/ssl/private/dhparam.pem;

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;

# modern configuration. tweak to your needs.
ssl_protocols TLSv1.2;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_prefer_server_ciphers on;

# OCSP Stapling ---
# fetch OCSP records from URL in ssl_certificate and cache them
ssl_stapling on;
ssl_stapling_verify on;

add_header X-Content-Type-Options nosniff;
add_header Accept "*";
add_header Access-Control-Allow-Methods "GET, POST, PUT";
add_header Access-Control-Expose-Headers "Authorization";
add_header X-Frame-Options SAMEORIGIN;
add_header X-XSS-Protection "1; mode=block";

proxy_cookie_path / "/; HTTPOnly; Secure";

Note: Depending on your requirements for other system you may need to include content security policy settings to satisfy CORS (Cross Origin Resource Sharing). In fact you MUST do this to allow Chrome and Firefox to work with Blast over HTML.

In our PCoIP client we add the new server as horizon.domain.tld and we get through the authentication and on to the selection of the available pools. So clearly the load balancing is doing the job. You can check the /var/log/nginx/access.log to confirm.

If you miss out the ip_hash directive for session stickiness you’ll find you can’t get past the authentication stage.

Advertisements
Syncing Config Files Between Servers —

Syncing Config Files Between Servers

Having setup a pair of load balancers I wanted to ensure the Nginx configuration from one system was replicated to the secondary where changes were made on the primary.

In this instance my weapon of choice is lsyncd. It seems quite old, but stable. It’s a layer over rsync and ssh that monitors a directory for changes and copies them over ssh to a target server.


Getting it working is pretty straight forward once you crack the configuration.

Install it using:

$ sudo apt-get install lsyncd

Then you’ll need to create the config file:

$ sudo mkdir /etc/lsyncd
$ sudo mkdir /var/log/lsyncd
$ sudo vi /etc/lsyncd/lsyncd.conf.lua

This is what my final config looked like:

settings {
  logfile = "/var/log/lsyncd/lsyncd.log",
  statusFile = "/var/log/lsyncd/lsyncd.status"
}

sync {
  default.rsyncssh,
  source = "/etc/nginx",
  host = "loadbal02.domain.local",
  targetdir = "/etc/nginx";
}

Outside of this config I needed to setup an ssh key for the root user from loadbal01 to logon to loadbal02. When you generate the key DO NOT specify a password for the key, or the process wont work.

$ sudo ssh-keygen
$ sudo cat /root/.ssh/id_rsa.pub

Then I copy the output and paste it into /root/.ssh/authorized_keys on loadbal02. Many ways of doing this scp, copy/paste, etc.

Then just to ensure it works by connecting at least once to the target host as root using ssh.

$ sudo ssh root@loadbal02.domain.local

This will ask you to trust and save the hosts id in the ~/.ssh/known_hosts file so it will be trusted in future. Make sure you connect to EXACTLY the same host name as you are putting into the config file eg. “loadbal02.domain.local” as the host id for “loadbal02″does not match the FQDN and you will get a service error like this:

recursive startup rsync: /root/sync/ -> loadbal02.domain.local:/root/sync/
Host key verification failed.

Start the service using:

$ sudo systemctl start lsyncd

and monitor the status and log file to make sure it’s doing what you expect.

$ sudo cat /var/log/lsyncd/lsyncd.status 

Lsyncd status report at Tue Nov 20 15:33:21 2018
Sync1 source=/etc/nginx/
There are 0 delays
Excluding:
nothing.
Inotify watching 7 directories
1: /etc/nginx/
2: /etc/nginx/sites-available/
3: /etc/nginx/modules-available/
4: /etc/nginx/modules-enabled/
5: /etc/nginx/sites-enabled/
6: /etc/nginx/conf.d/
7: /etc/nginx/snippets/

Restarting Nginx on the Secondary

After the files have copied, you need to tell Nginx on the secondary that the config has changed. Then it needs to reload the config so it’s running as up to date as the primary.

For this I use inotify-tools by installing them on loadbal02:

$ sudo apt-get install inotify-tools

Next I created a shell script that monitors the config and reloads the service.

I created a file called /usr/sbin/inotify_nginx.sh and set it as executable.

$ sudo touch /usr/sbin/inotify_nginx.sh
$ sudo chmod 700 /usr/sbin/inotify_nginx.sh
$ sudo vi /usr/sbin/inotify_nginx.sh

This is the content of my script:

!/bin/sh
while true; do
inotifywait -q -e modify -e close_write -e delete -r /etc/nginx/
systemctl reload nginx
done

It monitors the /etc/nginx/ folder (-r recursively) and any event like modify, close_write or delete will cause the script to continue and reload nginx, then loop around to wait for any more changes.

Next I made sure my script ran every time the server rebooted using crond.

$ sudo crontab -e

Added in a line to run the script at reboot:

@reboot /bin/bash /usr/sbin/inotify_nginx.sh

That’s it. Following a reboot the script runs happily. To monitor it you can look at the nginx error log (/var/log/nginx/error.log). This will show a process started event like:

2018/11/21 09:16:57 [notice] 736#736: signal process started

The only downside of this would be if I spanner the config on primary the service reload on the secondary will fail. eg.

2018/11/21 09:25:20 [emerg] 1819#1819: unknown directive "banana" in /etc/nginx/nginx.conf:1

This isn’t such a concern unless the primary happens to fail whilst you’re editing it. The most important part is:

Test your config before you reload the primary server!

$ sudo nginx -t
nginx: [emerg] unknown directive "banana" in /etc/nginx/nginx.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

Then fix any issues before doing a systemctl reload.

Laravel Routing Returns 302 Redirect — November 20, 2018

Laravel Routing Returns 302 Redirect

That was a frustrating few hours. I added a route into my api.php route and every time I visited it it triggered a redirect. I saw nothing in the browser and it was like my controller function wasn’t even being called.

I put the usual dd() and dump() into my function and it wasn’t being called. I even changed the function name so it didn’t exist and thought I’d get an error message, but nothing – still a redirect.

What is going on? I expanded out the call from “Namespace\Class@function” to put in a function () {} and that was being called and ran ok.

Clearly there was something wrong with my class somewhere. The other functions within it worked fine, just this new one didn’t even seem to be there.

Sure enough a bit of Duck-Jitsu and I found this: https://stackoverflow.com/questions/35020477/laravel-unexpected-redirects-302 – answer 3 is where I got my clue.

I looked at my class constructor and I was using the auth middleware to ensure the functions were not used unless authenticated.

public function __construct() {
  $this->middleware(['auth:api']);
}

For my function I wasn’t using any authentication, so of course I was getting a redirect. But because it’s an api call and the expected return in JSON under an XHttpResponse I wasn’t able to properly debug it. I was getting back a HTML page instead.

As a work around all I needed to do was add in an exception for my function:

public function __construct() {
  $this->middleware(['auth:api'], ['except' => [ 'myFunction' ]]);
}

Now all I have to do is go back and sort out the authentication for the function so I don’t need to bypass it.

Debian Upgrading from an ISO File — November 7, 2018

Debian Upgrading from an ISO File

Kind of an unusual situation, but I have a Debian jessie box that has a terrible <2MB Internet connection, no CD/DVD and the USB stick I have I don’t want to overwrite and make bootable – it already has things on it I need. But it does have the capacity to hold the Debian DVD ISO #1.

How do you upgrade Debian from an ISO without being bootable?

Mount the USB Sick

First mount the USB Drive into your stretch environment so you can use the ISO it contains. You may want to check the output of dmesg to see what device name your stick has been given.

$ sudo mkdir /mnt/usb
$ sudo mount /dev/sdb1 /mnt/usb

Now we can see the ISO in the /mnt/usb folder

$ sudo ls -lh /mnt/usb

total 12G
-r-------- 2 root root 3.4G Nov 7 11:25 debian-9.5.0-amd64-DVD-1.iso

Mount the ISO

We can then mount the ISO into another folder under /mnt

$ sudo mkdir /mnt/iso

$ sudo mount -t iso9660 -o loop /mnt/usb/debian-9.5.0-amd64-DVD-1.iso /mnt/iso

We have a mounted ISO

$ sudo ls -lh /mnt/iso

total 1.5M
-r--r--r-- 1 root root 146 Jul 14 11:27 autorun.inf
dr-xr-xr-x 1 root root 2.0K Jul 14 11:27 boot
...

Edit Your Installation Sources

Next we edit the file /etc/apt/sources.list so it only contains the path to our ISO to install from. Take a copy of the original one or just comment out the existing lines.

deb file:///mnt/iso stretch main contrib

You may also want to check any source list files under sources.list.d and move them out whilst you upgrade.

Carry Out the Upgrade

Just continue as you normally would using upgrade/dist-upgrade to deliver your new OS. Making sure you do an update first so you read your news sources file.

$ sudo apt-get update

$ sudo apt-get upgrade

$ sudo apt-get dist-upgrade

Because you’re not getting the install from the internet and apt isn’t able to trust the source, you will have to accept to install the upgrades by ignoring the security warning.

When you’re done make sure you uncomment/put back the sources.list to point at the internet and replace the version with the new one eg. change jessie to stretch.

Java Decompiler — November 6, 2018

Java Decompiler

Today I’ve been working on a problem where properties being loaded into our CRM system from a GIS DTF file are doing some strange updates. It appears that the vendor in their wisdom has decided to populate user defined fields that we already use!

The Loader is written in Java, I have a .jar file and can extract the .class files, but they’re all compiled.

A bit of searching reveals an online decompiler I can use: http://www.javadecompilers.com/jad

I started using it to decompile one .class file at a time and quickly became bored. So tried the whole .jar file. It happily decompiled the file and returned a single zip file containing the decompiled .class files as .javafiles.

Now I can scan the .java files to find out what other SQL calls they are making to fill other fields we might be using.

DBeaver – SQL GUI —

DBeaver – SQL GUI

I’ve used a few SQL GUI’s over the years, SQuirreL, DBVisualizer, HeidiSQL, MySQL Workbench, but the one that stands out recently is DBeaver.

It’s got a community and enterprise edition. The community does everything I need and connects to all the SQL servers we use, Microsoft SQL, MySQL, Postgres/PostGIS.

Being Java based it’s cross platform, so you can use it in Windows too.

RADIUS Testing — November 5, 2018

RADIUS Testing

We have a need to authenticate a couple of devices via our Wifi access points with a RADIUS server. Right now I wanted to test things out using a MAC address authentication process. But for some reason we can’t get it working on the AP’s.

How do I test the RADIUS authentication policies are correct?

I recall using a RADCHECK program in Windows many years ago and figured Linux would probably have something similar. Sure enough a quick search means I can install freeradius-utils which includes radtest and radclient.

I needed to pass a number of RADIUS attributes and values with my test call and this is how I did it:

$ cat << EOF | radclient -x [radisuserver] auth [supersecretkey]
User-Name = 6894244B56EB
User-Password = 6894244B56EB
NAS-Port-Type = 19
NAS-Port = 0
Calling-Station-Id = SSID
EOF

This spoofed an auth call to the RADIUS server using the specified MAC address as user name and password and pretended the call was from a NAS-Port-Type of Wireless - 802.1x (19). I got the table of values from here: https://www.juniper.net/documentation/en_US/junos/topics/concept/subscriber-management-nas-port-type-overview.html

Statement Option NAS-Port-Type Value Description
value

0–65535

Number that indicates either the IANA-assigned value for the RADIUS port type or a custom number-to-port type defined by the user
adsl-cap

12

Asymmetric DSL, carrierless amplitude phase (CAP) modulation
adsl-dmt

13

Asymmetric DSL, discrete multitone (DMT)
async

0

Asynchronous
cable

17

Cable
ethernet

15

Ethernet
fddi

21

Fiber Distributed Data Interface
g3-fax

10

G.3 Fax
hdlc-clear-channel

7

HDLC Clear Channel
iapp

25

Inter-Access Point Protocol (IAPP)
idsl

14

ISDN DSL
isdn-sync

2

ISDN Synchronous
isdn-v110

4

ISDN Async V.110
isdn-v120

3

ISDN Async V.120
piafs

6

Personal Handyphone System (PHS) Internet Access Forum Standard
sdsl

11

Symmetric DSL
sync

1

Synchronous
token-ring

20

Token Ring
virtual

5

Virtual
wireless

18

Other wireless
wireless-1x-ev

24

Wireless 1xEV
wireless-cdma2000

22

Wireless code division multiple access (CDMA) 2000
wireless-ieee80211

19

Wireless 802.11
wireless-umts

23

Wireless universal mobile telecommunications system (UMTS)
x25

8

X.25
x75

9

X.75
xdsl

16

DSL of unknown type