Building a high performance compute server on Azure and installing KenLM and Cuda/Kaldi with NVIDIA Tesla drivers.

About a week ago, I was asked to build a new server. This is going to be used for research purposes so the spec is quite high. 16 dedicated CPU cores, 110GB RAM and an NVIDIA Tesla T4 GPU. It’s running on Azure and the applications needed on it are a little different. So this was a lot of fun.

First the VM type: It’s a Standard_NC16as_T4_v3 server. You can’t just go buy one of these. You must create a support request with Microsoft so that they can release the number of cores required for this specific type of server. This is a painful process! There were 200 processor cores available in that subscription but obviously not at the right type. However, there is a very useful category when creating a support request in the Azure Portal for requesting additional cores. What isn’t so useful is the portal didn’t understand that I had enough cores. I needed the specific cores for this research server. I spoke to a HPC (High Computer Performance) specialist about something unrelated during the week and he knew what I was talking about right away. But it took over a week for Azure Support to understand what I was looking for then make the required changes.

Moving on, Once Microsoft did what they needed, setting up the new server wasn’t difficult. It was created within about 10 minutes after I finished with the VM creation wizzard.

The main requirements of this server are Cuda and KenLM and this is really what this post is about. I don’t spend every day in a Linux environment. So when I need to install something like this that I wouldn’t use often, I rely heavily on documentation. It’s not that I couldn’t go hunt down all the installation sources and dependencies. But that would be a waste of time. And time is not something I really like to waste.

I took notes during this process. These include the commands that I used to install everything and the various sources I read through to learn a bit more about what I was installing and how it could and should be done.

In case anyone copies and pastes the following lines, I am going to proceed my comments with #.

# First you need to determine the GPU that you have and the suggested driver. Fortunately, this is way easier than it used to be.
apt install ubuntu-drivers-common
ubuntu-drivers devices

# Do not use this next command. It installs way too much and will result in massive dependency issues when you go to install Cuda.
# ubuntu-drivers autoinstall

# After installing the GPU driver, you must reboot.
reboot now

# The following command will install the NVIDIA gPU driver. It will also install the unmet dependencies.
apt install nvidia-driver-470 libnvidia-gl-470 libnvidia-compute-470 libnvidia-decode-470 libnvidia-encode-470 libnvidia-ifr1-470 libnvidia-fbc1-470

# This will install all of the Cuda dependencies.
mv /etc/apt/preferences.d/cuda-repository-pin-600
apt-key adv --fetch-keys
add-apt-repository "deb /"
apt-get update
apt-get -y install cuda

# Add the Cuda binaries to your path:
echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' >> ~/.bashrc

# You can test that Cuda is installed and that the version installed is as expected as follows:
nvcc --version

# IF at some point, you need to start again, this one-liner will remove all the NVIDIA and Cuda packages that you might have installed using aptitude / apt-get.
# apt clean; apt update; apt purge cuda; apt purge nvidia-*; apt autoremove apt install cuda

# The following lines will install KenLM on Ubuntu 20.04.
apt-get update
apt-get install build-essential libboost-all-dev cmake zlib1g-dev libbz2-dev liblzma-dev
apt-get install build-essential libboost-all-dev cmake zlib1g-dev libbz2-dev liblzma-dev -y
git clone
cd kenlm/
mkdir build
cd build
cmake ..
make -j 4
make install

HomeServer updates 2022!

Oh this post could become very large. So I’m going to try to keep it brief. Perhaps I’ll pad it out with a few more posts over the next few days or weeks. But here goes.
My home server set up for 2022.
First of all, what’s all this for. Why do I need a home server? What is it used for?
My requirements for a home server have changed a lot over the past 20 years. Home servers for me started as Email and web servers then progressed into DHCP and DNS servers as well as firewalls with big noisy and powerfull beasts running under my stairs then running in self contained cabinits that were custom built for the task.
However, About five years ago, I decided I would move away from hosting my own DHCP and DNS servers and instead I would just go back to off the shelf solutions. Such as those provided by my ISP router and the Ubiquity controller for my wireless network. That has been fine. In fact, it has worked very well. However, it required a few small servers from time to time for testing technologies or ideas that I had. Raspberry Pi 4 has been my tiny compute platform of choice. But this started to get a bit messy. For example: I got more into home automation. So a Pi was dedicated to that. Previously, a Pi was running my Ubiquity Unifi controller and the code for some of my light automation. I was also frustrated a lot by the lack of decent customization in relation to DNS on the Fritzbox router. So here’s what I’m running right now.

  • PiHole for DNS. This is primarily working as an add blocker for all phones, tablets and computers on the network.
  • HomeAssistant. This handles all my home automation. I no longer even have a Philips or Aqara gateway / hub. I’m instead using a Combee ii USB stick as the Zigbee gateway. This will require some more explination.
  • The Unifi controller software for my Ubiquity wireless access points.
  • RClone. This is handling the processing and access to my cloud files.
  • Navidrone. This is my new audio server software. I’ll need to explain why that is needed in another post.
  • Bonob. This is a bridge between Navidrone and my Sonos. Used to let me play the media directly on my Sonos. Okay. I’m going to give you a quick overview of what I’m doing here because in my opinion, it’s kind of cool.

I’m running a large NAS in the house. But it’s getting old. It’s probably 8 years old by now. But it’s reasonably large. Running at 8tb usable storage space in RAID 5. Replacing that NAS isn’t something I’m very interested in doing for two reasons. Firstly. The cost would be huge. But second, it’s a big noisy thing. I could go for a quieter model but to get that kind of storage from solid state disks would cost a lot of money. So again. I suppose it comes down to cost. I’m going to need a NAS. That is unavoidable. But But thanks to an idea from a friend, I will need a lot less space.

So. How am I going to use less space while not removind a lot of files? Simple. Cloud storage. But that leads to another problem. How do you integrate cloud storage into your every day work flows and systems. For exmaple. If you store your music on Google apps or OneDrive, how does Sonos access it? It’s simple. It can’t. Not directly anyway. So here’s where for me it get’s interesting.

Firstly, understand that I wouldn’t just dump all the music up there. Because I have privacy concerns. I have aquired this music on CD over a very long time. It is mine but I would have a concern that if I start uploading 2tb of music, Microsoft or Google are going to start getting suspicious. Actualy, this is a founded concern. Paul Thurrott had this problem with OneDrive about four years ago. So I encrypt the files before sending them to the cloud service of choice. This really complicates things because now there’s really no hope of something like my Sonos reading the files because now they are on the cloud and they are also encrypted.

So. here’s how I get around it:

  1. I use RClone to encrypt and copy all files before I copy them from the old NAS up to the cloud storage.
  2. Now I mount the encrypted volume from RClone.
  3. I have set Navidrome up to look at this volume for it’s music
  4. Bonob then connects to Navidrome.
  5. Sonos is configured to use Bonob as a music service. Bonob is connected into Navidrome so the flow is: Sonos asks for musi c from Bonob. Bonob get’s that music from Navidrome. Navidrome get’s the file from the encrypted mountpoint by RClone. This encrypted mountpoint in turn goes to the cloud storage. All this happens within a maximum of four seconds. But although this sounds like a lot of time, it’s really not and also that 4 seconds is only really an issue when starting playback for the first time. When the Sonos is moving to the next track, it allows plenty of time to pre-cach the next track before playing it.

Have you read this far? Good. You’re officially a geek / nerd. Well done. I’m genuinely proud of you. There’s one more thing to just edge the geek factor up another notch.

Twenty years ago, this would have been running on several physical servers. Ten years ago it would have been running on one big beafy computer with several virtual machines dedicated to each function. In this generation of containers, this is all running on a mini-pc with an I7 processor, 16GB RAM and a 512GB NVME drive. Before the enterprise compute gurus jump out of your skin to tell me that there’s no redundancy here. You are absolutely right. But settle yourself down for a second. I’m going to talk about redundancy and backups now in a second.

Everything is running on Docker containers. So once I have backups do I really care if the computer dies? well, yeah. I would care because this little computer is really nice and it runs way faster than I had expected. But realistically if it dies, all I do is build a new host operating system, bring my docker containers back over to it, bring the containers up, configure networking and everything is back again. It’s not an enterprise environment with 100% up time. The main thing that matters is that it’s cheap to run, quiet, runs at a cool temperature and if something really goes wrong, I can update easily. I have the encryption password and salt saved somewhere safe completely disconnected from the server so once I can decrypt the encrypted backups, all is good. … I hope.

RDS / Terminal services: run multiple xming sessions on one server


Xming is an application that is used in conjunction with PuTTY on Windows to remotely access X11 graphical window interfaces on Linux. It’s an old application but a great one. Typically, it would be run as a single instance on one computer. However, my aim is to run it on a RDS session host server running Windows 2019. Several dozen students may need access to it concurrently. This


Xming is fine when running it as a single instance / a single invocation of the process on a single computer. But it wasn’t really designed to be run on a server where several dozen users might access it similtainiosly. There is a way of running several instances of it by specifying a server number. But the problem with RDS is that it would introduce a security risk if I was to allow end-users to specify their own command parameters when running an executable.


A quick Google search took me to this blog post from 2008. From there I found that there were command switches that enabled the process to run several times on the one server. However there are problems with implementing this in a user-friendly way for use on remote desktop services.

  1. Users shouldn’t have access to independently modify the parameters / arguments passed to any executable.
  2. A unique server number must be used for each server or the person already using that number would be kicked out.
  3. Incrementing the number based on the number of currently running instances of the xming process wouldn’t work. Take the example of 12 concurrent users of the process. User 5 logs off. There are now 11 users. A new user logs on so the count returns to 12. The number 12 is already in use but the user doesn’t know that so the person on number 12 is kicked off because the new person get’s 12 instead. Less than ideal!


The approach I took was to write a script that would associate every person who logs in with a unique number. Then each time they log in they will be allocated that same number.

$File = “c:\temp\tempuserlist.txt”
$Username = whoami
$CSVFile = import-csv $File | where-Object { $ -eq $Username }
If ($CSVFile -eq $null) {
$CSVFile = import-csv $File | Select-Object -Last 1
[int] $count = $CSVFile[0].Count
$Count ++
$details = [pscustomobject]@{
name = $Username;
count = $Count
$details | Export-Csv -Path $File -NoTypeInformation -Append
else {
[int] $count = $CSVFile[0].Count
Start-Process -FilePath “C:\Program Files (x86)\Xming\Xming.exe” -ArgumentList “:$Count -clipboard -multiwindow”

You now need to advertise PowerShell as an application, change the alias of PowerShell then also change the icon so that users think they are clicking on Xming. That’s easy enough as well. I assume that if you are reading this, you are already aware of how to set up an application in RDS. So I’ll focus on the step required to change the Icon of the PowerShell application so that it matches the icon of the Xming application.

  1. Open PowerShell as an administrator on the session host server where Xming is installed.
  2. Use the following command. Replace the connection broker and the collection name with the values from your environment.
    Get-RDRemoteApp -ConnectionBroker -CollectionName GenericStudent -alias “powershell” | set-rdremoteapp -ConnectionBroker -CollectionName GenericStudent -iconpath “C:\Program Files (x86)\Xming\Xming.exe” -IconIndex 0

The night before Christmas – 2021

Méabh and Rían continue to be the stars of the show. But I have tried to feature Emma and her mother in this more as let’s face it. Without their input, none of this would happen.

If you know us, or if you have followed these podcasts for the past 7 years, you will probably find that quite a bit changes from year to year. Sometimes not all for the good. This year, you’ll find that some of the dynamics change between Méabh and Rían. It’s nothing negative. Just an interesting alteration in their interactions.

Of course, in this podcast there’s always some fun. Some funny conversations and at some stage, someone will talk about poo.

I would sincerely like to take this time to with you and your family and friends the very happyist of Christmas. May 2022 bring you greater certainty, fortune, stability and safety.

Some music for a change.

I have a music related site over at so I rarely post my music related stuff here any more. But I had two professional videos recorded before the summer this year and I’m only figuring out that perhaps I should start getting them out to the public to get a bit of a return on my investment. 🙂

There’s another here.

Create a persistent alias in PowerShell

One of my pet hates about PowerShell is the /b switch isn’t available for get-childItem AKA dir or ls. I just want to read through a list of files in the directory. I don’t want all the other information. At least in the old dir command, the file and directory names were at the start of the line.

So to get around this, I simply just select the Name property from get-ChildItem. But I don’t want to type that every single time. Enter Profile.ps1 and set-Alias.

I have a simple command: List. This shows me exactly what I want to read and no more. You might have something similar for your situation.

You will notice that the alias list actually calls a function. This is because I’m piping the get-ChildItem command to the select command. This isn’t supported by set-alias. So adding this to a function get’s around that minor annoyance.

Oh, you will also notice that I’m changing the colour of the table row header to bright white using a new feature called $PSStyle. At some point, I’m going to script my screen reader to recognise these different colours and tell me what is a directory, what’s an executable, what’s a column header etc. This is the beginning of this process. Colouring the output of get-ChildItem is an experimental feature in PowerShell 7.2 through $PSStyle.FileInfo. I’m trying this out at the moment.
$PSStyle.Formatting.TableHeader = $psstyle.Foreground.BrightWhite
Function GetShortDirectory {
get-childitem | select Name

set-alias list GetShortDirectory

Check for high memory usage or hung status in Dell Boomi

I needed to add a check today to dell Boomi or as it’s now known as “Boomi” because it failed twice in the past few months. The problem was it failed but it didn’t actually stop the service. Because Boomi runs within a Java virtual machine, it doesn’t necessarily expose it’s problems to the host operating system. So monitoring systems such as Nagios don’t always pick up the accurate status.

The best way of determining if Boomi was not behaving as expected is to check the Boomi container logs. If there are memory errors, report an exit code of 3 to indicate to Nagios that there is a critical state or check for no logs written in the past two minutes to indicate again that there is a critical status as the Boomi Atom has hung.

Create the script on the Nagios Host and the Boomi Atom

Add this to the libexec directory. Probably /usr/local/nagios/
BoomiMemoryErrors="$(sed -n "/^$(date --date='10 minutes ago' '+%d %b %Y %H:%M:%S')/,\$p" /opt/Boomi_AtomSphere/Atom//logs/$(date +%Y_%m_%d).container.log | grep 'Low memory')"
if [ -z "$BoomiMemoryErrors" ]
echo "Boomi has no memory errors."
exit 0
echo "Boomi has encountered memory errors"
exit 2
AnyBoomiLogsWritten="$(sed -n "/^$(date --date='2 minutes ago' '+%d %b %Y %H:%M:%S')/,\$p" /opt/Boomi_AtomSphere/Atom//logs/$(date +%Y_%m_%d).container.log)"
if [ -z "$AnyBoomiLogsWritten" ]
echo "Boomi has stopped"
exit 2
echo "Boomi is Running correctly"
exit 0

On the Nagios server and the Boomi atom

First, find the commands.cfg file. Either use the locate command or use find -name. You need to add this to the bottom.

define command{
command_name check_boomi_memory
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c

define command{
command_name check_boomi_hung
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c

Add this check to the boomi Atom host file within your Nagios servers directory

I’m going to assume you know where that is. It’s usually in either /usr/local/nagios/servers or /etc/nagios/servers/

define service{
use generic-service
host_name #add your hostname
service_description check_boomi_hung
contacts #Add your contacts here.
check_command check_nrpe!check_boomi_hung

define service{
use generic-service
host_name #add your hostname
service_description check_boomi_memory
contacts #Add your contacts here.
check_command check_nrpe!check_boomi_memory

Update the NRPE config with these new commands

This file is likely in /usr/local/nagios/etc/nrpe.cfg


Verify that your new checks work

You will do this from the Nagios server. Make sure you reload the config first.

Quick tip:
Use the following command to check the validity of your config.

/usr/sbin/nagios -v /etc/nagios/nagios.cfg

Now to reload nagios, use the usual systemctl reload nagios.service

/usr/local/nagios/libexec/check_nrpe -H -c check_boomi_hung
/usr/local/nagios/libexec/check_nrpe -H -c check_boomi_memory

Coding on an iPad.

I really want a way of writing PowerShell and C# when on the iPad. I love this form factor and it’s versatility. But I haven’t found anything yet that does what I want. Not from a lack of trying. My search has lasted over a year now.

VSCode is my IDE of choice these days. There’s a project that will see this being ported to a web based tool. But for the moment it’s not near where I need it to be. Also, there’s that pesky need to be able to run or compile the code. It’s just not going to be possible on an iPad.

But I came across something today called CodeBlitz. It’s only really for client side development at the moment. Javascript type languages and HTML are it’s focus. But I took a look just to see how it behaved. I’m actually very impressed. If coding on an iPad is something your interested, this web based IDE is probably the approach to consider. OF course, this isn’t a new idea. The Arduino IDE transitioned away to a web based interface a few years ago. But still. It’s interesting.

Notice that on the right, the content of the quick test page that I created refreshed in real time? That’s kind of cool.

I was happy to find plenty of very responsive intelisense and movign around the UI was very efficient.

There are a few accessibility issues but they are minor. Namely unlabeled buttons on the top of the editor. Easy stuff to fix.

Frustrating encounters with external companies.

Frustrating encounters with external companies.

Support provider of a very expensive system hasn’t fixed a dodgy update that they released two weeks ago. I finally get them on the phone and 4 hours later, we have the following conversation:

External Company: There’s a socket error when we try to list the named instances on the database server.

Me: There are no named instances on that database server. It’s exclusively used for your database.

External company: But but wait. Microsoft say you need port 1434 open to list the named instances.

Me: Yeah. You’re right. Port 1434 is needed for named instance discovery. Lucky then we’re not using named instances isn’t it? Port 1434 isn’t needed.

External company: But look here. Telnet to 1434 doesn’t work! Look at this other support article from Microsoft that basically says the same thing from a few years earlier.

Me: Yeah. 1434 is UDP. Not TCP so telnet is never going to work for checking. But anyway. 1434 isn’t needed because as I said a moment ago: We! Aren’t! Using! Named! Instances!

Four hours later. We are going over the same song and dance. With every part of this. Yes. It’s a complex system. Yes. I shouldn’t get annoyed with an external company. But when systems cost tens of thousands of Euro and the support provider knows less about their own system than I do, I just get really really annoyed. Especially when it takes them two weeks to get back to me while the system is down and everyone is back to university.

The problem isn’t even with the database! It’s just one part of the entire system he needs to check while finding the actual root of the problem. Which I already know is corruption due to this badly implemented upgrade? Why and how do I know that? Because they have messed up these updates every year now for the past three years.


PowerShell Universal. Why I strongly discourage you from purchasing it.

PowerShell Universal should be a fantastic tool. A single language used for admin tasks, the UI, scheduling etc. I was delighted to find it!

But after a year of trying and trying again with some success but many many more failures, I’m finally giving up. I was willing to pass it off as a lack of experience with PowerShell when I started first. I’m certainly no master at it now either. But when something simply works in the console but refuses to work in PowerShell Universal over and over again, even I start to wonder if it’s not just me.

I won’t bombard you with examples. But here’s one:

$ApplicationFilter = @{
LogName = 'Application'
ID = 1000, 4005, 0
StartTime = [datetime]::Today.AddDays(-1)
EndTime = [datetime]::Today
$SystemFilter = @{
LogName = 'System'
ID = 2004, 6008
StartTime = [datetime]::Today.AddDays(-1)
EndTime = [datetime]::Today

$ServerList = Get-RDSessionHost -CollectionName mecheng -ConnectionBroker ((Servername.FQDN))
ForEach ($Server in $ServerList.SessionHost) {
try {
$Result = Get-WinEvent -ComputerName $Server -FilterHashtable $ApplicationFilter -ErrorAction Ignore | Select LogName, Level, ProviderName, TimeCreated, MachineName -ExpandProperty Properties

$Result += Get-WinEvent -ComputerName $Server -FilterHashtable $SystemFilter -ErrorAction Ignore | Select LogName, Level, ProviderName, TimeCreated, MachineName -ExpandProperty Properties
catch [Exception] {
if ($_.Exception -match "No events were found that match the specified selection criteria") {
Write-Host "No events found";
Return $Result

Simple enough. This fetches application and system events from RDS session hosts in a specific collection on a given RDS connection broker. It doesn’t use any kind of custom object. It just returns the output in a standard object. But still, although this works perfectly in PowerShell 5 and 7, it doesn’t work in PowerShell Universal. I get a completely strange error relating to the connection broker.

It’s not just this script. I have at least another dozen that work perfectly fine in the console but don’t work at all in PowerShell Universal.

Today I found that I couldn’t get any output at all from these scripts unless I configured them to use the default environment. Not PS5 or 7. That’s new.

Ironman software decided last year that they could no longer support me. Here’s the message from back then.

It’s becoming apparent that this solution is not working for you and I do not have the ability to continue to support your installation of it.

I realize that you have had lot’s of problems but that is what is making me realize that you should investigate another vendor. I apologize that we did not suite your needs.

I hate to be so negative about a small company. I have nothing against them personally. But I think the feedback I’m providing to them and to you is just the responsible thing to do. I see so many people with requests here and so many answers to the effect that there’s a bug, or something will work again in the next version. There’s no doubt in my mind that the product has improved in the past year. But I think it’s just too complicated in it’s design to be reliable. In my opinion, they should stop releasing new features for about 6 months and just focus on making things work and updating the documentation.

If I thought this had a chance of working , I would have put in more hours. I’ve already put in hundreds of hours trying to make this work for me. I see the potential. But without good support from Ironman software and with the inconsistency in so many parts of the tool, I need to just conclude that this is just a failed test.

Some of the features that drew me to this tool are:

  • Easily create API’s with PowerShell
  • Run PowerShell scripts on a schedule using various environments and credentials
  • Use centrally stored variables
  • Store the output of various jobs
  • Easily integrate with active directory for authentication
  • Quickly build UI’s

An example of where I was able to use this effectively last year was during several security sprints. I scheduled a script to go around all of our servers and collect the inbound firewall rules. It stored the job results so that I could then compare job 1 with job 50 for example to see what had changed. I then provided a UI with the number of inbound firewall rules for each server and gave people the ability to drill down to each server to specifically see what firewall rules were listed.

I missed the first two security sprints because I had problems where Universal wasn’t working as expected. But when it worked, it worked really well.

So I’m not saying that there isn’t a need for this product. If there wasn’t a need, I wouldn’t be so frustrated writing this post.

The fact is, it’s just not reliable. I have written a rather large module for me and others in my team to use. Each function produces an object. It works flawlessly. The way I usually plan out what would be nice to have in Universal is I’ll write my functions in this module, let the senior admins use them for a while then I’ll create scripts that call them in PowerShell Universal so that they can either be scripted or a UI can be wrapped around them for more junior people to use. A great example is the above script. I have that so that I can quickly check the status of these RDS session host servers. But my plan was to make a quick UI for it. I would run the script on a schedule then display the output from the last execution. This means the UI opens much faster resulting from the script running a few moments previously in the background. But then something senseless happens that throws a completely unexpected error in PowerShell Universal and what should take 15 to 20 minutes ends up taking me hours.

I urge caution if you are buying this. It’s not very expensive. But the hours I spent trying to make it work cost a lot more. It took me a long time to come to the decision to give up and longer then to decide to post this negative review in conclusion. It’s the first time I have ever felt so compelled to do so. This is a reflection of the frustration I have encountered with this product.