The blog of Vaughan Hilts

Installing and running HorizonXI on Linux

2022-06-19T21:17:00+00:00

The following is for installing HorizonXI, a FFXI private server on Linux. Specifically, I am installing with the following Arch Linux System:

touma@setsuna:wine/x86_64-windows $ neofetch
                   -`                    touma@setsuna 
                  .o+`                   ------------- 
                 `ooo/                   OS: Arch Linux x86_64 
                `+oooo:                  Model: B450 AORUS PRO WIFI 
               `+oooooo:                 Kernel: 5.12.9-arch1-1 
               -+oooooo+:                Uptime: 6 days, 19 hours, 29 mins 
             `/:-:++oooo+:               Packages: 1672 (pacman) 
            `/++++/+++++++:              Shell: zsh 5.8 
           `/++++++++++++++:             Resolution: 2560x1440 
          `/+++ooooooooooooo/`           DE: GNOME 40.2 
         ./ooosssso++osssssso+`          WM: Mutter 
        .oossssso-````/ossssss+`         WM Theme: EvoPop-Azure 
       -osssssso.      :ssssssso.        Theme: Materia-light-compact [GTK2/3] 
      :osssssss/        osssso+++.       Icons: Numix [GTK2/3] 
     /ossssssss/        +ssssooo/-       Terminal: gnome-terminal 
   `/ossssso+/:-        -:/+osssso+-     CPU: AMD Ryzen 7 3800X (16) @ 3.900GHz 
  `+sso+:-`                 `.-/+oso:    GPU: NVIDIA GeForce GTX 1080 
 `++:.                           `-/+/   Memory: 5913MiB / 32116MiB 
 .`                                 `/
                                                                 

Install HorizonXI - `wine-staging` (non steam-deck; steam deck below)

Prepare a version of Wine with the proper patches

**tldr; install wine-staging if you want something simple (such as yay wine-staging) and you’re good to go.

You need a version of WINE that won’t crash when accepting the EULA, presumably like most XI installs. There are a couple options here.

“Easy, recommended”: Use a variant of WINE that has the patch integrated into it already, such as wine-staging (here). If you are on Arch Linux, you can do this using pacman -S wine-staging (or any other helper you use). You can find wine-staging in some other repos as well, such as the Ubuntu ones. You can find more on the WINE install page. There are other versions that include that patch in their definition – any of them will do.
“Easy, works in a pinch”: You can download a patched imm32.dll from the web such as https://github.com/bluffnix/ffxi-wine/tree/master/linux and replace your system library with it. You can find some directions on the web to do this – I would recommend you don’t do this because every time you update the operating system and the wine package it’s going to get replaced with a new binary. However, I wanted to write this here since it’s shown on the web a lot as a drag and drop solution from a lot of users.
“Hard”: You can compile a version of WINE that has the proper fix. The patch in question is this. If you are an advanced user, you can do this if you can’t replace your system wine. If you do this, make sure you compile WINE with 64 bit support since you will need it. If you need help with this, leave a comment but for most users this shouldn’t be needed and should only be done if you can’t use the other options.

At the time of writing, this worked with:

touma@setsuna:tidus $ wine --version
wine-7.22 (Staging)

Note: A lot of places on the Internet tell you to use a specific prefix architecture for Final Fantasy XI; this does not work for many private servers – so please follow the directions very closely. If in doubt, follow step by step and start with a fresh prefix.

Prepare a new prefix for HorizonXI using the following command:

WINEPREFIX=~/.wine-horizonxi-64 winecfg

Check that “Windows 7” is selected as the version.

Moving along, let’s run the installer inside of WINE:

WINEPREFIX=~/.wine-horizonxi-64 wine "~/Downloads/HorizonXI-Launcher-1.0.1.Setup.exe"

From this point, you can click “Install HorizonXI” and things will begin working:

An install directory, you can leave that as a default but it defaults to your virtual “Users” drive. I left it there since there is no harm (virtualized drive).
You do not need to login to install the game from what I can tell.

That’s it as far as we know right now.

Install HorizonXI - Linux Desktop using Lutris (non steam-deck; steam deck below)

Lutris is meant to be an all-in-one installer, however there are additional steps required at present to make it work. If you want a quick solution that will work use the process listed above. Otherwise follow the instructions below. Once new Lutris runners are shipped with Lutris i’ll update the script and this guide. Should be early next year for a new runner.

Requirements

You need to have wine-staging installed.

ProtonUp-Qt and runner “lutris-GE-Proton7-42-x86_64” installed for lutris.

Install ProtonUp-Qt on your distribution and open it up.
1. At the top make sure Lutris is selected.
2. Then hit “Add Version”
3. Select “Wine-GE” for compatability tool.
4. Select “GE Proton 7-42” for version.
5. Hit install and once it’s complete you’re done with ProtonUp-Qt.
HorizonXI Lutris Script Downloads the script from sarca571ca/horizonxi-lutris repo. Be sure to extract the YAML files inside the archive.

Installation

Open Lutris and hit the (+)plus in the top left.
Select install from YAML file, select the Horizon-XI.yaml file, and follow the steps in the installer.
Once completed hit Close not Launch.
Now your free to launch HorizonXI you will get a launch prompt asking for either HorizonXI or the 4GB Patch. Select HorizonXI and install like normal.

Note: You might get an error about missing the prereqs.zip, this can be ignored as lutris installed all pre-reqs automatically for you. If you experience the “black void” while changing areas you can lower your textures some in the HorizonXI Launcher. You can also use the 4GB patch i’ve included.

4GB Patch

Check the FAQ section or click here

Windower 4

If you want to install Windower follow the README.md provided in the archive you downloaded earlier. Unless otherwise requested by Vaughan I won’t clog this guide up with that install process.

Version

Updated September 22, 2023
- Added new requirement for ProtonUp-Qt
- Updated install process for a smoother install.
- Supports Horizon Launcher v1.2.1
Initial guide December 22, 2022

Guide by: Ex1L3(Asri) in the HorizonXI discord, feel free to ask question etc.

Install HorizonXI - Steam Play (Steam Deck / other systems)

There is an video that goes with this guide if you would prefer: https://www.youtube.com/watch?v=4pWbxU025Ak

If you are new to Proton or Linux, it may be valuable.

The following is for Steam Deck users:

Automatic installation

Download Trent’s script and execute in konsole.

Follow the prompts instructions, and report any bugs in the Steam Deck Thread tagging @trent with a snippet including the konsole logs.

For those who click through without reading the basic idea is to install the base game with v1.0.1, then when that is fully completed rerun update-horizon.sh and it will update the installer for you to the latest.

Manual installation

Boot your Steam Deck into “Desktop Mode”
Install ProtonQT from Discover (link)
Open up “Firefox” and “Steam”
Download the installer from the website. It will end up in ~/Downloads
Open a terminal and do the following:

Note: At the time of writing, the installer from the website may not work due to some breaking changes. You may need to install from https://github.com/HorizonFFXI/HorizonXI-Launcher-Binaries/releases/tag/v1.0.1 first, and then upgrade later. The install process seems broken on Linux on the newer launcher

mkdir ~/horizon-xi
cp "/home/deck/Downloads/HorizonXI-Launcher-1.0.1.Setup.exe" ~/horizon-xi/installer.exe
cd ~/horizon-xi
7z x installer.exe
7z x HorizonXI_Launcher-1.0.1-full.nupkg

OK, that’s done.

Now:

Launch ProtonQT-Up
Install “GE-Proton7-42”
Restart Steam / deck if you can’t restart Steam from desktop

It should look something like this:

Now, open “Steam” and add a ‘Non-Steam Game’ and navigate to ~/horizon-xi/lib/net45/HorizonXI-Launcher.exe for the executable (you may have to change “.desktop” at the bottom to all application types)
Right click the new entry, hit Properties > Compatibility
Change to the new version of Proton GE you just installed

Launch the game!

Begin installing. You should pick C:\Program Files when prompted for an install path. You should not pick anything else. You can move compatdata to an SD card later if needed.

The launcher has some bugs. Continue down to FAQ down below to find any problems you might have. You may want to read them all before clicking install.

When install is complete you can run:

sudo find /home -name "HorizonXI.zip" -type f | sed 's/ /\\ /g' | xargs -i rm {}

… to clean up the ZIP to get some space back. You may want to save a copy if you want to install somewhere else in the future.

Gamepad

The gamepad needs configuring to work. In game mode:

Open the launcher open the Gamepad Configuration
Enable XInput

In desktop mode:

Open Dolphin to /home/deck/.local/share/Steam/steamapps/compatdata/<prefix_id>/pfx/drive_c/users/<user>/AppData/Roaming/HorizonXI-Launcher/ (protontricks can help find the ID)
Open config.json
Make the following edits to the config:

"padmode000": {
    "name": "Gamepad Settings Controls",
    "key": "padmode000",
    "value": [
        1,
        1,
        0,
        0,
        1,
        1
    ]
}

and:

"padsin000": {
    "name": "Button Mappings",
    "key": "padsin000",
    "value": [
        8,
        9,
        13,
        12,
        10,
        0,
        1,
        3,
        2,
        15,
        -1,
        -1,
        14,
        -33,
        -33,
        32,
        32,
        -36,
        -36,
        35,
        35,
        6,
        7,
        5,
        4,
        11,
        -1
    ]
}

Gamepad should now work in game mode. Credit goes to Sabriel@ for the gamepad directions.

FAQ for Linux

The launcher keeps crashing while trying to install the game

… and it’s wiping all my progress!

Mandatory warning from the HorizonXI staff:

Please note: We will not offer support for users that bypass the launcher with this setup, it’s a work around for users having memory issues due to having less than 16gb ram on PC.**

Still with me? OK, next.

There is a memory leak in the launcher as of launch day. The launcher does not save progress. The staff are working on fixing it but in the meantime, you can follow the below steps to get it working:

magnet:?xt=urn:btih:4eecae8431428820347314bc002492e210f29612&dn=HorizonXI.zip&tr=udp%3a%2f%2fopentracker.i2p.rocks%3a6969%2fannounce&tr=https%3a%2f%2ftracker.nanoha.org%3a443%2fannounce&tr=https%3a%2f%2ftracker.lilithraws.org%3a443%2fannounce&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a6969%2fannounce&tr=https%3a%2f%2fopentracker.i2p.rocks%3a443%2fannounce&tr=udp%3a%2f%2ftracker1.bt.moack.co.kr%3a80%2fannounce&tr=udp%3a%2f%2ftracker.torrent.eu.org%3a451%2fannounce&tr=udp%3a%2f%2ftracker.tiny-vps.com%3a6969%2fannounce&tr=udp%3a%2f%2fpublic.tracker.vraphim.com%3a6969%2fannounce&tr=udp%3a%2f%2fp4p.arenabg.com%3a1337%2fannounce&tr=udp%3a%2f%2fopen.stealth.si%3a80%2fannounce&tr=udp%3a%2f%2fopen.demonii.com%3a1337%2fannounce&tr=udp%3a%2f%2fmovies.zsw.ca%3a6969%2fannounce&tr=udp%3a%2f%2fipv4.tracker.harry.lu%3a80%2fannounce&tr=udp%3a%2f%2fexplodie.org%3a6969%2fannounce&tr=udp%3a%2f%2fexodus.desync.com%3a6969%2fannounce&tr=udp%3a%2f%2f9.rarbg.com%3a2810%2fannounce&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337&tr=udp%3a%2f%2fexplodie.org%3a6969

Launch the installation once
Let it begin downloading some of the files
Kill the launcher
Grab the magnet with your favourite torrent client – you can use a PC for this and transfer the file to the deck or just install something like Transmission / Deluge / qtBitTorrent on your Deck to use.
Wait for the download to finish. It won’t crash :)
Open Dolphin (file manager) and navigate to /home/deck/.local/share/Steam/steamapps/compatdata/2237253119/pfx/drive_c/Program Files/HorizonXI/Downloads/.
The above path is an example – your randomly generated number after compatdata is probably different and the install path depends on what you picked.
Place the ZIP in this folder
Open Dolphin (file manager) and navigate to /home/deck/.local/share/Steam/steamapps/compatdata/2237253119/pfx/users/<user>/AppData/Roaming/HorizonXI-Launcher/config.json
Open config.json and modify the file:

		"baseGame": {
			"downloaded": false,

should be replaced with:

		"baseGame": {
			"downloaded": true,
            

If you open the launcher, it should now finish installing the game.

How can I update the launcher?

If you followed the guide, do the following:

Find the launcher version you want to update to from https://github.com/HorizonFFXI/HorizonXI-Launcher-Binaries/releases
Run the following commands in the terminal one-by-one in Desktop mode again. This is just like the initial install:

cp "/home/deck/Downloads/HorizonXI-Launcher-1.1.12.Setup.exe" ~/horizon-xi/installer.exe
cd ~/horizon-xi
7z x installer.exe
7z x HorizonXI_Launcher-1.1.12-full.nupkg

Make sure to replace the version number in the file paths to match the version of the launcher you just installed.

That’s it.

Launch the game, it should be updated.

My game is asking me to install again. What happened?

This is typically because something has happened to storage.json. If you just want to fix this, do the following:

Reboot your Steam Deck and do NOT launch the game at all.
Boot into desktop mode
Head to /home/deck/.local/share/Steam/steamapps/compatdata/<PREFIX_ID>/pfx/users/<user>/AppData/Roaming/HorizonXI-Launcher/storage.json on the deck via Dolphin or some other file manager. Just like all other Proton installed software, the prefix ID will be random. You can use protontricks to find it or just sort by “recently modified” or something else to try and find the folder.
Open the file. Inside of this, you will find a bunch of JSON with several sections. The following section should be present.

		"installPath": {
			"name": "installPath",
			"description": "Location of HorizonXI Game install.",
			"path": "C:\\Program Files\\HorizonXI\\Game"
		},
		"downloadPath": {
			"name": "downloadPath",
			"description": "Location of HorizonXI zip download.",
			"path": "C:\\Program Files\\HorizonXI\\Downloads"
		}
	},

If you have followed the guide, you should have a copy of the game at C:\\Program Files\\HorizonXI. Some folks may have also installed it to C:\\. If you are not sure where it is, go back to the drive_c inside of the pfx folder (up a few directories from this config file) and find the location. The path will be whatever it is under drive_c, so for example if drive_c/HorizonXI exists, then your path is probably C:\\Program Files. The double slash is intentional, it’s called escaping.

Modify the paths to point to the copy of the HXI game you have located. If you can’t find it, then that means your install is either lost or you cannot locate it. You will need to reinstall.
In addition, there is a section of the config file that looks like this:

		"currentVersion": 3,
		"latestVersion": 3,
		"baseGame": {
			"downloaded": true,
			"extracted": true
		},
		"updater": {
			"downloaded": 3,
			"extracted": 3
		},

You can modify the values to match this. If you don’t know if your old install is fully patched or not, I would recommend replacing currentVersion with 2 or 1 to redownload any DATs needed.

Launching gamepad configuration prevents my game from launching anymore

Sorry about that. It looks like a bug. Windows users are reporting the same.

You have two choices:

Re-install and don’t click Gamepad configuration – the bug should be sorted “soon”
If you need gamepad support (Deck / using gamepad out of the box), you can follow this guide: https://www.youtube.com/watch?v=0UYdFoaVnOE&lc=UgywlUbnFvFWxBUOzW54AaABAg. It’s not supported by the team, but has been confirmed to work on the deck.

A patch is supposed to come soon for gamepad players.

I can’t see buttons on the UI

The launcher has a fixed size. You can open the Steam keyboard and use tab to move around controls you can’t see, dock your Steam Deck to a montior / TV for more space, or use KDE window management features to move it around (google: “kde move window with mouse”).

The launcher is showing all black

DXVK can cause this on some older cards. Disable it.

I’m getting told I can’t run it as admin

You’re probably trying to run the installer as-is from the website. Don’t do that; follow the directions above closely and make sure the item you added in Steam is the extracted HorizonLauncher and not the setup.

4GB Patch

Allows the game to use 4gb of RAM instead of the default 2gb. The disclamer in the discord says VRAM but Thorny(the Aurthor of the patch) says it increases RAM allowance of the executable.

If you are using the Lutris install script then:

Launch HorizonXI in Lutris seclect 4GB Patch option and hit ok.

If using the base wine install then:
> Download the Patch here and extract the archive.
> Run the patcher inside your prefix

WINEPREFIX=~/.wine-horizonxi-64 wine "~/Downloads/LargeAddressAwarePatcher/LargeAddressAwarePatcher.exe"
Navigate to C:\PATH\TO\HorizonXI\bootloader\horizon-loader.exe
Hit Patch, OK, and close the Patcher.

Where can I get artwork for my install?

You can find some at assets.

These are offical art assets cut for the Deck. Thanks to trent@ and Aku@.

Finding all the chest messages in "Trails from Azure"

2021-10-26T21:17:00+00:00

There is a fun tradition from back in Trails in the Sky for Trails in the Chest that involves hiding fun messages inside of the treasure chests in some of the Kiskei games. You can find some of these chest messages here, such as the Trails in the Chest series but no comprehensive list seems to exist anywhere. It turns out they are not too hard to extract from the game, so I thought it would be a fun evening project to extra the ones from Trails to Azure since I just finished playing it and it was fantastic! I’ll write about that later. For now, let’s move on to getting those chest messages!

First, let’s begin with just finding the Trails in the Chest messages since they’re easier to find as they have well known bounds and text.

Let’s start with the following:

I ran strings on the game scene files to create a dump from all the strings I could find. This is pretty easy to do since the files are inside of a single data directory and they’re not compressed. Something like find . | xargs strings > ~/workspace/dump
From there, we could do all sorts of transformations on this data A quick peek in less ~/workspace/dump shows us the following (mild spoilers from Azure):

#5CCongratulations on reaching the end.
You've got a long road ahead of you still,
but I know you can do it.#0C

This is one of the messages from the game. We can see that it has some codes between the text. These are probably font colour op-codes since it is shared with some things that aren’t chest messages as well and they appear in a different font in game. For example:

Skimming through the file reveals that some other non-chest messages inside of the file. However, for now there is no need to worry about that since we’re just looking for the “Trails in the Chest” messages. They have some characteristics that make them easy to find:

They start with [1/xx] etc, etc, one for each party of the story
There is a fixed number of them

With this, we can just use awk to find them in the dump: *cat ~/workspace/dump | awk '/#5C\[.\*\/.\*\]/,/#0C/'*. This gives us all the messages of which you can find a sample of them below:

#5C[55/61] In Bose, cramped between two of the most popular
shops on the block, was a tiny, tiny building. Last year,
it had been in a terrible state. Cobwebs once adorned the
windows, and insect droppings were commonplace.#0C
#5C[1/61] You remember reading the teaser in Trails in the Sky the 3rd
for 'Trails in the Chest, Pt. II,' but you didn't think it'd ever come.
Yet suddenly, you have a craving for more Trails...in the Chest.#0C
#5C[6/61] The day of their wedding, she, beautifully pristine
to a distressing degree in his eyes, had marched down
the aisle at a stiff pace. When he lifted her veil, he'd
seen no joy in her expression.#0C
#5C[44/61] 'Oh, SHUT UP!' Genevieve hopped just enough to
grip it and pulled it downward. Both had their hands on
the chest now, and neither one appeared to have any
intention of letting go. And then, to Genevieve's shock...#0C
#5C[11/61] True love! Something out of fairy tales, yes, but
Cyril's heart always did pang with longing while reading
them. He was a romantic at heart, and now here he sat,

We could do some cleanup to make this easier to read but it’s some simple regular expressions that we don’t need to cover here.

Now, how do we find all the chest messages? This is a bit trickier and we need to observe the following things about the string dump:

The chest messages can span at most 3 lines
The chest messages tend to adjacent to each other in the string dump (since they are located close together)
They always have the same font colour in the game

With this, we can write a short program to try and get the data out of our strings file:

const fs = require("fs");
const b = fs.readFileSync("/home/touma/Downloads/ao/dump");
const msgs = b.toString();

let pageBuffer = undefined;
let pageCount = 0;
const pages = [];

const lines = msgs.split("\n");

for (const line of lines) {
  if (line.startsWith("#5C")) {
    pageCount = 0;
    pageBuffer = {
      msg: "",
    };
    pageCount++;
  }

  if (pageCount > 4) {
    // Give up on this message, lost cause and probably "garbage"
    pageCount = 0;
    pageBuffer = undefined;
  }

  if (line.includes("#0C")) {
    // End of the line, we can include this line and then start over
    if (pageBuffer) {
      pageBuffer.msg += line.replace("#0C", "").replace("#5C", "") + "\n";
      pages.push(pageBuffer);
    }
    pageCount = 0;
    pageBuffer = undefined;
  }

  pageCount++;
  if (pageBuffer) {
    pageBuffer.msg += line.replace("#5C", "") + "\n";
  }
}

for (const page of pages) {
  const { msg } = page;
  console.log(msg);
}

console.log(pages.length);

This gives us pretty much a perfect listing with some manual cleaning needed to remove some false positives.

The full output of this can be found here.

Playing Playstation remotely over Wireguard

2021-10-26T21:17:00+00:00

tldr; if you want to play over a VPN, consider something like Chiaki

I’ve been going out more lately – especially now that COVID in Canada has been subsiding and things are returning to some sense of normalcy over the past few months. I’ve also been playing some games like Trails of Cold Steel IV that are quite long and have been wishing I could take them on the go. Looking for solutions, there are a few to play over the Internet:

PCs: You can use the remote play application from Sony, but unfortunately it only works under Windows on desktops
Mobile phones: You can use the official apps from the store but they lack good gamepad support outside of the Dualshock series of controllers

I run Android and use a stereoscopic controller for mobile. I use Arch Linux for my desktop operating system. This makes both of the above solutions problematic. The above solutions also have some other problems for both clients:

Sony limits the bandwidth you can get from over the Internet through their relay
Their relay introduces some latency

We then come up with the following needs list:

Supports whatever resolution I want
Should be able to play without relying on Sony’s server as much as possible
Should support whatever game-pad I have
Should work under Linux without trouble
Ideally, it would support my already existing Wireguard install

Running official apps over a VPN

The first obvious thing to do is try the applications that Sony provides over a VPN and to see if we make some tweaks to them to get them working for our needs. Unfortunately, this does not work well at all with Wireguard and other Layer 3+ VPNs. The reason is mostly that they use multicast to try and find devices over local networks. You need Layer 2 networks to do this.

I found this out after doing some sniffing around and reading this good [https://github.com/williampiat3/ImprovingPSRemotePlay

. In my case, I didn’t want to setup a Layer 2 VPN or change my existing networking topology just to accommodate this. However, it does not easily support Linux nor does it meet some of our requirements.

Running aftermarket over a VPN

The thing that ended up working for me is using two other applications with more support than the Sony applications that also happen to support Layer 3 VPNs due to the nature of allowing direct IP connection:

Chiaki
PSPlay for Android

You can setup direct connections for both of these and I can attest they work perfect under Wireguard. I setup some static addresses via DHCP reservations inside of my router and then I can use those to connect when routed through my VPN and it all works well: controller support + good resolution support.

It is worth noting that Chiaki didn’t seem to fully support my gamepad like PSPlay did, so that’s why I ran with it for Android. This is at the time of writing – perhaps in the future it will look a bit better.

Installing and running WingsXI on Linux

2021-06-19T21:17:00+00:00

The following is for installing WingsXI, a WoTG FFXI private server on Linux. Specifically I am installing with the following Arch Linux System:

touma@setsuna:wine/x86_64-windows $ neofetch
                   -`                    touma@setsuna 
                  .o+`                   ------------- 
                 `ooo/                   OS: Arch Linux x86_64 
                `+oooo:                  Model: B450 AORUS PRO WIFI 
               `+oooooo:                 Kernel: 5.12.9-arch1-1 
               -+oooooo+:                Uptime: 6 days, 19 hours, 29 mins 
             `/:-:++oooo+:               Packages: 1672 (pacman) 
            `/++++/+++++++:              Shell: zsh 5.8 
           `/++++++++++++++:             Resolution: 2560x1440 
          `/+++ooooooooooooo/`           DE: GNOME 40.2 
         ./ooosssso++osssssso+`          WM: Mutter 
        .oossssso-````/ossssss+`         WM Theme: EvoPop-Azure 
       -osssssso.      :ssssssso.        Theme: Materia-light-compact [GTK2/3] 
      :osssssss/        osssso+++.       Icons: Numix [GTK2/3] 
     /ossssssss/        +ssssooo/-       Terminal: gnome-terminal 
   `/ossssso+/:-        -:/+osssso+-     CPU: AMD Ryzen 7 3800X (16) @ 3.900GHz 
  `+sso+:-`                 `.-/+oso:    GPU: NVIDIA GeForce GTX 1080 
 `++:.                           `-/+/   Memory: 5913MiB / 32116MiB 
 .`                                 `/
                                                                 

There a few guides on the Internet that explain how to get this working as things are today but none of them cover WingsXI specifically which has a few quirks, so this guide will run through it. You can skip to each section if you want to know about specific parts of WingsXI or just installing Final Fantasy XI on Linux in general. All the commands here are given for Arch based distributions but you can find commands (such as replacing yay with something else such as apt-get for your system if you are using something else)

Downloading WingsXI

You can visit the WingsXI Discord from their site to get the latest binary. At the time of writing, this is this file – however you should check Discord and check out the #first-time-setup to ensure you’re getting the latest version. It’s a torrent – so ensure you have some sort of torrent software to grab it. I use Deluge.
Sign up for an account.
Activate your account via the e-mail that you provided.

Prepare a version of Wine with the proper patches

You need a version of WINE that won’t crash when accepting the EULA. There are a couple options here.

“Easy, recommended”: Use a variant of WINE that has the patch integrated into it already, such as wine-staging (here). If you are on Arch Linux, you can do this using pacman -S wine-staging (or any other helper you use). You can find wine-staging in some other repos as well, such as the Ubuntu ones. You can find more on the WINE install page. There are other versions that include that patch in their definition – any of them will do.
“Easy, works in a pinch”: You can download a patched imm32.dll from the web such as https://github.com/bluffnix/ffxi-wine/tree/master/linux and replace your system library with it. You can find some directions on the web to do this – I would recommend you don’t do this because every time you update the operating system and the wine package it’s going to get replaced with a new binary. However, I wanted to write this here since it’s shown on the web a lot as a drag and drop solution from a lot of users.
“Hard”: You can compile a version of WINE that has the proper fix. The patch in question is this. If you are an advanced user, you can do this if you can’t replace your system wine. If you do this, make sure you compile WINE with 64 bit support since you will need it. If you need help with this, leave a comment but for most users this shouldn’t be needed and should only be done if you can’t use the other options.

At the time of writing, this worked with:

touma@setsuna:tidus $ wine --version
wine-6.10 (Staging)

You must wait until the download of Wine and WingsXI to continue on from this point.

Preparing a prefix for WingsXI

Note: A lot of places on the Internet tell you to use a 32-bit prefix for Final Fantasy XI; this does not work for WingsXI so please follow the directions very closely. If in doubt, follow step by step and start with a fresh prefix.

Prepare a new prefix for WingsXI using the following command:

WINEPREFIX=~/.wine-wingsxi-64 winecfg

If you get prompted for installing Gecko, you can accept without trouble.

This will generate a new prefix. You should check:

That the configuration window shows up
Windows 7 is selected as the version

Moving along, let’s run the installer inside of WINE:

WINEPREFIX=~/.wine-wingsxi-64 wine ~/Downloads/Install\ Wings\ v3/Installer.exe

From this point, you can click through the prompts and you should get:

An install directory, you can leave that as a default and the rest of the guide will assume you did. It will be a Windows path that is relative to your prefix.
A bunch of prompts that will install various Visual Studio redistributable packages. Let that run.

It should complete without trouble. Next, unfortunately WingsXI bundled with an installer that does not setup the path to the boot command by default. We can fix this with a quick sed replacement, so go ahead and run this:

sed -i 's/C:\\Users\\alexs\\Desktop\\Installer Package\\Ashita\\ffxi-bootmod\\pol.exe/C:\\WingsXI\\Ashita\\ffxi-bootmod\\wingsloader.exe/g' "~/.wine-wingsxi-64/drive_c/WingsXI/Ashita/config/boot/New Configuration 1.xml"

(If the above does not work, you can edit the file manually as well)

Running WingsXI

To run the game, you can use the following command:

cd ~/.wine-wingsxi-64/drive_c/WingsXI/Ashita/
WINEPREFIX=~/.wine-wingsxi-64 wine ~/.wine-wingsxi-64/drive_c/WingsXI/Ashita/injector.exe "New Configuration 1.xml"

You should get a login screen that looks something like:

touma@setsuna:WingsXI/Ashita $ 19/21 21:48:47] DarkStar Boot Loader (c) 2015 DarkStar Team
[06/19/21 21:48:47] Git Repo   : https://github.com/DarkstarProject/darkstar
[06/19/21 21:48:47] Modified for use with the Wings Project (c) 2021 Wings
[06/19/21 21:48:47] Version: 1.02
[06/19/21 21:48:47] Website    : https://www.wingsxi.com
[06/19/21 21:48:47] Git Repo   : https://gitlab.com/ffxiwings/wings
[06/19/21 21:48:47] ==========================================================
[06/19/21 21:48:47] Checking for updates...
[06/19/21 21:48:48] Already using the latest version.
[06/19/21 21:48:48] Connected to server!

From the login:

Press “1” for login
Type your username from the registration site
Type your password from the registration site

You should login. If you are having trouble, consult the following common issue list:

The game crashes before I can see anything. You are probably running a 32 bit prefix or something else that is not configured wrong. Open a bug on WINE or attach a crash report here so I can help you.
The game crashes after I accept the EULA. You don’t have the X11DRV patch or it has stopped working. Try the WINE version listed here, make sure you’re using staging, or ensure you’re running your patched compiled version.
The login boot loader keeps repeating the menu over and over and I can’t login. There exists a bug in WINE at the time of writing that prevents you from using the bootloader IO console. You can work around this by editing the boot command from WingsXI with your own command. If you have modified nothing else in the XML configuration, you can simply open the file “~/.wine-wingsxi-64/drive_c/WingsXI/Ashita/config/boot/New Configuration 1.xml” in your favourite text editor and under the boot_command tag you can add the following arguments –user USERNAME –pass PASSWORD` to the command.
When I tab out of the game, I can’t move / my macro palette keeps coming up. This is due to your ALT key being stuck as being read. You can address this in a couple ways but the simplest way to do this is run winecfg per the initial command at the start of the guide again (WINEPREFIX=~/.wine-wingsxi-64 winecfg and turn on “Virtual Desktop” to get it’s own window frame). Alternatively, when tabbing back in simply press “Alt” again and it should work again.

Assuming a successful login, you should be able to login with a character and get going without any trouble now. You should have access to all Ashita plugins like you should under Windows as well.

You can create a shell script to run this command if you want / make a desktop shortcut out of it.

Correcting gamma

There exists an issue in XI that for some users when you launch the game it will request a gamma shift to play the opening movie but this gamma shift becomes permanent while the game is running. To fix this:

If you are running GNOME or some other DE, you can simply reload the shell after launching to revert the change. For example, for GNOME you can do Alt+F2 and then type “R” and Enter. KDE, etc have similar commands you can look up.
You can change some registry settings to shut this off. Since changing the registry can be dicey, I won’t provide a one liner for this but you can find information online once you understand you have to change.

You’ll know the gamma has changed when you see it.

Configuring XI / Running the Ashita GUI

The Ashita GUI itself will not run (or I could get it to run) under a 64 (auto) prefix. It does, however, work, in a 32 bit prefix with one caveat: the game won’t launch from the launcher due to some other stubbed behaviour of fetching values from the registry.

I would recommend you open the configuration file at ~/.wine-wingsxi-64/drive_c/WingsXI/Ashita/config/boot/New Configuration 1.xml and simply modify it yourself. The syntax is very straightforward and you can edit things like default.txt with a simple text editor to get most of the control that that the GUI provides.

However, it IS possible to run Ashita in a 32 bit prefix if you want to, configure things there, and then copy the config to your 32 bit prefix. To do this:

Install wine-staging if you didn’t already
Create a 32-bit prefix such as WINEPREFIX=~/.wine-wingsxi-32 WINEARCH=win32 winecfg
Install WingsXI per the directions above
Install .NET 4.0 and 4.5.2 into the prefix: winetricks dotnet40 dotnet45 WINEPREFIX=~/.wine-wingsxi-32 WINEARCH=win32 (you may need to install winetricks if it was not included in your wine package such as pacman -S winetricks)
You can now run Ashita WINEPREFIX=~/.wine-wingsxi-32 wine ~/.wine-wingsxi-32/drive_c/WingsXI/Ashita/Ashita.exe
Make your config changes and then save them out to the other prefix

You may be able to run Ashita from the old install from a the 32-bit wineserver but I’ve not tried it. Editing config files is something you only have to do once, so I did mine by hand. I would seriously recommend just doing that.

Addressing performance

The following section is true of Windows as well but it not documented well on the web so I will document it here.

Performance in FFXI is fine in 30FPS mode and even in 60FPS for most setups. However, in cities with a lot of people in them performance can suffer at times due to the single threaded nature of the game and using DX8 which WINE support for it crummy.

You can do a few things to make this a lot better:

Turn off shadow using /localsettings shadows off in game.
Run the game in 30FPS mode using /fps 2 if you can’t maintain 60FPS
Installing a wrapper to convert D3D8 calls into D3D9 or Vulkan calls.

There are many things that can do this:

dgVoodo
Ashita’s proxy
crosire

I use crosire’s proxy. You can install it by:

Downloading the latest DLL from the release page
Copying d3d8.dll it into the various spots:
1. ~/.wine-wingsxi-64/drive_c/WingsXI/Ashita/ffxi-bootmod
2. ~/.wine-wingsxi-64/drive_c/WingsXI/SquareEnix/FINAL FANTASY XI
3. ~/.wine-wingsxi-64/drive_c/WingsXI/SquareEnix/PlayOnlineViewer
Set winecfg to use the native override
1. Run WINEPREFIX=~/.wine-wingsxi-64 winecfg
2. Set the override in the library tab
Run the game per normal

The process is very similar for all the other proxies. If one does not work well for you, you can try others. Some users have had luck with layering proxies to translate into Vulkan (for lower CPU draw call overhead, such as dxvk) ultimately but I get a smooth frame rate with a single one, so I’ve not had the need. If you have steps for this, leave a comment and I can include them.

“Make FFXI beautiful 2021”

The following are a couple notes you may find useful. I will update them as time goes on into full out guides if there is interest:

https://www.reddit.com/r/linux_gaming/comments/b2hi3g/reshade_working_in_wine_43/
https://www.youtube.com/watch?v=ADjRYUdSyXA
https://nocturnalsouls.net/getting-setup/high-definition-dats/

The path of least resistance for WingsXI for HD mod replacements is DAT replacement directly. You can snapshot your prefix folder to get easy rollback – or use a brtfs snapshot if you want, that makes this trivial to roll back for updates (something WIndows users can’t do)

Did I miss something?

If you can’t get it working, you can find me on the Wings Discord or you can leave a comment below. I’ve tried to cover everything you would have to do and will flesh out things like HD Mods as I have time to type it out but in the meantime, contributions are welcome.

#windurst

Investigating how my local home router is resolving hostnames automatically

2020-04-20T21:17:00+00:00

I have a boring standard home router, though it’s a bit on the expensive side (Or it was when I got it many years back). It runs AsusWRT and it’s the RT-N66U from Asus.

It has a neat feature that automatically resolves hostnames on my local network that I never really thought about. I can do something like:

ping promathia

… and get a response back from a machine on my network that has the name promathia. A lot of home routers don’t actually work this way and most of them just forward all DNS requests to get their upstream provider. Of course, you could always add these entries to a host file but you would have to manage that static mapping everywhere and that’s kind of annoying.

I use this feature on a weekly basis but the thing that bothered me was that I could never set another DNS server in my router configuration page or this feature would break. I’d have been wanting to try out Cloudflare’s 1.1.1.1 or Google’s resolver because benchmarking showed they were faster in some cases. Unfortunately, this would break the hostname resolution over the entire network when I had tried it. That’s when I had realized I had no idea how any of it worked. Given the above change as well, it was clear this was not something like Netbios, Bonjour, or Avahi. Something else was going on.

Using various tools such as dig and nslookup, you could verify that the resolution was going to the default gateway to get the answers. The question remaining now was “how”?

I first actually looked to see if there was something in the DHCP specification to do this, since it seemed logical. I made a bunch of queries to Google including things like “DHCP add hostname” but I could not find anything about the specification doing this. I did find that many implementations did this though, such as Microsoft’s DHCP but I am running mostly Linux systems and I am certainly not running that on the router as far as I could tell. (Are there routers out there that run this?)

I got some other complicated answers like setting up DDNS and bind with some keys. I remember doing neither of those things and they looked a lot more complicated than some home router setup.

I supposed the first thing to figure out was to understand what was running on the router itself. Luckily, AsusWRT supports telenet (which is not super secure, but for this purpose is fine) and I could go into the machine and check it out. I knew of a few DNS packages, so I figured I would go check for those first (some of the answers on Superuser helped me round up some of the items that would be likely)

admin@RT-N66U:/tmp/home/root# ps | grep dns
  357 nobody    1212 S    dnsmasq --log-async
20032 admin     1620 S    grep dns

This reveals that we are running dnsmasq for our DNS server and DHCP. So, then the question left remaining was: how is this handled by dnsmasq? It turns out if you read the documentation you can find out this happens by default. You can even add a domain (--expand-hosts and friends) to have the hostnames expanded. However, I don’t have those settings form what I could tell. The question remaining became how does this work? I had some ideas but I would need to read some source code to know for sure.

It turns out the key and simplicity is in how dnsmasq does both DNS and DHCP.

Figuring out how `dnsmasq` would get the hostname and add it to DNS

First of all, let’s take a step back and figure out how a hostname would even get to dnsmasq in the first place. It is clear that some point you would need to send it and that cannot happen without some kind of network traffic or statically configuring. I certainly did not do the latter and that would defeat the purpose of an automatic process. So then, what else could do it? A multi-cast DNS system for sure, but I had none of those either.

The answer lies then in the process that happens on every boot of an Internet connected system and the thing that happens when you need to configure your NIC to get access (typically, you could be statically configuring but I only have a handful of those and those work too – more on that later) That would be DHCP.

You send your hostname as part of the DHCP process, specifically it’s client option 12 and allows you to send this as part of the process. By the way, client options are just pieces of metadata information that are sent as part of the process. I read quite a bit about it – but there are better explanations online. You’re better off reading those. Now, let’s remember that dnsmasq is a DHCP server and DNS server; that means it has the ability to read out these options. It can also serve DNS query responses.

The next part is understanding the fact that the DHCP protocol does not specify any sort of specification about DNS updates but like the Microsoft DHCP server, you would be free to implement something on top as long as it didn’t violate the specification. In this case, adding DNS entries based on the option 12 was looking like a good bet. From here, we had to look at some source code to figure out if that’s exactly how it worked though since information on this via the web is kind of vague and you can’t get an exact answer. I found a good mirror on Github here and checked it out that’s a bit older but would serve well for the purpose of viewing how things work.

When getting a lease from DHCP (from a client) you can find that the function lease_update_dns (link) is called and has something like this in the code:

	  if (lease->fqdn)
	    cache_add_dhcp_entry(lease->fqdn, prot, (struct all_addr *)&lease->addr, lease->expires);
	  
	  if (!option_bool(OPT_DHCP_FQDN) && lease->hostname)
	    cache_add_dhcp_entry(lease->hostname, prot, (struct all_addr *)&lease->addr, lease->expires);

So, in other words we get the hostname from the lease and add it to some cache. You can dig through the code some more and find that many of the functions are looking through the DHCP cache, such as check_for_local_domain (link). This gives you the answer you need – and that is that it is directly taking those leases that you generated and maintaining the cache from those. Then, the DNS server is using the items in that cache to resolve things.

The last important to thing to note is that – unless a client has configured otherwise – DHCP will provide a DNS server to the client to use. You don’t have to use this one to connect to the Internet, though most clients will. dnsmasq is a DNS server – so you when I am told to contact my default gateway for DNS, that would be dnsmasq under the hood. This then later forwards to the ISP upstream later.

That’s basically it; there are other interesting things in the implementation (such as how it implements round robin – which is through the cache get mutating the list internally) but they aren’t required to understand how my all in one home router was operating – it was running a local copy of dnsmasq that does most of this heavy lifting already.

Incidentally, for the things with a static IP (things like my NAS, virtual machine server and others) the reason their hostname resolution works is because I use DHCP reservations on the router instead of on a static IP client. If I had done it on a client, then the hostname would have never have been set as part of the DHCP request (no request would have been sent if I had manually configured the interface all together). Since the static mapping is setup server side, the client will still make a DHCP request per normal and dnsmasq will take care of giving it the same IP each time. Win win. There’s other advantages to doing it this way as well.

So, then why did changing the domain server break things? Simple: it was handing out a different DNS server from the local one as part of DHCP. You can verify this easily on Linux at least just looking at your resolv.conf briefly:

touma@setsuna:/tmp/test $ cat /etc/resolv.conf
# Generated by resolvconf
nameserver 192.168.1.1

vs.

touma@setsuna:/tmp/test $ cat /etc/resolv.conf
# Generated by resolvconf
nameserver 1.1.1.1

Mystery solved. Maybe someone will have the same question as me in the future and this will bring some answers to them a bit faster.

Installing OpenMediaVault without a network connection at time of install

2019-04-20T21:17:00+00:00

My current media server is running unRAID. While I have some gripes about it (that I won’t get into here) it has been running rock solid for quite some time and has serviced all my needs, especially with the Docker container support. I have a backup server in my home as well that I sometimes use for mirroring data, which was running Debian for a while. I wanted to replace it with a flash drive OS and remove the 2TB drive being used as a host OS drive, since I wanted to use it for something else.

Since it was mostly just a rsnapshot drone + HDD enclosure for the backup drives, I figured something simpler that could be installed to a flash drive with some persistence would do the trick.

The only issue is that my workbench (the backup server has no monitor or keyboard near it) where I service my machines (it has a small monitor + KB mouse combo) had no network cable there yet since I had moved. I didn’t really want to install that one at the time since it would be a pain and I was moving soon anyway. With no WiFi card and understanding it would hopefully pick up network config later when I plugged it back in headless later, I decided to try and install it without being connected to the network since it was supposed to be supported.

Well, that’s what the Internet claimed. I wouldn’t be writing here today though if it was so simple. For the record, life is easier if you just install while connected to a network. But it’s not too bad to fix the issues you come up with if you can’t, this post is hoping to get people on the same page about that.

Installing the system

This is pretty straight-forward. You can install it with the directions from the site. There are a few changes that I will list below…

The network detection is going to fail. This is to be expected. When it tries detecting networks and tries to do DHCP, you can just cancel it. It will warn you that nothing is configured. That’s OK.
The network sources for the Debian repos will not be configured properly. It will warn you to let you know. This will cause issues later, so we will take care of that later.
You won’t get hostname validation. So make sure the hostname is unique.

If you’re like me, you’re going getting through the install, reboot and login. Things will appear to be working well, locally with no network. You will then proceed to login via the prompt and verify things look good.

Then, you unplug from the bench and plug into where it’s going to live. You load up your workstation and try and hit the host name in your browser, for example like so (where the hostname I picked was lalafell) and you would get something like…

(This is pulled from the web – it would the hostname of your machine in the error)

And if you try and ping it, you would get…

[setsuna@setsuna ~]$ ping lalafell
ping: lalafell: Name or service not known

It turns out that it won’t auto detect the new network cable you plugged in and you have to set it up… ugh.

Resolving network connectivity

You can solve this using omv-firstaid. If you have a monitor and cable, you can just login and run that command and be good to go. Of course, that was a bit trickier without lugging a bunch of crap to my workstation. You can run it headless, just do the following:

Turn on your machine and wait a few minutes to make sure it’s good to go.
Type your username in and hit enter. Then type your password in and hit enter. Do this carefully to ensure you get it right, especially since you can’t see feedback (no monitor)
Type omv-firstaid and hit enter.
Hit “Enter” again after about 5 seconds. The default option is for network connections.
Hit “Enter” again. It’s selecting an interface.
Hit “Enter” again. It’ll begin configuring.
Wait 1 minute.

Then, you can ping the machine and you will get:

[setsuna@setsuna ~]$ ping lalafell
PING lalafell (lalafell) 56(84) bytes of data.
64 bytes from lalafell: icmp_seq=1 ttl=64 time=1.84 ms

You’re good to go. You can do the rest from your workstation.

I figured this out by installing in a VM and then just walking through it in the VM and then reproducing the steps inside of the machine. A bit reckless but for a machine with nothing on it yet this was fine. And it worked out.

Fix the broken sources from the install process

By default, the source list was not installed right given the current installer I used at the time of writing when done offline. You can follow this guide to to fix that. This is pretty easy to figure out when you know this is the problem, but how to figure out this is the issue?

You might get something like this…

Failed to execute command 'export PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin; export LANG=C; export DEBIAN_FRONTEND=noninteractive; apt-get --yes --allow-downgrades --allow-change-held-packages --fix-missing --allow-unauthenticated --reinstall install openmediavault-flashmemory 2>&1' with exit code '100': Reading package lists...

when trying to install the flash memory plugin like I did. You’re going to need to do the above if you get this. The following Github issue is also relevant. It took me a few queries to get this – so hopefully this saved you some time.

Checking everything else out

Be sure to run a system update since I had some updates to get. Not sure if this was due to the lack of network during install or what, but there were some.

Make sure to check network connectivity is as you wanted.

That’s all I had to do. Smooth overall; just had a couple road bumps. I hope this saves someone some time.

Replacing the fans on a Zotac GTX 970 (Dual Fan Model)

2019-03-21T21:17:00+00:00

My Zotac GTX 970 is 3.5 years old or so. Or was at the time of writing this post. About three weeks ago, the fans from it started making a whining noise. Some might call it almost a squeak. Or it sounded like perhaps it needed to be lubed. But life was busy and a little fan noise or “coil whine” never killed a card… immediately… right?

Well, they died about two weeks later. Other issues I had leading into those two weeks (for those coming from Google):

I had a faint chemical odour being emitted from my PC case. Turns out this was most likely from the fans
The fans would start and stop at random on the GPU. And then the whining would go away
You would hear “clicking” noises coming from the GPU
I would try and play games like Move or Die or Warframe with friends and the temperatures would get stupid high. I would then eventually lose display after a few minutes of gameplay. This often happened around the 100 degree mark. Nvidia cites around 105 being the max – so it sounds about right.

So, I had to figure out what was wrong. To make matters worse: it died completely with the 4th symptom during a few days I booked off at home to get some stuff done at home + with the intent at least to play some games!

Diagnosing the fans vs. the fan controller

I wasn’t sure if the fans were dead or not. I mean, they didn’t work but it could have been a dead fan controller. Luckily, it looks like the GPU Fan Controllers just use standard case fan headers so I tested a case fan from the computer in the video card header.

You can see the header above.

I gave the card some power and the fans began to turn on and rotate. Looks like we just had some dead fans.

After checking the specs of the fans and trying to find a comparable part, I settled on these. Amazon estimated it would take two days to get them to me.

Getting me through the days

I still wanted to play while I was waiting on my new fans, though. Of course, not in a reckless way but if I could get something working…

Believe it or not, this was enough to cool the card to play games. It would drop the idle temperatures from 75 to the mid 50s which is not great but was more than serviceable – and light to mid gaming worked well, if a bit toasty overall.

Of course, this was temporary. :D

Installing the new fans

Sorry, I don’t have packaging photos. Forgot to take some. Here’s the best one I have:

Now, onto the steps of what I did. I more or less followed this guide. However, I will outline some issues / things I had to do with mine that the video didn’t cover.

Remove the splitting connector

Mine came with this adapter holding the two together. You don’t need this since this GPU has two pins – and it’s just going to get in the way. You can remove it.

Clean out the dust

Mine had a ton. You might want to consider removing some of this since you’re here. I doubt this is strictly needed but I doubt it would hurt, either.

Make sure the fans spin right before assembling; make sure wires won’t touch the blades

It sounds obvious but I had this problem when I initially installed them. I hadn’t retained the wires under the clip perfectly so I had a small problem where the fan blade was hitting up against the wires sometimes.

Make sure the wires are down tight and you rotate the fan without trouble.

Pay attention to how the casing came off

I didn’t and it took a few minutes to figure out the orientation of putting it back on.

Done!

Success!

Who knows if it’s done 100% right? But it works well. I probably should give some other stuff in the case some TLC as well.

The idle temperatures are honestly a bit on the warmer side. 39 or so. Writing this post without much else going (just driving a 1440p display x2) we’re looking at:

I’ve seen better. But I had seen much worse just last week!

In some games, load causes it to go up to about 80, sometimes 85. That’s a bit toastier than I think would be ideal – but honestly, I plan on replacing the card in the next couple of years anyway so as long as it serves me right till then. ;)

Hope this helps someone!

Writing a fast (mostly) Binox solver

2019-03-07T00:00:00+00:00

We’re going to look at solving Binox puzzles in this post today. The game has also been known to go under the popular name Tic-tac Logic if you are familiar with that.

If you just want to try it, you can find that here.

What is the game?

Binox is a binary game with a board looking like this:

The objective is to fill the entire board with either “X” or “O” given some rule scheme – which varies in difficulty. The rules are as follows:

No row or column may have more than three characters that are the same in a row. For example, this is illegal:

… since there is 3 of the same character in a row. However, something like ` XX00XO` is legal

No row or column may have an unequal amount of characters. This means the above image would also be illegal as well.

Corollary: The puzzles width and height must be even in length.

Proof: Suppose some board exists where n, the length of the board is odd. Let x be the number of characters assigned “X” in a complete board. Let y be the number of characters assigned “O” in a complete board. Given the restriction x = y and x + y = n then consider the following cases,
- x is odd, y is odd. The result of two odd numbers added together is always even.
- x is even, y is even. The result of two even numbers added together is always even.
- Any other combination implies breaking the constraint x = y since a number cannot be even and odd at the same time.
Each row and column in the entire puzzle must be unique. This means you cannot repeat the same row or column anywhere.

With these restrictions, solving the puzzle becomes a bit more than just throwing random “X” and “O”s at the grid. Let’s look at an algorithm that can solve this.

The simplest thing that can solve this

I tried a few things that were “clever” and “complicated” such as generating permutations of rows and comparing to other possibilities for other rows. We won’t go over those – since it turns out the best solution I found is one of the simpler ones! You can read about general strategy on the web but the example I’m going to step through below is a general approach I came up with.

The algorithm is as follows:

Let Q be the set of indices on the grid that still need to be found (that is, they are neither “X” or “O” yet; often denoted as blanks). Initialize this to all possibilities.
Find an entry, any entry, that when you would place a X or 0 there (but not both! think… XOR) the following criteria is true:
1. The column and row associated with this entry does not contain a streak of 3
2. Modifying this entry does not make it impossible in the future for other to-be-determined entries in that row or column to respect the rules. The main one to worry about is the quota surrounding equality. If n is the row or column length, then by placing on this entry a specific token, the number of “X”s or “O”s must not exceed n/2 since it would then be impossible for other to meet the criteria.
Remove this entry from the set, since it is now placed
Continue this process until all entries are placed
If you at some point cannot find such an entry in the set, the puzzle has no solution.

Let’s break down why this is the case.

Why XOR for choices? This one is pretty simple – if you have two choices and you can only make one of them without breaking the rules at any given step, then that choice is obvious. It’s the ideal scenario.
Why the count guard? This is also pretty simple – if you let a row or column exceed the length divided by two – the others cannot meet the criteria of being a valid “slice” as explained above
Wouldn’t we run out of things with only a single choice? As you place more entries, other entries become more obvious as to what they must be since the decisions of other entries from the set are now dictating their possibilities. Each choice made this way only enables more information for future moves – preventing you from ever barring off future choices for other moves.

Automating it all

Of course, this is easy to describe so we can encode it inside of an application. I won’t go over all the implementation details but you can describe the entire main algorithm like so, in Javascript:

class PuzzleSolutionSolver {
  solve(board) {
    let activeBoardPointer = board;
    const pointsToSolveFor = activeBoardPointer.getIndiciePairsThatAreNotFilledIn();
    let iterationsBeforeGivingUp = pointsToSolveFor.length;
    const originalIterationsBeforeGivingUp = iterationsBeforeGivingUp;

    while (pointsToSolveFor.length > 0) {
      const pointToSolveFor = pointsToSolveFor.shift();
      const theoryWithOne = this._isTheorySafe(activeBoardPointer, pointToSolveFor, '1');
      const theoryWithZero = this._isTheorySafe(activeBoardPointer, pointToSolveFor, '0');

      if (theoryWithOne ^ theoryWithZero) {
        const tokenToProceedWith = theoryWithOne ? '1' : '0';
        activeBoardPointer = activeBoardPointer.withPointChangedTo(
          pointToSolveFor,
          tokenToProceedWith
        );

        // Let it cycle around the list a few times. This could be dangerous in some cases since
        // you might let things spin for a long time but this should only happen on VERY large boards. So it's not a huge deal
        iterationsBeforeGivingUp = originalIterationsBeforeGivingUp;
      } else {
        // Both were doable, so we cannot make a choice since we are unsured yet
        // so we place it back on the queue in hopes that something other mutation
        pointsToSolveFor.push(pointToSolveFor);

        iterationsBeforeGivingUp--;
      }

      if (iterationsBeforeGivingUp === 0) {
        throw new Error(
          'Reached the max iteration count. Could not converge on a solution. Giving up...'
        );
      }
    }

    return activeBoardPointer;
  }

  // helper functions
  _isTheorySafe(board, point, token) {
    board = board.withPointChangedTo(point, token);
    const slices = board.getSlicesForPoint(point);

    const horizontalSlice = slices[0];
    const verticalSlice = slices[1];

    const isMaintainingSliceQuotaIntegrity =
      !horizontalSlice.isExceedingQuota() && !verticalSlice.isExceedingQuota();

    const isMaintainingRunIntegrity =
      horizontalSlice.isNotSlidingWindowLargerThanThree() &&
      verticalSlice.isNotSlidingWindowLargerThanThree();

    return isMaintainingRunIntegrity && isMaintainingSliceQuotaIntegrity;
  }
}

You could even cut some of this down a bit as well. You can dig into the actual implementation of some of these rules in the full implementation on my Github. It’s a short read – the entire thing (with all the tests, which is most of the code) clocks in at less than 1000 lines of code. It should be easy to digest.

Let’s first examine the puzzle posed above.

X	-	-	-	-	-
-	O	-	-	-	X
-	-	X	X	-	-
-	-	X	-	-	O
-	-	-	-	X	-
-	X	X	-	-	-

Using the above rules, we can then run the program and find:

  console.log src/PuzzleSolutionSolver.js:26
    Going to proceed to change 0 | 2 with 0

  console.log src/PuzzleSolutionSolver.js:26
    Going to proceed to change 1 | 2 with 0

  console.log src/PuzzleSolutionSolver.js:26
    Going to proceed to change 1 | 3 with X

  console.log src/PuzzleSolutionSolver.js:26
    Going to proceed to change 2 | 1 with 0

...

If we step through these picks, we can see the rationale.

If you were to add a “X” in that spot, there would be more X’s than allowed (since you could never possibly 4 O’s there). Therefore, the only logical placement is to put a “O” there.
The same rationale for a row below – you can fill it in immediately.
This has to be an “X” since otherwise, you would have 3 “O”s in a row. This is because the item to the left was filled in from the previous iteration, as we knew what it had to be
You can’t have a streak of 3 – no prior info is needed since those “X”s are static and already placed.

… and there are more iterations. The number of choices grows smaller with each pick so the queue grows smaller until it reaches zero. There are more boards it has been tested on; you can check them out in the tests. I have some plans to add more tests from the PDFs provided. However, I tried to do a couple “cURLS” to get the PDFs and stopped being able to access the author’s site from home. Huh.

I’ve probably been blocked. Not sure why.

Further Optimization

Some heuristics could be developed to pick the points to prevent cycling through the list to find a valid one. For example, in the case of “exceeding quota” in a specific column and “forcing” a single token such as in example #1 and #2 above, we could have easily filled in the entire column. The solver began to do this but once it reached another point it could solve it just did that one first instead. The current implementation just uses a dequeue and pushes things to the back if i t can’t find something it can place immediately.

There’s probably other speed improvements to be made too: like not writing this in Javascript. But the solver runs so fast, already even with larger boards that I’m not that motivated to do much else. It was a fun afternoon project – now it’s time to say goodbye for now.

Thanks for reading.

Swapping from macOS and Linux, work to home... things I miss from each platform

2019-03-07T00:00:00+00:00

I work as a Software Developer doing mostly front-end work but I do some back-end work as well. Where I work right now, we mostly develop using Macbook. Up until recently, there were a few people using some Linux machines to get their work done but the standard issue machine is a Macbook. Even if I could swap to something else, I wouldn’t and here’s why:

I want to be on the same platform as my peers so when there is a problem I can receive help in the best way and give them the best help as well.
Development tools are more or less platform agnostic but rarely is anything free. If we were split 50/50, that would be one thing. But being one of the only outliers? That’s painful
Hardware and IT support is going to be a lot better with a standard stock macOS install
Macbook hardware has bad support for it unless you are going for older hardware – and that’s just a no no when you are trying to make the make use of everything you have.

That being said, it’s not like I like macOS. Even though some people are reporting software quality problems more and more (and I have noticed some things) there are a lot of things I like about the macOS environment. I figured I’d name a few here and hope that by putting it down, I even might be able to improve my workspace in both.

Things I have in my Arch Linux setup I wish I had there

Tiling Window Manager

I have a tiling window manager here. I don’t have to use a clunky terminal with it’s own keybindings like iTerm on macOS. I use i3 and I won’t try and sell you on the benefits, but I will say is I never have to reach for my mouse to resize random windows and everything is just fullscreen and driven by a keyboard by default. For some non-work workflows, this is a bit clunky (looking at you – Steam) so I have GNOME for some other stuff. It’s not my favourite but for getting work done, I can’t beat the i3 setup.

There’s a nice guide for getting started here on that.

My software just upgrades without much trouble

I run a Pacman upgrade every once in a while and then reboot when I need to. Nothing nags me. Nothing is in the way of trying to prevent me from doing what I need to. macOS has brew and the AppStore but it’s not enough. There is a bunch of stuff that just does not work and hangs. I was a victim of something like this earlier. And there are more. brew is also moving towards being a binary package manager where it can. This is not so bad, but this can be a real hassle sometimes. For example, consider the time I had to recompile something w/ SSLv3 for something I had to support, Arch made this a joke to do but macOS… well that would have been an ordeal.

brew loves to choke sometimes as well on random things. pacman, well I’ve had issues but you can usually find the solutions on the front page since they’re expected.

Native Docker

Holy crap. This is such a big deal and I didn’t even realize why until I really figured it out. People have been complaining about it for a bit but until I tried using it on macOS for other stuff, I didn’t realize what a difference there was. That combined with the fact that I can poke things inside of Linux land without going through a ton of hoops. That’s a miracle. I get performance without running on top of a giant hypervisor that’s sluggish and clunky.

That’s not to say it’s unusable – but when I learned Docker from scratch, I did it on the Linux host for a reason. There was no competition.

Nothing gets in my way

No iCloud pop-ups. No notifications to reboot. No jarring software prompts. No force Windows Updates. I update on the same schedule weekly, on my own time and I run it manually. If there is something urgent, I do it again. I do my server @ the same time and control the schedule so I can control downtime as needed.

I can use the same tools to adminster my server as a workstation

This is self explanatory. I have a home server and I love being able to ssh and use all the things I am familiar with to modify it. macOS has a ton of these tools as well but often they don’t work quite the same.

Being able to choose when to adopt new things

USB-C is awesome.

I am also not ready to move everything over to it. I think Apple thinks I’m ready, though. Considering they removed my HDMI. This has been covered a ton by the press but seriously: it’s annoying.

Suffice to say, I have some older hardware and I don’t mind. It’s mature and most things are working now.

Things I miss coming home from my macOS machine

Performance

There’s not inherently slow about Linux. In a lot of stuff, it’s even faster and works better on a raw level… for example if we’re talking Vulkan support, there’s no contest. And for other stuff like some parallel computing, some of the schedulers are on another level. Or some of the machine learning stuff going on. However, general compute performance is slower. i3 is just laggier sometimes on a similar specced machine. I get random stalls sometimes as things hiccup. The desktop experience just is not as polished. It’s not like things are unusable – not by a kilometer. However, things are just generally more sluggish.

Software Support for “some mainstream things”

Office? Google Suite has replaced a lot of the need for that stuff for me. Typing documents is a basic need for me and I’m not strapped for having a ton formatting options.

Photoshop?

The experience is better here. This is still a sore spot. I wish I had these. I might have to get a VM passthrough going or something to get better results over here in Linux land. I’m using Pinta for basic editing over here and I’ve tried GIMP but it’s not the same. Once you’ve learned Photoshop skills, it’s pretty hard to retrain for something as simple as “OS loyalty”. I am sure the job can get done in other spots but I don’t know want to relearn when it’s not the primary objective of my workstation.

Hardware Support

People keep saying that Linux desktop is plug and play. We either live in a different world or some of them are more optimistic than I. I’ve plugged a lot of things in and they don’t work as well as I would like. I still can’t find a nice way to adjust scrolling speed in Linux for Chrome and other apps to be faster to my liking. xpad and xboxdrv often have competing goals and clobber each other. My options for mice and keyboards are a bit more limited; I pick according to what is going to work in my system.

##Searching for a problem? You can narrow it down – it’s a macOS thing probably

Running an Arch Linux distro + a ton of other stuff that is non-standard, I get a shit ton of problems I have to sort out myself. Origin in WINE not running? It does not play nice with tiling window managers. Switch to GNOME for now. Steam hanging for some random reason? Break out the debugger.

When looking for macOS stuff, there’s usually only a handful of hits that apply to most systems. No guesswork for involved. As I get better at debugging, I can pinpoint problems faster but there’s less annoyance in this case.

Installing Final Fantasy XI and dual boxing on the Nasomi Private Server (Linux)

2018-10-27T00:00:00+00:00

I recently wanted to give the Nasomi XI Private Server a try as I had played a lot of Final Fantasy XI when I was younger and was feeling a bit nostalgic. Since the population is a bit smaller than it was then but still very respectable (1300 players during prime time) there are quite a few dual-boxers. These are people who play more than one character at the same time. For those not familiar, Final Fantasy XI is a game that is very heavy on co-operation and a lot of tasks require at least two players and often a lot more. Most people do this dual boxing because..

It is allowed here and encouraged where it was not on the retail servers.
It makes a lot more content more accessible to you without needing to rely on others for EVERYTHING.

The only problem was my platform of choice these days is a Linux x86 PC where as I had played on a PS2 during the early 2000s era of Final Fantasy XI. Nasomi is not supported on the PS2 – so that just left getting the game running on Linux or running it in Windows.

**Spoilers: ** It runs inside of Linux very nicely – we just need to put in a bit of effort to do so.

The Guide

Let’s get started by just downloading the official installer for Windows from here. You can pick whatever method you want – there is no difference in the contents.

Next, you’re going to want to unzip the package somewhere. You can use a GUI tool or the unzip command for this but it does not matter much. if you don’t have WINE or have no idea what it is, you can read about it more here. We’re going to make use of it – there’s a trick, though… the official wiki recommends some weird hacks to get it running and we will still need to do this but it applies them to the installed version of wine on your machine since the wiki suggests butchering your system install of WINE.

There’s a couple snags I ran into and I will list them here:

It asks you to override a DLL to play – and this is going to get overwritten every time you have to an update, or at least it would be on Arch Linux which is my OS of choice for playing games.
It does not account for the fact that there is an open bug in newer versions of WINE that prevent Final Fantasy XI from being playable. There are a few things that can be wrong – but if you have a Nvidia card like me you are likely to run into this at the time of writing if you are running a rolling release. This might not be everyone but there are enough of us that someone is going to have a bad time.

It does mention Ashita won’t work but it does not mention how to get dual boxing support. I’ve had not found a way to get Ashita v2 working yet reliably but there are ways of getting dual-boxing working.

Thus, there are two things we can do to get around this and have a much better experience.

We will grab a local copy of WINE and use this for our FFXI – and do the DLLs overrides in a local copy
We will then be able to apply the patch as well on top for this, allowing us to apply any sort of bug fixes we need to get the same running (see: #2)

Let’s get started. I’ll assume you have copied the installer into a folder, such as ~/nasomi and we are working inside of there.

Grab the WINE source code (or wine-staging) if you prefer that and clone it with
```
git clone https://github.com/wine-mirror/wine
```
Go into the source directory (cd wine) and then download the patch from before that we need, apply it and commit it. **If you are using Wine 3.19 or a later trunk, you may need to use this path instead https://bugs.winehq.org/attachment.cgi?id=62646&action=diff&context=patch&collapsed=&headers=1&format=raw as the below patch may not apply clean
```
wget "https://bugs.winehq.org/attachment.cgi?id=62460&action=diff&context=patch&collapsed=&headers=1&format=raw" -O ffxi.patch
git apply ffxi.patch
git commit -am "Applied FFXI patch for NVIDIA cards"
```
We’re going to compile WINE. Inside the wine directory (you can check you are in the right directory by running git status and making sure it reports as found) run…
```
./configure
make -j 2 # You can make this the number of processors you have
```
As it’s compiling, move onto the next step.
Now, grab the DLL override. The wiki has a copy of it here however, I will provide a copy of the same DLL here as well on my own server in case it ever goes offline, since the link is to another player’s server. For example, you can just do this to get the file (or use the URL from my server if that is ever needed):
```
wget http://nabutso.com:4438/FFXI/imm32.dll.so 
```
We need to replace the DLL now inside of the WINE folder. We can do this with a simple copy command:
```
cp imm32.dll.so dlls/imm32/imm32.dll.so
```
from inside the WINE directory. This should give you the ability to run without problems now.
We are ready to begin installing the game. There’s a setup EXE or batch file (depends on the version of the game) and you can run it by doing the following:
```
WINEPREFIX=~/ffxi-custom-first WINEARCH=win32 ./wine/wine "./installer.exe"
```
A couple things to note above:
- You should replace the WINEPREFIX without whatever you want – in my case, I have two separate prefixes for both players so I can have completely different settings and I have the diskspace to support it. And you should change the installer EXE to whatever the name of the setup installer is of the version at the time of running.
- FFXI is a 32 bit game so we are going to be running in 32 bit mode.
You should get the installer with some music. Just install with the defaults and move on. This took about 5 minutes for me on an SSD – time will vary.
The game is installed – so it’s time to run it. By default, the settings are a bit weird so it might be worth just configuring them quickly. You can do by running the following command:

WINEPREFIX=~/ffxi-custom WINEARCH=win32 ./wine/wine "/home/touma/ffxi-custom/drive_c/Program Files/NasomiXI/SquareEnix/FINAL FANTASY XI/ToolsUS/FINAL FANTASY XI Config.exe" 

… and you can change some things like the resolution, bump settings, texture resolution etc. When are you done, save your changes and we can move onto playing the game.

You can run the following command to play:

WINEPREFIX=~/ffxi-custom WINEARCH=win32 ./wine/wine "/home/touma/ffxi-custom/drive_c/Program Files/NasomiXI/SquareEnix/Ashita/ffxi-bootmod/boot.exe"

and then you are good to go. The game should launch. If you want to dual box, you would do the same thing as you just did but with another copy of WINE – checked out from the source code again. This will give you two WINESERVERs and prevent the crashing on startup you would get from trying to launch two clients at once under normal circumstances.

None of this will have Ashita running, though since that does not work in v2 with Nasomi

Other things you may want to know

FFXI benefits from the esync patches – so if your CPU is a bit on the weaker side like mine you may want to get a WINE Build with them included.
Create launch scripts. It sounds obvious but don’t forget you can create simple bash scripts to wrap some of the commands for launching the game. I have a script for each of my characters so I can run whatever one I want to login to.
Periodically update WINE. There are performance benefits you can get from updating your copy (git pull) from the mirror from time to time. You may want to do this from time to time.
The patch from above may stop working at some point – you may need to reapply it from time to time with a new one. Or it might not be needed at all. You can pick a stable WINE tag if you want to avoid problems. For example, if you are using 3.19 as a base you will notice there is a different patch here for that.
This has only been tested on the below configuration, I have not tested on anything else:

touma@setsuna:~ $ neofetch
                   -`                    touma@setsuna 
                  .o+`                   ------------- 
                 `ooo/                   OS: Arch Linux x86_64 
                `+oooo:                  Kernel: 4.18.16-arch1-1-ARCH 
               `+oooooo:                 Uptime: 10 days, 6 hours, 34 minutes 
               -+oooooo+:                Packages: 1393 (pacman), 4 (dpkg) 
             `/:-:++oooo+:               Shell: zsh 5.6.2 
            `/++++/+++++++:              Resolution: 2560x1440 
           `/++++++++++++++:             DE: GNOME 3.30.1 
          `/+++ooooooooooooo/`           WM: i3 
         ./ooosssso++osssssso+`          Theme: Paper [GTK2/3] 
        .oossssso-````/ossssss+`         Icons: Numix [GTK2/3] 
       -osssssso.      :ssssssso.        Terminal: terminator 
      :osssssss/        osssso+++.       CPU: Intel i5-2500 (4) @ 3.700GHz 
     /ossssssss/        +ssssooo/-       GPU: NVIDIA GeForce GTX 970 
   `/ossssso+/:-        -:/+osssso+-     Memory: 7723MiB / 16002MiB 
  `+sso+:-`                 `.-/+oso:
 `++:.                           `-/+/                           
 .`                                 `/

The rest you can figure out from the Nasomi page and is standard stuff. This ought to work for any server or set of people who cannot run Ashita or Windower as well, such as macOS users but your steps may vary.

Using Huginn for profit and fun; finding price errors on the Internet

2018-04-26T00:00:00+00:00

Those are not faked (read: photoshopped) – that is how much I paid for those items sometime in 2014. Those are really good deals for those items, especially in Canadian Dollars. Most of those retailed for about over $100 CAD over here.

These are all known as price error items as the merchant did not intend to sell them for this much. However, sometimes logistics software screws up and there’s a community of people who want to scoop up these things.

They actually come up quite a bit:

https://www.theguardian.com/money/2014/dec/14/amazon-glitch-prices-penny-repricerexpress
https://www.google.ca/search?ei=su3kWpLWFKTCjwTJ8JuoBg&q=price+error+redflagdeals&oq=price+error+redflagdeals&gs_l=psy-ab.3…2939.4442.0.4541.0.0.0.0.0.0.0.0..0.0….0…1c.1.64.psy-ab..0.0.0….0.6Tp0J06Cgms
https://www.telegraph.co.uk/finance/personalfinance/money-saving-tips/11288449/I-spot-and-exploit-pricing-errors-for-a-living.html

.. at least a few times a week in most cases. Of course, not all items are of interest.

The process of ordering something that was priced in error

To get in on this, you need to…

Notice and find out about the error. Usually communities like RedFlagDeals or SlickDeals have sub-forums where people tell each other about these things. There are also services that notify people (more on this later)
You need to order and act quick.

You need a quick reaction time as these are often fixed within an hour, sometimes even less than that. There’s other factors as well that affect the quality. In Canada, if you can do in an in-store reservation at that price and then go in to get it the same day – you have better odds than having it shipped, since orders are sometimes canceled on-line but fulfillment rates are high in store.

Since short of reading every thread frequently is tedious, we need automation. Some tools are available already off the shelf to do this which we will dive into first to motivate building something new.

The manual process of finding some of these

Generally, there are a few ways of doing this:

Subscribe to items on LightningDrops and have them notify you.
Check communities such as Redflagdeals and then monitor threads
Keep track of certain items on the individual vendor sites

No single strategy is going to get you 100% of the way there and some services provide automated ways of doing this already for you to increase your chances. I was a frequent user of method #2 for a while but it takes up a lot of time, since often they are the best sources of some of the more obscure deals and involve reading pretty much every thread. Wouldn’t it be nice if we could just get a notification when someone posts something of interest?

Slickdeals has something to do this and so does RedFlagDeals.

I’ve used them. They both have some problems, though:

They’re a bit slow – it can sometimes take up to 15 minutes to get the e-mail and that’s a bit too slow in some cases (despite saying “instant” on their pages for “Notification Speed”)
They deliver through their own mechanism, which means e-mail in most cases and perhaps not what we want. Or their application. Or IFTT’s application.
They don’t support complicated queries or Regular Expressions, which as a developer, I know is pretty important in some cases. ;) We’ll see some example thread titles later that show why this is the case, but image things like HOOOOT or HOTTT!! which are things that won’t be picked up by a basic string matches but something humans might type, with exaggerated letters.

We know the inflexibility problem in this case is because the “data source” and the notification system (and querying of the data) are coupled to each other. We have no way of querying the data ourselves, the communities themselves provide one-stop shops.

Our solution? To decouple the stream of data from the notifications and allow us to create more customized ways of filtering on the data. IFTT is an inspiration for Huginn and kind of solves this problem. IFTT actually has a widget for Slickdeals here but it’s not very customizable. It’s likely you could create your own using IFTT with some work but you won’t be able to get #3 for free that easily and you will have trouble getting 2. IFTT is quite fast, thought.

If you don’t have your own infrastructure, it’s not a bad way to go. But let’s look at alternatives using Huginn to do this. I’m not going to cover the installation, so check out their install guide (I recommend Docker if you can use iti) or my install guide using unRAID if you want to follow along. Otherwise, time to get started.

If you don’t know what Huginn is you can read more about it here. It’s essentially a system that allows you to pull data from various sources, aggregate it into events, query it, and then act on it. That sort of vague, right? But it’s kind of exactly what we need…

Setting up a plan

This is our general plan of attack:

Monitor the RSS feed of a particular forum and collect new posts only since our last check
Aggregate them all into a shared pool
Run some regular expressions against each entry to filter down to the ones we’re interested in as they come in
Dispatch to a notification daemon and have it notify me in real time if there’s a match

Our notification stack is going to be Pushbullet for a few reasons:

I already have an account
It can mirror between phone and desktop, so I’m going to get it quick no matter where I am.
It supports link types with one click opening of links.
Huginn has a native agent for it installed by default.

We’ll pull our data from boring Atom RSS feeds – reliable, updated fairly quick and standardized. Huginn also supports web scraping for more speed (since RSS feeds can sometimes lag behind) and for sites that do not provide federated access to their data. We won’t touch this here but it could be pretty easy to implement and swap into.

And in the end, our goal is to look something like this:

in response to something like this:

That’s the plan. Let’s figure out how to execute.

A motivation for Huginn

This is all just some basic HTTP requests and data extraction! I can write something working in a couple hours, I don’t need Huginn!

Some Developer before attempting this, probably. (2018)

If you had this reaction, then read the requirements below. Explicit and implicit requirements are dissected and include what complications might arise:

You need to make sure you only pull new posts – that means you need some kind of in-memory storage at the very minimum and need to write some code to track the ID and only serve up in new ones into a pool. If you want fault tolerance (if your app crashes, you would lose all events) then you might even need something like Redis running in persistent mode. Things just got a bit hairier.
**If you want to do more than one type of alert, say, SlickDeals afterwards ** … then you need generalized event types. If you need more than one type of notification backend, you need to implement that.
You need to implement some kind of reliable scheduler, visualizer if you want them This can be cron but now you have to manage cron jobs.
You need logging of events for debugging.
You need to integrate with a notification service – this one is bit easier. Most languages have something to do this. For example. here’s one for Node. But you also have to write our own fault tolerance.
If you want to use this data for other things, you need to make sure you create a generalized abstraction over top to query the data for other purposes.

I mean, it’s not that you can’t write all that but it’s a matter of if you should or need to. Setting this all up can easily take a few hours if you’ve never done it before and even an experienced developer is going to take a bit – and then you have to deploy it. So, that means some kind of deployment technology needed.

Maybe read on first and then come back and see if this is still appealing.

Getting the data into our stream

Huginn uses things called events to pool data into a queue that can be processed by other agents. As a first step, let’s take a look at using this RSS Feed to get out information.

We’ll create a new RSS Agent and set it up like so:

A few explanations in order:

Scenario: This is just a way of grouping agents together in Huginn. We’re using the default one for now but it would be a good idea to group them later on so you know how they all interact with each other.
Keep events: We’ve picked 3 days but even less is fine. We’re just using this for triggers which are near real-time, remember. However, if we had other plans for this data more would be in order.
**Schedule **: 1 minute might be a bit eager, but remember we’d rather generate a bit of extra traffic than miss a deal.
Expected days: You can read more on the official docs – but this is how Huginn will notify us of broken agents. i.e: we expect data to come in here at least somewhat frequent or something is not right.

With that saved, go ahead and run it and you should generate some activities in the events stream:

Perfect, we have something that now runs and gets the data and serves it up for us. Let’s get a filter in front of that now.

Filtering the data using regular expressions

We now have our own copy of the data using RSS. It could be through scraping the HTML using the scraper agent, too, though. It does not matter – what does is that we now have events. We don’t want to send every Hot Deal to our devices since that’s just going to result in too many posts. Probably at least three dozen per day, minimum and we don’t want to have to keep checking these.

First, let’s do a little bit of research using Google to figure out what we can expect.

I’ll save you the trouble if you don’t want to search yourself through pages of data and let you know what I came up with:

The title often contains: price error, price mistake or just the word error.
The title often contains the word dead if the deal is dead. Or expired
People sometimes use “HOT” as well when a really good deal is up – sometimes this is not a price error at first but people discover it is later.
People sometimes exaggerate these words as hinted about before – so just searching for “HOT” as a literal is is likely to miss things, since it may be !!HOOOOT!!! or some variant.

Let’s get a corpus of data going and then try writing some Regular Expressions to match the examples we want. Using a basic script, we can get some of it out:

const cheerio = require('cheerio')
const fs = require('fs')

const data = fs.readFileSync('./file.xml');
const $ = cheerio.load(data, {
    xmlMode: false
})


// Get all the title tags
const $titles = $('title')
console.log($titles.first().text());
$titles.each(function (index, titleNode) {
    const $titleNode = $(this);
    console.log($titleNode.text());
})

(There’s probably a way to do this in Shell as well – but this got me the results in just a few minutes, so good enough)

Our next step is then to write a regular expression that can match as many of the items in the corpus as we can without tripping too many false negatives. Consider the following regular expression below as an example:

price.(error|mistake)|(h.*o.*t(!))|error|hot/gmi

It covers a few of the cases here without triggering the false positives on the items above. There’s a few items in the list I might personally be interested in (the pre-order for that video card…) but let’s keep our task focused so that we don’t have scope creep.

With this basic regular expression (in practice, I use a slightly more complicated one but this is pretty good) we now have a good basis to filter against. In practice, we could tweak this is a bit more but it gets us a lot of value with low effort (this cruddy regular expression only took 4 minutes to write – yay!)

Creating a new Trigger Agent with the following configuration would give us what we want:

(Trigger Agents allow us to filter data and then generate other events based on these. You can think of the raw data as the base fact and then this is a refinement on those facts. Specifically, it generates additional events we are interested in real time information for. We could still use the raw data for other things since it does not transform it but rather makes a new event for consumption – for example, a daily digest of all the Hot Deals for the day at the end of a day.)

You can use the “Dry Run” functionality to inject some test data to verify it works – but our corpus data should at least give us some confidence. If you wanted to take this to another level, you could have a small toy app that had two test fixtures – valid and invalid and have a script that runs the regex against each test case when modifying it to make sure you don’t have any that are invalid. For now, my regular expression has been running so I have no need to modify it…

Time to get that info into the cloud.

Notifying yourself when it happens

The last part is the easiest part – we just need to get those events we generated into our Pushbullet devices. We can go ahead and create Pushbullet Agent and make sure you get an API key from them.

You can configure it similar to below:

.. and you do need to fill in your API key. After that, you should have everything you need. Since this is a simple dumb relay system, it’s also why I’m comfortable introducing a cloud dependency here. It could easily be swapped for anything else since it lives at the end of the flow.

Profit???

That’s it! With that, we should now have real-time updates to our devices like we showed above. It all works but you don’t have to take my word for it – you can create a manual agent and then trigger some events to watch the entire thing work together. You could also try finding a deal and posting it as well and that would also trigger your new test.

With this, we now have completely automated the process of getting notified of price errors and other interesting deals. I’ve been running a variant of this for a while and scooped up at least a few things such as Google Home Mini’s for $30 to due a coupon error. The original thread title used to say HOT but it was modified since the deal has been expired. It’s not perfect, so I often actually combine this with digests at the end of the day to comb through “not error” things but things I still may be interested in.

Happy savings & hope you learned something!

Got a better idea for a more comprehensive regular expression? Are there better platforms? Just want to say hi? Let me know in the comments.

Setting up Huginn on unRAID

2018-04-26T00:00:00+00:00

I recently became intrigued with using Huggin for a few automation tasks I had been doing manually up to this point. However, it looks like nobody to this point had bothered to really post how to set this up with unRAID. It was not too tricky since you can use Docker with unRAID but there were a few things that slowed me down.

I’m documenting them here for anyone else who wants to do this, you can hopefully be saved some time.

Get an unRAID Huginn template

As the first step, you’re going to to need a template to do this since unRAID makes use of Docker templates for setting up things quickly. I created one you can use right here if you don’t want to create your own. That template assumes you have a appdata mount point already existed for your other applications. If you don’t have one already (this is your first docker app in unRAID) then you can read this to get more info.

If you are using my template, you can follow the below directions:

Go to the “Docker” tab in the unRAID interface
In the “Template Repositories” at the bottom of the page, add my repository which is at https://github.com/hilts-vaughan/unraid-docker-templates/tree/master/unraid-hilts-vaughan-templates and then hit “Save”
Click “Add Container” which is a button on the Docker Page
Select my template from the dropdown above, to be prompted with something like this:

You can accept the defaults and let it pull down – which should give you a working instance once it comes up at (by default) tower:3000/
Login with the default credentials of admin and password – make sure you have saved these.

Getting email working

I had to do a couple things in unRAID land to get this working properly as it was not working out of the box.

Configuring the user in Huginn

Once signed in, head to http://tower:3000/users/edit (you can change the tower if you changed the name of your unRAID box)
You should see no e-mail in the box – so you can configure one, like so:

Then, you can hit “Update”

Configuring unRAID with environment variables

If you plan on using GMail for the e-mail in Huginn, you may want to read the below section first. To configure environment variables, and email specifically for Huginn, you can use the following steps at the time of publication (tested for GMail):

Go to your “Docker” tab
Click edit on the image
Using “Add another Path, Port, Variable or Device” at the bottom of the page, configure something like this:

If you want to configure anything else at this point, you can refer to this environment variable documentation to do so.

Configuring GMail

If you are using GMail, there are additional steps go through to generate something secure and that is usable for Huginn. I was getting some interesting errors like:

huginn rbuf fill

… and authentication errors among other things. In practice, if you plan on using this for any amount of time I would advise you sign up for something like Amazon SES or something with a very basic (or free) SMTP server you can use and then have them send your emails for you. This will give you a layer of insulation against having your e-mail credentials served.

Otherwise, do this:

Enable two factor authentication for your GMail account. This is not strictly needed but this guide will use it since it allows you to generate app specific passwords which are needed to keep your unRAID box from knowing your actual credentials.
Generate an app password
Use this password in the above steps, where it says “PASSWORD” use this disposable password. It’s slightly more secure and lets you bypass some of the “Use less secure apps” mode that some articles talk about. Do not use this mode if you can help it. It’s insecure, as the name implies.

Testing it

To test this, I would just run a couple of the default agents and see if it works. The easiest way to do this is:

Run the iTunes Trailer Source
Run the Afternoon Digest agent
Check if your e-mail was sent successfully

That should be it. I hope this saves you a couple hours. I wish someone had told me about a couple of these things when I started (and that a unRAID docker template existed!)

Why are we still not enforcing our invariants in 2018?

2018-04-13T03:11:00+00:00

This post is a work in progress; it’s mostly a scratchpad for my thoughts

Motivation & why you should read on

Before you read much more, here’s a summary of what you can expect:

At least in some circles, we need to figure out how to enforce the invariants of our code better
Often, libraries (even very popular ones) do this very badly. We can do better – and it’s not that hard to do. It might just require some education on the tools available to us.
We’re going to look at some real case studies of where poorly enforced invariants cost me real time. That’s real money for someone.
We’re going to look at why you might as well document assumptions and decisions as hard enforced constraints

Preface

It’s no secret that I work a lot in JavaScript. If you look at my GitHub profile, you will find no shortage of projects there that I’ve written in JavaScript and I’ve been writing it professionally for a number of years. Let that be a preface – I have worked in .NET, Java and other languages as well but I want it to be known that this very well might be a rant against the state of Javascript. I definitely know some languages handle this better

I…

Have written many commercial enterprise applications and features in it
Used a ton of libraries, frameworks, APIs and the like that interact with the ecosystem as whole in it
… and I’m a huge shill for static type systems since I come from a .NET background before delving into the web

In other words, I’ve worked with other peoples code a lot. I’ve invoked a lot of API(s). I’ve poked and invoked tons of external systems. I’ve played the integration system.

More importantly, though:

I’ve also written a lot of code in .NET (quite a bit of it, I wrote a few games, game servers, and a couple non-trivial back-end web applications in .NET Core as well)
I’ve been on the end of being frustrated by poor API(s) that suck a bunch of my time as I try and debug what I did wrong (or the library in some cases is not documented properly)

The last point is the one I want to focus on today since it’s probably wasted hundreds of hours over my life. Specifically, I want to talk about poor type systems, invariants, why we don’t enforce them, why we should, and how we can do better even in languages such as JavaScript that don’t support them.

Let’s get started.

Introduction

If you already know about invariants, I implore you to skip this. It’s boring stuff. You can skip to the examples below of insanity. The definitions below are far from Academic and they’re drawn from experience. Some places, such as the Code Contracts documentation will point out the difference between Preconditions, Post Conditions, and Invariants. I am not interested in such differences for the purpose of this post: I just care about valid state, all the way through.

I don’t want this post to be too long but I do want to spell out what it means to violate an invariant. Invariants are things we expect to be true about the lifetime of some unit of encapsulation. There are more advanced definitions but there are some simple examples we can derive. We’ll talk about functions since they are applicable to all languages without needing to worry about the specifics of OOP, Prototype Inheritance, Functional etc. Most languages have functions. Not all of them have classes – and it’s the smallest unit of work we can deal with.

A word on validation vs invariants: Validation can definitely enforce invariants, preconditions and more. However, often for a piece of code it might be too late. Validation usually implies some kind of deferred handling – which is sometimes too late as we will explore after the Introduction. We are more interested in immediately, observable

Nullability and buffer lengths. Let’s talk about these since they’re so easy to pick on. Validate your buffer lengths and the optionality of something. A large portion of the world does this now but still we have a ton of exploits involving these on a daily basis. Still, modern languages such as Rust, Swift and friends make this very hard to mess up. We’re moving forward.
Proper data type. The most basic (and arguably, this is a loose definition but it should be easily digestible) is validating your data types for a parameter into a given function. If you have an add function, you probably want it to take some integers. It might be an array, two integers, a variable length splat of integers (if your language supports such a construct) or any other set of parameters. However, the important thing is it’s decided what is supported and what is not. If we are to enforce this, we enforce the type.

There are a couple ways this is done. Static typing or run-time assertions are just two possible options.
Single Domain invariants. I classify these as things that are specific to something about the domain of how a piece of data is being used. A simple example is Percentages. If you have some floating point number (or even fixed point arithmetic operation) that you need to bound between some values (probably 0 and 100, if you are dealing with standard ones) you should go ahead and apply some invariants here. This means the value should not exceed 100 or be less than 100.

There are a few ways of accomplishing this. For example, Ada has a range construct you can use to create bounded integer value. In Javascript, the only choice we might have is to write something like this:
```
if (value < 0 || value > 100) {
  throw new OutOfRangeError('The value given ' + value + 'was out of range');
}
```
Code Contracts for .NET provides something similar and a bit more automated. We’re not going to dive into the specifics here.
**Class invariants. ** This is a classical OOP type – we touch upon it quickly here just because so many people are used to OOP. Wikipedia has a pretty good article on this but the gist is things that must be preserved across the larger unit of works lifetime. For example, if you had a class that held a startDate and endDate and it should enforce that the startDate should always be before endDate (the reverse is true, too) but you exposed two setters, setStartDate and setEndDate – then you would want to make sure this rule held and would need to examine both pieces of state in most cases.

As an aside, there a lot of different ways of tackling invariants. Some design methodology (looking at you Domain Driven Design actually gives some recommendations on how to do this depending on who you ask. We’re going to focus more on why you should do this more than how you should do it and how it can save a ton of time. At the end of the day, we’re going to be talking about documenting your assumptions in code. This might sound obvious but let’s look at what happens when we don’t do these things with some real code that has cost people time and money.

We can look at solutions later, I find the problems will convince people to go “Ooo, I’ve experienced that before! I thought this was just normal”.

The cost of assuming without documenting

Buffer overflows

The first kind of assumptions that we don’t validate that cost us all money that we encounter all the time is the classic buffer overflow. Sure, this can only happen in a subset of languages in most circumstances but it illustrates a good example of where your bad assumptions can cost real money. Let’s look at all the damage buffer overflows have caused over the years. In no particular order:

The “unpatchable” exploit that makes every current Nintendo Switch hackable - 2018 (details unclear, but definitely some kind of overflow attack)
Buffer Overflow Found in PSP Firmware v2.0 (this wasn’t the last either…)
The PS4 was partially comprimised through buffer overflow via Webkit

These focus on game consoles and they are only some of the most basic examples. I talk about them since a lot of them will and did in the past open the door to piracy. No matter your stance, the impact is not zero on the industry. Money is spent on legal departments, patching, PR, and more handling these things.

Depending on the systems, not handling some basic checks like this cost a ton.

Null pointers, null references, and friends

There’s a lot of things wrong with null pointers as we have learned over the years. There’s some good articles on-line (here’s a decent one) that explain why this is the case and Tony Hoare himself even indicates that:

This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.

Tony Hoare, inventor of ALGOL W.

Oftentimes (arguably always) these just occur because something was assigned null (or defaulted to) when people assumed they would not be. Simple to validate. Not often done. Tons of damage. Poorly handled null references can take crudely designed systems down. The most robust ones will behave wrongly.

This is an interest read on gaining root with null.

**OK, extreme examples, commonly known problems aside, let’s talk about the other large problems. **

Buffer exploits. Null pointers. Improper authentication. All bad things. These are bad from security but for a typical CRUD app running in your browser, maybe no big deal. But there’s a larger story here.

What can we do about it?

Design good APIs that easy to use – but this is just a first step.
Document your assumptions and fail fast when someone breaks the promise
Use languages with good support for these invariants

A word on tests

I just wanted to make this a section on it’s own really fast since I often get this argument. Specifically, I want to attack Unit Tests. Often, I will get the argument: “The tests pass – and the tests do the validation and document the values that are valid. If you want to figure out how to use it, you can look there. “

I just want to make a few arguments. If you disagree, this might not be the article for you and that’s OK.

If your tests are just testing known, good input, but do not cover if something is handling bad data (in the absence of a proper type checker at the most basic level) then you still have not solved the problem with users of your API who misuse it without realizing. That leads me into…
Users do not read your tests. They read your documentation, if you are lucky. In a lot of cases, the documentation either does not exist, is old, incomplete, does not work properly, or worse: it’s incorrect. Your code will not be incorrect (hopefully)
In some languages with a good enough type system, tests that probe the behavior of some these are actually just brittle and harmful. It does not excuse writing good code.

To be continued?

Automating adding missing require statements for globals using jscodeshift

2018-04-13T03:11:00+00:00

Sometimes, working on some older legacy codebases they are using some Frankenstein build systems they created in house or a port to something like gulp, especially in some older open source projects. As part of getting them over to something else and to make use of new tech. such as “module bundlers”, part of the challenge is to make sure all imports are valid and dependencies are clear. Unfortunately, in some of these projects this is not made clear and it’s hard to tell.

Generally, you would add require or import statements to let developers know you are importing some code for usage. In some cases, the system is setup with some kind of loader already such as RequireJS and it’s just being used improperly. For example, take the following pieces of code:

class A extends Backbone.View {
    render() {
	   this.$el.html(_.random(0, 10)) // set the view to something between 0-10
	   return this;
	}
}

This has a dependency on something like lodash … or maybe underscore. If the code base just uses one, then it’s fine to be able to just import it and move on, if you know what it is. In this case, this is a lodash function.

So, what if we have hundreds of files like this? We can analyze the source code and make good guesses.

Use a regular expression to search the source code, then add them…?

If we were going for quick and easy, we might be tempted to use a regular expression. This would work but for things like lodash which uses _, there’s a few problems with that making it hard to identify when it’s actually in use:

You could have something like _(items) to wrap the entire collection for example, so you cannot just search for _.
.. but because of that, you can’t just search for _ either since you’re going to get way too many hits and next thing you know – every module depends on this! Might as well have just had a global.js and had it declare it.
Even if you can write something to look out for the various forms, such as combing something like this:

… you immediately notice comments are a problem. You could probably add something to parse the line as well and make sure it’s not a comment. I share the same sentiment as Jeff Atwood:

I love regular expressions. No, I’m not sure you understand: I really love regular expressions.

… but that blog post summarizes what’s happening:

Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems.

Guess we should go fix them by hand, huh? Or should we charge on ahead? Or if you want, you play with the regular expression here first from above. Then do it for every single library… :(

If you eventually keep patching up cases, to succeed, you end up out of necessity with a parser. So, then…

Using a parser such as Babylon, esprima or other friends

You can read a lot more about projects such as Babylon and esprima as you would like but there’s just a couple things you need to know for this blog post:

They parse a chunk of code into syntax elements that make sense for programatic consumption – this means being able to differentiate between say, a comment, an assignment (const foo = bar()), statements (something) and just other things in Javascript land.
… and you can use jscodeshift plus one of the parsers to manipulate the Abstract Syntax Tree they produce in order to suit your needs using Javascript scripts.

So, let’s recap:

We can find the difference between comments, statements, assignments, etc. This means we can tell apart a real usage of a token aside from a supposed usage.
We can script it and programming languages are generally Turing Complete, where Regular Expressions are not. This means we can edit the AST as well, such as adding these require statements.

This is enough for us to be able to parse out what is being used. So, let’s get our hands dirty and figure out how to do this:

Install jscodeshift – npm install -g jscodeshift if you use npm
Write a transform to do this, and then run it against files in your code

How to get acquainted?

There’s a great site called ASTExplorer we’re going to use to get used to things. We’ll be using it to explore the code snippet from above. A couple housekeeping items…

Make sure your language is set to “Javascript”
Parser should be set to “Babylon” or something similar – since jscodeshift uses the babel parser by default.
Transform tool should be set to jscodeshift – that one should be obvious as to why.

If you load up your snippet code, then you should get something like this;

You will notice I have some code that logs out identifiers. First, let’s define what an identifier is and then let’s look at why we care about them.

What is an identifier?

Identifiers are value pairs / nodes that reference or define the name of a node. Often, they are the names of expressions, statements, class names, etc However, they are only for code. They have a couple important properties we care about:

They don’t include any comments, which is undesirable for our case
They’re entire words, separated by delimiters defined by the language syntax, making it easier to tell them apart, for example: Backbone.View is tokenized into something like:
- Backbone
- View
They can be boiled down into a list of identifiers that is nice and easy to iterate over and then map to statements.

So what about that output?

We get an output that looks something like:

A
VM120:14 render
VM120:14 $el
VM120:14 html
VM120:14 _
VM120:14 random
VM120:14 value_case
VM120:14 _
VM120:14 items
VM120:14 Backbone
VM120:14 View

For the sake of argument, we can say this output is exactly what we need, since it parsed out the _ and other identifiers we needed. If you add a comment with something else, like perhaps:

// $('div')

you will notice that nothing changes in the log changes. There are a couple flaws to this approach that aggressively would import more than require in some of the cases for some global tokens, but our goal is not perfect, just to get us most of the way there.

With token info in hand, there’s only one a couple things left to do. Let’s outline our plan of attack:

Creating a script to import something like lodash that was being used globally

We can start with something simple, given our previous input:

Input

class A extends Backbone.View {
    render() {
	   this.$el.html(_.random(0, 10)) // set the view to something between 0-10
	   const value_case = _(items)
		
       // and then something here _(items) .. and then something here...		
	   // and then $
	   return this;
	}
}

We want something like this:

_ - require('underscore');
class A extends Backbone.View {
    render() {
	   this.$el.html(_.random(0, 10)) // set the view to something between 0-10
	   const value_case = _(items)
		
       // and then something here _(items) .. and then something here...		
	   // and then $
	   return this;
	}
}

We will need a couple things here:

A way of parsing out identifiers, which we already looked at.
A way of injecting require statements into the code
A way of checking if a require statement is already there, so we don’t duplicate it (this is optional – you can always run something like rm-requires after to clean this up)
A way of outputting the source code

A way of parsing out identifiers, which we already looked at.

This is already complete – we looked at it above. We know how to load in a source code file and then loop over each identifier.

A way of injecting require statements into the code

There’s a few ways of doing this. There’s this package that promises you can do this. However, for something simple we can do something like this instead assuming we want them at the top of the file:

j(file.source).get().node.program.body.unshift(requireStatement)

This is basically getting the current file and then adding a line to the top.

Some ASTs might have something to do this with a specific node type, for example ts-simple-ast for TypeScript has this, but the parser we’re using does not for require statements, at least. After all, the only standardized module system is “ES2015 Modules” and if you were using those, you probably would not be in this problem.

A way of outputting the source code

This is built-in if you use the CLI tool – since it modifies files in place. You could also go to stdout as well, which is handy if you are doing dry runs. You just need to call toSource on the end of your fluent API.

Checking for duplicates

Since the above body variable from the injecting step is an array, we can just use includes

Plan of attack

That should be everything. Now:

Iterate over all identifiers
Finds one matching known problematic globals
Inject a require statements, checking if it already exists
.. and then terminate and print out the source code

Putting it all together, you get something like this

const containsStatementAlready = (builder, statement) => {
	return builder.get().node.program.body.includes(statement);;
}

// Press ctrl+space for code completion
export default function transformer(file, api) {
  const j = api.jscodeshift;

  const builder = j(file.source);
  return builder
    .find(j.Identifier)
    .forEach(path => {
      const statement = "_ - require('underscore');";
      if(path.node.name === '_' && !containsStatementAlready(builder, statement)) {
         builder.get().node.program.body.unshift(statement);  
      }
    })
    .toSource();
}

With this, you get the output we wanted when run in ASTExplorer! Pretty simple and you can automate most of this, with some manual code review after to make sure things weren’t added in error. Don’t forget to save this as a index.js script and then use jscodeshift to run this against as a sample file locally as well – that’s where the real power comes and where you can dot this en-mass.

The best part of JSCodeShift is it’s just Javascript – so you’re scripting your automation. If wanted to, you can automate entire sets of globals using something as simple as a hash map (read: Javascript Object for simplicity) I’ll leave running this against your entire code-base (a zsh glob will probably do the trick…) and for multiple global tokens an exercise.

I actually did multiple tokens right here if you need something now.

I’m not sure this would be ready for a large project but it certainly should get you on the way. Most importantly, we learned a bit about what we can do with parsing an actual language that becomes a lot harder with something like a simple Regular Expression.

Where does this fail?

There is going to be partial matches, for example: _.$.retry - if this was a function, then we might (correctly) identify the identifier $ but then mistakenly classify it as a global – even though it is not. These are safe in that they will just require in things that are not needed.
You might miss things if you’re not careful – for example, it possible to do something like window['$']' and this is technically a string literal referring to the global. In practice, I find this is pretty rare so they can be solved by hand if required. However, we could definitely solve this problem with more code. This would become even harder to do with Regular Expressions – since context becomes so important and knowing the semantics of the language.

So, next time before you reach for a regular expression to solve a problem involving language rules – grab for a parser first instead, it might make your life a bit easier.

Steam hanging when clicking "Login" on WINE Staging

2018-02-16T00:00:00+00:00

touma@setsuna:repos/hilts-vaughan.github.io ‹master›$ wine --version
wine-2.21 (Staging)

I was installing Steam in WINE to play a game. I’m not sure if this is a recent problem or not but I was installing WINE and ran Steam, punched in my credentials, hit “Login” and then expected the Steam Guard.

Instead my window hung. I could not click anything else, Steam was locked up. I ended up having to xkill it. What to do? The WINE output had something like:

 x11drv: Can't allocate handle for display fd

… ok. Out of file descriptors? Let’s see how many we have open…

os -aux | grep wineserver
touma     8792 62.2  0.1  27760 16884 ?        Rs   Feb15 814:28 /usr/bin/wineserver
touma@setsuna:repos/hilts-vaughan.github.io ‹master*›$ lsof -p 8792 | wc -l
4060

Well, that’s approaching my ulimit of 4096, so I wonder what’s going on? Does it stop when it realizes it will not be able to allocate more? Let’s see what’s opened:

wineserve 8792 touma 4050r      REG                8,5    97664  7349547 /usr/share/fonts/TTF/YanoneKaffeesatz-ExtraLight.ttf
wineserve 8792 touma 4051r      REG                8,5   112472  7347499 /usr/share/fonts/TTF/YanoneKaffeesatz-Light.ttf
wineserve 8792 touma 4052r      REG                8,5   114376  7353990 /usr/share/fonts/TTF/YanoneKaffeesatz-Regular.ttf

Looks like WINE / Steam / someone is scanning all my fonts, or something. YanoneKaffeesatz is part of the Google Fonts bundle, which I tried a while ago but never really use anymore. So, let’s get rid of them and see if it helps:

touma@setsuna:~ $ yaourt -R ttf-google-fonts-git

Run Steam, login, and it works.

A bug with Steam? I’m not sure. I hope this helps someone. At least Linux gives us the tools to debug our problems.%

Playing Trails of Cold Steel I && II on Linux using WINE

2018-02-16T00:00:00+00:00

Guide and Remarks: Playing Trails of Cold Steel I && II on Linux using WINE

If anything here becomes out of date, please email me.

I’m a huge The Legend of Heroes fan and love playing all the games from the series. XSEED Games has done a great job releasing the ports for the PC where available and where they can but they only port to Windows. The games are more on the niche side, so it is understandable, but I love them so and still want to play them on Linux as well. I got it working after tinkering for a few hours so I wanted to save any other random Internet go-ers the trouble and write this up so other people can enjoy this as well and other Linux users can give XSEED their money so they continue to bring us this great series. ;)

I purchased on Steam version but since they are reported to all be DRM free this should work for any version. You can just skip the Steam instructions if you bought on GOG or something.

Step 1: Preparing a WINE prefix, getting WINE, etc

Install WINE

You need a copy of WINE Staging (or Wine 3.2 that has CSMT enabled by default if your distro is up to date) to get started. Depending on your distribution, you might have to find it elsewhere. I’m on Arch Linux, so you can yaourt to get it there if you’re like me, yaourt -S wine-staging (or Pacman since apparently it’s not a AUR package).

If your distro makes it hard to get a copy of WINE Staging (I tested on 2.21) then you can install PlayOnLinux or Lutris which provide local copies of different versions of WINE outside of the system version. Once it’s installed, run:

touma@setsuna:Falcom/ed8_2 $ wine --version
wine-2.21 (Staging)

… and this should show you that it’s running fine. We can move on.

Configure WINE Prefix

I had used WINE in the past for some one-off’s but had nothing of value. You can use WINE prefixes but to make this bulletproof I decided to just make a new .wine and save myself the trouble. i.e:

mv ~/.wine ~/.wine.backup
winecfg

and this should generate a new ~.wine for you. You should go check. If you do this with an already existing WINE prefix, you run the risk of having problems. You can either look up how to use WINEPREFIX or you can just move the backup back into place when you are done. This probably only works for the System Wine, if you installed through a third party tool such as Lutris you will need to configure out how to run the configurator for that version. When the dialog comes up, make sure you go to the Staging tab and check off “CSMT” mode as well since this will be needed later. Photo below:

If you do not have this option, either:

You’re not running Wine Staging, make sure your version is as such (for example, make sure you are running the right wine configuration for the proper version)
They have merged the CSMT patches and you’re using WINE 3.2+ (which has this enabled by default – yay!)

We will also be using winetricks – your distro should have it somewhere or you can follow the manual instructions here. It should be fairly simple to get.

With this, we can move on to installing Steam. If you have something else, you can probably skip this step.

Step 2: Installing Steam

I referenced this test run at time of writing and it seems to work. For brevity and to prevent link rot, I will repeat roughly the steps you need to perform below:

winecfg # Setup a prefix (we already did this, so you can ignore)
winetricks corefonts
winetricks steam
winetricks d3dx9_42
winetricks vcrun2008
winetricks xact
winetricks vcrun2015

.. one of these failed but it didn’t matter. Or it was one of the commands below. I think it was a package that did not exist or was typo’d. For me, it made no difference. So, don’t worry about it.

After you have DSound as well installed, you will need to change the DSound library from native to builtin to avoid crackling and latency problems. You can do this in winecfg like we did with CSMT, you can a screenshot below. Just hit “Edit” on dsound to get the desired builtin

I also know from other Falcom titles that these are good to have in hand and won’t hurt Cold Steel, I don’t know if they are strictly needed but better safe than sorry in this case since all we care about is Cold Steel. If this was a shared prefix, we would have to be more careful.

winetricks amstream
winetricks dsound
winetricks gdiplus
winetricks quartz
winetricks xvid

That should be everything. You can go ahead and then start Steam if you have not already (you need to use wine Steam.exe -no-cef-sandbox to do this, inside the steam folder), login and start installing Cold Steel. If you have problems with Steam hanging on login (you hopefully will not), you can check out this other post wrote on the topic which is not documented very well on the web.

Step 3: Configuring Cold Steel, and then running it

If everything goes well, Steam should be running and you should be able to hit Play to begin the game. You can configure it first. I have the following configuration:

Just a few notes on my configuration, which I hit “Maximum” and then tweaked from there:

MSAA: Yeah, this is prretty much the only thing I could not get working. It crashed on startup if I had this set at all. The game looks very good without it, so whatever.
Mouse and Keyboard: I tested the gamepad and it worked fine but I prefer the mouse and keyboard.
Fullscreen: I tried window mode as well, it worked fine.
Everything else seemed to work fine, including turbo.

YMMV. At this point in time, you may want to check all the “Skip video” checkboxes as well, since it will crash initially if you let them play. There’s more on that in another section to get that working, detailed below.

If you want, you should be able to launch the actual game with my configuration and start playing.

Step 4: Getting movies working

If you can either just watch the video files as you need to from the game folder (there’s seriously almost no videos in the game, mostly cutscenes) or you can install LAVFilters in your WINE to get the videos working. This will vary on system to system, so YMMV.

Step 5: Loading up your clear save data, importing a CS2 save file, or someone else’s from the Internet

If you are playing Cold Steel II, then you would want to import some data. There’s some cool stuff if you do. If you’re like me and played on the Vita or PS3 first and just want to bring your new save into CS2, then you can download save files from the web. I have one right here that is a Level 99 Rean NG+ with Alisa selected as the dance partner. Other people have posted collections on-line, such as here. If you do not care, you can skip this step.

Importing the saves

OK, so on the web you will see a lot of people saying you need to place these in C:\Users\<your_username>\Saved Games\FALCOM\ed8 (or ed8_2 for Cold Steel II) which under WINE would be /home/touma/.wine/drive_c/users/touma/Saved Games/FALCOM/ed8 – however, if you are like me, placing your saves here will not work. Instead…

touma@setsuna:~/.wine $ find . -name "ed8_2"
./drive_c/users/touma/Application Data/Falcom/ed8_2
touma@setsuna:~/.wine $ pwd
/home/touma/.wine
touma@setsuna:~/.wine $ 

.. and you will realize that there is a folder sitting inside of Application Data. For Cold Steel II import, just create a folder called ed8 and then inside, place your save files:

touma@setsuna:Falcom/ed8 $ ls -la
total 1248
drwxr-xr-x 2 touma touma   4096 Feb 15 22:22 .
drwxr-xr-x 3 touma touma   4096 Feb 15 21:46 ..
-rw-r--r-- 1 touma touma 169016 Feb 15 22:08 save000.bmp
-rw-r--r-- 1 touma touma 460992 Feb 15 22:08 save000.dat
-rw-r--r-- 1 touma touma 169016 Feb 15 22:08 save001.bmp
-rw-r--r-- 1 touma touma 460992 Feb 15 22:08 save001.dat

… and you should be able to load them. Start the game. Hit “New Game” and when prompted to load clear data, hit yes. The files will be invisible (a problem on Windows as well) but just click slot 0 or 1 and you should load up no problem.

Similar, if you have Cold Steel II Data, just drop it there:

touma@setsuna:Falcom/ed8_2 $ ls -la
total 9536
drwxr-xr-x 2 touma touma   4096 Feb 15 23:56 .
drwxr-xr-x 4 touma touma   4096 Feb 15 22:04 ..
-rw-r--r-- 1 touma touma 493976 Feb 15 23:56 autosave00.dat
-rw-r--r-- 1 touma touma   1536 Feb 15 23:56 autosave00_t.dat
-rw-r--r-- 1 touma touma 493976 Feb 16 00:03 autosave01.dat
-rw-r--r-- 1 touma touma   1536 Feb 16 00:03 autosave01_t.dat
-rw-r--r-- 1 touma touma 493976 Feb 16 00:10 autosave02.dat
-rw-r--r-- 1 touma touma   1536 Feb 16 00:10 autosave02_t.dat
-rw-r--r-- 1 touma touma 493976 Feb 16 00:15 autosave03.dat
-rw-r--r-- 1 touma touma   1536 Feb 16 00:15 autosave03_t.dat
-rw-r--r-- 1 touma touma 493976 Feb 16 00:26 autosave04.dat
-rw-r--r-- 1 touma touma   1536 Feb 16 00:26 autosave04_t.dat
-rw-r--r-- 1 touma touma 493976 Feb 16 00:31 autosave05.dat
-rw-r--r-- 1 touma touma   1536 Feb 16 00:31 autosave05_t.dat
-rw-r--r-- 1 touma touma 493976 Feb 16 00:50 autosave06.dat
-rw-r--r-- 1 touma touma   1536 Feb 16 00:50 autosave06_t.dat
-rw-r--r-- 1 touma touma 493976 Feb 15 23:50 autosave07.dat
-rw-r--r-- 1 touma touma   1536 Feb 15 23:50 autosave07_t.dat
-rw-r--r-- 1 touma touma 493976 Feb 16 00:50 save000.dat
-rw-r--r-- 1 touma touma 493976 Feb 15 22:50 save001.dat
-rw-r--r-- 1 touma touma 493976 Feb 15 22:50 save002.dat
-rw-r--r-- 1 touma touma 493976 Feb 15 22:50 save003.dat
-rw-r--r-- 1 touma touma 493976 Feb 15 22:50 save004.dat
-rw-r--r-- 1 touma touma 493976 Feb 15 22:50 save005.dat
-rw-r--r-- 1 touma touma 493976 Feb 15 23:03 save006.dat
-rw-r--r-- 1 touma touma 493976 Feb 15 23:03 save007.dat
-rw-r--r-- 1 touma touma    120 Feb 16 00:50 save255.dat
-rw-r--r-- 1 touma touma 368640 Feb 16 00:50 sdslot.dat
-rw-r--r-- 1 touma touma 169014 Feb 16 00:50 thumb000.bmp
-rw-r--r-- 1 touma touma 169014 Feb 15 22:50 thumb001.bmp
-rw-r--r-- 1 touma touma 169014 Feb 15 22:50 thumb002.bmp
-rw-r--r-- 1 touma touma 169014 Feb 15 22:50 thumb003.bmp
-rw-r--r-- 1 touma touma 169014 Feb 15 22:50 thumb004.bmp
-rw-r--r-- 1 touma touma 169014 Feb 15 22:50 thumb005.bmp
-rw-r--r-- 1 touma touma 169014 Feb 15 23:03 thumb006.bmp
-rw-r--r-- 1 touma touma 169014 Feb 15 23:03 thumb007.bmp
-rw-r--r-- 1 touma touma  43256 Feb 16 00:50 thumb255.bmp

.. as you can see, I’ve saved a few times. ;)

Eventually, you should have a game running. Minor spoilers below…

… That should be everything you need to play! There are a couple other things that I noticed as well but they might not affected everyone. I’ve attached some of that info inside of the misc section below. You can read it if you have problems.

Step 6: Misc. other things

DLC

I noticed I had a bunch of DLC unlocked, I was not sure if this was intentional or a WINE bug.

The outfits:

is for owners of the Trails Series on Steam and other platforms, which I have. So, that makes sense. The rest, I am unsure. Speaking of outfits…

Crashes with costumes

On Windows as well at the time of writing, wearing costumes sometimes can cause the game to crash. For example, early on when you enter a certain hot spring, the game will crash wearing the costumes given from the DLC, which I experienced.

Camera spinning on startup

Do you have xboxdrv installed? I did. systemctl stop xboxdrv.service should do the trick while you’re playing. You an always start it when you are done.

Performance notes, audio, wine debug

If you have problems with performance / audio, make sure you set the WINE Debug flags accordingly to get your performance back.

Reference System

touma@setsuna:Falcom/ed8_2 $ neofetch
                   -`                    touma@setsuna 
                  .o+`                   ------------- 
                 `ooo/                   OS: Arch Linux x86_64 
                `+oooo:                  Kernel: 4.15.3-1-ARCH 
               `+oooooo:                 Uptime: 13 days, 16 hours, 23 minutes 
               -+oooooo+:                Packages: 1620 
             `/:-:++oooo+:               Shell: zsh 5.4.2 
            `/++++/+++++++:              Resolution: 1920x1080, 1920x1080 
           `/++++++++++++++:             DE: GNOME 
          `/+++ooooooooooooo/`           WM: GNOME Shell 
         ./ooosssso++osssssso+`          WM Theme: Adwaita 
        .oossssso-````/ossssss+`         Theme: Paper [GTK2/3] 
       -osssssso.      :ssssssso.        Icons: Numix [GTK2/3] 
      :osssssss/        osssso+++.       Terminal: gnome-terminal 
     /ossssssss/        +ssssooo/-       CPU: Intel i5-2500 (4) @ 3.7GHz 
   `/ossssso+/:-        -:/+osssso+-     GPU: NVIDIA GeForce GTX 970 
  `+sso+:-`                 `.-/+oso:    Memory: 6230MiB / 16005MiB 
 `++:.                           `-/+/ 
 .`                                 `/                           

I play games on GNOME since it works better for me than i3wm which is my development WM. I leave this here just so people can see what I was using at the time.

Ramblings

I’ve only played about 4 hours by now but I’ve not noticed any game crippling issues. Game seems to run great, with the occasional minor hiccup but things are good. I’ll edit this article if I come across anything that is a deal breaker and how I managed to work past it.

Fixing "playing games under WINE in Steam, the camera rotates in one direction "

2018-02-16T00:00:00+00:00

Just writing this down here in case anyone else stumbles across this. On my Arch Linux system, many games running under WINE were having their camera spinning over and over or it would look like a joystick was held down. For example, in Trails of Cold Steel II the camera would spin but in Recettear: An Item Shop’s Tale the main menu would scroll through all the options over and over. Something is sending joystick input.

Turns out, for me, it was xboxdrv – and you can turn it off while you’re playing using your distributions service manager. For Arch Linux (and probably any systemd system, which most are by now at time of writing), just:

systemctl stop xboxdrc.service

… or just uninstall xboxdrv if you don’t need it.

Setting up a TF2 Server with Custom Maps using Linux Game Server

2018-02-06T00:00:00+00:00

Setting up a TF2 Server with Custom Maps using Linux Game Server

I was recently trying to play some custom maps with some friends on Team Fortress 2 but with some custom maps. There are tons of community servers that run certain maps on rotation but these are not always the ones you want to play, especially if you want to play the older maps. I had a bit of trouble setting the server up since there is very little up to date documentation on how to get it running with custom maps specifically so I figured I would try and fill the gap for those who stumble across this in the future.

I’ll leave the guide here and commentary to the end, in case you are interested in ramblings. This is for Linux, whether on a personal Linux host or on your cloud provider. Some of the lessons here can definitely be used on a Windows machine, such as the files you need to place and where but the commands, scripts, etc are very Linux centric and not portable. You may have luck with the Linux Subsystem for Windows 10, but I have not tried it.

Step 1: Getting the Linux Game Server, aka a stock install

This much is pretty simple, you can follow the directions on their site and that’s all you have to do for setup. For completion sakes and because link rot is sometimes a thing, I will include the exact directions here:

adduser tf2server
passwd tf2server
su - tf2server
wget -N --no-check-certificate https://gameservermanagers.com/dl/linuxgsm.sh && chmod +x linuxgsm.sh && bash linuxgsm.sh tf2server
./tf2server install

This should get you a working install, after working through that stuff. There are a few other things you may want to change, you can read the rest of the page there to find out more. The download is fairly big and can take a bit, so you can can read on and be prepared for when it’s done. You can even complete Step 2 while this is going in the background.

Step 2: Expose it to the Internet

Home Connections

This will sound pretty obvious if you’ve ever ran a game server but I will mention it for completion sakes. You will need to open the port that inbound connections will be coming on if you sit in a typical residential network with a gateway in front of your server.

You can refer to you routers manual for specifics but you need to setup Port Forwarding for the inbound port, 27015 and there is no sense in going over a tutorial on this here, since there are tons of guide on the web.

Cloud Providers, Digital Ocean

If you are using Digital Ocean or some other cloud provider, the port is probably already available if you have a dedicated IP for yourself. You may need to allow the port as well, depending on the distro you chose to run. If you are running something running ufw then you can use something like: sudo ufw allow 27015 to open inbound traffic on that port for yourself. Otherwise, open the port on the firewall however you choose.

You can read more about ufw on the DigitalOcean site or elsewhere on the web, if you need to deal with it.

Checking connectivity…

There are tools you can use such as CanYouSeeMe for local installs to check a port. Linux Game Server has something built in for this, so there’s no need though. Just run ./tf2server monitor and you can check your status. If it says OK then you should be good to go.

Step 3: Load up custom maps into the directories needed

You can find a ton all over the web, these are just a few places:

GameBanana
Wiki
TF2Maps
… or any other BSP files you can find on the web that is supported

You can find a ton of them in various indexes available on the web as well, such as here. Once you have a few, you can follow the below directions.

You can begin by entering your directory of your server as the user you created when using the TF2Server installer. If you run cd serverfiles/tf/maps/ you will end up in a directory full of bsp files. In here, you can place all the files you downloaded and want to play. There are a couple ways to do this….
1. If it’s local, you can just use a File Explorer, of course.
2. If it’s remote, you can look at something like scp, ftp, mount your file system as SSH or whatever you prefer. You can use Google to find this information…
Then, you can cd - to go back to the root directory, where you were and your ./tf2server script should reside. You need to add the maps that you have to your config now, so you can this by executing a bash one liner. This was the part that confused me the most, since a lot of places on the web talk about maplist and mapcycle and to edit it to include the new maps you want to play. If you are like me, you would not have had these files at this point and would be unable to find them using find . -name "*maplist*" and it would turn out that it does not. You need to create it. To do this…
```
ls serverfiles/tf/maps/ -a | sort | sed -e "s/\.bsp//g" > ./serverfiles/tf/cfg/maplist.cfg

ls serverfiles/tf/maps/ -a | sort | sed -e "s/\.bsp//g" > ./serverfiles/tf/cfg/maplist.txt

ls serverfiles/tf/maps/ -a | sort | sed -e "s/\.bsp//g" > ./serverfiles/tf/maplist.txt
```
To be honest, I am not sure you need to duplication but I have not gotten around to testing it yet. However, this should be at least a working install. You can throw this in a script and call it, say updateMaps.sh if you want so that you can re-run it every time you add new maps.
For some maps, there are some additional settings to set as well to set. You can find those below, make sure to read them if you need them.
Restart your server. You can do this using the Linux Game Server script, by running ./tf2server restart and then waiting a few minutes for the server to come back up.

There’s also some additional configuration to do depending on the map you play. For example, some maps have bots that occupy slots and need a large number of slots for them, such as Mann vs. Machine and perhaps other modes. Some maps are just big. In our case, we needed to do this:

 nano lgsm/config-lgsm/tf2server/tf2server.cfg

… and then add these lines

maxplayers="32"
net_maxfilesize="64"

This will let you play most maps without trouble. You will need to restart your server after any of these changes.

Some places on the web will let you know you need to also configure a FastDL server, this is false at the time of writing this in 2018. You only need this if you want to serve files that are large at a quick download time. If you are just setting up a local server to play with some friends and have custom maps rotate out, this is overkill and your friends can wait the 5 minutes it needs to download a map, once.

Step 4: Connect and then load the map

In Team Fortress 2, you can then click “Find Game”:

.. and then click “Community Servers”. Then, you can click the “Favourites” tab and then “Add Server” button in the bottom right. You can then add the IP of the server. You can find the public IP pretty easily by requesting it from the outside world by asking Google or if you know there’s no gateway in front you can just do ip addr to figure it out.

If all went well, you should be dumped into whatever the default map is when you connect. If not, time to troubleshoot.

To change maps, the easiest way is through rcon and you can find most of this through the Linux Game Server documentation. The gist is, run ./tf2server console and then you will be dropped in a console. Then…

You can run maps* to get a listing of all the installed maps. Yours should be listed now, without the bsp suffix since that’s the convention.
You can run changelevel to change the map that is currently being played, using the name of the map from the above command or just use the filename of the map without the extension (.bsp)

There are other rcon commands and there are various guides on the web with interacting with it. I reference this http://ozfortress.com/showthread.php?p=415211 but there are probably other guides out there that are better / more detailed.

Ramblings

I found it pretty frustrating that most of the information on the web was pretty incomplete in terms of what to do to get a fully working install with nothing. Google gives a lot of guides on Windows which are incomplete and talk about files that are not even there and does a good job of misleading you through what you need as a minimum. For example, the first Steam Link there insists you setup a web server and FastDL and spends most of the post on that. There are other interesting problems aside from FastDL. I kind of wish there was a better way and I am kind of surprised that the folks at Linux Game Server don’t have a way to do this since they have a mod installer but not a map installer.

It took a pretty large amount of time to get everything right and it could have been a lot easier with a proper write up and instructions that detailed the custom map portion. So, I leave this in hopes it will help someone else.

Happy playing! If there are questions, you can drop them below.

You need to learn to navigate code

2017-11-22T00:00:00+00:00

You need to learn to navigate code

Often when we talk about productivity of developers, we will measure various outputs. In the worst places, this metric might be lines of code. In better ones, it will be bugs density, features completed, support requests answered. Often, to get better at these we will reference things like better software design, maintainable code, unit tests, and more. Some of these focus on letting other people read our code in a more efficient manner and others just help us write better code for ourselves. The best will do both. There is another side of the equation, though. We write readable code because we know that eventually someone else will read it, even if that “someone” is ourselves. The thing we neglect to also talk about sometimes is that being able to read your code in an efficent way is almost as important.

Learn your IDE shortcuts and make use of them

This might sound obvious. It might even seem like it goes without saying. However, I have witnessed many developers that simply do not use any hot keys or know the shortcuts to navigate their editor. They do not use any shortcuts. When you are reading a dense chunk of code, this can cost you minutes that will add up in a very long session. These days, I do mostly web development so I will be speaking mostly from that perspective. I use Atom and Visual Studio Code as my main editors and feel comfortable talking about these. Even though I prefer Visual Studio Code more nowadays, I focus a lot on Atom since I think a lot more developers still use it. It’s very popular and because of that, I feel like a lot of people use it without actually configuring it properly. It is is possible that this phenomenon is exclusive to web developers but I have a feeling it can be applied to others, too. I feel like developers with full IDEs are more aware of these kinds of things, especially in languages like Java where the tooling has been very good for a while now.

These are some things you need to learn to use, else you are severely missing out:

**Jump to method **. If you are pressing Ctrl + F to find a method inside of your code, you are doing it wrong. Most IDEs and even text editors, such as Atom (with the proper language server / support) can jump straight there. For example, in Atom, you can do this with Ctrl + R (at least on Linux) There are numerous advantages
- The search is often fuzzy, so you only need to recall a few characters
- You can see all the symbols in one view, so you can flip through them fast if you need to often
- You can get there by just typing some characters instead of scrolling and then positioning the cursor
- You only get definitions and not all the usages as well and then need to read each result you get if you were to have done a search
Go to file / class. This is also a big one that I see a lot of people use already and it is rare that I do not see someone use it. Hunting for a single class or file in your structure is painful and could require dozens of seconds to get right, depending on how large your project is. Ctrl + Shift + F is not the right answer == instead, use something like Ctrl + P in Atom or Ctrl + O in Jetbrains IDEs. … and now you get something that can help you find a file very quickly.
If your tooling supports it, learn your quick navigation tools such as “Go to class” or “Go to definition” when you have a word selected. Visual Studio Code has good support for this.

In fact, there are a ton more. These are just some of the very basic ones that kill me I see someone not using them. Atom does not support much out of the box but Visual Studio Code supports a ton more. I do not need to repeat much more of what is already available. Just read the documentation for your IDE if you have not already. Scour the web, people have summarized their findings for you.

I’ll put some links below to some good articles on getting the most out of your IDE for code navigation:

Visual Studio Code
IntelliJ
Visual Studio, random shortcuts from SO
.. and there’s more. You can find them all from a simple search with “your favourite IDE” + “code navigation”

Just learn your keyboard shortcuts.

Spend some time setting up your environment to your workflow

These are just a few things I have done to my Atom install. For each thing, I identified where time was being sunk into and then make changes to make things less cumbersome.

Select the file in the tree view you are looking at by default ( Settings > Tree View > ‘Auto reveal’): If you are exploring new code, there is a chance you will be looking at files grouped in a similar package or module. If you are using your keyboard to hop between files as you find them you might want your file finder to follow you as well. Some Jetbrains IDE have this feature as well which is disabled by default. This will save you time from hunting down the file or executing the “Show in tree view” command (once you figure it out…)
Look into tools that can provide you extra value. ** For example, in **Atom we do not have a fully fledged IDE available but there are plugins that will make a big difference, such as TernJS for Javascript which allows us to get some “type hinting” on code that is not annotated.
Rearrange Windows if you have multiple monitors, make use of them. I was guilty of this one for a while. I would be making use of only one monitor when developing and using the other for documentation. On a 1080p monitor, I thought of better things to do with a 2nd monitor then holding a browser I would look at ‘sometimes”. The stock configurations for most environments are not suited for dual monitors, let alone multiple.
1. Unit Test Feedback that I can glance at from time to time to make sure nothing is broken
2. Console ouput that can be monitored
3. The running application when doing web development, so I can see the change in front of me
4. … anything else related to the current task’
Find a colour scheme for you that makes it easy to differentiate things This one is simple but makes a big difference. Make sure the colour scheme you are using is providing you value and you can tell things apart.

There are tons of other things you can do. The web has covered them all everywhere and there is no reason to repeat everything that has been blogged to death. Instead, I beg, notice what you are doing and what is taking forever. Then, you should find a way to fix it. Just by changing up a few things, I saved myself a lot of frustrating and got some very helpful hints while reading code.

Yep, that’s it

Yeah, that is really all I wanted to say. All this information is already available elsewhere if you are looking for it but nobody talks about this in academics. In school, you are not taught to learn about your tooling all the time. (At least, I was not) If you have good co-workers, they will probably point out your inefficiency when they see it. If you are self employed, you will need to discover this on your own. You know, most people already know this stuff. But I hope this will reach someone out there whom never considered it.

Happy reading and navigating.

Honest Perspective - A year on Arch Linux coming from Windows and macOS

2017-11-20T00:00:00+00:00

Honest Perspective: A year on Arch Linux coming from Windows and macOS

A little background about me before we begin fully:

I’m a Software Developer in Canada, I’ve been writing web applications professionally for a number of years
Most of my development is done in Java, C#, and Javascript. So, this will be written from the perspective of someone who works in those technologies where applicable. However, in general, my experience with development was very pleasant, so there is not much to write about!
My previous Linux experience amounts to working on a remote server with a few basic *NIX commands and using macOS
I like a little bit of punishment when I do things wrong and understanding why my decisions were poor – this ties into my distribution choice later on

A little over a year ago, I switched from Windows to Linux completely on my journey to finding a better development environment that would serve me on my home machine. At the time, the Windows Subsystem for Linux did not exist. There were a few reasons I wanted to switch over:

There were some development tools that I used a lot that I just wish I could use hassle-free under Windows. They were mostly Linux tools or software. These includes a lot of command line utilities and other goodies grep, dd, zsh , git and a host of other things. Some of these worked fine under Windows, such as git, but the experience was not as good.
I was sick of being unable to properly diagnose my machine under Windows with full understanding. The kernel logs were not as easily exposed and things were always a bit more arcane when they went wrong, with less hope of debugging them on my own. I take pride in being able to debug software – why not my operating system when I need to be able to?
I wanted to try a new workflow other than the one Windows could provide me with. This included new file managers, new desktop environments and new ways of navigating my machine. After using a Mac for a while, I had come to appreciate some of the features macOS had that Windows did not natively, such as brew and workspaces (before Windows had any!). (Yes, I know Chocolately and there are other 3rd party workspace managers, but none of them were tightly integrated with the OS and felt bolted on through some “not-designed-for-this” APIs)
I wanted to play around more with Docker a little more and “Docker Machine” was a pain and it was not the “native experience” that Docker was claiming to fame. I played around with it in a Virtual Machine a bit but I wanted to experience the full thing inside of a native operating system.
It was something new and cool – who doesn’t like cool and new things?

Choices & Setting up

I spent a long while deciding on what distribution to install and I eventually settled on Arch Linux. There were a few reasons for this but the main decision points just boiled down to this:

The AUR is awesome and had almost everything I could have wanted when I researched it, this seemed like a great perk.
I wanted up to date drivers and video hardware. At the time, I was running an AMD card and I heard the bleeding edge drivers were a lot better. As it would turn out later, better was not even close to good enough at the time.
I wanted some a little less abstract so I could learn the nuts and bolts
I needed something that out of the box would let me do all the customization I needed. For example, I already knew I wanted i3-gaps and wanted to try some alternative menus such as dmenu. This made it harder to choose distros that would prepackage their own environment which I would then have to dissect.
I needed to be able to install all the developers tools I needed with no friction, and the AUR was a huge help with this.
.. and it was hyped up to be the best thing since sliced bread. Might as well play the hype machine, right?

The runner ups were Ubunutu and Manjaro. Ubuntu was too stock and in the way of what I wanted to do, so it was out. Manjaro? Well, I read something about a SSL certificate that went something like, not once, but twice:

Hello!

Our SSL certificate has once again expired. We are waiting for a new one to be issued (while also looking at more sustainable alternatives, i.e. Let’s Encrypt). If you’re having problems accessing any of the sites please use a different browser profile, or Private/Incognito Browsing. You will then be able to add a temporary exception.

Example (Firefox 46)

https://manjaro.github.io/SSL-Certificate-Expired/

.. and it left a bad impression. So, that was out. I figured I might as well for the full authentic experience.

So, I followed the Beginner’s Install Guide and was installed in a couple hours with GDM and i3-gaps with a multi-monitor setup using arandr to configure myself. I did the initial run with gnome since I figured it would give me the least amount of problems and at least a working environment to boot into. These are the notes I left for myself when I had installed:

arandr did not work out of the box and I had to manually create my xorg file. This was kind of annoying but considering it worked under the GNOME Environment that I initially installed, I assume it was a bug with arandr or the way I was using it.
The video drivers for AMD were bad. I was unable to play most games on Steam with acceptable frame rates. My 7770 was already getting pretty old and I was eying a 970 anyway, so I bit the bullet with the understanding that: NVIDIA drivers were better on Linux than AMD and that I would be able to play things and the monitor setup would be easier. Turns out, both were true. Success!
The media keys were not working out of the box on my keyboard along with some other things taken for granted. I did want a DIY approach and I think some of this worked under GNOME when I tried it briefly, but it would have been nice if there pre-packaged sane defaults that I could install. I ended up installing i3-gnome which gave my some of these “sane defaults” I wanted by running some of the GNOME subsystem. Other things missing which I did not realize I had missed: easy screen snipping, volume management, and well thought out settings GUI applications for the day to day stuff.
I had to jump through some hoops to customize my K70 keyboard again. I just configured it and left it alone..
I had a problem where I noticed fonts sucked really bad out of the box. I had to install some additional packages to make it look better.
Some of my games and applications would not launch due to missing libraries. I had to learn how to ldd… I guess I asked for it.

Note: At the time of writing this, it looks like the Beginner’s Guide no longer exists. It seems it has been consolidated. I cannot comment on the quality of this new guide.

The final setup looks something like this after all is said and done. I think the only thing I have done since is added another 8GB of RAM to keep things happy, especially since I needed the Windows VM from time to time now.

'                   -`                   touma@setsuna 
                  .o+`                   ------------- 
                 `ooo/                   OS: Arch Linux x86_64 
                `+oooo:                  Kernel: 4.13.11-1-ARCH 
               `+oooooo:                 Uptime: 7 days, 6 minutes 
               -+oooooo+:                Packages: 1649 
             `/:-:++oooo+:               Shell: zsh 5.4.2 
            `/++++/+++++++:              Resolution: 1920x1080 
           `/++++++++++++++:             WM: i3 
          `/+++ooooooooooooo/`           Theme: Arc-Dark [GTK2/3] 
         ./ooosssso++osssssso+`          Icons: HighContrast [GTK2/3] 
        .oossssso-````/ossssss+`         Terminal: terminator 
       -osssssso.      :ssssssso.        CPU: Intel i5-2500 (4) @ 3.7GHz 
      :osssssss/        osssso+++.       GPU: NVIDIA GeForce GTX 970 
     /ossssssss/        +ssssooo/-       Memory: 7041MiB / 16013MiB 
   `/ossssso+/:-        -:/+osssso+- 
  `+sso+:-`                 `.-/+oso:                            
 `++:.                           `-/+/ 
 .`                                 `/ 

Package managers are great

There’s nothing much to say here. I always liked brew on macOS even if it was a bit clunky to use and the method of installing it was, shady, at best. After using pacman for a while, I can say that I never want to go back to manual installers infested with AdWare ever again. You get so much for free with pacman, some of which is not exclusive to it, but just a part of package managers in general in most distros:

Dependency Management
Versioning (rollbacks, if needed)
Visibility into what is about to be installed
Installs in seconds in some cases

Seriously. They are awesome. I wish Windows was more careful about this kind of thing and that macOS supported something a bit better. I guess the application stores are getting there, but they are a little more cumbersome and not really “developer oriented”.

I won’t go into this more. There are dozens of articles on the web detailing why they are great.

Picking your own file manager is great, better selection

There are replacements for the file managers on Windows as well but they are not as well integrated and a lot of them I find are pretty clunky. I have found I actually really like pcmanfm. It has very few animations on my setup, the search is very fast, I can hop directories with little trouble, it integrates well with my dark theme, and it still has enough features to not get in my way. For example, with gvfs it plays with automounts fine, script executing, and good selection capabilities. And it’s integrated with my system with no additional trickery. I also tried dolphin and nautilus and found them okay as well but not quite my taste.

In Windows and macOS, changing the file manager is pretty tricky across the entire system from my experience. Windows Explorer I find is pretty good. I found “Finder” almost unusable and just using MacRanger on macOS for the small amount of times I need a file manager there.

It was refreshing to pick something I like and being able to zip through folders at high speed.

Web development is great as well

There is also not much to say here, either. For my development work there were a few immediate benefits in most cases:

Reduced CPU usage in the case of some “file watcher” implementations due to better native support.
The support in general was better for a lot of tools – including Apache and Docker. I was able to spin up tons of projects and keep them running with minimal resource usage. This was awesome – Docker Machine was unable to do that.
The Git CLI was a first class citizen. It worked great with some ZSH extensions and made me appreciate what the CLI could do that Windows was unable to do. I was used to this from macOS but it was nice to see it not exclusive to there.
Scripts other people wrote just worked. You found something on StackOverflow for git or some other tool you wanted to run? There is a chance it is written in bash or perl They ran with no problem without any additional tools in most cases. When they did not, pacman -S got me sorted in a few seconds.

Overall, a much better experience than Windows and even a little better than macOS. I’m looking at you, long polling implementations for file watching under macOS.

Java & C#

**C#: ** Project Rider and Visual Studio Code do a wonderful job of giving you a competent environment to write code for this language. I was able to maintain the handful of console applications and the .NET Core applications I needed to without much trouble within these environments. They ran pretty good. Visual Studio Code is a bit laggy but much more responsive than Atom. Considering writing C# code on Linux was not even an option not so long ago, this is something I am excited about.

**Java: ** IntellIj worked out of the box pretty much the same. No comment here.

All things considered, the experience in these were on par with what I was getting before. Thus, no complaints.

Games are still a problem in some cases, but I was pleasantly surprised how well things ran when they did work

I use Steam to play most of my games. It’s not a secret that if you want to access the largest collection of games possible, you need to have an account here. I also happen to like playing games with some of my friends on this platform. These are a few notes from that experience.

A surprising amount of games have ports that run pretty well. For example, Terraria had a good port and Minecraft runs very well. Civilization works well, as well. Borderlands 2 was a nice surprise, as well. Some of the games that did not have ports would run fine under WINE. For example, the first Borderlands was played from start to finish over the Internet with a Windows user. I had a hard crash or two but other than that, everything went fine. However, there are a few times where I spent way too much time trying to get things to work. For example, “Duck Game” was not working in my prefix for some various reasons which I eventually resolved. It’s kind of annoying to have friends ping you to ask you to play – just to have to turn them down because you cannot start the game. Ouch. Nowadays, we mostly play Linux compatible titles with more and more companies releasing for the platform. For single player games, I just do not buy games that do not support my platform of choice or I just play on a console.

I made an exception for the Trails in the Sky series. The series is amazing and the XSEED team did an amazing job bringing it to us. If you have not read the struggle and immense amount of work that went into localizing the game I suggest you read it here if you have some time. The article is a bit long but it comes from the heart. I’ve played the latter two games inside of WINE with no problems since they are older.

Performance is just okay. There are a few instances where the performance has clearly taken a hit in order to be playable on Linux. There were problems unrelated to ports, too. When I first installed dolphin-emu from the AUR, I was unable to play any emulated games of my choosing from my ripped library with adequate performance. Most of them would lag pretty hard. I was not the only one, either. Check out these quotes from the AUR:

Dolphin-emu is still super slow with the new Nvidia driver 364.16 (it was all good months ago). -Neros

I’m also using NVIDIA and noticed that the performance is very impaired in some cases. I don’t know since when exactly, because I’m not playing regularly. -Martchus

There are other cases as well. At the time of writing, the problem no longer exists. It resolved itself after a routine update sometime a few months ago. However, I am documenting it here since it was real and it stopped me from playing some of my older games without unpacking my Gamecube.

In other cases, some emulators worked better under Linux in terms of compiling and setting up. Simply put, the experience is not the same. However, for me, the experience was close enough that I am happy with it in 2017.

Tiling window managers are awesome!

I’ve been using i3 for a about a year now. And when Wayland hits, I think I will also check out sway (link) and continue to use the same kind of layout for a while. These are the primary reasons I wanted to start using it:

I found myself maximizing a lot of windows by hand anyway, such as Chrome and most of the windows I was “working with” and then laying them out side by side
I loved being able to snap Windows in interesting ways. This is one of the features macOS is missing out of the box that annoys me every day. The fact that I have to find my mouse and drag a window to fit how I want is annoying enough – this could be achieved with other window managers, but i3 has a whole other level of control in this category.
I wanted a keyboard driven work flow. I already spent most of my time in a terminal in Windows and in my text editor, an environment that would allow me to spend more time on my keyboard and less on the mouse was a plus.
I wanted something that would let me build something of my own, “ricing” seemed kind of fun. :)

I followed this set of screen casts to learn the basics and then after and that I went on to read the documentation to put a few more customizations in place that I required (mostly floating windows for my file manager and a few keyboard repeat settings and color changes) and then I used the setup for about a year. These are some things I found awesome about tiling window managers, and i3 in particular:

The learning curve was not that bad at all. I think after some custom configuration I was productive using it in as little as a couple weeks. Within a month, I was able to accomplish some tasks faster than using the old window modal method.
I was able to create my own “custom development environment” with no trouble. I could tile a code editor, such as Atom, and a terminal on the bottom. This allowed me to use the best native bits of my platform when it was called for without needing to do a bunch of dragging on the screen. No interrupted thoughts!
I was able to create custom launch configurations and layouts for frequent tasks.
I always knew exactly where everything was. Browser was always on Desktop 1, no more flipping through windows. My development environment was always on 3. Scratch workspaces always on 0. Everything has a home.
Multi-monitor support worked in the way I wanted it to. To the above point, certain workspaces were always on the other monitor. This was super handy for flipping to a documentation window or when playing a game with friends and I needed to pull up a Wiki page. (Need focus on that Terraria wiki page over there? Super + Right will get you sorted out there without guessing how many “Alt-Tab”s you would need to get there. Your hands are already on the keyboard anyway, right?)
Yet, it remained able to do things I needed it to. In the Tabbed Mode of i3, I was able to replicate the old Alt + Tab workflow where it was needed. I could stack a bunch of things in sequence and get it done.
Minimalistic and less space taken up. i3bar and i3 with no window chrome allows me to focus on what I am doing. I’m typing on typora right now and everything is clean and with little distraction. I can full screen any application with the press of a few buttons. There is nothing to get in my way of authoring content.
Paired with Vimium for Chrome, it was a real joy to be able to drive everything with the keyboard!
It looked great and it was mine.

What was not so hot?

File Managers were kind of harder to use. I don’t always to stack my file manager. So, I made it a popover scratch window bound to Super + - and this lets it float over top of the other windows. This is handy when I need to just recall something very fast and don’t want to have to place it somewhere. However, it’s not the only application…
Opening things outside of the normal work flow could be annoying and take some forethought when it used to be effortless. For example, opening that JPEG of that puppy? Well, it’s going to split your screen in some way to make room for it. However, now you can only see half the puppy! Better waste some time resizing… ah well. I’ve worked around this by specifying manual overrides for some windows to launch in full screen or to open in specific workspaces. In some extreme cases, I make them floating since they just work better like this. It’s a bit jarring but it’s not often I need to break out of this workflow since I normally use my PC these days to work on the same type of tasks.
Some games are not very cooperative with it. For example, sometimes they will spawn at quarter resolution in the middle of the screen and I have to hunt them with my mouse and then full screen them. Annoying. But livable.
Other people have no idea how to use my computer. My life partner knows how to use it when she needs to from time to time but I had to show her a few things. It’s not very friendly to people who have no idea what they are doing. However, in the end it’s my machine to work done.

I will also say this: The i3 developers have done a great job making things robust and stable. I have not had to fix or tweak my .i3config much since I created it. Things have just been working. Kudos! If you have not tried a tiling window manager and you have similar needs to me, it would be worth checking out at least. I was skeptical at first but it was a game changer for me.

Misc moments where I thought: “This is amazing!”

Forgot to sleep the PC before bed and already tucked in? Within 30 seconds, I was able to SSH in to the already installed and configured daemon (from a prior setup) and simply sleep it from the CLI.
Native SSH is awesome for administering a file server. Sure, there is Putty on Windows which is widely used but it’s just not nearly as nice. Nothing is as good as using your terminal from your own computer with all your configuration already setup.
rsnapshot is amazing and I love backing up my server with it. Part of this is the great filesystems provided and available with Linux.
Dotfiles synced with Git to my laptop is nice. Keeping configs in sync on other platforms is less than fun without some kind of cloud storage.
Being able to modify the source code of a package, and then have it still managed by my package manager. This is a great feeling, even if a bit of a hack. You can read more on my .NET Core experience to know why this was a boon but being able to just modify an “installer” that was open and fix my problem is great.
Just running youtube-dl against a site and noticing that I was already running the most up to date version and that I would not have to waste time making sure I was up to date. The same goes with other packages.
Hey, I can run the tools my server is running!
I didn’t like the color of some things on the my GTK theme. So, I opened up the theme overrides and applied some CSS. Done.
Docker is working the way I wanted it to!!
When was the last time I was nagged to do a system update, again?
Stability: I have had up time that was longer than I should admit because I really should have patched my kernel.

So, what are some of the things that bothered me more?

There’s a few things that bother me and they might have something to do with me running an odd setup. However, I wanted to give an honest recollection from someone who just dove right in.

January 2017: Infinality fiasco

I did a full system update like any other day. Then, I got a bunch of errors about repositories that I had been using being discontinued. I guess “Infinality” had been discontinued and it was now up to me to decide what to do. I was not able to update until I resolved it.

I probably spent a couple hours resolving things while I was trying to figure out what was going on. At the same time, I really needed a newer version of a package and this was kind of in the way. Arch has a pseduo-rule that you should never do partial upgrades, so that was out of the question.

In the end, I found some advice to migrate away and managed to get my system into a better state.

April 2017: Microsoft Word stops working in WINE

A day in April I did a full system update, using pacman and then tried to run my wine-prefix containing Microsoft Word. Yeah, I was using this product since I was used to it and needed something to ease the transition. It stopped working, WINE was a pain to debug and in the end I discovered I really only needed Google Documents and LibreOffice for the occasional thing. So, I never got to the bottom of this. However, it is annoying that things stop working “randomly”.

I treat this as a minor incident but I wanted to point it out since a lot of the time you will hear the rumour that “Arch does not break! It is user error if your system stops working.”

User-space applications certainly do break sometimes.

August 2017: .NET Core

I was working on a new client project and had inherited some .NET Core code that was already running in production. I had the goal of adding some new features to and re-writing the UI. I estimated this would probably take a dozen hours or so, of which maybe 10 of them would be billable. The rest of them would be new environment configuration that I was responsible for since I had not worked in .NET Core before. I was pretty excited to start especially since Microsoft had announced recently that .NET Core support would be coming to Linux. They had even announced a cool new editor that is now loved by many, known as Visual Studio Code. Alas, things were not so easy…

Arch Linux was not a supported problem. I ran to the AUR which is my response to these kinds of things to see what I could find. I found a package (that is no longer available now) and installed it. The package itself installed fine without many problems and I wrote a couple “Hello World” applications. I started up a basic ASP.NET core project as well and everything seemed fine. Success! or so I thought. As it would turn out, making any sort of database query would cause a native exception to be thrown. You can read about how I ended up resolving this here but in the end it ended up costing me over 6 hours to debug this thing and trace it down. I doubt I am the only one either, since if you end up searching for my post, it’s linked around the web a few times as well as the solution to the problem for the .NET Core 1.x series. I have not followed up to see if the .NET Core 2.x series has this problem.

In the end, it was not so bad but on Windows this probably would have worked out of the box considering it’s Microsoft’s own platform. Then again, I have vague memories of installing Visual Studio and it not starting and having no recourse to debug…

November 2017: Full Machine Breakage

I ran a full system update and there was an OpenSSL change. I took the upgrades and followed the mailing list advice but still ended up with a bricked machine. A little disappointing but it was probably my fault somehow. Still, bricking a machine is no fun. The journal had indicated something about failed updates and failure to rollback – so I left it for a while.

I never did figure out the exact problem, however, when I did ssh into the machine from a different PC and ran another full system update the next day, the problem resolved itself. +1 for being able to read your logs, huh?

Huh. I still need to dig into this one.

Reverse Engineering Tools still aren’t all the way there

In one of my previous posts, I reverse-engineered the game “Trails in the Sky” and provided an introduction to Kaitai Struct inside of it. You will notice I used a lot of NIX* tools in that guide. However, what is glossed over, is that a lot of the debugging was done on Windows with a debugger there. This is partly because the game is a Windows Executable and partly because the tools for debugging them were just better over there. There’s a couple reasons for that but it all stems to the ecosystem:

There are better memory editors on Windows, such as Cheat Engine that aided in probing the data
There are better debuggers for Windows executables there. The reason should be obvious: most developers debug Windows Executables on Windows
I was able to actually able to run the executable there without needing to have WINE running as an abstraction over top of it

GIMP is no Photoshop

Some people might try and convince you that GIMP is a fine replacement but they were probably not power users. It’s true, if you just need to move some things around on a canvas and trim them, it is probably fine. Even for basic effects, it is probably fine. However, I found when loading complicated images and trying to do operations, things were just too different. Could I have done them with GIMP? Yeah, I probably could have. For now? I will stick to the free edition of Photoshop CS2 that was given out and use my time for other things that are of more value for me. There are only so many hours in the day and re-learning an image editing tool for the simple tasks I need to do are not worth my time. For the record, I found even basic stuff like: scaling layers, adding text effects, actions, and blending modes – a bit cumbersome. I think it was a combination of unfamiliarity and the clumsy UI. It also did not cooperate very well out of the box with i3wm and I had to do some setting tweaks and set it to full screen to get it to work even remotely well.

PlayOnLinux is not all that good in a lot of cases

For some of the Windows applications I was holding onto, I was trying to run them in a wine prefix under PlayOnLinux. I figured that since I only had a couple applications that I wanted to run, as the the old Photoshop CS2, I could just use this and not worry about learning the details. Unfortunately, in a lot of cases the scripts are older and use a very old WINE version. In some extreme cases, they do not work at all anymore. I assume this is due to changing systems and things probably do not look like they did when they were written. I learned fairly quick that backwards comparability in some parts of the ecosystem is not a priority. In the end, I decided to just use a virtual machine for the occasional thing I needed and it worked great.

Misc. things that annoy me

Atom randomly bugs out and there are visual artifacts. The tree view will go away, sometimes.
Occasionally, a Linux kernel will break something on me. I am not this user, but you are not alone, user!
When a package is unsupported by my distro, it becomes up to me. The diversity of the distros is also a weakness.
Screen tearing is still a thing sometimes along with audio and video desynch on YouTube. I’ve heard “Wayland” might fix these problems – so I wait patiently.

Going to jump in? Here’s some main take aways

Make sure you know what your distribution is like and how likely you are to end up with a system that could be broken. If you can’t tolerate your machine being down for a day and needing to resolve a few issues here and there, rolling release distributions are probably not for you, at least not of the Arch based variety. I have heard good things about Solus.
Embrace the change. Do not let yourself become restricted to something Windows-esque and instead let yourself try some new paradigms. You may just fall in love. Try a new window manager. Use your package manager a lot. Automate tasks with powerful shell scripting.
Be prepared for some frustrating nights. There will be problems and a learning curve to conquer. Linux is still not perfect even in 2017
Not all your games are going to work in WINE and you will have to just accept that it is not a main goal of the Linux ecosystem.
Follow some forums on the web and become engrossed. There are several on the web, but /r/linux and /r/archlinux for me in particular were helpful on Reddit in order to keep up on how things were doing. You will be happier if you are aware of what is going on.

What’s next?

Arch Linux was cool and it let me learn a lot. I think Linux in general is pretty awesome and I want to continue using it as my daily driver. I think will probably continue to do so for a while. That being said, there are some very cool distributions that are cropping up lately that are now on my radar. I like the “DIY” experience but not the breakage that comes with it some cases. Perhaps I can find something that can meet me in the middle, but that is the joys of the Linux ecosystem, I can choose…

With the Windows Subsystem on Linux now available as well, I plan on giving that a try and seeing how much of the tools that I am using can be used there now. I have a feeling that some of the pros I have listed will become the gold standard there now, such as the NIX* tools working out of the box. Games will work a heck of a lot better there, too, I am sure. And despite the random complaints on the Internet sometimes, I would probably get less random breakages in some cases, since there are less choices to make.

But in the end? I think I will stay where I am for now. There is much more to learn and so much more to gain. Compared to the pain, I think I am still way ahead.

Misc Development Tip - Move through blocks of text faster by simply increasing key repeat rate

2017-10-16T00:00:00+00:00

Tired of slow scrolling through code? Try changing your key repeat rate

As part of an ongoing attempt to keep my development speedy and find those little micro optimizations that will make my workflow just a little bit faster, I found something I noticed while browsing some programming streams today. Specifically, some editors have very fast scroll speeds.

When I hold my arrow keys to navigate from code block to code block (when not using the navigational shortcuts) there are times where it seems like the cursor just drags on forever.

If you’re using Linux like I am, trying reduce your key repeat delay and then increase the amounts of repeat using xset:

xset r rate 150 55

It makes a huge difference in terms of being able to scale through a ton of text effectively.

Dotnet Core Getting Around System Cryptography Initialization Error

2017-02-20T00:00:00+00:00

`System.Security.Cryptography.Native` error on a web request? Upgrade `System.Net.Http` for .NET Core

I downloaded a sample .NET Core project off the web to play around with it. It was making use of web requests to proxy some data around the web. Everything was running fine until it would make an actual web request on my Arch Linux machine…

System.DllNotFoundException: Unable to load DLL 'System.Security.Cryptography.Native': The specified module could not be found.
 (Exception from HRESULT: 0x8007007E)
   at Interop.Crypto.GetMaxMdSize()
   at Interop.Crypto..cctor()

Ugh! This would happen every time I would make an outbound network request on 4.1.0 of System.Net.Http which is the version that was being used. The solution? I simply upgraded to the latest version, which as of writing today, is 4.3.0 The problem went away.

I am not sure if this was a bug being triggered or not but it seemed to have worked. Frustratingly, this was originally a red herring for my other problem that I had a few weeks ago – oh well. :(

I hope this helps someone else.

Dotnet Core Global Json Make Sure It Is At Your Root

2017-02-19T00:00:00+00:00

.NET Core: Make sure global.json is at the root of your repo

I was recently writing some .NET Core code and I was getting some strange error messages. It was something like…

"Can't parse project.json(1,1)"

This was when I was deploying my application to Azure. As it would turn out, if you do not specify a global.json then Azure will automatically pick the latest version for you. This is a recipe for disaster considering I had not upgraded to the ABSOLUTE latest, as of yet.

As it would turn out, it would seem that in .NET Core 1.1 had migrated to kill off the project.json format in favour of .csproj style a la MSBuild. I was not ready to migrate yet.

The fix is simple: just add a global.json specifying your SDK version. Or it would be, anyway if your file structure was that your app was at the root of the repo. Alas, that was not me so I got the above error message over and over.

The solution was to simply drop global.json into the root directory of my source control folder (and not in the same directory as the application). Bam, everything worked.

I hope this helps someone out.

Recompiling OpenSSL with SSLv3 Support on Arch Linux for use with .NET Core

2017-02-06T00:00:00+00:00

NOTICE: As of .NET Core 2, this is no longer a problem. If you can upgrade your project and you were considering it already, you might as well do it to avoid this problem if you are having it.

I was recently working on a project for a client and was trying to use the new .NET Core technology by Microsoft. Of course, this is not a supported platform for my operating system of choice. However, that did not stop me from trying to run with it.

A lot of stuff works but Entity Framework does not out of the box. You will get the following error trying to run a query:

/opt/dotnet/dotnet: relocation error: /opt/dotnet/shared/Microsoft.NETCore.App/1.1.0/System.Security.Cryptography.Native.OpenSsl.so: symbol SSLv3_method, version OPENSSL_1.0.0 not defined in file libssl.so.1.0.0 with link time reference...

As it turns out, Arch Linux removed SSLv3 in this commit. There are some solutions like recompiling your application that is using the SSLv3 stuff and stripping it off. Considering my stuff was from the AUR and I had other packages that had this problem and I did not feel like recompiling them all to get my work done, I decided to just add the support back myself until I could decide how to tackle it. If anyone else is having this issue and needs SSLv3 back to get this working, you can follow the directions below:

We’ll apply patches to the PKGBUILD provided by the Arch repos to include SSLv3 support. This is important to note as some advice is to downgrade your package on various parts of the WWW. This is generally actually a bad idea since OpenSSL often has security vulnerabilities. Being several years behind on security patches is far worse.

Install customizepkg-git from the AUR.
Create a definition for the openssl package to patch. To do this…

sudo nano /etc/customizepkg.d/openssl

**NOTICE / EDIT: ** It has come to my attention after upgrading my Arch Linux box, that the OpenSSL v1 has been replaced with OpenSSL v1 Compatability library from the AUR. So first of all, make sure you have this package.. Secondly, make sure you edit the the libopenssl-1.0-compat file instead, i.e:

sudo nano /etc/customizepkg.d/libopenssl-1.0-compat

instead…. continue on. :)

and then put the following contents into the above file (most likely the compatabiltiy file if you are reading this on an updated system post June 2017)…

replace#global#no-ssl3-method#zlib
replace#global#no-ssl3#zlib

Then, just do a system update or just reinstall SSL (yaourt -S openssl). You should see the diff shown and be able to just continue installing with support.
If this fails, you may need to install the gpg key for the package, i.e: gpg --recv-keys D9C4D26D0E604491
Once the build is finished (took a couple minutes on my i5 2500) you should have SSLv3 support

… and that should enable quite a few applications on Arch Linux that have issues with this to begin working, including .NET Core. If you don’t want to enable this system wide, you can consider recompiling .NET Core’s corefx assemblies to accomplish this same kind of goal.

For me, this is enough for a development machine. I hope this helps others.

Unlinking MKVs on Windows, OSX, and Linux - Native C# Port

2016-11-27T00:00:00+00:00

You may have seen my other post on unlinking those pesky segmented MKVs that I posted a while back. Lots of users have tried it out and it works pretty well – however, the original Perl port had a few issues with it. Mainly:

It required a lot of dependencies that Windows users weren’t used to having, such as “Strawberry Perl” since it’s not something that runs on Windows “natively”
It had a couple bugs that were hard to fix, since I am not a native Perl programmer. This made it hard to make it work for every release I wanted.
It made some assumptions about the process that have since been fixed.

Since I was still getting bug reports well into 2016 about certain MKV files not working with the process and I was having some bugs myself with certain files, I decided to rewrite a quick native port in C# of the actual merge logic in order to work around these three issues.

Brute-Force Linear Ordering

The biggest bug I had to deal with was files that looked something like this:

mediainfo file.mkv
Menu
00:00:00.000                             : en:Prologue / en:Opening / en:Ending
00:05:57.019                             : en:Episode
00:22:40.059                             : en:Preview

MKVUnlink (Perl) was handling these pretty good but would occasionally trip up on these. If you look at the order inside the actual MKV itself, the chapters are presented in sequence but their timecodes are all wrong. I’m not sure if there is something I am not understanding or if this is just a bug in the way they were split out. A lot of media playback software that actually supported segmented playback actually worked find with these files so I was not sure. However, the algorithim I was using and that UnlinkMKV was using simply did not work sometimes for these files.

The solution is to linearize these, you see the implementation online at Github but the basic premise is to just take the order the chapters and for each:

Measure the duration of the current clip
Set the start timecode of the current chapter to the end timecode of the previous
Then, make the ending equal to the duration + the start of the old timecode

This will make everything nice and linear. The process only has to be carried out if the sequence is kind of out of order. Either way, this fixes a few issues with a couple MKV files that were giving me grief.

You can download this new release on Github today here which contains quite a few of the bugfixes. I hope in the future, every player can just support the segmented format and we won’t have any need for software like this. For now, if you find any bugs please leave an issue open on Github.

To use the new “Native Mode”, just tick the “Native Mode” in the GUI.

Any feedback? Experienced a similar issue? Feel free to post about it below in the comments.

Experimentation with Reverse Engineering - Trails in the Sky (FC / SC) Extracting Sprite Data w/ Unix Tools & Kaitai Struct

2016-11-16T00:00:00+00:00

I recently began and finished the great Trails in the Sky series and was amazed at some of the technical skill that was used in such an old game (developed originally in 2001) and remastered for the English release. At the same time, I was interested in the internals of reverse engineering and how such things were performed. After reading an article I found on Reddit that covered the process of decompiling a Japanese visual novel game, I decided that now would be as good of a time as ever to give it a try.

The below post chronicles what I had to do to get it prepared for examination, including unpacking and my thoughts while reverse-engineering the file format(s) provided by the game. I’ll be doing this all on Arch Linux with the aid of a Windows VM for limited functionality but you all the information here should be applicable on any UNIX system and most of it should apply to Windows with some substitution.

Examining the game folder and the data files within

The first thing we should do when examining a piece of software we want to decompile is to see what we have to work with and what seems interesting. I’m going to work with Trails in the Sky (FC) in this case but the sequel shares a lot of traits it would seem. After installing the game from the Steam client, I entered the directory and ran a quick ls to figure out what I had to work with.

touma@setsuna:common/Trails in the Sky FC $ ls -la
total 1986388
drwxr-xr-x  5 touma touma      4096 Oct 20 23:19 .
drwxr-xr-x 10 touma touma      4096 Nov  5 20:43 ..
drwxr-xr-x  2 touma touma      4096 Nov  5  2015 BGM
drwxr-xr-x  4 touma touma      4096 Nov  5  2015 _CommonRedist
-rwxr-xr-x  1 touma touma    787968 Feb  7  2016 Config.exe
drwxr-xr-x  2 touma touma      4096 Nov  5  2015 dll
-rw-r--r--  1 touma touma    165311 Oct 20 20:08 dump
-rw-r--r--  1 touma touma  35621809 Feb  7  2016 ED6_DT00.dat
-rw-r--r--  1 touma touma     18448 Feb  7  2016 ED6_DT00.dir
-rw-r--r--  1 touma touma   2468101 Dec 16  2015 ED6_DT01.dat
-rw-r--r--  1 touma touma     36880 Dec 16  2015 ED6_DT01.dir
-rw-r--r--  1 touma touma    199759 Nov  5  2015 ED6_DT02.dat
-rw-r--r--  1 touma touma      9232 Nov  5  2015 ED6_DT02.dir
-rw-r--r--  1 touma touma   8696670 Nov  5  2015 ED6_DT03.dat
-rw-r--r--  1 touma touma     18448 Nov  5  2015 ED6_DT03.dir
-rw-r--r--  1 touma touma  18093622 Nov  5  2015 ED6_DT04.dat
-rw-r--r--  1 touma touma      9232 Nov  5  2015 ED6_DT04.dir
-rw-r--r--  1 touma touma  24291816 Nov  5  2015 ED6_DT05.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT05.dir
-rw-r--r--  1 touma touma  11260428 Nov  5  2015 ED6_DT06.dat
-rw-r--r--  1 touma touma     18448 Nov  5  2015 ED6_DT06.dir
-rw-r--r--  1 touma touma  66609144 Nov  5  2015 ED6_DT07.dat
-rw-r--r--  1 touma touma     72016 Nov  5  2015 ED6_DT07.dir
-rw-r--r--  1 touma touma 226823581 Nov  5  2015 ED6_DT08.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT08.dir
-rw-r--r--  1 touma touma 152468135 Nov  5  2015 ED6_DT09.dat
-rw-r--r--  1 touma touma     46096 Nov  5  2015 ED6_DT09.dir
-rw-r--r--  1 touma touma 222776216 Nov  5  2015 ED6_DT0A.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT0A.dir
-rw-r--r--  1 touma touma 121100311 Nov  5  2015 ED6_DT0B.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT0B.dir
-rw-r--r--  1 touma touma  26658261 Nov  5  2015 ED6_DT0C.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT0C.dir
-rw-r--r--  1 touma touma   6742488 Nov  5  2015 ED6_DT0D.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT0D.dir
-rw-r--r--  1 touma touma     91156 Nov  5  2015 ED6_DT0E.dat
-rw-r--r--  1 touma touma      9232 Nov  5  2015 ED6_DT0E.dir
-rw-r--r--  1 touma touma   1164468 Nov  5  2015 ED6_DT0F.dat
-rw-r--r--  1 touma touma      9232 Nov  5  2015 ED6_DT0F.dir
-rw-r--r--  1 touma touma    221582 Nov  5  2015 ED6_DT10.dat
-rw-r--r--  1 touma touma     18448 Nov  5  2015 ED6_DT10.dir
-rw-r--r--  1 touma touma    189950 Nov  5  2015 ED6_DT11.dat
-rw-r--r--  1 touma touma     18448 Nov  5  2015 ED6_DT11.dir
-rw-r--r--  1 touma touma    441915 Nov  5  2015 ED6_DT12.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT12.dir
-rw-r--r--  1 touma touma  13191634 Nov  5  2015 ED6_DT13.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT13.dir
-rw-r--r--  1 touma touma 428173213 Nov  5  2015 ED6_DT14.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT14.dir
-rw-r--r--  1 touma touma 254700304 Nov  7  2015 ED6_DT15.dat
-rw-r--r--  1 touma touma     73708 Nov  7  2015 ED6_DT15.dir
-rw-r--r--  1 touma touma  15105502 Nov  5  2015 ED6_DT16.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT16.dir
-rw-r--r--  1 touma touma  77669890 Nov  5  2015 ED6_DT17.dat
-rw-r--r--  1 touma touma  85893124 Nov  5  2015 ED6_DT18.dat
-rw-r--r--  1 touma touma      9011 Nov  5  2015 ED6_DT19.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT19.dir
-rw-r--r--  1 touma touma  47053132 Nov  5  2015 ED6_DT1B.dat
-rw-r--r--  1 touma touma     73708 Nov  5  2015 ED6_DT1B.dir
-rw-r--r--  1 touma touma   3992401 Nov  5  2015 ED6_DT1C.dat
-rw-r--r--  1 touma touma     18448 Nov  5  2015 ED6_DT1C.dir
-rw-r--r--  1 touma touma    581636 Nov  5  2015 ed6_logo.avi
-rw-r--r--  1 touma touma 176822844 Nov  5  2015 ed6_op.avi
-rwxr-xr-x  1 touma touma   2019776 Feb  7  2016 ed6_win.exe
-rw-r--r--  1 touma touma    215120 Feb  7  2016 steam_api.dll
-rw-r--r--  1 touma touma     81768 Nov 27  2015 xinput1_3.dll

There’s a few interesting files in here for sure, but the one we’ll cover first off are the ED6_DTXX.(DAT|DIR) files as they’re the most interesting to us off a glance. Looking at the file size, it looks like these are probably archives for a bunch of game files. The DIR files also sound like a case of some kind of virtual filesystem.

For those who were not familiar for the reason this is done, it’s a very popular strategy in PC and console games. You can read a great answer on the Game Development Stack Exchange right here. To briefly summarize:

Faster loading times since there’s less seek
Full control of data, no OS
Compression can be more effective, especially if data is packed tightly

A few Google queries reveals that the company who originally developed the game, FALCOM, has been known to use a specific archive format for a while and there are well known decompression programs available. Unfortunately, I was not be able to find much in the way of documentation. One of interest for us in particular is the “Falcom Data Archive Conversion Utility”, or just falcnvrt for short.

As it would turn out, this application can decompress the DAT files into directories:

If we go ahead and instruct it to encode them into PNGs for us, we will have them converted to a friendly format. This is because as it turns out, Falcom uses a custom raster bitmap format that isn’t readable by standard programs. Since we’re mostly interested in data right now and not the raster format, we’ll skip over how this is done and just chalk it up to the good work of the community.

With that, let’s dive deep…

Locating some sprites and their metadata

Let’s take a look at what we would get from unpacking:

touma@setsuna:~/unpacked $ find . -name "*.png*" | grep -E -o "ED6_DT.." | sort | uniq
ED6_DT06
ED6_DT07
ED6_DT09
ED6_DT20
ED6_DT23
ED6_DT24
ED6_DT25
ED6_DT26
ED6_DT27
ED6_DT29
ED6_DT2F
ED6_DT33
ED6_DT35
ED6_DT36
ED6_DT3A
ED6_DT3C

OK, looks like these are some PNGs in those directories for us to take a look at. If we poke around at some of these files, we’ll find some interest things but what we can start with is ED6_DT07/CH00000.png which represents the sprite of our Heroine, Estelle. More importantly, there seems to be an accompanying file named aptly CH00000P.SCP

Let’s get some metrics on these files:

touma@setsuna:unpacked $ file CH00000.png
CH00000.png: PNG image data, 256 x 16384, 8-bit/color RGBA, non-interlaced
touma@setsuna:unpacked $ file CH00000._CH
CH00000._CH: data
touma@setsuna:unpacked $

The latter is not very interesting but the former has some information to use in the way of sizing that would be informative to us. Using a little bit of knowledge of spritesheets we can guess that each frame is 256x256. This is because often in video games we use textures and sub textures that are powers of two and square because traditionally, GPUs can render these kinds of textures better and more efficiently. There’s no science to this guess then aside from some educated guesses.

Let’s verify this claim by loading it up into an image editor and setting some grids. I will be using Photoshop as it is a tool I am familiar with but you could really use anything that you have handy. By setting up sub-grids of 256x256 across the entire sprite sheet, we can visually determine if the spritesheet would have a high likelihood of satisfying our guess.

The above is done by using Photoshop “Grid & Slices” set to 256x256 to visualize the boundaries.

Yep, the frames seem to fit snuggly.

Before going forward, it helps to know characteristics of the file we are attacking and what we might look for. In the case of a sprite file, there are a few desirable characteristics that we should fine.

A definition file will likely define the width and height of a frame if it’s not fixed
The number of frames may be present somewhere in the file
Timing and animation information may also be present
Misc headers. which may / may not be useful to us

We can keep this in mind when examining the file.

Extracting metadata

OK, next up… let’s dump the metadata and see what we can gleam on first glance. Most files start with a header, so let’s begin there to see what we can see:

touma@setsuna:unpacked/ED6_DT07 $ hexdump CH00000P.SCP | head -n 4
0000000 0040 ffff ffff ffff ffff ffff ffff ffff
0000010 ffff ffff ffff ffff ffff ffff ffff ffff
*
00000b0 0000 0001 ffff ffff ffff ffff ffff ffff

OK, let’s keep in mind that list from before. I would expect a number of frames to be near the start of a file if it was going to be declared because it’s easy to read immediately and determine how many frame blocks in this file may potentially follow. 0x004 is interesting immediately because it is 64 in decimal – remember “16384” from the height? Pick your favourite calculator – I use the Python IDLE / REPL and compute 16384/256 … which is 64, which would make sense. So, our first 8-bit block here is clearly the number of frames (note: there is no rigorous proof for this – it’s an assumption I’ve made given the information available to me. Disassembly would be the only way to know for sure.)

We should record this information. If you’re trying to follow along, you might consider installing Kaitai Struct now as I will be using it to browse data. However, you can record your notes in whatever you want – including markdown. I use Kaitai Struct because the visualizer allows me to view my assumptions and visualize the hexdump in a DSL that makes it easy to browse and test definitions. Then, when it comes time to import them into a programming language to manipulate, the compiler can take the DSL and generate an object definition for most major programming languages I use on a daily-basis. This makes it easy to interact with my new model.

You may also be interested in browsing the wiki here to be briefly familiar, though this is not required as all definitions are provided.

So, here’s what our notes will look like for the sprite.yml that we will create. This will be a definition file that is used by the Kaitai Struct Compiler and the Visualizer. By writing our notes and observations of the game data inside of Kaitai Struct, we gain the ability to visualize it quickly. For example, here is what we’re aiming to build here:

Note: Yukino in the background is my terminal background – not part of KSV

We can navigate a tree of data in semantic blocks that make sense to us rather than some homogeneous data blob. Yes, some hex editors have this ability but none of them are human readable like this and capable of generating object definitions from their DSL.

And you have to pay for a lot of them. So, what would a definition for this look like?

meta:
  id: scp
  application: TiTS Engine - Sprite Format
  endian: le
seq:
  - id: header
    type: header
types:
  header:
    seq:
      - id: frame_count
        type: u1
      - id: magic_zero
        contents: [0]

How does this work? Meta is mostly describing ceremony. id and application are user-defined fields and do not affect parsing. Endianess is declared for how the bytes should be read. If you do not understand why we picked “le”, read the notes below and read up on Little Endian vs Big Endian.

After this, we really get into the meat of the definition. seq is declaring that are about to declare a bunch of sequential blocks from the current point in the file. Since this is the outter-most level of the YAML, it is the “root”. We declare something with the id of header, which is the name which will appear on object defintions and the visualizer, and the type. A type is analogous to a class or struct. It is a container and defintion for the block of data. In our case, we know for sure the file starts with some kind of header. It also probably contains some “frame data” as the screenshot before alluded to, so to make things easier to read we declare a type of header in the types below. Then, we just declare the nested seq block and we’re good to go.

With this, you would have the header portion of the above screenshot if you ran the following command:

ksv CH00000P.SCP sprite.yml

And you should see 64 listed as the frame_count. We could even compile with ksc now to get a working class but we don’t have anything worthwhile to compile just yet.

A couple of notes of other notes:

We use Little-Endian here because this is an x86 processor game – which happen to use Little-Endian.
We declared the frame_count as 8-bits and then some “magic zero” afterwards which we state is always zero. This very well could be some 16-bit number so that frame counts can exceed 2^8… but we’re not sure on that yet without looking at more files so we play this safe.

Then, there’s the rest of these huge blocks of data. If we load this into a hex editor or just use hexdump we see a lot of FF indicating empty space in the file. It would be unlikely for a header be padded with a huge amount of FF so we can assume that perhaps the header is just 2 bytes long. Hm… looks like no frame height or width information? It seems unlikely that it’s encoded here.

If we run find . -name "*.png*" | xargs file {} we’ll notice a lot of 256 wide PNG files. Actually, all of them in this folder seem to be like this. Furthermore, if we run our ksv tool and markup against some of these SCP files all their frame counts seem to match the heights, for example a small output:

./CH01593.png: PNG image data, 256 x 2048, 8-bit/color RGBA, non-interlaced
./CH00435.png: PNG image data, 256 x 8192, 8-bit/color RGBA, non-interlaced
./CH01033.png: PNG image data, 256 x 2048, 8-bit/color RGBA, non-interlaced
./CH02280.png: PNG image data, 256 x 16384, 8-bit/color RGBA, non-interlaced
./CH02130.png: PNG image data, 256 x 16384, 8-bit/color RGBA, non-interlaced
./CH00128.png: PNG image data, 256 x 14848, 8-bit/color RGBA, non-interlaced
...

OK, probably safe to assume that this data is hard-coded into the game engine somewhere and not stored here. Or it’s stored somewhere else.

That leaves us with a large file with some sparse data, though. When you see a lot of sparse data like this, you should think fixed length records … and I bet some frame data is encoded in these records. However, we should verify that this seems plausible and where it starts and ends.

stat the file to get a size and get started. i.e: stat CH00000P.SCP to get a size of 32770, in bytes. Now, it’s time to think.. if there was a record for each frame, how big would each record have to be? (32770-2)/64 in this case (remember: we sliced off two bytes for the header) which is a nice, round, 512. Remember CH01593 above? If we stat this file we get 4098… if we use the same logic then we get (4098-2)/8 = 512. It’s likely, but maybe not completely true, that we are looking at 512 byte records if we follow this train of thought. Let’s draft it out in Kaitai Struct and see what it would look like:

meta:
  id: scp
  application: TiTS Engine - Sprite Format
  endian: le
seq:
  - id: header
    type: header
  - id: frame_entries
    type: frame_entries
    repeat: eos

types:
  header:
    seq:
      - id: frame_count
        type: u1
      - id: magic_zero
        contents: [0]
  frame_entries:
    seq:
      - id: junk
        size: 174
      - id: data1
        type: u1
        repeat: expr
        repeat-expr: 4
      - id: junk2
        size: 334

A couple new syntax elements here. The most important in my opinion is the repeat: eos which is just saying create frame_entries until the file ends. This makes sense since we asserted that after the header, it would be all frame data. I have filled in some junk and data fields which might not make sense right now but they will once we load things up in ksv. I arrived at these offsets through dumping and examination but it will be more obvious where they come from once you load it up…

ksv CH00000P.SCP sprite.yml

Huh. Do you notice incrementing sequence of numbers in each block that are the same “offsets” apart? There’s a bunch of increasing sequences in the data actually if you look at it through the visualizer like this. The visualizer is really handy in letting you scope your search to what you believe to be localized data. I wonder what those could be for… :) On first glance, it feels like they could frame numbers or some kind of counters. Considering they’re the only data in here, one would have to imagine that perhaps they layout the sequence of the frames when played back. There’s a lot of possibilities – to figure it out, we will need to do one of two things:

Once again, consider the characteristics of this file and what we think it might contain
Disassemble the executable and read the resulting ASM code to how a loading routine for this file would marshal data around

When reverse engineering, it is a good idea to examine other test subjects time to time as well to verify claims. In this case, if we run ksv against CH01593P.SCP we’ll actually realize that “data1” is full of FF. Doh! And there are sequential clusters as well again but they do not start from 0 like the other definition. To learn more, we should study the makeup of each spritesheet and image how it would need to jump from frame to frame to be able to visualize this properly.

However, this post is getting pretty long as it stands, however, so we will wrap things up here. I wanted to provide a practical way of reverse-engineering data structures in Trails in the Sky using UNIX tools and Kaitai Struct and for that, I think I have succeeded. In the future, we may load these sheets up into an image editor to examine what is different and similar between them and their data files. (Hint: Things that move in all four directions like a PC Avatar are represented the same but different than say, a spell animation which flows in one linear sequence rather than jumping around in one of four directions like a PC Sprite.)

For a follow-up, would you prefer to see us reason this out again with more UNIX tools or an attempt to attack this file with static analysis by disassembly? Let me know in the comments – or feel free to just leave feedback or questions.

Scraping the web with X-Ray utilizing Node.js - An example using Google

2016-06-04T00:00:00+00:00

I recently was working on a project for a client that required going through some data on the web and extracting some information, known in most circles as data scraping. Normally when I need to do something like this, I will use Cheerio alongside something like Superagent or just request to download pages and then extract information with a jQuery-like API. This is normally great for dealing with single pages or a list of similar pages that I want to get information from. For example, consider scraping some Wikipedia articles from a known list. Then, we can just run requests to grab the HTML contents and then use Cheerio to pick out what we want. However, I had some extra requirements on this project that made this approach not as straight-forward:

I needed to be able follow links on pages given some criteria and then easily extract information out from
I needed to be able to paginate these pages, that is follow a series of links until my results were exhausted
I needed it to do the heavy lifting for me (Cheerio is purely a DOM-API library) to save time and get the job done.
I needed rate-limited, which I have built before in the past, but it would be nice if something handled it for me.

That’s where X-Ray comes in – it so happens that it uses Cheerio and request as dependencies, as well. It’s an abstraction against these two and makes scraping the web easy. Below, I’m going to illustrate how to scrape Google search results and follow their links for more information. —

Setting Up

Getting started is pretty easy, just download the package using npm:

npm install x-ray@2.0.2 --save

And then, inside of your Node.js file just do:

var Xray = require('x-ray')
var x = new Xray()

You’re set!

Why 2.0.2?

At the time writing, there is an open issue for the newer versions (Issue #65) that prevents some of the things we want to do. You should try the later versions if you are reading this in the future.

Querying Google

You can read the syntax on the X-Ray page to get a better idea of what is going on but I will try to explain below, what is going on. First, our goal:

We want to be able to programmatically run a Google Search and extract the URL of each of those links, and then in turn, we want to be able to visit each of them and extract:

The title of the page it links to, after viewing it
The very first link on each page, that is the first a tag and it’s href attribute

Surprisingly, this is pretty easy to in X-Ray, we can do it like so:

x("https://www.google.ca/search?site=&source=hp&q=spice+and+wolf&oq=spice+and+wolf", '.g', [{
   link: 'a@href',
 }]).paginate('#pnnext@href').limit(5)
 ((err, obj) => {
   if(err) {
     callback(err);
   }
   else {
     obj.forEach((item) => {
       var start = item.link.indexOf('q=')
       var end = item.link.indexOf('&sa')
       item.link = item.link.substring(start + 2, end)
     })
     callback(null, obj) // do something with the objects?
   }   
 })

A couple interesting observations here:

There is no explicit request code, no load or anything l ike that into a Cheerio object.
There are no temporary variables or forming of an object manually, like we may do with Cheerio
All the error handling code for elements is missing. As it would turn out, X-Ray handles this and forwards it to the err parameter in the callbacks.
.paginate is present and so is .limit. These are actually part of the fluent API that is provided to us – we can simply provide a selector to the “clickable next button” on the page and x-ray will internally paginate the results, iterating over the next 5 pages in this case, adding even more results to our internal array.

The 2nd parameter in the x function is just the scope in which selections should be done on. The 3rd is kind of strange – it’s an array of objects wrapped. By wrapping in square brackets, we are instructing X-Ray to iterate once for each .g, without the brackets, it would execute once. So, the flow goes something like this.

  Download the page
  For each element in the set of .g ...
    Extract the first *a* element's href
  Return the result

The only annoying portion is the link stripping and that is because of the way Google formatted the links at the time of writing. It’s actually kind of a hack but I am sure there is a better way to extract it out if you want to.

Then, the only step left is to visit the links and process them. As it would turn out, you can compose in X-Ray to make this a snap as well. Consider the following:

x("https://www.google.ca/search?site=&source=hp&q=spice+and+wolf&oq=spice+and+wolf", '.g', [{
   link: 'a@href',
   details: x('a@href', {
      title: 'title',
      link: 'a@href',
      linkText: 'a@text'
   })
 }]).paginate('#pnnext@href').limit(5)
 ((err, obj) => {
   if(err) {
     callback(err);
   }
   else {
     callback(null, obj) // do something with the objects?
   }   
 })

Notice the new details section? This will execute for each page and extract out:

The title tag
The first link location of an a tag
The first link a tag’s text

We’d have to write a lot of boilerplate spaghetti to do this without our framework. So, if we then go ahead and print out of some this using a basic loop, (printing title and link URLs for now), we get something like:

Watch Spice and Wolf Anime Episodes Streaming on FUNimation :::: http://www.funimation.com/loadModal/index/forgot_password_modal
Spice & Wolf Wiki - Wikia :::: http://spiceandwolf.wikia.com/#WikiaArticle
http://www.google.com/ :::: https://www.youtube.com/watch%3Fv%3DI4Ztnt5ej20#
Ookami to Koushinryou (Spice and Wolf) - MyAnimeList.net :::: http://myanimelist.net/
Page not found :::: http://hachettebookgroup.com/
Spice and Wolf (TV) - Anime News Network :::: http://www.animenewsnetwork.com/
Ookami to Koushinryou - Anime - AniDB :::: https://anidb.net/animedb.pl?show=main
Ôkami to kôshinryô (TV Series 2008– ) - IMDb :::: http://pubads.g.doubleclick.net/gampad/...

Cool! We could evidently extract more information from the pages if we needed to. Since we decided to maintain a link on the top-level of each object as well, we could run this through another X-ray parser to fetch site specific data depending on the URL, if we needed to. For example, if we detected a YouTube video / a video site, we might also be interested in, say it’s view count.

Conclusion

So, if you have some requirements that I had – you should definitely check out X-Ray. Specifically, if you need pagination, easy robust requests, rate limiting and a nice execution flow you will get it here. If you want to see an example of this code and chaining requests, you can find it on my Github page.

Setting up zeroRPC on Windows with Node.js

2016-05-30T00:00:00+00:00

zeroRPC is a really cool messaging system for distributed communication between server and client applications. In the past, I have used it to have my Node.js applications offload their workload to other machines. You can already do this with Cluster in Node.js land or something like Node Workers – but the super cool part about RPC libraries such as zeroRPC is that it’s just a protocol in the end. One handy feature of this is that the client and servers not need be written in the same language – only speak the same protocol.

You can read more about zeroRPC and why you might want to use it on their site here. On this post, I am going to focus on a much more sinister issue I had to deal with when setting it up on Windows. I typically run a Linux workstation, but recently a few collaborators I was working with were on Windows machines. This machines I had to make the install and build process iron-clad with Windows machines, too. zeroRPC probably was one of the more tricky packages to install. Below is a small series of memo’s to myself and any future users who may need to get this working.

Setup Instructions

Install zeroMQ

zeroMQ is a messaging library and zeroRPC is built on top of it. You’re going to need the latest windows installer and use that. You can grab the latest windows installer here. You will probably want “x64 build for Vista, 7, 8, Windows Server 2008 R2 and newer.”

Install a connector of your choice

On the offical site, there is a couple options, mostly Python and Node.js. If you plan on using Node.js like I did, then just run:

npm install zerorpc --save

Installing Visual Studio 2015 for MessagePack

zeroRPC uses MessagePack and this is not documented very well on the home-page (most NIX* systems would just pull this as a dependency). Windows users will need Visual Studio 2015 with C++ installed in order to be able to build it during the Node.js connector npm install Important: Visual Studio 2015 does not install C++ by default. You will need to select it manually during the install process.

Migrating from older Node.js versions

If you are using an older box like I was with some other installs and an older version of node-gyp, it is likely you will need to reconfigure your node-gyp to point to compiling with the latest MSVS. Some native libraries will require you compile with a newer compiler to work. To do this, execute the following:

npm install -g --msvs_version=2015 node-gyp rebuild

Python Bindings

zeroRPC is usually used to talk and communicate with other applications. If you plan on using the Python version and you are running Windows, you will likely find this link at Stackoverflow a neccessity, as I did: http://stackoverflow.com/questions/13395458/how-to-install-zerorpc-python-on-windows/28386060#28386060

And that should be it…

That’s what it took for me. I hope this helps somebody out!

Unlinking MKVs on Windows, OSX, and Linux

2015-04-30T00:00:00+00:00

For the longest time, the video format of choice was XVID or other types of encodings held in AVI containers. Howevever, more and more internet content is being distributed in Matroska (MKV) containers. While being more efficent and typically being used with some very good codecs like H264; there has been a price to pay for some slashing file sizes. A large trend of late has been to cut media files into several files and make use of the MKV linking feature to insert segments of video into mutliple files, while only storing it on the disk once. This sounds pretty great in theory but some of my media players of choice like Plex do not support that part of the spec.

So, I set out for a way to unlink and put these files back together. There’s a great project called UnlinkMKV written in Perl that did what I wanted to do. It does not run on Windows or OSX properly completely at time of writing, so I set out to port it and create a pull request. I didn’t want to settle for just some command line script port, though so I went overkill.

I developed a full GUI interface overtop that can be used by anyone to solve this pesky linking problem. Details on how to use it below.

Using UnlinkMKV-GUI

It’s a portable application running on the .NET Framework / Mono, you can grab the latest release here and read the setup requirements here for your operating system. Setup basically boils down to having Mono, a working Perl install, and MKVToolNix. If you know you have these already, you can move along. Otherwise, feel free to follow the guide for your operating system in the README.

If you’re on OSX or Linux, after downloading the release you can just run it with mono UnlinkMKV-GUI.exe

Using the application

If everything is good, you should see a screen like below:

Select the folder where your segments (linked video, along with the connecting segments) are for the “input folder” and then select “output folder” to be the folder where you want the merged files to be output. Then, just click “Unlink”!

With any luck, just a bit of waiting will be all that is required to have your files waiting in output. If it has some problems, the following options below are available and described. If they still cannot fix your problem, cut me an issue on GitHub and I will try and get it fixed up.

Fix Audio/Fix Video: These will transcode the video into a new format. This isn’t really reccomended and only required in a few cases. The unlinking will take a long time and you will lose some quality.
Fix subtitles: If the subtitles don’t appear in the finished product, try ticking this.
Ignore default flag: Same as above, try ticking this.
Ignore missing segments: You can use this if you’re sure you want to skip missing segments
Verbose Output: You should use this mode when attaching a log for a bug report, will include important developer information

Thanks to Garret Gnoling for the original release. Pending pull request approval, the original script should be updated for cross platform, too.

Installing Jekyll on Windows with Ruby 2.2.2

2015-04-15T00:00:00+00:00

Before migrating my blog over to Jekyll, I did some research and found Jekyll was probably a fairly matured static generator that had a fair degree of running on Windows with no issues. Actually, it was this guide in particular that I followed mostly. Though, if you have as much bad luck as me sometimes, well you’ll probably run into a snag using the latest version of Ruby; for me that’s the 2.2.x revisions.

Issues with Hitimers

The issue

If you receive the following error when trying to run jekyll serve:

C:/Ruby22/lib/ruby/2.2.0/rubygems/core_ext/kernel_require.rb:54:in 'require': cannot load such file -- hitimes/hitimes ( LoadError)

Then, chances are you just need to run the following commands in your command prompt:

gem uni hitimes

gem ins hitimes -v 1.2.1 --platform ruby

Explanation

It would turn out that typically, there is a fat binary that is rolled into the gem installation of hitimes. However, hitimes has only been around since January and the authour hasn’t yet released a fat binary for it yet. Basically, there is no Windows support. Running the above code, we’ll get around the issue until hitimers has an update.

(Credit to this Stackoverflow answer for the solution.)

The blog of Vaughan Hilts

Installing and running HorizonXI on Linux

Install HorizonXI - wine-staging (non steam-deck; steam deck below)

Prepare a version of Wine with the proper patches

Install HorizonXI - Linux Desktop using Lutris (non steam-deck; steam deck below)

Requirements

Installation

4GB Patch

Windower 4

Version

Install HorizonXI - Steam Play (Steam Deck / other systems)

Automatic installation

Manual installation

Gamepad

FAQ for Linux

The launcher keeps crashing while trying to install the game

How can I update the launcher?

My game is asking me to install again. What happened?

Launching gamepad configuration prevents my game from launching anymore

I can’t see buttons on the UI

The launcher is showing all black

I’m getting told I can’t run it as admin

4GB Patch

Where can I get artwork for my install?

Finding all the chest messages in "Trails from Azure"

Playing Playstation remotely over Wireguard

Running official apps over a VPN

Running aftermarket over a VPN

Installing and running WingsXI on Linux

Downloading WingsXI

Prepare a version of Wine with the proper patches

Preparing a prefix for WingsXI

Running WingsXI

Correcting gamma

Configuring XI / Running the Ashita GUI

Addressing performance

“Make FFXI beautiful 2021”

Did I miss something?

Investigating how my local home router is resolving hostnames automatically

Figuring out how dnsmasq would get the hostname and add it to DNS

Installing OpenMediaVault without a network connection at time of install

Installing the system

Resolving network connectivity

Fix the broken sources from the install process

Checking everything else out

Replacing the fans on a Zotac GTX 970 (Dual Fan Model)

Diagnosing the fans vs. the fan controller

Getting me through the days

Installing the new fans

Remove the splitting connector

Clean out the dust

Make sure the fans spin right before assembling; make sure wires won’t touch the blades

Pay attention to how the casing came off

Done!

Writing a fast (mostly) Binox solver

What is the game?

The simplest thing that can solve this

Automating it all

Further Optimization

Swapping from macOS and Linux, work to home... things I miss from each platform

Things I have in my Arch Linux setup I wish I had there

Tiling Window Manager

My software just upgrades without much trouble

Native Docker

Nothing gets in my way

I can use the same tools to adminster my server as a workstation

Being able to choose when to adopt new things

Things I miss coming home from my macOS machine

Performance

Software Support for “some mainstream things”

Hardware Support

Installing Final Fantasy XI and dual boxing on the Nasomi Private Server (Linux)

The Guide

Other things you may want to know

Using Huginn for profit and fun; finding price errors on the Internet

Setting up a plan

Getting the data into our stream

Filtering the data using regular expressions

Notifying yourself when it happens

Profit???

Install HorizonXI - `wine-staging` (non steam-deck; steam deck below)

Figuring out how `dnsmasq` would get the hostname and add it to DNS

`System.Security.Cryptography.Native` error on a web request? Upgrade `System.Net.Http` for .NET Core