• About

Brian Richardson's Blog

  • Secure your webapp with RBAC using KeyCloak and .NET 6

    December 1st, 2022

    I’ve spent a bit of time working with KeyCloak lately. It’s been some time since I looked in the Open Source world for an OIDC/OAuth2 solution, and when I found KeyCloak, I thought, “How did I miss this?”. I’ve been working with an ancient OIDC framework available for .NET, whose name escapes me right now. Later on, I came across IdentityServer4, now IdentityServer5, available as Duende IdentityServer under commercial license.

    But KeyCloak was developed quietly by Red Hat, and seems to have gained some traction. Indeed, it is a highly capable authentication server, supporting the OIDC protocol, SSO complete with user provisioning via SCIM, and a complete OAuth2 implementation for more advanced scenarios. For this article, I’ll discuss the more basic approach of RBAC using KeyCloak and the built-in authn/authz available in .NET 6.

    Install KeyCloak

    KeyCloak is relatively easy to install:

    1. Download the binary package from https://www.keycloak.org/downloads.html
    2. Unzip/untar the binary package in a system directory somewhere
    3. Create a service that launches the KeyCloak server from the distribution

    I’ll give instructions for Linux, but I imagine it should work equally well on any machine with a reasonably recent version of Java. I untarred under /opt, which creates a directory /opt/keycloak-20.01. I link this to /opt/keycloak.

    We have to bootstrap the service before we can install it. We will need an initial username and password, and those will be set in the environment variables KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD respectively. Before we start that, though, we need to install and configure Java and MySQL. I’ll leave this for the reader, as it’s usually just a simple matter of installing the openjdk-18-jdk and mysql-server packages.

    Next, we need to modify /opt/keycloak/conf/keycloak.conf as follows:

    # postgres is supported too if you prefer
    db=mysql
    
    # Create this user
    db-username=keycloak
    db-password=keycloak
    
    # The full database JDBC URL. If not provided, a default URL is set based on the selected database vendor.
    db-url=jdbc:mysql://localhost:3306/keycloak
    
    # HTTPS - requires root privileges or change of port
    https-protocols=TLSv1.3,TLSv1.2
    https-port=443
    
    # The file path to a server certificate or certificate chain in PEM format.
    https-certificate-file=${kc.home.dir}/conf/server.crt.pem
    
    # The file path to a private key in PEM format.
    https-certificate-key-file=${kc.home.dir}/conf/server.key.pem
    
    

    Start the service manually as follows:

    # KEYCLOAK_ADMIN=admin KEYCLOACK_ADMIN_PASSWORD=changeme /opt/keycloak/bin/kc.sh start
    

    This will enable you to login to the service at http://localhost:8080 with the username and password set in the environment. Your first order of business should be to create a new administrative user with a secure password, and disable the admin user.

    You can now stop the running service (Ctrl-C in the terminal in which you ran the kc.sh command. It is time to replace it with a proper service file and have KeyCloak start automatically on boot.

    # vi /etc/systemd/system/keycloak.service
    

    Use the following contents:

    [Unit]
    Wants=network-online.target
    After=network-online.target
    
    [Service]
    Type=simple
    User=root
    ExecStart=/opt/keycloak/bin/kc.sh start
    
    [Install]
    WantedBy=multi-user.target
    
    

    Finally, set KeyCloak to auto-start:

    # systemctl enable --now keycloak.service
    

    Configure KeyCloak

    The first thing we will need is a new realm to manage our application’s users. Create a realm by logging into the server (hopefully you took the time to configure HTTPS!). In the top left corner, there is a dropdown that will list all the current realms:

    You can add a new realm from the button on the bottom. Give it a name, and save it. This will bring you to the main configuration screen for your new realm:

    There’s a lot here to configure, but don’t worry about most of it for now. A lot of the options are related to security policies and automatic enforcement of scopes and permissions using OAuth2 Resource Server flow. This is an advanced topic that this article will not cover.

    For our purposes, we will configure just the name. We will use the Client settings to configure our RBAC. So, create a new Client by selecting Clients on the left navigation, and clicking Create. Fill in a name, leave the protocol on openid-connect. You don’t need to fill in the Root URL, but you can if you like.

    Now you are at the main configuration screen for your new client:

    We are only interested in roles. Go to the Roles tab and add any roles you might need (I used Administrator and User, making Administrator a composite role that contained User as well). You can then assign these roles to individual users in their details screen.

    So adding users with roles is easy enough. How do we inform our application of those roles? We need to put a claim in the access token that will declare our roles to the application. KeyCloak’s built-in mapper for User Client Role will put those roles in a JSON block within the token as follows:

    resource_access: {
        bookstore: {
            [ "Administrator", "User" ]
        }
    }
    

    Unfortunately, .NET 6 won’t interpret these roles out-of-the-box, so we need to give it a little help. Help is provided in the form of a class extending AccountClaimsPrincipalFactory<RemoteUserAccount>. The base class provides a virtual method, CreateUserAsync(), that will construct the ClaimsIndentity given an access token (well, more specifically a token accessor – more on that below). The entire class looks like this:

    public class KeycloakClaimsPrincipalFactory : AccountClaimsPrincipalFactory<RemoteUserAccount>
    {
        public KeycloakClaimsPrincipalFactory(IAccessTokenProviderAccessor accessor) : base(accessor)
        {
        }
    
        public override async ValueTask<ClaimsPrincipal> CreateUserAsync(RemoteUserAccount account, RemoteAuthenticationUserOptions options)
        {
            var user = await base.CreateUserAsync(account, options);
            if (user.Identity is ClaimsIdentity identity)
            {
    
                var tokenRequest = await TokenProvider.RequestAccessToken();
                if (tokenRequest.Status == AccessTokenResultStatus.Success)
                {
                    if (tokenRequest.TryGetToken(out var token))
                    {
                        var handler = new JwtSecurityTokenHandler();
                        var parsedToken = handler.ReadJwtToken(token.Value);
                        var json = parsedToken.Claims.SingleOrDefault(c => c.Type == "resource_access");
                        if (json?.Value != null)
                        {
                            var obj = JsonConvert.DeserializeObject<dynamic>(json.Value);
                            var roles = (JArray?) obj?["bookstore"]["roles"];
                            if (roles != null)
                                foreach (var role in roles)
                                    identity.AddClaim(new Claim(ClaimTypes.Role, role.ToString()));
                        }
                    }
                }
            }
            return user;
        }
    

    Note that we use the TokenProvider provided by the base class. This is an IAccessTokenProvider, which will use the IdP token endpoint to fetch a fresh access token. This is important to note, because if we are not yet authenticated, we obviously cannot get an access token, hence the need to ensure that we are receiving a valid token response prior to proceeding.

    The key line here is var roles = (JArray?) obj?["bookstore"]["roles"]. A JArray works very much like a Javascript Array, and can dereference multiple levels of a hierarchy using array notation. Once we have the roles, we simply add the claim to the identity using the expected claim type and return the updated identity.

    Now that we have an access token with the proper claims, we should be able to simply use the following service declaration:

    builder.Services.AddOidcAuthentication(options =>
    {
        builder.Configuration.Bind("OIDC", options.ProviderOptions);
    }).AddAccountClaimsPrincipalFactory<KeycloakClaimsPrincipalFactory>();
    

    (Note – this is Blazor WASM. You will need to use an appropriate package for Blazor Server to do the same thing). You will also need an appsettings.json with the following content:

    "OIDC": {
        "Authority": "https://dev-keycloak.example.com/realms/Bookstore/",
        "ClientId": "bookstore-dev",
        "RedirectUri": "https://localhost:7004/authentication/login-callback",
        "ResponseType": "code"
      }
    

    Now the final step is to use AuthorizeAttributes to control access:

    @attribute [Authorize(Roles = "Administrator")]
    

    That’s it! You can now use the .NET 6 RBAC to interact with KeyCloak.

    Advertisement
  • What’s So Great About Remote Development, Anyway?

    November 8th, 2022

    You may have come across one or more of these products in recent months: GitHub CodeSpaces or JetBrains Space. So what, you say, why would I care about coding in my browser? I have a good computer, why do I need to screw around with remote connections, etc.? As it turns out, this journey for me started from a desire to code in my browser…

    I like my iPad. It’s really good at many, many things, but it simply doesn’t have any native coding tools, nor is it likely to ever have them. Originally, I came across a project on GitHub called code-server. I think this project has now morphed into a couple of different forks now, and I don’t even know if the original still works since it seems to have forked into commercial offerings. In any case, this program allowed me to run VSCode full screen in Safari. That seemed to work reasonably well, but it felt pretty clunky.

    CodeSpaces was a lot better than VSCode. I could add individual projects to my iPad home screen and get a near-native experience that was quite usable. I more or less left it at that until I came across JetBrains Space. Space offered the same thing, but for JetBrains IDEs. But the client is no longer a simple HTML5 client. It’s a full thick client written in Java. Again, this will never run on the iPad. I put it aside since it didn’t seem to serve any immediate purpose.

    But then secure coding discussions came up. How do we prevent keep source code from leaving the organization? How can we support developers as the infrastructure components of our growing application grow accordingly? I remember when I first implemented RabbitMQ and provided instructions to developers on how to install it locally, there was a lot of grumbling. And, to be fair, they had a point. “Local Development Workstations” are not really a thing anymore given that many resources required by cloud applications don’t even have local analogues, or have very limited local analogues.

    I thought about this for a long time. Remote development seemed like a good fit, but it just felt so clunky. Now there was a reason for it, though, so I tried again. After I figured out what the roles of the various JetBrains products are (Space Desktop, Gateway, IntelliJ-based IDEs), it actually seemed like a pretty good solution. I was able to run Rider IDE on an Azure D2. Latency through OpenVPN hovered between 70-100ms with the occasional spike over 100ms. System load for the IDE with a basic project is very reasonable for the debugging cycle. I could see needing a D4 for larger projects, though. Still, 4 cores is a relatively small VM and reasonably inexpensive when running Linux. (< $200 CAD/mo).

    It’s important to put aside concerns about the idle desktop PC. That is not the problem we are trying to solve. We’re not even trying to figure out how to code on the iPad anymore (though that is a nice-to-have). The problem we are trying to solve is one of security and technical support. Security and Support have requirements that impose on developers slightly. It’s our job to make sure that imposition is as painless as possible.

    One of the ways we can do that is to build shared infrastructure that developers no longer have to support. In my RabbitMQ experience above, I ended up having to do just that: build shared infrastructure so I didn’t have to support multiple developer workstations. Instead, I only had to support a single development cluster and come up with a way the application could share a single configuration file and cluster, while still allowing for easy developer onboarding.

    So, our massive local development workstation turns into a series of smaller purpose-built VMs and PaaS services and a netbook (or a developer’s personal device, if the corporate security can accomodate it). I’m not going to pretend this is cheaper than buying a laptop every couple of years. Indeed, a quick estimate shows that the cloud environment is about twice as expensive. This is a trade-off that you’ll have to decide on yourself. In our circumstances, the security and infrastructure support concerns win out for us, ymmv.

    It took me a long time to see the benefits of remote development. For the individual, beyond the very narrow use-case of coding in a browser, it’s not really there. It’s not until you start worrying about your code and data walking out the door, or how you’re going to support that developer in India that any of these costs start to make sense. But, remote development does provide solutions to both of those concerns that are not easily satisfied by issuing physical hardware to developers.

    To conclude, it looks like Microsoft has hopped on this bandwagon too with their vscode-server, currently in private preview. If it’s anything like code-server, it should provide a similar in-browser experience to code-server, with hopefully less of the clunkiness. Having used github.dev, vscode.dev, JetBrains Space and Gateway, I think I can say I’ve covered most of the gamut of possibilities. The JetBrains offering definitely feels the most integrated right now, but doesn’t run on the iPad. Maybe one day I’ll get both!

  • Arch Linux for Ubuntu users

    October 27th, 2022

    Software Management

    I’ve spent some time now in the Steam Deck CLI, and gotten a good flavor of Arch Linux from the perspective of a long-time Ubuntu user.

    Honestly, I like it a lot. It’s really a unique OS that offers a lot to many different users. For your average Steam Deck user who dabbles in the desktop, the Discover software center, based on Flatpak, will be all you need. For the power-user, though, it’s a little more complicated based on what you need. There are 4 programs that you can use to install software: the Discover software center, the pacman tool, the yay tool and the makepkg tool. All of these serve different purposes.

    The basis of all of these tools is makepkg. Without going into technical detail, makepkg works by reading a build specification and producing a binary package. This is similar to other distributions that provide a “build-from-source” mechanism such as source RPMs. pacman and yay seem to have a fair bit of overlap. pacman reads only from system repositories though. I like to think of pacman as the super-user yay. For installing system libraries and upgrades, it is ideal. Crash course:

    pacman -Ss <search> – search for a package
    pacman -S <packagename> – install a package
    pacman -Sy <packagename> – update spec from repository and sync package
    pacman -Syu – update all specs from repository and upgrade all packages

    The Arch User Repository (AUR) contains contributed binary packages and build specs from around the world. These packages typically install from source code, and are based on build specs instead of binary packages. You install these packages with the yay tool, which has the same invocations as pacman, but is not invoked with sudo. You will be prompted for sudo credentials if required.

    Finally, you can make your own packages with makepkg. From a directory with a build spec, simply type makepkg -si (no sudo required). The spec will allow a binary package to be built and installed.

    Overall, I find the AUR to be too unstable. I know I harp on Neovim, but it’s an important tool for me. The “stable” release of Neovim (i.e. from pacman) is only at 0.6. I need a minimum of 0.7. Theoretically the “unstable” release of Neovim (neovim-git), is at 0.7. But because it is a git repository, the package information contained in the AUR is wrong – the git repository has since been updated to 0.8. So, yay -S neovim-git doesn’t work. yay -S neovim is too old. That leaves only flatpak, which thankfully has Neovim 0.8.

    From all of this, I’ve learned that the Flatpak repository is probably the best way to go for software. While the pacman system repositories are stable, the AUR most definitely is not. I’d consider it a last resort before building it from source yourself. Flatpak is also stable, and allows installation of important userspace applications.

    System Services

    Arch Linux is also built on systemd and uses systemctl to enable, disable, start, and stop services.

    Desktop

    While there’s no “official” desktop for either distribution, SteamOS defaults to KDE Plasma 4, and Ubuntu to GNOME 3. Arch Linux itself is not a graphical distribution, and allows installation of many desktops. This is the first time I’ve used KDE Plasma, and my first impressions are very good. I haven’t used KDE in a long time, and I’m surprised to see how stable and shiny it is.

    Support

    I think that if you require commercial support, you’ll want to stick with Ubuntu. Arch Linux is very much a DIY OS, and isn’t really advisable from a business perspective. However, most sites that have Linux software provide instructions for Arch Linux; or, the AUR contains a build spec.

    Conclusion

    Arch Linux goes on my “approved” list. I have a good deal of flexibility in how to get software onto the machine. It is similar enough to Ubuntu that I didn’t get lost, but is clearly a unique OS that is meant more for power users than casual users. It’s a decent choice on Valve’s part, one that has turned a great purchase in the Steam Deck into a very viable development PC.

  • Arch Linux on the Steam Deck?

    October 23rd, 2022

    I think I am obsessed with Neovim. After finding out that the default Neovim currently distributed with SteamOS is only 0.6, I became determined to install the current git version, 0.8. This led me down an interesting rabbit hole, which while ultimately unsuccessful, may be possible in the future.

    The idea is simple enough: install Arch Linux and use the linux-steamos AUR package to get the full hardware support. But it just didn’t work out that way. Installing Arch Linux is simple enough. Just image a USB and boot from it. On the Steam Deck, this is accomplished by plugging a bootable USB into the dock and holding down the volume down key while powering on until you hear the chime.

    Following the Arch Linux installation instructions was easy enough too, once all the required packages were installed. First, yay-bin from GitHub, then use yay to install update-grub (This is apparently just a convenient alias for grub-mkconfig).

    Getting it to boot the default OS is easy enough, and so is installing the KDE Plasma desktop (yay -S plasma xorg sddm plasma-wayland-session). Unfortunately, the current Arch Linux kernel doesn’t support the Steam Deck audio or bluetooth hardware. This, of course, makes it quite unusable for gaming.

    So, there’s a package in the AUR called linux-steamos. This is theoretically the answer to the problem above, since it would have all the necessary hardware support. However, after several hours, I find I am unable to get the system to boot using the new kernel.

    My first attempt, I used an ext4 root file system, but continued to get error messages about an unknown root file system type. And indeed, there’s no ext4 module for grub to insmod. I recalled that the default SteamOS is using btrfs. So attempt #2 was to use btrfs. Again, I got complaints about unknown filesystem type, even after dropping into the grub shell and manually entering the boot commands:

    set root=(hd0,gpt2)
    linux (hd0,gpt1)/vmlinuz-linux-steamos root=/dev/nvme0n1p2 rw
    initrd (hd0,gpt1)/initramfs-linux-steamos.img
    insmod btrfs
    boot

    This has me stumped, and I’ve given up and gone back to stock SteamOS. I guess I’ll have to make do with Neovim 0.6 until the next SteamOS update. For posterity, I had one note about building linux-steamos, which is that the build will fail on linking vmlinux (i.e. right at the end, after 2 hours!) unless you modify the file scripts/link-vmlinux.sh with change discussed here: https://lore.kernel.org/bpf/20220916171234.841556-1-yakoyoku@gmail.com/

    So, it’s possible to boot Arch Linux on kernel 6.0, but the hardware support for all parts of the Steam Deck isn’t there. If you try to install the package linux-steamos, I can’t seem to get it to boot no matter what I try. So, for now at least, stock Arch Linux is out of the picture. For the most part, this doesn’t matter as Valve maintains their packages. However, this does mean that you must wait for reasonably recent packages. Consider this blog post my only complaint to Valve about the Steam Deck 😛

  • Microsoft quietly releasing systemd for Windows 11 WSL2

    October 5th, 2022

    For those of us who have been using WSL for a long time, we have all been quite frustrated by the lack of a functioning systemd. I’ve personally resorted to a package called systemd-genie that did some gymnastics to fool WSL into thinking it had booted properly (with systemd at PID 1). This mostly worked but was exceedingly brittle. Trying to remember whether you were in or out of the bottle was a pain (VS Code integrated terminal – out of the bottle). I changed my shell to always be in the bottle, but it just didn’t quite work out.

    To my surprise, I was following through my WSL setup instructions with another developer, and pointed him at systemd-genie, only to find the message on the page: Microsoft is previewing the completed systemd!

    So. The bad news. Your WSL installation is probably borked beyond repair if you’ve been using systemd-genie or similar. Back it up and blow it away. Get the latest release from the [Microsoft WSL Github repository](https://github.com/Microsoft/WSL/releases). Run wsl –update to get the latest updates, and install one or more distributions from the Microsoft store. Boot the distribution, and edit the file /etc/wsl.conf. Add the lines:

    [boot]
    systemd=true

    Restart the distribution. When you boot back up, you’ll find that systemd is now running. Try snap install … or start some services with systemctl. It’s been a long journey, but Microsoft Windows 11 is now Linux!

  • Back to the CLI

    October 1st, 2022

    My fingers hurt. I’ve spent the last few hours bemoaning the fact that I left my notebook charger at the office 30km away, and that the replacement won’t arrive until tomorrow. Not content to sit around, I realized that while most iPadOS remote desktop/VNC clients suck, there are plenty of good SSH clients. So, I set about learning how to use Neovim, and a dash of LUA for good measure. I won’t post the file, since there’s already plenty to search for, but I will point out why Neovim is better.

    So, first: LUA. If you’ve tried to make any kind of useful VIM configuration, you’ve probably seen that it can get ugly fast. LUA allows for modular configuration and is a rather pleasant configuration language. It reads well, indents well and generally communicates intent pretty clearly. Configuration with LUA is much cleaner, so point #1 for Neovim.

    I wanted to get the most basic setup possible, spend as little time configuring things, and end up with a very specific result: autocompletion, syntax checking and navigation. I’ve struggled with getting all these to work together in VIM, but Neovim uses fewer plugins to accomplish the same task, and thus has fewer configuration issues.

    But, this is not about Neovim, as awesome as it is, but more of a philosophical reflection on how hard it is to get away from the command line. It’s where I started so many (I won’t even tell you!) years ago. And it’s where I’m going back to. Somewhere around when Windows Vista came out, I made a concious effort to learn how to use a GUI, because my fingers hurt then too.

    But, it turns out that you need language, not gestures, to express complex thoughts. And so interfaces go back to the CLI. And though I can still navigate certain pieces of complex software (VS, e.g.) extremely well with the GUI, notice that all of them have added back a text interface to allow for more complex commands (the command palette in VS Code, e.g.). And there are options that exist only in the Azure CLI that you can’t find in the Portal. No, no matter how hard I try, the CLI just keeps coming back. And now my fingers hurt again. Where’s my voice-controlled computer that understands spoken C# and bash? 😂

  • An Intro to Apache Kafka

    September 26th, 2022

    I’ve been evaluating queues and storage for an event sourced system lately, and I seem to have found what I am looking for in Apache Kafka. Kafka is used in a surprising number of places. I only learned today, for example, that the Azure Event Hub has a Kafka surface, and can be used as a Kafka cluster itself.

    I have the following requirements:

    • Topic-based subscriptions
    • Event-based
    • Infinite storage duration
    • Schema validation
    • MQTT connection for web and mobile clients

    I’ve tried out a number of different solutions, but the one I am thinking about right now is based on the Confluent platform, a cloud-managed Kafka cluster. It is relatively easy to set up a cluster with the above requirements. Confluent has a nice clear option to turn on infinite storage duration, and provides a schema registry which supports multiple definition languages, such as JSON schema, Avro and ProtoBuf. Schema registries are a nice way of ensuring your event streams stay clean, preventing buggy messages from even entering the queue.

    I say “event-based”, but really we just need to be able to identify the schema type and use JSON. That’s pretty standard, but it needed to be mentioned.

    MQTT is a little bit of a challenge, but not really. I’d recommend checking out CloudMQTT, a simple site for deploying cloud-based Mosquitto instances. Setting up the MQTT broker took 2 minutes, and then it was off to Kafka Connect to hook it up. Adding the MQTT source is really easy as expected: provide the URL and credentials and the rest just happens automatically. You can additionally subscribe to topics to push back to MQTT. This works perfectly for web and mobile clients, whose tasks are to push events, and to receive notifications. MQTT will allow for a very nice async request/response mechanism that doesn’t use HTTP and doesn’t have timeouts.

    Finally, as I mentioned, Azure Event Hub has a Kafka surface, so you can even push certain topics (auditing, e.g.) out to Azure Event Hub to eventually make its way into the SIEM. There’s a number of useful connectors for Kafka, but I haven’t really looked at them yet except to note that there’s a connector for MongoDB.

    Kafka is a publish/subscribe based event broker that includes storage. This makes it ideal for storing DDD aggregates. Having the broker and the database in the same place simplifies the infrastructure, and it’s a natural role for the broker to fill.

    The net result of this architecture is that we no longer need to talk HTTP once the Blazor WASM SPA has loaded. All communication with the back-end system is done via event publishing and subscribing over MQTT.

    I’m happy with this architecture. Confluent seems to be reasonably priced for what you get (the options I have chosen run about $3USD per hour). CloudMQTT is almost not worth mentioning price-wise, and Kafka Connect leaves open a lot of integration possibilities with other event streams. As it is a WASM application, a lot of the processing is offloaded to the client, and the HTTP server backend stays quiet. The microservices subscribe to their topics and react accordingly, and everything that ever touches the system is stored indefinitely.

  • Some Thoughts on k8s on WSL2

    September 23rd, 2022

    It’s getting to the point now where I rarely touch a Powershell prompt. My WSL2 system has taken over the computer, and it thinks it runs Linux now. The last thing for me to get running on my local “Linux” dev environment is Kubernetes. I use the docker-desktop Kubernetes cluster, since I am already using Docker Desktop and it’s as simple as checking a box. Minikube comes recommended as well as k3s.

    I initially installed the snap for kubectl as per the suggestion from the CLI when I typed the non-existent kubectl command. I’d suggest installing it from the package repository instead. In any case, obtaining kubectl is not that hard; just don’t use the snap.

    The real difficulty with running a local k8s environment is that your localhost interface fills up awfully fast with ports. It would be much easier if you could assign your external IPs to an aliased interface and then you could not worry about port collisions as much. Enter MetalLB, a virtual load balancer for your k8s cluster. Installation instructions are at https://metallb.org/. MetalLB will create the necessary virtual interfaces for you, and assign all of your LoadBalancer ports to one or more address pools. This is great – all of your load balancers in one place.

    The only problem is that Windows can’t reach them by default. At least our MetalLB configuration is based on static addresses, because our WSL address will move every time WSL2 starts up. Not only that, but we’ll get an address in a different network! This makes setting up static routes a little difficult, but thankfully not impossible:

    $wsl_gw=$($(netsh interface ipv4 show addresses "vEthernet (WSL)" | findstr "IP") -replace "^.*?(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).*$","`$1.`$2.`$3.`$4")
    
    try {
        route delete 10.100.0.0/24 > Out-Null 2>&1
    } catch {
    
    }
    
    try {
        route add 10.100.0.0/24 $wsl_gw > Out-Null 2>&1
    } catch {
    
    }
    

    The magic comes from the first line, a one-liner for grabbing the IP address of the WSL gateway interface. The regex is excruciatingly correct, but it works every time. It’s probably worth unpacking the declaration a bit:

    Show the address information for the interface named “vEthernet (WSL)”. This is the default name of the virtual network installed to host WSL. The output looks like:

    Configuration for interface "vEthernet (WSL)"
        DHCP enabled:                         No
        IP Address:                           172.17.192.1
        Subnet Prefix:                        172.17.192.0/20 (mask 255.255.240.0)
        InterfaceMetric:                      5000

    Now that we have the output, it seems that the address we are interested in is all by its lonesome on the line labelled “IP Address”. This sounds like a job for a regular expression. So, we match the output against an entire line containing a single IP address and replace the line with the IP address. We use findstr to narrow the output to a single line in the first place.

    And finally, we fire and forget a couple of network commands that should (occassionally) give us a route from our Windows machine to our MetalLB address pool.

    There’s a lot that can go wrong here. Most notably, we cannot run anything related to WSL without the user logging in. So, we’re stuck with startup scripts in the Windows Startup folder to get the job done, and the one above goes in there.

    Second, we need to trigger the Linux-side scripting on startup too. One clever thing you can do is to install systemd-genie and then start terminal upon login. Set Ubuntu to your default terminal and logging in will cause systemd to kick in and start those services. With any luck, both the route and the additional interface will be allocated without issue. But if you expect it to work without occasional (frequent) intervention, you’re crazy. Microsoft, if you’re listening, please have your systemd guy give us a real systemd! 🙂

    But, as expected, other than the fiddling with IPs and ports, k8s runs very well on “WSL2” (remember – it’s running in a different Linux container on your Windows host). The Docker Desktop implementation of Kubernetes is fine.

    One last thought. Deployment to Kubernetes is usually achieved using a CI/CD pipeline. I’d suggest you do the same for your local k8s cluster as well. Install an Azure DevOps agent in WSL2 and connect it to your DevOps project. Now you can manage your k8s builds from DevOps instead of tinkering from the command line. This turns out to be excellent for other reasons – it lets you break the build dozens of times in a pipeline whose stats don’t matter. Then you can simply clone the pipeline and update your variable groups for test/prod.

    As usual, I’m late to the party on Kubernetes. But I believe it’s a key architectural component that allows us to avoid vendor lock-in. Building cloud-native apps for Kubernetes increases agility and reduces lock-in because you can easily deploy your application to other clouds, giving you the ability to rapidly move from cloud to cloud with minimal disruption. This leaves you in a much better negotiating position with your current cloud vendor.

  • A Deeper Dive with Terraform

    September 19th, 2022

    Having spent the weekend building infrastructure scripts, I can now say that I like Terraform. My initial foray into the Infrastructure-as-Code (IaC) arena was using Pulumi. This appealed to me because it supported declarations in C#. After not very long, however, I found that Pulumi’s C# support was rather lacking, and put IaC on the shelf.

    IaC was resurrected when my boss told me that all infrastructure I built for this project would have to be scripted. One of my team members dove into Terraform, and recently gave me a crash course. The language itself is easy, but I think the development environment itself is very good as well.

    As I posted earlier, I’m using Terraform Cloud both as a state provider for my development workspace, and as CI/CD for my dev/test and production workspaces. This produces a very nice development workflow.

    1. Create a feature branch
    2. Make the feature branch work
    3. PR to QA branch
    4. Terraform Cloud (TC) plans the updated code
    5. Confirm plan in TC
    6. PR to …
    7 …

    Infrastructure workflows are the same as Git workflows! With all of the features that Azure DevOps provides, including work item association and approvals. The pipeline itself is triggered upon push. But that’s ok. The code review process was required first, since the merge to the QA branch required approval. Then, TC will not automatically apply changes unless instructed to. Making infrastructure changes requires confirmation from a user with the appropriate access.

    That’s the development workflow, and it works very well, and was not that difficult to set up given the extensive Terraform documentation. Personally I run it on WSL2 using VS Code with the Terraform extension. Infrastructure dev environments should probably run some flavor of Unix (WSL2, MacOS, Linux) and nothing more than VS Code is required.

    Code organization is very important. Terraform is a declarative language with very few opportunities for reuse and almost no control structures. This makes for the possibility of unreadable code very quickly. Organizing into workspaces and modules is the only way to keep larger projects under control. It’s probably easiest to explain with an example, so let’s put down some requirements.

    The project in question has 4 total environments, 3 non-prod and 1 prod. The 3 non-prod environments share a single Azure Dev/Test subscription, and the production environment has its own subscription. Costs should be minimized without sacrificing code readability. Development should plan and apply from the CLI or Web UI only. QA should plan and apply upon push to qa branch, and UAT should plan and apply upon push to uat branch. Finally, production should plan and apply upon push to main branch.

    The first thing to note from these requirements is the cost minimization. It’s always easiest to duplicate, and it would be much easier to simply build three copies of all infrastructure in dev/test. But that’s expensive: the search service alone costs $300/mo per instance. And having multiple key vaults when one will suffice is kind of annoying. And you don’t need multiple storage accounts, multiple containers is fine. Upon analyzing the infrastructure requirements, it was clear that the Redis cache, the Key Vault, the Storage Account and the Container Repository could all be shared among the three non-prod projects.

    So how to achieve this? My first attempt simply included the same module in multiple projects. Alas, this simply led to cycles of create/recreate as updates were made to the various branches. Creating and recreating the core infrastructure is absolutely the opposite of what we want. So, the first step was to split out the shared infrastructure into its own folder, and added the Terraform files. Then, the shared infrastructure modules moved, and the root Terraform definition updated. Finally, the shared infrastructure main.tf and variables.tf are updated.

    Now, the workspace layout. First, we need a workspace to hold the shared infrastructure. I created a single workspace with a -devtest suffix and initialized it. From the shared directory, run terraform plan and terraform apply. This will create the shared infrastructure in a separate workspace so that there are no conflicts between the environments. This workspace should be CLI-driven. The development environment should always be CLI-driven, and the shared infrastructure for development also belongs to QA and UAT. Therefore, no other workspace is needed for Dev/Test core infrastructure. Note the need to create a corresponding production workspace later.

    Next, we need workspaces and variable sets for each environment. The dev workspace will be marked as CLI-driven, and the other two (QA and UAT) will be attached to Azure DevOps Git and trigger runs upon push. The variable sets will be created and associated accordingly.

    That’s the gist of it. Terraform really saves a great deal of effort and reduces mistakes, as well as providing a lot more process around infrastructure development. Good use of workspaces and modules can make for reasonably organized layouts for even more complex infrastructure requirements.

  • An Introduction to Terraform Cloud

    September 17th, 2022

    I’ve been given a crash course in Terraform lately, and the first thing I did was to get a Terraform Cloud account to help manage all of the variables and workspaces. The general workflow of Terraform remains the same:

    terraform init
    terraform plan
    terraform apply
    

    However, we can optionally connect Terraform Cloud to our Git repository and trigger runs based on commits to specific branches. So, the workflow I used was as follows:

    Create an organization and workspace connected to my Git repository

    Switch the workspace to Local

    Configure main.tf as follows:

    terraform {
      required_providers {
        azurerm = {
          source  = "hashicorp/azurerm"
          version = "2.86.0"
        }
      }
      cloud {
        organization = "my-organization"
        token = "my-api-key"
        workspaces {
          name = "my-workspaces"
          #tags = ["my-app"]
        }
      }
    }
    

    Create a Terraform API token with the required permissions. Note that we have to put the api-key in this file 😦 I’d really rather not do that

    Initialize Terraform

    Develop your Terraform plan and verify that it is working as usual. I’d suggest that you do NOT commit your code with the token value above (remove it before committing and provide the API key to each developer). There isn’t a way to pass this value in the environment or on the command line, so be careful! Code reviews are a good thing.

    Switch your Terraform Cloud workspace from Local to Remote

    Commit your code

    Trigger run on Terraform Cloud, either automatically upon push, or manually

    You no longer have to manually trigger runs locally; you can simply push your code and trigger the run in the Cloud console. I’m no stranger to a CLI, and Terraform has a pretty nice one. But I prefer the web interface. Additionally, we get teams, users and permissions. We can, for example, assign a different team to the production environment than the development environment.

    Think of this as the CD for your DevOps for Infrastructure. Using Azure DevOps (or similar), you will be able to enforce code reviews before committing to branches used by Terraform Cloud. Terraform will automatically execute once the code is committed, saving you the management of a DevOps pipeline to run your code.

    There’s a lot of advanced features here that I haven’t looked at yet. For example, you can apply organization-wide policies that are checked when any developer in the organization checks in. Terraform will fail if these policies are breached. My initial thoughts are that this will simplify the workflow and management of an infrastructure DevOps process.

←Previous Page
1 2 3 4
Next Page→

Blog at WordPress.com.

Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use.
To find out more, including how to control cookies, see here: Cookie Policy
 

Loading Comments...
 

    • Follow Following
      • Brian Richardson's Blog
      • Already have a WordPress.com account? Log in now.
      • Brian Richardson's Blog
      • Edit Site
      • Follow Following
      • Sign up
      • Log in
      • Report this content
      • View site in Reader
      • Manage subscriptions
      • Collapse this bar