Nabeel Valley

HTML in WebGPU Shaders in Canvas

Sun, 12 Apr 2026 00:00:00 GMT

import Example from './Example.astro'

I made some minor updates to my Shader Web Component to add support for the proposed HTML-in-Canvas API. So you can see the following in a Chromium based browser with the flag chrome://flags/#canvas-draw-element enabled

How Does It Work?

The HTML in the above example is rendered within the canvas and is - if/when supported - exposed to the accessibility tree

As per the documentation, this makes use of the following:

The layoutsubtree attribute on the canvas element to tell the browser that we want to render the contents
The copyElementImageToTexture function on the WebGPU GPUDevice.queue to copy the texture
Use the canvas.onpaint handler to run the copyElementImageToTexture function with the contents

Usability Issues

The proposal is still just that - a proposal - so it has some issues and is still very experimental.

The above example should show the inner content if supported. I've noticed that there are some quirks around how the content focus, etc. work since the click targets aren't aligned as expected, etc. which creates some weird usability issues

Some challenges I see here are around keeping positions in sync so that interactive elements remain correctly interactive and that basic parts of the web continue to work as users expect

The example above originally suffered from the fact that the size of the canvas content doesn't match the canvas and so things are not quite in the right place. Matt Rothenberg also points this out in their HTML in Canvas post and while it's definitely doable it does require a fair amount of wrangling to make work nicely

I've solved the resizing issue in the above example using CSS but it's also possible to do using the drawElementImage function as per the API to do the necessary alignment

Technical Challenges

I also ran into some VERY annoying bugs (or is this just how WebGPU is meant to work - I don't know) along the way, the main one was to do with figuring out how to use the GPU bindings

In particular, I saw the following error about a million times:

In entries[0], binding index 0 not present in the bind group layout.
Expected layout: []
 - While validating [BindGroupDescriptor ""textures""] against [BindGroupLayout (unlabeled)]
 - While calling [Device].CreateBindGroup([BindGroupDescriptor ""textures""]).

This was because the bindings in my shader related to the textures were not used:

// unused
@group(0) @binding(0) var htmlSampler: sampler;
@group(0) @binding(1) var htmlTexture: texture_2d<f32>;

// used
@group(1) @binding(0) var<uniform> uTime: f32;

The respective texture and sampler were not used, and so the auto layout for the shader doesn't seem to detect that they exist. I find this very confusing - the error here could be better

There's a good example on explicitly defining the layouts on this WebGPU Bind Group Layouts article and a handy calculator for providing the bind group layout config. This ended up eventually being how I realized the bind group was being left out since for some reason the entries in @group(0) were being left out even though their example - when updated to use "my" bindings for the texture and sampler - worked just fine.

Anyways, ensuring that the sampler and texture are used meant that everything worked fine again. So conditionally adding the bindgroup works which adds some ugliness to my shader rendering code but I think it's fine for what it makes possible

The snippet for the above example can be seen below:

<site-shader-canvas>
  <canvas width="1000" height="1000" style="width: 100%" layoutsubtree>
    <div class="html-in-canvas" style="width: 100%; height: 100%; color: green; display: flex; flex-direction: column; align-items: center; justify-content: center; gap: 8px;">
      <h2 style="margin: 0;">Hello from HTML!</h2>      

      <label for="input-in-canvas">Input in canvas</label>
      <input style="display: block;" id="input-in-canvas" />      
    </div>
  </canvas>

  <script type="text/wgsl">
    struct VertexOutput {
      @builtin(position) position: vec4f,
      @location(0) texcoord: vec2f,
    };

    @vertex fn vs(
      @builtin(vertex_index) vertexIndex : u32
    ) -> VertexOutput {
      const pos = array(
        vec2( 1.0,  1.0),
        vec2( 1.0, -1.0),
        vec2(-1.0, -1.0),
        vec2( 1.0,  1.0),
        vec2(-1.0, -1.0),
        vec2(-1.0,  1.0),
      );

      var vsOutput: VertexOutput;
  
      let xy = pos[vertexIndex];
      vsOutput.texcoord = pos[vertexIndex] * vec2f(0.5, 0.5) + vec2f(0.5);
      vsOutput.position = vec4f(pos[vertexIndex], 0, 1);

      return vsOutput;
    }

    @group(0) @binding(0) var htmlSampler: sampler;
    @group(0) @binding(1) var htmlTexture: texture_2d<f32>;

    @group(1) @binding(0) var<uniform> uTime: f32;

    fn flipSample(fsInput: VertexOutput) -> vec4f {
      // the texture must be flipped since the shader coord system is the
      // opposite of the HTML one
      var pos = vec2f(fsInput.texcoord.x, 1 - fsInput.texcoord.y);
      return textureSample(htmlTexture, htmlSampler, pos);
    }
    
    @fragment fn fs(fsInput: VertexOutput) -> @location(0) vec4f {
      var red = abs(sin(uTime/10.0)) * fsInput.texcoord.x;
      var blue = abs(cos(uTime/5.0)) * fsInput.texcoord.y;

      return vec4f(red, 0.0, blue, 1.0) + flipSample(fsInput);
    }
  </script>
</site-shader-canvas>

You can also take a look at the source of the web component and respective shader renderer if you're interested

Bad Bots

Fri, 03 Apr 2026 00:00:00 GMT

Shh ... you've found an RSS Club post. I'll be using this to post small ideas or share things from around the web that I find interesting but don't intend to write an entire post about

I have a website that kept randomly going down every few weeks - turned out that I hadn't enabled a healthcheck so the auto-restart didn't know to that it had to restart. So after some annoyance I added a basic check and I think it's fine now - probably ...

Anyways, not the point - I looked into why it kept going down and seemed like it would randomly get thousands of simultaneous requests from bots and I decided that I don't want that anymore?

I remembered reading a post a while back about blocking AI crawlers which upon some searching turned out to be by Matthais Ott. And so I did some reading and here's a list of some related content on how to stop the LLMs:

I particularly resonate with the last paragraph of the last link in that list:

"If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts." - Please stop externalizing your costs directly into my face - Drew DeVault

None of these are Traefik so I'll probably put together a bit of an implementation once I settle on something and put in in my Traefik notes

Symbols on the Ground

Tue, 31 Mar 2026 00:00:00 GMT

import Gallery from '../../../../components/Gallery.astro'

I came across some random murals on the along the path in Utrecht today. A while later when walking by I stopped to take some pictures of them - I saw 15. But according to Mozaiek Monumenten there are 17 - I guess I the remaining two were in the construction site just past the 15th

They're a reference to the Sustainable Development Goals. The reference from Mozaiek Monmumenten can be seen below with the descriptions of each of them:

You can see the pictures I took of them below:

The two that I don't have are:

Freedom, Safety, and Strong Public Services
Partnership for the Goals

Okay that's it, I should walk around with my eyes open more often

Photos from the Nederlands Fotomuseum

Sat, 28 Mar 2026 00:00:00 GMT

import Gallery from '../../../../components/Gallery.astro'

We went to the Nederlands Fotomuseum which was really great. I resisted the urge to take any pictures in the museum but when I got to the Cyanotype exhibit I couldn't resist. You can see some of the pictures below

Also, they had this really cool digital gallery thingy so here's a video of that!

I also saw this silly bus which turned out to be one of these water bus things

Then on the train ride home I watched about 8 people look at a seat, have a weird facial expression and walk away, not sitting on the chair

Zahrah pointed out that there was a sticky note on the chair saying it was wet (I couldn’t see the note)

At some point an old man came along, looked perplexingly at the note, and decided to touch it and was like - this is not wet?

The man across said, yeah it doesn't look wet but I don’t know

The old man sat on the seat next to the labeled-as-wet one

People continued to walk past and look at the sticky note and walk away

The old man moved the sticky note away, this was either to state that the seat is in fact not wet - or as an effort to prank someone by making them sit on a wet seat

We do not know

Zahrah had some comments about how everyone was appealing to the authority of the sticky note and how much that says about the fragility of the social contract

IDK I was just sitting and laughing every time anyone walked past because it was funny

Anyways ... here are the pictures then!

Quick and Dirty Object Access in Go

Fri, 27 Mar 2026 00:00:00 GMT

Assumed audience: Developers/technical people who use Go as a programming language

This probably isn't too difficult to write but it's recursive and I had fun putting it together so here it is

Basically, I was trying to access a deeply nested object that was parsed from BurntSushi/toml in some generic fashion and it was getting annoying to constantly cast things and do the necessary checks at every level so I made two utilities that can access nested data a bit more conveniently

First off, here's a helper type to sprinkle around:

type dynamic = map[string]any

Then, here's the version that will panic if it hits something it's not expecting:

func unsafeIndex(data dynamic, path ...string) any {
	if len(path) == 0 {
		return data
	}

	inner := data[path[0]]
	if len(path) == 1 {
		return inner
	}

	return unsafeIndex(inner.(dynamic), path[1:]...)
}

And the version that returns the appropriate errors along the way:

func safeIndex(data dynamic, path ...string) (any, error) {
	if len(path) == 0 {
		return data, nil
	}

	inner, ok := data[path[0]]
	if !ok {
		return data, fmt.Errorf("Error indexing path %v for %v", path, data)
	}

	if len(path) == 1 {
		return inner, nil
	}

	dyn, ok := inner.(dynamic)
	if !ok {
		return dyn, fmt.Errorf("Error indexing into inner struct %v for %v", path, inner)
	}

	return safeIndex(dyn, path[1:]...)
}

Using them is also fairly normal, as seen below:

// whatever you're doing to decode the data. e.g. JSON/TOML parser
var output dynamic
decodeData(FILE_CONTENT, &output)

dynamicSources, err := safeIndex(output, "some", "deeply", "nested", "property")
if err != nil {
    panic(err)
}

Async TUIs using Bubble Tea

Thu, 26 Mar 2026 00:00:00 GMT

Assumed audience: UI developers, people who program in Go, or anyone just generally interested in making computers to things using code

There was a little bug I ran into a while back but hadn't been important enough for me to fix until yesterday when it started to slow me down

On Tri - a TUI app I built that is something like if tree was searchable and had previews likefzf - I (knowingly) didn't implement async tasks upfront. At the time I was mostly focused on getting the implementation to a good level of UX and free of bugs. As such, there was a clear slowness when navigating the UI while running slow tasks, such as a git diff in a large repository

I sat down yesterday to make the app run previews run in the background - this turned out to be really easy and I thought I'd write about it just to mention why that was the case

Most TUIs I write in Go use the excellent suite of libraries by Charm - and in this particular case, the Bubble Tea TUI framework

Elm Architecture

The Bubble Tea framework is based on the Elm Architecture which is a functional style pattern for building UIs. I think understanding the Elm architecture is a generally useful and the documentation is worth a read for developers building any kind of user interface (even if it's not in Elm)

The core idea is this:

All UI flows from a Model
Messages are used to perform Updates on the Model
A View converts the Model into UI

This is also known as the MVU pattern (Model -> View -> Update)

Using this pattern, we can build a simple implementation of an app that has two bits of independent UI - a counter that increments when the user presses space, and a task runner that runs some heavy tasks triggered by pressing enter

A Sad Implementation

A naive implementation of this using Bubble Tea has the following bits that matter for discussion:

In the Update function, when we press space we increment the counter, and when we press enter we run some tasks:

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
	switch msg := msg.(type) {

	case tea.KeyPressMsg:
		switch msg.String() {
		case "ctrl+c", "q":
			return m, tea.Quit

		case "space":
			m.counter++
			return m, nil

		case "enter":
			m.running = true
			m.tasks = doTasks()
			return m, nil
		}
	}

	return m, nil
}

This method then updates the model and returns the updated model - for context, the doTasks function looks like so:

func doHeavyWork() {
	t := rand.IntN(5)

  // irl we'd do something other than sleep
	time.Sleep(time.Duration(t) * time.Second)
}

func doTasks() []task {
	var tasks []task

	for i := range 10 {
		doHeavyWork()
		tasks = append(tasks, task{i, true})
	}

	return tasks
}

We can see this running below:

The problem with the above implementation is twofold:

Bad performance - The UI is blocked while the tasks are running, so the counter doesn't update until after the tasks are run
Sad UX - There isn't a way to update an in-progress task, would be nice to not have to wait

<details> <summary>You can see the full V1 implementation if you'd like</summary>

package v1

import (
	"fmt"
	"os"
	"time"

	tea "charm.land/bubbletea/v2"
	rand "math/rand/v2"
)

type task struct {
	index int
	done  bool
}

type model struct {
	counter int
	running bool
	tasks   []task
}

func (t task) string() string {
	status := "busy"
	if t.done {
		status = "done"
	}

	return fmt.Sprintf("Task %d [%s]", t.index, status)
}

func sleepRandomly() {
	t := rand.IntN(5)
	time.Sleep(time.Duration(t) * time.Second)
}

func doTasks() []task {
	var tasks []task

	for i := range 10 {
		sleepRandomly()
		tasks = append(tasks, task{i, true})
	}

	return tasks
}

func initialModel() model {
	return model{
		counter: 0,
		running: false,
		tasks:   []task{},
	}
}

func (m model) Init() tea.Cmd {
	return nil
}

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
	switch msg := msg.(type) {

	case tea.KeyPressMsg:
		switch msg.String() {
		case "ctrl+c", "q":
			return m, tea.Quit

		case "space":
			m.counter++
			return m, nil

		case "enter":
			m.running = true
			m.tasks = doTasks()
			return m, nil
		}
	}

	return m, nil
}

func (m model) View() tea.View {
	count := fmt.Sprintf("count = %d", m.counter)
	if !m.running {
		return tea.NewView(count + "\nPress space to increment counter\nPress enter to start tasks")
	}

	tasks := ""
	done := true
	for _, t := range m.tasks {
		if !t.done {
			done = false
		}
		tasks += "\n" + t.string()
	}

	title := "Running Tasks"
	if done {
		title = "All done"
	}

	return tea.NewView(count + "\n" + title + "\n" + tasks)
}

func Run() {
	p := tea.NewProgram(initialModel())
	if _, err := p.Run(); err != nil {
		fmt.Printf("Alas, there's been an error: %v", err)
		os.Exit(1)
	}
}

</details>

A Happy Implementation

The solution that's provided by Bubble Tea is to move the IO based work into a Command. A command is used to make things async and is handled by the framework

A command looks like so:

// its type is tea.Cmd
var cmd tea.Cmd

// its value is a function that returns tea.Msg
cmd = func () tea.Msg {
  return  SomeMessage{}
}

So in order to make our work async, we simply need to return a tea.Cmd in our Update function instead of actually doing all the work

Instead of defining a function that does the work, we can define one that returns a tea.Cmd that will do the work:

type taskDoneMsg struct {
	index int
}

func makeTasks() ([]task, []tea.Cmd) {
	var cmds []tea.Cmd
	var tasks []task

	for i := range 10 {
		tasks = append(tasks, task{i, false})
		cmds = append(cmds, func() tea.Msg {
			doHeavyWork()
			return taskDoneMsg{i}
		})
	}

	return tasks, cmds
}

This will offload the work and we'll receive a taskDoneMsg message when the work is done. This also has a nice side effect - by decoupling the creation of the task and the actual execution of it, we can now track the status of each task as it completes

We can do that in the Update function by handling the taskDoneMsg message as well as returning the []tea.Cmds that comes from the makeTasks function instead of actually doing the work upfront

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
	switch msg := msg.(type) {

  // handle updating the model when the task is done
	case taskDoneMsg:
		m.tasks[msg.index].done = true
    return m, nil

	case tea.KeyPressMsg:
		switch msg.String() {
		case "ctrl+c", "q":
			return m, tea.Quit

		case "space":
			m.counter++
			return m, nil

		case "enter":
			m.running = true
			tasks, cmds := makeTasks()
			m.tasks = tasks

      // batch the new cmds for bubbletea to handle
			return m, tea.Batch(cmds...)
		}
	}

	return m, nil
}

And with that, we've now got a responsive UI that lets the counter work even while the tasks are running as well as makes it possible for us to track task state:

<details> <summary>You can see the full V2 implementation if you'd like</summary>

package v2

import (
	"fmt"
	"os"
	"time"

	tea "charm.land/bubbletea/v2"
	rand "math/rand/v2"
)

type task struct {
	index int
	done  bool
}

type taskDoneMsg struct {
	index int
}

func (t task) string() string {
	status := "busy"
	if t.done {
		status = "done"
	}

	return fmt.Sprintf("Task %d [%s]", t.index, status)
}

func doHeavyWork() {
	t := rand.IntN(5)
	time.Sleep(time.Duration(t) * time.Second)
}

func makeTasks() ([]task, []tea.Cmd) {
	var cmds []tea.Cmd
	var tasks []task

	for i := range 10 {
		tasks = append(tasks, task{i, false})
		cmds = append(cmds, func() tea.Msg {
			doHeavyWork()
			return taskDoneMsg{i}
		})
	}

	return tasks, cmds
}

type model struct {
	counter int
	running bool
	tasks   []task
}

func initialModel() model {
	return model{
		counter: 0,
		running: false,
		tasks:   []task{},
	}
}

func (m model) Init() tea.Cmd {
	return nil
}

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
	switch msg := msg.(type) {

	case taskDoneMsg:
		m.tasks[msg.index].done = true
		return m, nil

	case tea.KeyPressMsg:
		switch msg.String() {
		case "ctrl+c", "q":
			return m, tea.Quit

		case "space":
			m.counter++
			return m, nil

		case "enter":
			m.running = true
			tasks, cmds := makeTasks()
			m.tasks = tasks

			return m, tea.Batch(cmds...)
		}
	}

	return m, nil
}

func (m model) View() tea.View {
	count := fmt.Sprintf("count = %d", m.counter)
	if !m.running {
		return tea.NewView(count + "\nPress space to increment counter\nPress enter to start tasks")
	}

	tasks := ""
	done := true
	for _, t := range m.tasks {
		if !t.done {
			done = false
		}
		tasks += "\n" + t.string()
	}

	title := "Running Tasks"
	if done {
		title = "All done"
	}

	return tea.NewView(count + "\n" + title + "\n" + tasks)
}

func Run() {
	p := tea.NewProgram(initialModel())
	if _, err := p.Run(); err != nil {
		fmt.Printf("Alas, there's been an error: %v", err)
		os.Exit(1)
	}
}

</details>

Summary

That's it - no Goroutines or channels needed, a pretty good abstraction on the side of the framework - and minimal effort needed from us

A small aside, I've started working on what will probably be a fairly sizeable side project. As such, I've gotten absolutely nothing done on that while somehow managing to put together a few blog posts (this one + another one about eid + another one about tri), update my photo galleries (general nl and stuff from last winter) on my site, and fix a bunch of random things in other random side projects just this week - oh the power of procrastination

Murder Mystery in Git

Thu, 26 Mar 2026 00:00:00 GMT

Shh ... you've found an RSS Club post. I'll be using this to post small ideas or share things from around the web that I find interesting but don't intend to write an entire post about

I had a dumb idea, what if you could make a game in a git repo. First thought is some kind of text-based murder mystery where you have to follow git tags as evidence in some silly way to find the killer

Could be funny, someone who's more creative than me could probably make it work. But yeah free idea, use it if you want

Git Tricks with Tri and Difft

Thu, 26 Mar 2026 00:00:00 GMT

Assumed audience: Developers/technical people who use git and/or enjoy terminal UIs (TUIs)

So I finally got tri running super fast which I already talked about today and I came across something mildly annoying but not all bad so thought it would be nice to write down

I use difft for my git diffs. This works really nicely as it's got some syntax awareness and integrates really well into other tools I use such as lazygit

Now, I'm a Unix Philosophy dude so I build my tools to do the same - so naturally I wanted to use difft with tri - so, here's how to do that

The below examples are using Nushell so your shell's exact syntax may vary but the idea is the same

git diff HEAD --name-only | GIT_EXTERNAL_DIFF="difft --color=always --display=inline" tri --preview `git diff HEAD -- `

Okay, braindump definitely. Realistically, since I want difft to always behave like this, It's probably worth setting the environment variable elsewhere, but the steps are basically:

Use the GIT_EXTERNAL_DIFF environment variable to have the difft command, this causes git to use the entire command and not just execute the binary provided as with diff.external, so I can provide the flags to format difft as desired
Then just use tri as normal

Assuming you've set step 1. within your shell, the command is more simply:

# step 1.
$env.GIT_EXTERNAL_DIFF = "difft --color=always --display=inline"

#step 2.
git diff HEAD --name-only | tri --preview `git diff HEAD -- `

Which is just the normal way that tri works so yay

And that's it okbye

Update 14 April 2026

I've added this function to my nu config which basically do the above:

def "g diff tri" [range = "HEAD..master"] {
  git diff ($range) --name-only
  | GIT_EXTERNAL_DIFF="difft --color=always --display=inline" tri --preview $"git diff ($range) -- " --flat
}

Update 04 May 2026

I've wanted to do more complex preview behavior with tri - particularly making it possible to generate custom commands. I've recently added support for that so it now makes for some really interesting stuff, like a command for example the below command which lets you browse the input files' history

Using it looks like this:

^find **/*.go | g log tri

And the definition is a bit messy, but not too complicated I hope:

# Expects a list of paths as an input
def "g log tri" [] {
  $in
  | lines
  | par-each {|p| git log --pretty=format:"%as %h %f" -- $p | str replace -m -a --regex ^ $"($p)/" }
  | to text
  | GIT_EXTERNAL_DIFF="difft --color=always --display=inline" tri --preview "git diff $4..$4^ -- $1" --pattern `^(.*)/((\d|-)+) (\w+)`
}

This is also probably very inefficient. So be selective about the paths you provide since this does a log for all given paths and can be very slow on a large repository

Mildly interesting photos from Eid

Sun, 22 Mar 2026 00:00:00 GMT

import Gallery from '../../../../components/Gallery.astro'

It was Eid on Friday (20 March). We spent it with some friends which is so different to when living near family. It's quieter and far less chaotic. The Cherry Blossoms are in bloom and the sun was out though - so I took some pictures

Also a few fairly random things happened so I figured I'd list them for amusement's sake

When we went to the Masjid for the Eid Salaah Shaheed thought that the drink being served was Boeber (which of course, it wasn't - because why would it be at a Moroccan Masjid in the Netherlands?). Anyways - it was fun watching him realize it was in fact just a cup of milk.

In the afternoon, on the way to Jummah I saw a giant hole being dug next to the shopping center so I made a mental note to take a picture of it on my way home

On the way to the Masjid still, my path was blocked by a school of kids cycling with bright orange vests and teachers shouting "Trappen voor je leven!". There were like a lot of them

Then on the way back from the Masjid, after taking my picture of the giant hole - I stopped and took a few of the cherry blossoms. The old man sitting outside his apartment building in the sun shouted at me to get some from the other side where the sun was coming from to which I obliged

Anyways, all that to say - here are some random pictures from the past week

Update 27 March - I was cycling back from the masjid today and saw an old man stop to take a picture of the big hole. 'tis truly universal innit

Join the Webring!

Thu, 12 Mar 2026 00:00:00 GMT

Assumed audience: People with a personal website or anyone interested in the indie web

I've wanted to start a Webring for a while now and finally got around to it. Click the links in the footer to explore!

Would You like to Join?

That's great! You need to do two things:

Add the following HTML to your site's footer (or some variation of those 4 links) - style it however you like!

<section>
  <a href="https://webring.nabeelvalley.co.za"><em>Webring</em></a>

  <ul class="links">
    <li><a href="https://webring.nabeelvalley.co.za/previous">Previous</a></li>
    <li><a href="https://webring.nabeelvalley.co.za/random">Random</a></li>
    <li><a href="https://webring.nabeelvalley.co.za/next">Next</a></li>
  </ul>
</section>

Make a pull request adding your site to the webring repo at src/config.gleam

And that should do it! Welcome to the web

Web Component for Making Patterns

Tue, 17 Feb 2026 00:00:00 GMT

I made a small web component for defining geometric patterns, you can see the component in action here:

(inspect the JS if you'd like to see what's happening, though it's just some bad SVG stuff)

<site-pattern scale="20" pattern-x="0 1 0 0 1 0 1 1" pattern-y="1 0 0"> <svg style="width: 100%; aspect-ratio: 3/2"></svg> </site-pattern>

CSS Anchor Positioning

Wed, 11 Feb 2026 00:00:00 GMT

I was reading a pull request the other day and someone suggested the use of position-anchor which was something I hadn't seen before and so I thought I'd write something small about it

In putting together this example, I came across a few interesting points that I wanted to talk about - but before getting into those, here's a small example first

Example

For the example we'll use some HTML and CSS in order to place some content relative to some specific anchor. The result we see is that the "Content for Anchor N" is placed at the top right of the "Anchor N" element, but this could be many other placements or layouts as desired

<section> <h3 class="anchor" anchor-name="--anchor-1">Anchor 1</h3> <p class="anchored" anchor-name="--anchor-1">Content for Anchor 1</p>

<h3 class="anchor" anchor-name="--anchor-2">Anchor 2</h3> <p class="anchored" anchor-name="--anchor-2">Content for Anchor 2</p> </section>

.anchored { position-anchor: attr(anchor-name type(<custom-ident>)); position: fixed; bottom: anchor(bottom); right: anchor(right); } </style>

The above example uses a few interesting bits that I'd like to discuss before throwing a wall of code at you

CSS Identifiers

Though it does a pretty good job of making everything look like a string - CSS does indeed have data types (<cite>CSS Data Types - MDN</cite>). Usually we don't need to think too hard about these but they end up being necessary in the above example when using the position-anchor attribute and attr function

An <ident> is the type used to represent an identifier in CSS (<cite>CSS <ident> - MDN</cite>)

A <custom-ident> is a case-sensitive string that can be user-defined and used as CSS identifier (<cite>CSS <custom-ident> - MDN</cite>)

The position-anchor attribute requires a <dashed-ident> which is a <custom-ident> that starts with -- (<cite>CSS <dashed-ident> - MDN</cite>). These will never be defined by CSS and are only ever defined by the user

In our example, we're receiving our <dashed-ident> via attr(). Since a <dashed-ident> cannot be defined by CSS, the type we will receive from attr() is a <custom-ident>. Note that this must start with -- in order to work in the place of a <dashed-ident>

Reading Attributes in CSS

The CSS attr() function lets us read the value of an element attribute and use it in CSS (<cite>CSS attr() - MDN</cite>). This means that we can effectively provide variables for use within CSS by way of HTML attributes. This is really handy and has tons of use cases

In the example above, we're using the attr() to read the value of the anchor attribute. Some ways of using attr() can be seen below:

/* basic */
attr(attribute-name)

/* with a unit or type*/
attr(attribute-name unit)
attr(attribute-name type(<css-type>))

To actually use this, we can specify an attribute on an HTML element like so:

<h3 class="anchor" anchor-name="--anchor-1">Anchor 1</h3>

And we can then read an attribute, for example - the anchor-name attribute as a <custom-ident>, using attr() as seen below:

.anchor {
  anchor-name: attr(anchor-name type(<custom-ident>));
}

attr() can be used to read data from an attribute and use it in CSS pretty much anywhere that we'd normally use hard-coded data

Anchor Positioning

Now that we've covered what the attr and <custom-ident> syntax is all about, we can dive into anchor positioning

Anchor positioning is a set of CSS attributes and behaviors that make it possible to connect the positions and sizes of different elements (<cite>CSS anchor positioning - MDN</cite>). For the sake of this example, we'll use the different parts of the anchor positioning module to place an element at the top-right of another element. Making this work requires us to define the anchor element. This is done using the anchor-name property on said element (<cite>CSS anchor-name - MDN</cite>):

<h3 class="anchor-1">Anchor 1</h3>

<style>
.anchor-1 {
  anchor-name: --anchor-1
}
</style>

And then, we can reference this anchor-name from any elements that want to position themselves relative to this with the position-anchor property which specifies the name of the anchor element (<cite>CSS position-anchor - MDN</cite>)

<p class="anchored-1">Content for Anchor 1</p>

<style>
.anchored-1 {
  position-anchor: --anchor-1;
  position: fixed;

  bottom: anchor(bottom);
  right: anchor(right);
}
</style>

In the above, we specify position: fixed to place this element relative to the position-anchor and out of the normal document flow (<cite>CSS position - MDN</cite>)

The CSS anchor() function is used to return the length relative to the anchor element and allows us to position the anchored element relative to the anchor (<cite>CSS anchor() - MDN</cite>)

Putting It Together

The example above is fairly repetitive. For each element having to redefine a CSS rule sets the anchor-name and position-anchor can become tedious. There are two solutions that will give us some kind of reusability here, namely:

Using a CSS custom property that is set on the element
Using a custom HTML attribute and reading that in the CSS

Lately I've been liking the use of custom HTML attributes as a hook for styling - this is inspired by the TAC CSS Methodology which uses Tags, Attributes, and Classes - in that order - as styling hooks

As such, the below example uses the anchor-name attribute on the HTML element to define the values to use for anchor-name and position-anchor, which we then read in the CSS using attr():

<h3 class="anchor" anchor-name="--anchor-1">Anchor 1</h3>
<p class="anchored" anchor-name="--anchor-1">Content for Anchor 1</p>

And then, define generic CSS rules that can work with any provided anchor-name attribute and read them accordingly:

.anchor {
  anchor-name: attr(anchor-name type(<custom-ident>));
  margin: 50px;
}

.anchored {
  position-anchor: attr(anchor-name type(<custom-ident>));
  position: fixed;
  bottom: anchor(bottom);
  right: anchor(right);
}

Putting that together gets the behavior we're after in a pretty neat and reusable way

The Big Example

Putting all the above together, we can get back to the code for the original example, which can be seen once more below:

<section> <h3 class="anchor" anchor-name="--anchor-3">Anchor 3</h3> <p class="anchored" anchor-name="--anchor-3">Content for Anchor 3</p>

<h3 class="anchor" anchor-name="--anchor-4">Anchor 4</h3> <p class="anchored" anchor-name="--anchor-4">Content for Anchor 4</p> </section>

.anchored { position-anchor: attr(anchor-name type(<custom-ident>)); position: fixed; bottom: anchor(bottom); right: anchor(right); } </style>

The respective HTML and CSS is as follows:

<section>
  <h3 class="anchor" anchor-name="--anchor-3">Anchor 3</h3>
  <p class="anchored" anchor-name="--anchor-3">Content for Anchor 3</p>

  <h3 class="anchor" anchor-name="--anchor-4">Anchor 4</h3>
  <p class="anchored" anchor-name="--anchor-4">Content for Anchor 4</p>
</section>

<style>
  .anchor {
    anchor-name: attr(anchor-name type(<custom-ident>));
    margin: 50px;
  }

  .anchored {
    position-anchor: attr(anchor-name type(<custom-ident>));
    position: fixed;
    bottom: anchor(bottom);
    right: anchor(right);
  }
</style>

Accessibility Notes

Be careful with this idea, though it's useful to move elements around the screen arbitrarily but it's probably undesirable from an accessibility standpoint. Try to ensure that the elements are still related logically and ordered so that screen readers and other accessibility tools work as you'd intend

Generator Generation

Wed, 17 Dec 2025 00:00:00 GMT

So I was just going to write a short post about this handy function I made, but I thought it would be a nice opportunity to build a little bit more of an understanding around the topic of generators more broadly

Our end goal is going to be to define an abstraction that will allow us to convert any Callback-Based API into an Async Generator. Now, if those words don’t mean much to you, then welcome to the other side of JavaScript. If you’re just here for the magic function, however, feel free to skip to the end

Promises

Before diving into the complexity of generators, we're going to quickly kick off with a little introduction to Promises and how they relate to async/await and callback-based code

Promises are used to make async code easier to work with and JavaScript has some nice syntax - like async/await that makes code using promises easier follow and understand. They're also the common way to represent async operations which is exactly what we're going to use them for

Before we can dive right into implementing our function for creating an AsyncGenerator from a callback based API, it's important to understand how we might go about wrapping a callback based API into a Promise

Creating Promises from Callbacks

Often we end up in cases where we've got some code that is callback based - this is a function, for example setTimeout, that will invoke the rest of our code asynchronously when some task is done or event is received

A simple example of a callback based function is setTimeout which will resume the execution of our code after some specified amount of time:

setTimeout(() => {
  console.log('this runs after 1000ms')
}, 1000)

A common usecase is to convert this to a Promise so that consumers can work with this using async functions and awaiting the relevant function call

The basic method for doing this consists of returning a Promise and handling the rejection or resolution within the callback. For example, we can create a promise-based version of setTimeout using this approach:

function sleep(timeout: number) {
  return new Promise((resolve, _reject) => {
    setTimeout(resolve, timeout)
  })
}

This promise will now resolve when setTimeout calls the resolve method that's been passed to it. This allows us to turn some callback based code like this:

setTimeout(() => {
  console.log('done')
}, 1000)

Into this:

await sleep(1000)
console.log('done')

Granted, this isn't a huge difference - the value of this comes from when we have multiple of these kinds of calls nested within each other. Callback code is notorious for its tendency towards chaos. My rule of thumb on this is basically "less indentation is easier to understand". And if we can avoid indentation and keep our code flat we can focus on the essential complexity of our application and not the cognitive load that comes with confusing scope, syntax, and callbacks

This is such a common problem in the JavaScript world, that Node.js even has a builtin function node:util called promisify that converts Node.js style callback functions into promise-based ones

Async/Await

When working with promises, it's useful to define our methods using the async keyword, this allows us to work with a Promise using await and not have any callbacks

So we can have our function below, which can await the sleep function and it would be defined like so:

async function doWork(){
  await sleep(5000)
  console.log('fine, time to work now')
}

The doWork function returns Promise, this is because the async keyword is some syntax sugar for creating a Promise

Promises Vs Async

For the sake of understanding, all that the async keyword does allow us to remove the Promise construction from our function - async functions are simply functions that return a Promise - these are alternative syntax for the same thing - so, the following two functions are the same:

Using async:

async function getNumber() {
  return 5
}

Using an explicit Promise:

function getNumber() {
  return new Promise((resolve) => resolve(5))
}

Promise.withResolvers

Another pattern that often comes us is the need to reach into the Promise constructor and grab onto its resolve and reject methods and pass them around so that we can "remotely" compelete a Promise, as per MDN, the common pattern for doing this looks something like so:

function withResolvers<T>() {
  let resolve: (v: T) => void = () => {}
  let reject: (error: unknown) => void = () => {}

  new Promise<T>((res, rej) => {
    resolve = res
    reject = rej
  })

  return {
    resolve,
    reject,
  }
}

This method is also a recent addition to the Promise class via Promise.withResolvers, but for cases where it's not - the above should serve the equivalent purpose

Now that we've got an understanding of Promises, it's time to talk about Iterators and Generators

Iterators and Generators

Iterators and generators enable iteration to work in JavaScript and are what lies behind objects that are iterable by way of a for ... of loop

Iterator

An iterator is basically an object that will return a new value whenever its next method is called

A simple iterator can be defined as an object that has a next method that returns whether it's done or not.

function countToIterator(max: number): Iterator<number> {
  let value = 0

  const iterator: Iterator<number> = {
    next() {
      value++
      return {
        value,
        done: value === max,
      }
    },
  }

  return iterator
}

JavaScript uses the Symbol.iterator property to reference this object and therefore makes it possible for the language for ... of loop to iterate through this object. We can use the countToIterator's returned iterator to define an Iterable

What a mouthful right?? - But the implementation is actually easier than the description:

function countTo(max: number): Iterable<number> {
  const iterator = countToIterator(max)

  return {
    [Symbol.iterator]() {
      return iterator
    },
  }
}

Once we've got this, we can use the countTo in a loop:

// counts from 1 to 5
for (const v of countTo(5)) {
  console.log('count', v)
}

Could this be an array? Maybe. Arrays are really just a special case of an iterator. More generally, iterators are cool because they don't have to have a fixed endpoint, or even a fixed list of values. For example, we can create a different iterator that counts until a random point by modifying how the next function works:

function randomlyStopCounting(): Iterable<number> {
  let value = 0

  const iterator: Iterator<number> = {
    next() {
      value++
      return {
        value,
        done: Math.random() > 0.8,
      }
    },
  }

  return {
    [Symbol.iterator]() {
      return iterator
    },
  }
}


for (const v of randomlyStopCounting()) {
  console.log('count', v)
}

This is used the same as above, but the point at which this will return isn't really known beforehand. This dynamic behavior can come from lots of different places and not just from Math.random and it can allow some really interesting behaviors

Generators

Defining the above iterators is fun and all, but it's quite messy. The higher-order syntax for defining these kinds of iterators is using Generators

(I know, what's with all these words right??)

Okay, so generator functions are functions that return a special iterator called a Generator. If we weren't into the territory of weird syntax already - generators are defined using the function* keyword, and use the yield keyword to provide the next value. So we can rewrite our countTo function using a generator function and it would look like this:

function* countTo(max: number): Generator<number> {
  let value = 0

  while (value < max) {
    value++
    yield value
  }
}

That's actually way nicer to read right? When we yield a value the flow is delegated to the loop body, just like in the case of a normal iterator, which means the consumer can actually end the iteration early, like:

for (const v of countTo(5)) {
  console.log('count', v)
  if (v == 3) {
    break
  }
}

This also applies for the iterators above, I just find it so much more interesting in this case because there's no concept that someone is going to "call the next method again" which is so transparent in the iterator example above

Async Generators

Now, we're taking one more step - what if I wanted to do some long running task between each yield? This could be anything from waiting for a Promise to resolve, or a network request, or some user event (oh wow - there's an idea for multistep forms!)

Async Generators enable us to use promises in our iterators. Let's take a look at how we might define an async version of our countTo generator above:

async function* countToAsync(max: number): AsyncGenerator<number> {
  let value = 0

  while (value < max) {
    await sleep(1000)
    value++
    yield value
  }
}

Almost exactly the same right? Aside from the sneaky async keyword and the await in the loop, this is pretty similar to the sync version above. Using this also has a new little twist, can you spot it?:

for await (const v of countToAsync(5)) {
  console.log('async count', v)
}

Interesting right? We're now using a for await ... of loop. If you were to run this, you'd also notice that there's a little pause between each value being logged

Unwrapping the Generator

Now that we've seen what the inside of an iterator looks like - it's time to open the box and see what generators have inside

Let's start with the sync version

Inside a Sync Generator

So if we redefine our countTo generator without using the function* and yield syntax sugar, we'll see something like this:

function countTo(max: number): Generator<number> {
  let value = 0

  const generator: Generator<number> = {
    [Symbol.iterator]() {
      return generator
    },
    next() {
      value++
      return {
        value,
        done: value === max,
      }
    },
    return() {
      return undefined
    },
    throw(e) {
      throw e
    },
  }

  return generator
}

This looks very similar to an iterator - and that's because it is!

The Generator type inherits from the Iterator and needs an additional return and throw methods. The return method allows the generator to handle any cleanup once the consumer is done iterating. The throw allows any handling of errors and any other cleanup tasks

Inside an Async Generator

The async version of the above is almost identical - but we just sprinkle the async keyword around to make the respective methods all return a Promise as this is what the AsyncGenerator requires:


function countToAsync(max: number): AsyncGenerator<number> {
  let value = 0

  const generator: AsyncGenerator<number> = {
    [Symbol.asyncIterator]() {
      return generator
    },
    async return() {
      return undefined
    },
    async next() {
      await sleep(1000)
      value++
      return {
        value,
        done: value === max,
      }
    },
    async throw(e) {
      throw e
    },
  }

  return generator
}

for await (const v of countToAsync(5)) {
  console.log('async count', v)
}

Just like when defining the Async Generator before, we're just calling sleep between each return value. We've also made the return and throw methods async as well

Creating Generators from Callback Functions

Well, it's been a long way, but we finally have all the tools we need to turn a callback based method into an iterator. So far, we've been using setTimeout for our callbacks, but generators return multiple values. We're going to create a little modified version of setInterval for this so that we can play around

The version we'll define is called countInterval and will emit a new number until the given value and then stop, this looks like so:

function countInterval(
  max: number,
  onValue: (num: number) => void,
  onDone: () => void,
) {
  let value = 0
  const interval = setInterval(() => {
    if (value === max) {
      clearInterval(interval)
      onDone()
      return
    }

    value++
    onValue(value)
  }, 1000)
}

And the usage looks like this:

countInterval(
  5,
  (v) => console.log('count interval', v),
  () => console.log('count interval done'),
)

Assume for whatever reason that we want to be able to take functions like this and turn them into generators. In general, we can go about the process of manually defining a generator, but that's a little tedious and requires managing a lot of internal state. It would be nice if we could do this without having to manage the intricate details of a custom generator. We basically want to define countInterval such that it looks like this:

function countIntervalGenerator(max: number): AsyncGenerator<number> {}

For now, let's assume we've got a method called createGenerator that returns everything we need in order to hook up a generator and return it, this looks something like this:

function countIntervalGenerator(max: number): AsyncGenerator<number> {
  const { generator, next, done } = createGenerator<number>()

  countInterval(max, next, done)

  return generator
}

And our new generator can be used just as usual:

for await (const value of countIntervalGenerator(5)) {
  console.log('count interval generator', value)
}

Defining this wonderful createGenerator function combines what we've learnt about promises and generators to get this:

function createGenerator<T>() {
  let current = Promise.withResolvers<T>()
  let final = Promise.withResolvers<void>()

  const generator: AsyncGenerator<T> = {
    [Symbol.asyncIterator]() {
      return generator
    },

    async return() {
      const value = await final.promise

      return {
        done: true,
        value
      }
    },

    async next() {
      const value = await current.promise
      return {
        done: false,
        value,
      }
    },

    async throw(e) {
      throw e
    },
  }

  const next = (value: T) => {
    current.resolve(value)
    current = Promise.withResolvers()
  }

  const done = () => {
    final.resolve()
  }

  return {
    next,
    done,
    generator,
  }
}

This isn't doing anything that we haven't covered before. The only interesting bit (in my opinion) is how the next handler creates a new promise on each iteration - this implementation probably has some weirdness due to that so in practice you probably want to handle that edge case somewhat. This could maybe be done by updating the next function to take a parameter to indicate if it will emit a new value or not but in practice it's not always easy to figure that out - and in many cases you just don't know

That all being said, I think this implementation should make it clear how these pieces all fit together - there's a lot more detail that can be had in this discussion since each of these topics are fairly deep - but I hope this post was - if not useful - then at least interesting

References and Further Reading

Parsing Helix logs in Nushell

Tue, 16 Dec 2025 00:00:00 GMT

A super quick one today - I was doing some debugging on a Language Server for Helix and it was getting really annoying trying to find the data I was looking for so I wrote this quick Nushell command:

cat ~/.cache/helix/helix.log | 
    lines | 
    parse "{date} helix_lsp::transport [{type}] {source} <- {data}"

This will parse the Helix logs using the given format.

The data portion is also JSON, so adding the JSON parsing can be done with:

cat ~/.cache/helix/helix.log | 
    lines | 
    parse "{date} helix_lsp::transport [{type}] {source} <- {data}" |
    each {update data {|e| $e.data | from json}}

I also prefer normal JSON to the helix data view for this kind of data, so you can tac on | to json | jq to pipe it to jq for some nice JSON rendering and querying:

cat ~/.cache/helix/helix.log | 
    lines | 
    parse "{date} helix_lsp::transport [{type}] {source} <- {data}" |
    each {update data {|e| $e.data | from json}} |
    to json | jq

Coat Rack

Sun, 09 Nov 2025 00:00:00 GMT

Coat Rack is a platform that provides users with the necessary tools to own and share their data within their physical space. Enabling non-technical users to take control of their data ownership is the foundation on which the platform is built.

This enables users to take their daily utilities off the cloud and move their data to something they own while providing a user experience that is on par with that of cloud-based applications.

I've been working on the project with Logan Dam since early 2024. I recently gave a presentation on the project and though this may be a good time to mention it on my site as well. You can also find the coat-rack repo on GitHub if you'd like to contribute or just browse

Shader Web Component

Sun, 09 Nov 2025 00:00:00 GMT

import Example from './Example.astro'

So I want to use some more shaders and I want to also migrate everything over to use wgsl instead of glsl but it's kinda annoying to set them up every time so I made a little web component for using them, this is very much a work-in-progress, but here's it in action:

Using the component looks like so:

<script type="module" src="/web-components/shader-canvas.js"></script>

<site-shader-canvas>
  <canvas />
  <script type="text/wgsl">
    struct VertexOutput {
      @builtin(position) position: vec4f,
      @location(0) texcoord: vec2f,
    };

    @vertex fn vs(
      @builtin(vertex_index) vertexIndex : u32
    ) -> VertexOutput {
      const pos = array(
        vec2( 1.0,  1.0),
        vec2( 1.0, -1.0),
        vec2(-1.0, -1.0),
        vec2( 1.0,  1.0),
        vec2(-1.0, -1.0),
        vec2(-1.0,  1.0),
      );

      var vsOutput: VertexOutput;
  
      let xy = pos[vertexIndex];
      vsOutput.texcoord = pos[vertexIndex] * vec2f(0.5, 0.5) + vec2f(0.5);
      vsOutput.position = vec4f(pos[vertexIndex], 0, 1);

      return vsOutput;
    }

    @group(0) @binding(0) var<uniform> uTime: f32;
    
    @fragment fn fs(fsInput: VertexOutput) -> @location(0) vec4f {
      var red = abs(sin(uTime/10.0)) * fsInput.texcoord.x;
      var blue = abs(cos(uTime/5.0)) * fsInput.texcoord.y;
      return vec4f(red, 0.0, blue, 1.0);
    }
  </script>
</site-shader-canvas>

The component code is:

// @ts-check
import { setupCanvas } from './shader.js'

class ShaderCanvas extends HTMLElement {
  static observedAttributes = ['centered', 'highlight', 'large']

  /** @type {MutationObserver} */
  #observer

  /** @type {HTMLCanvasElement} */
  #canvas

  /** @type {HTMLScriptElement} */
  #script

  constructor() {
    super()
    this.#observer = new MutationObserver(() => this.#initialize())
    this.#observer.observe(this, { childList: true })
  }

  disconnectedCallback() {
    this.#observer.disconnect()
  }

  connectedCallback() {
    this.#initialize()
  }

  async #initialize() {
    console.log('here')
    const initialized = this.#canvas && this.#script
    if (initialized) {
      return
    }

    const canvas = this.querySelector('canvas')

    /** @type {HTMLScriptElement} */
    const script = this.querySelector('script[type="text/wgsl"]')

    if (!(script && canvas)) {
      return
    }

    this.#observer.disconnect()

    this.#canvas = canvas
    this.#script = script

    console.log(canvas, script)

    const render = await setupCanvas(this.#canvas, this.#script.innerText)

    function renderLoop() {
      requestAnimationFrame(() => {
        render?.()
        renderLoop()
      })
    }

    renderLoop()
  }
}

customElements.define('site-shader-canvas', ShaderCanvas)

And the code for actually doing the shader rendering pipeline is and is a heavily simplified version of what I'm currently using for my Image Editor

// @ts-check

/**
 * @param {HTMLCanvasElement} canvas
 * @param {string} shader - WebGPU Shader
 * @returns {Promise<((saveTo?: string) => void) | undefined>} renderer function. Will be `undefined` if there is an instantiation error
 */
export async function setupCanvas(
  canvas,
  shader,
) {
  // @ts-ignore
  const adapter = await navigator.gpu?.requestAdapter()
  const device = await adapter?.requestDevice()
  if (!device) {
    return
  }


  /**
   * @type {any}
   */
  const ctx = canvas?.getContext('webgpu')
  if (!ctx) {
    return
  }

  // @ts-ignore
  const format = navigator.gpu.getPreferredCanvasFormat()
  ctx.configure({
    device,
    format,
  })

  const module = device.createShaderModule({
    label: 'base shader',
    code: shader,
  })

  const pipeline = device.createRenderPipeline({
    label: 'render pipeline',
    layout: 'auto',
    vertex: {
      module,
    },
    fragment: {
      module,
      targets: [
        {
          format,
        },
      ],
    },
  })

  const uTime = device.createBuffer({
    size: [4],
    // @ts-ignore
    usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
  })

  let curr = 1

  /**
   * @param {string} [saveTo]
   */
  function render(saveTo) {
    curr += 0.1


    // https://stackoverflow.com/questions/70284258/destroyed-texture-texture-used-in-a-submit-when-using-a-video-texture-in-ch
    // render pass descriptor needs to be recreated since this doesn't live very long on the GPU
    const renderPassDescriptor = {
      label: 'render pass descriptor',
      colorAttachments: [
        {
          loadOp: 'clear',
          storeOp: 'store',
          clearValue: [0, 0, 0, 0],
          view: ctx.getCurrentTexture().createView(),
        },
      ],
    }

    const bindGroup = device.createBindGroup({
      layout: pipeline.getBindGroupLayout(0),
      entries: [{
        binding: 0,
        resource: { buffer: uTime }
      }],
    })

    const encoder = device.createCommandEncoder({ label: 'command encoder' })
    const pass = encoder.beginRenderPass(renderPassDescriptor)

    pass.setPipeline(pipeline)
    pass.setBindGroup(0, bindGroup)

    device.queue.writeBuffer(uTime, 0, new Float32Array([curr]));

    pass.draw(6) // call our vertex shader 6 times
    pass.end()

    const commandBuffer = encoder.finish()
    device.queue.submit([commandBuffer])
    if (saveTo) {
      // saving must be done during the render
      downloadCanvas(canvas, saveTo)
    }
  }

  return render
}

/**
 * @param {HTMLCanvasElement} canvas
 * @param {string} name
 */
function downloadCanvas(canvas, name) {
  const data = canvas.toDataURL('image/png')
  const link = document.createElement('a')

  link.download = name.split('.').slice(0, -1).join('.') + '.png'
  link.href = data
  link.click()
  link.parentNode?.removeChild(link)
}

Snippet: Rachets, not Levers - Chris Krycho

Mon, 03 Nov 2025 00:00:00 GMT

I came across this quote on Chris Krycho's Website that I thought was worth sharing:

"The real wins, then, are tools which do not require everyone to be at their best at every moment: ratchets, not levers. Levers let you move things, but if you are holding something up with a lever, you have to keep holding it — forever. A ratchet lets you drive forward motion without slipping back as soon as you let up on the pressure."

Week 37, Server Management

Sun, 14 Sep 2025 00:00:00 GMT

Off topic

Something off-topic but I just came across this explainer on the different task queues and related methods in Node.js that I thought would be a nice share

What I Found

I've been looking into/exploring some solutions for self hosting and had a little exploration/review of some tools with Kurt Lourens on some of the tech that's available

Proxies

Traefik is a reverse proxy that integrates really nicely with Docker by letting you configure endpoints via the container configuration in your environment
Caddy is a web server and reverse proxy that supports TLS out of the box and is really nice to configure

Application Management

Portainer is a management layer and UI for Docker and Kubernetes
Arcane is another UI for managing Docker containers
Komodo is a tool that simplifies the building and deployment of software across many servers

Côte d'Azur

Wed, 10 Sep 2025 00:00:00 GMT

import Layout from '../../../../layouts/PhotographyPortfolio.astro' import Gallery from '../../../../components/Gallery.astro' import PlacesLinks from '../../../../components/PlacesLinks.astro' import ReadingWidth from '../../../../layouts/ReadingWidth.astro'

<Layout title="Côte d'Azur, September 2025"> <section slot="gallery"> <ReadingWidth> <h2>Nice, France</h2>

  <p>
    We spent some time in the South of France recently.
    The weather for the entire trip was pretty fabulous, warm, sunny, beachy days.
    Most of our trip was spent in Nice where we stayed right on the border of the old city, filled with lovely markets and great food.
    The short walk up to the waterfall - Cascade du Château - was also fantastic and had a wonderful view of the coastline
  </p>
</ReadingWidth>

<Gallery path="places/2025-09-cote-d-azure/nice" />

<ReadingWidth>
  <h2>Menton, France</h2>

  <p>
    We also spent a little time exploring the region by train, my favourite stop was Menton - the entire city was pretty much just yellow and orange.
    Just a short train ride from Nice - lovely beaches and a lot less busy
  </p>
</ReadingWidth>

<Gallery path="places/2025-09-cote-d-azure/menton" />

<ReadingWidth>
  <h2>Monte Carlo, Monaco</h2>

  <p>
    I wasn't a huge fan of Monaco. Not very pedestrian friendly. The architecture was really interesting though, and the water was clearer than anything I've ever seen
  </p>
</ReadingWidth>

<Gallery path="places/2025-09-cote-d-azure/monaco" />

<ReadingWidth>
  <h2>Ventimiglia, Italy</h2>

  <p>
    Deciding to explore a bit further we took the first stop in Italy, Ventimiglia. A quiet coastal town. It's weird how just crossing the border completely changes the vibe of a town
  </p>
</ReadingWidth>

<Gallery path="places/2025-09-cote-d-azure/ventimiglia" />

<ReadingWidth>
  <h2>A side quest</h2>

  <p>
    I also wanted something to keep my eye on my surroundings so I could hunt for some interesting pictures,
    what drew my attention was the presence of laundry hanging on windowsills or balconies pretty much everywhere in the region -
    I suppose the sunny weather had something to do with that
  </p>
</ReadingWidth>

<Gallery path="places/2025-09-cote-d-azure/laundry" />

<ReadingWidth>
  <p>
    I also just want to say - damn, Fuji colours, amiright? -
    that's about it, thanks for reading!
  </p>
</ReadingWidth>

<br/>
<hr/>
<br/>

</section>

Week 36, Creativity

Sun, 07 Sep 2025 00:00:00 GMT

I've kind of always wondered how people make those crazy art installations with loads of lights and stuff, turns out there's software that exists for that

What I Found

Relatively obscure but here are a couple of generative art tools, one for music and one for everything?? They're both kinda fascinating, you should take a look

Strudel

Strudel is an a programming based music performing software that also looks super sick

TouchDesigner

While diving into some art on the internet this name kept coming up. TouchDesigner is a node based application for creating generative art and art installations that work with loads of different input and output types

Use a password manager, they said

Mon, 01 Sep 2025 00:00:00 GMT

I just had to boot up my computer to download Microsoft Edge because half way through the day I found out that the Passwords tab has been removed from the Microsoft Authenticator app

A few months ago the app notified me that Autofill would stop working - I was okay with this, I just started copying and pasting passwords. Next, the ability to add passwords was removed altogether - well okay, I haven't added a new password in a while so that's also fine. But then today I see that the passwords tab has been straight up removed

Turns out that Autofill doesn't just refer to Autofill. But the entirety of password management

This announcement outlines the details but I absolutely love the solution

Basically, you have to download Edge in order to get passwords to autofill at the OS level. But what if you don't want to use that but just want your passwords? Well you can still use Edge since that's now where passwords live

Okay. So I downloaded Edge on my phone (where I was using Authenticator) with the hope that I'd be able to export my passwords. Nope.

Get this - since the Android app does not support password exporting you're going to have to do this on desktop

So then I downloaded the desktop app, and after installing the malware that is Microsoft Edge on my computer, I can now export my passwords.

Okay. Got the passwords. All good.

Technically, yes. But my issue here isn't just technical.

Let's first address my reason for using Authenticator. So when 2FA was becoming a thing, companies wanted you to have it. At some point, I had to install it so that corporate 2FA would work. And, well, I wasn't going to use multiple 2FA apps - and to be completely honest, I was just a lot less critical of the companies making some of these tools at the time

So anyways, that's how everything got in there to begin with

Well, my browser is where I'm drawing the line. It's one thing to copy and paste passwords out of an app, it's something else for most of my online interactions to go through the giant data collection service that is Edge

The concern I share is broader though. I'm technical enough to find the documentation and get my data migrated over. But what if you don't know how do do all that export/import jazz, or you don't even have a desktop. What if you don't really understand the implications of having a single organization tracking you across all your interactions

It's as simple as pushing you towards a specific web browser, or search engine, or forcing you to log into Windows with your Microsoft Account - because that's the only way to use the computer that you own

Let's also not forget about Copilot that was recording every action users did along with literal screenshots

Passwords are pretty important these days. It kind of sucks that big organizations are continuously pulling us into these closed systems that become harder to escape at every step. I struggle to suggest a good solution at this point other that just finding another password manager that's not controlled by a big corp - or well, write all your passwords on a piece of paper I guess

There are also some other solutions like Yubikeys or WebAuthn or other FIDO stuff but they're going to be hit or miss depending on what services or devices you're accessing and I don't really know what support for these kinds of technologies is like so I don't think a 100% migration is even really possible

Week 35, Documentation matters

Sun, 31 Aug 2025 00:00:00 GMT

Trying to get back into some regular blogging rhythym since if I don't pay attention multiple months pass by between posts

What I'm working on

Wrote a bit about the impact Go has had on how I write software recently
Wrapped up work on Déjà vu which is a command line app to bring documentation to developers when they need it
Learning SwiftUI, basically watching pretty much everything Paul Hudson and Karin Prater have made

What I found

Mask

Something I spotted a while back and kind of just skimmed over, mask is a task runner (like make) that uses markdown for documenting commands. It seems great and is something I'd like to play around with in future

Gauge

On the topic of markdown, Gauge uses markdown docs to specify test specifications that are then backed by framework/language specific implementations to actually run those specs

Pages CMS

Pages CMS is a content management system I've been using for a while that provides a nice UI for working with markdown and JSON content and is actually what I'm writing this post on right now

Problems are better left solved

Wed, 27 Aug 2025 00:00:00 GMT

I don't enjoy writing Go. I like languages that challenge me, that make me think about processes, model the domain, and force me to be as complete as possible in my thinking. Go does none of this, but it seems to be the only language I program in lately

Corporate work, as you might imagine (or, more likely, understand to your core), is not fun. It tends to be a lot of talking around problems instead of solving them, and solving around metrics instead of people

I've been struggling to find ways to add value and have been moving my focus around a bit. Between people throwing AI at anything that will stick and pushing for MORE OUTPUT constantly, there's little room to make a meaningful impact. Most people are flooded with work and don't really have time to talk about big problems or to find solutions to anything real

I think small frustrations are often undervalued - particularly ones that people think are unique to them. These tend to be quite easy to solve, but since they're perceived as isolated problems we don't really bother to solve them

I've recently noticed a lot of issues popping up where the solution is "read the documentation"

It's something we emphasise a lot as technical people. What always falls by the wayside however, is what documentation? Where do I find it? How do I know what I'm doing is even documented - why would it be if it's a problem I'm only experiencing?

At the moment I'm on a team that develops and supports libraries that other teams use. A large chunk of this involves working with developers to help understand where and how to integrate with our libraries

When working with lots of developers you start to notice patterns. In the way people work, how they use their tools, and the types of issues they run into

Developers in some team may struggle with a bug for days, when across the hall, another team knows the fix

Sure, community channels help, but if the message isn't phrased in a certain way or if the right person doesn't see it then it's of no use

Often, we've documented the solutions, but no one seems to know.

The problem isn't that developers don't read the documentation, it's that we've failed to show it to them at the right time and in the right place

Usually when developers run into an issue, the first place they look is their terminal - do I see any errors? Is there a weird warning of some kind?

I think this space that occupies so much of our time and energy is underutilized, searching for error messages online is hit or miss, and good luck if it's something to do with your company's specific setup

Well, computers read faster than people. I had an idea that I could tie the terminal and documentation together - so I wrote an app that does just that

Déjà vu is a little app that scans the output of terminal commands in real time to find documentation that may be relevant for issues that developers are facing

It works using existing documentation and is a little program I put together in a few hours using Go

So I'm not a fan of Go.

I've been writing a lot of it recently though. It's easy to set up, has a great ecosystem, is pretty fast, and it just works. Things that would take days to solve in other languages are an evening of work with Go

Recently, the problems I've been trying to solve have revolved around other people. Hacky scripts and complex shell setups are not things they have. Solutions need to be free standing and "just work"

When it comes to other people, I tend to prioritize getting a solution quickly. Small solutions become less valuable as time passes, and people become more annoyed too.

Iterating across people is a lot slower than iterating with yourself, so being able to start conversations early is important.

All of this basically to say - Go helps me solve problems quickly. So what if I don't like it.

Scripting Text Manipulation

Fri, 01 Aug 2025 00:00:00 GMT

This is kind of just a brain dump of some things that I think would be interesting to explore

So, I've been using Zed a bit recently and have particularly enjoyed how it's MultiBuffers work. MultiBuffers basically allow you to simultaneously edit multiple files at once while also having things like syntax highlighting and language server support

I really wanted to add Zed to my toolchain because of this feature but it doesn't quite seem to be a good fit in lots of other ways

That being said - I really want multi buffer editing now so naturally my mind has shifted to thinking about how building something like this that meshes well with my workflow could look

There are two levels that I could see something like this working on and are basically what I'd like to explore at some point, the ideas are basically:

1. Literally just implementing MultiBuffers

Yup. Basically have some kind of method that lets you do a search and open that using a utility that can the flush the changes back to the original files

Imagine I want to edit all usages of the word "hello", I could have a buffer that's opened in my code editor via my-search 'hello' | my-edit | my-save or something like that, where the result of the search command would give us something that looks like this:

--- start my/file/path1.txt:120-122
some context lines
hello world
some more context
--- end my/file/path1.txt:120-122

--- start my/file/path2.txt:10-12
some context lines
hello world
some more context
--- end my/file/path2.txt:120-10-12

Making any changes to that would then update the ranges in the underlying files. This is nice because we can possibly do something like apply a macro to all instances of the word hello across what looks like a single file. Which can then be saved an applied

This could also be a simplified git patch view which is tweaked to make editing nice, the advantage of this is that applying this patch could be done just using git if it can be normalized back to a patch when saving

Now, there are still some challenges here that I'd still like to solve, like searching but getting the containing function instead of just some random range of files, which is why I think the search tool for producing these ranges also needs to be sufficiently smart

There are lots of other quality things that do come up, for example getting LSP support within these edit-blocks, but I think that's possibly doable. VSCode has something called Embedded Programming Languages but if something like this could make it's way through to the Language Server Protocol at large there could be a really cool user experience. But anyways, that's not part of this solution

Note that idea here isn't to solve this for any specific editor but rather to create a composable solution that can work with any text editor and provide a nice mechanism to efficiently enable multi file editing

2. A language for text manipulation

I've mentioned macros, in the context of code editors this typically refers to some kind of key you can press that triggers a predefined set of other keypresses (hopefully not recursively) that enables some relatively complex editing behavior. This is really powerful with tools like Vim, Emacs , or Helix which allow users to repeat certain sets of changes, for example I may want to tell the editor to go to first word of line, change it to X, go to end of line, add semicolon and then I'd record a macro which may be interpreted as I<esc>wdiX<space><esc>A;<esc> in Helix

The sequence can look scary but remember that it's an artifact of what we did, and not something we set out to create from the start

Now, what if I wanted to create these sequences from scratch and have them applied to a file? Well, the Emacs people have solved this by the looks of it, there are two tools for doing it, namely:

TXR which seems really complicated? IDK how this is even meant to be used
elmacro which looks like it's got a really nice API for writing macros but seems to run inside of Emacs which isn't what I'm looking for

The elmacros API seems really nice, here's the snippet from the docs:

(defun upcase-last-word ()
  (interactive)
  (move-end-of-line 1)
  (backward-word 1)
  (upcase-word 1)
  (move-beginning-of-line 1)
  (next-line 1 1))

I think this is a great example of a language for manipulating text

So my proposal is then - can we make something like this that can be run in isolation and has context awareness to some extent while still being generic enough to avoid language-specific things (like AST based text transformers)?

Basically take some file like this:

<h1>hello world</h1>

<p>
  <span>hello bob</span>
</p>

And use a script which in psuedo code may look like this:

# assume the "cursor" is at the start of the file
for-each search `hello`
  # goes to 'world' or 'bob'
  go-to-next-word-start
  
  # selects 'world' or 'bob'
  select $id
    go-to-end-of-word
  end select

  # searches for the containing 'h1' or 'p' tag
  go-to (previous `<h1`) or (previous or `<p`)

  # writes some text using the previously selected text
  append `id="$id"`
end for-each

To turn it into this:

<h1 id="world">hello world</h1>

<p id="bob">
  <span>hello bob</span>
</p>

This would allow us to define a macro for doing the manipulation above and could allow us to make mass changes to a single file

The code focus of this macro-style script would be that it's quick to write and modify, but doesn't have to scale since it's intended for on-the-fly, once-off modifications, much like how you'd use macros normally

It could also be really cool to make it possible to extract this from a Vim/Helix macro which can then be re-run in isolation or over a bunch of files

Bringing it together

So these two ideas seem quite different. One of them provides a method for editing many files at once in an interactive manner, and the other automates edits to a single file. I've proposed these two because I think that they've both got some interesting strengths

The first method for multi buffers allow us to do changes that may be more random or complex over multiple files and are great for where interventions are needed due to inconsistencies when doing large refactors

The second method allows us to automate predictable changes - the goal with this is to minimize the time needed to write a script for manipulating text files, which is often quite time consuming and difficult to understand. In doing so, it should make large changes more efficient and approachable for teams while lowering the barrier to entry overall

I think there's also room for composability between these two methods, and possibly even executing a script on a multi buffer using something like Helix's :pipe command

Why?

Okay so sure, we can edit files in some weird ways "fast". So what?

My idea here is that this can be a neat way to automate large code changes. Especially if those changes are macro-able. And I think the extension of a general purpose language for working with these kinds of syntax-aware macros can be really powerful. It could also just be a neat find-and-replace tool which is always useful

Also, it just sounds kind of fun to build and seems like it would fit well into my current text editing workflow. So even if there aren't any other users there would at least be one - which I think is perfectly fine for small software

but but but AI

No.

Update 14 April 2026

Just came across ast-grep which makes it possible to search in an AST aware way which could be very cool

Update 15 April 2026

Another interesting tool for this kind of text manipulation is fastmod which allows for regex based replacements in an interactive fashion for manual intervention

Web Authentication API

Tue, 29 Jul 2025 00:00:00 GMT

The Web Authentication API is a public/private key based authentication mechanism that is built into modern browsers and enables us to easily develop authentication solutions on the web

It consists of a few simple APIs, namely navigator.credentials.get and navigatior.credentials.create. A simple example that uses these APIs can be seen below:

Example

<script> let creds;

const createButton = document.createElement('button'); createButton.innerText = "Create Account"; createButton.onclick = async () => { const newCreds = await createCredentials(); alert("Created credentials are" + JSON.stringify(newCreds)); creds = newCreds; };

const loginButton = document.createElement('button'); loginButton.innerText = "Log In"; loginButton.onclick = async () => { if (!creds) { throw new Error("Credentials not created"); } const gotCreds = await getCredentials(creds.rawId); alert("Loaded credentials are" + JSON.stringify(gotCreds)); };

document.querySelector('#authn-example').appendChild(createButton); document.querySelector('#authn-example').appendChild(loginButton);

function getCredentials(id) { return navigator.credentials.get({ publicKey: { challenge: new Uint8Array([117, 61, 252, 231, 191, 241 /* … */]), rpId: window.location.host, allowCredentials: [ { id, type: "public-key", }, ], userVerification: "required", } }); }

function createCredentials() { return navigator.credentials.create({ publicKey: { challenge: new Uint8Array([117, 61, 252, 231, 191, 241 /* … */]), rp: { id: window.location.host, name: "Nabeel Credential Testing" }, user: { id: new Uint8Array([79, 252, 83, 72, 214, 7, 89, 26]), name: "jamiedoe", displayName: "Jamie Doe", }, pubKeyCredParams: [{ type: "public-key", alg: -7 }], }, }); }

function sleep(ms = 2000) { return new Promise((res) => setTimeout(res, ms)); } </script>

Code

And the code that implements the above can be seen as follows

let creds: Credential | null

const createButton = document.createElement('button')
createButton.innerText = "Create Account"
createButton.onclick = async () => {
  const newCreds = await createCredentials()
  alert("Created credentials are" + JSON.stringify(newCreds))

  creds = newCreds
}

const loginButton = document.createElement('button')
loginButton.innerText = "Log In"
loginButton.onclick = async () => {
  if (!creds) {
    throw new Error("Credentials not created")
  }

  const gotCreds = await getCredentials(creds.rawId)
  alert("Loaded credentials are" + JSON.stringify(gotCreds))
}

document.querySelector<HTMLDivElement>('#authn-example')!.appendChild(createButton)
document.querySelector<HTMLDivElement>('#authn-example')!.appendChild(loginButton)

function getCredentials(id: BufferSource) {
  return navigator.credentials.get({
    publicKey: {
      challenge: new Uint8Array([117, 61, 252, 231, 191, 241 /* … */]),
      rpId: window.location.host,
      allowCredentials: [
        {
          id,
          type: "public-key",
        },
      ],
      userVerification: "required",
    }
  })
}

function createCredentials() {
  return navigator.credentials.create({
    publicKey: {
      challenge: new Uint8Array([117, 61, 252, 231, 191, 241 /* … */]),
      rp: { id: window.location.host, name: "Nabeel Credential Testing" },
      user: {
        id: new Uint8Array([79, 252, 83, 72, 214, 7, 89, 26]),
        name: "jamiedoe",
        displayName: "Jamie Doe",
      },
      pubKeyCredParams: [{ type: "public-key", alg: -7 }],
    },
  });
}

function sleep(ms = 2000) {
  return new Promise((res) => setTimeout(res, ms))
}

Exploring the CSS Paint API

Thu, 10 Jul 2025 00:00:00 GMT

The CSS Painting API is part of the CSS Houdini group of APIs which provide low level access to the CSS engine

The CSS Houdini APIs use the idea of "worklets" which are basically JS files that the CSS engine will use as part of it's rendering pipeline

For the sake of this example, I'll be looking specifically at the CSS Painting API to build a custom CSS paint worklet

The CSS `paint` function

Before getting into specifics around how to implement a worklet, it's nice to see what we're trying to get to. Worklets are effectively JS functions that we can "call" from our CSS code to modify how an element is rendered. The CSS Painting API exposes a paint function in CSS. Let's assume we have a worklet called myCustomPainter

We would be able to apply this to some html code like so:

<h1 class="fancy">Hello world</h1>

And we can style this using the CSS paint function with the name of our worklet:

.fancy {
  background-image: paint(myCustomPainter);  
}

This will invoke our worklet to paint a custom background for our element

The Methods Available

The CSS Paint API exposes a few different methods and bits of functionality to us

Firstly, we have the methods needed for defining a worklet, this is the global CSS.paintWorklet.addModule method:

namespace CSS {
  declare const paintWorklet: {
    addModule(url: string): Promise<void>
  }
}

Next, we also have the registerPaint function which is a global function in the Worklet scope that is used to register a class that handles painting:

declare function registerPaint(name: string, paintCtor: PainterOptions): Promise<void>

The actual PainterOptions consists of two parts, a Paint class that handles the actual painting, and a static PainterClassRef that specificies some metadata about the Paint class:

declare interface Paint {
  paint(ctx: PaintRenderingContext, size: PaintSize, styleMap: StylePropertyMapReadOnly): void
}

type PaintCtor = new () => Paint

declare interface PainterClassRef {
  /**
   * CSS Properties accessed by the `paint` function. These can be normal or custom properties.
   */
  inputProperties?: string[]

  /**
   * Specififes if the rendering context supports transparency
   */
  contextOptions?: { alpha: boolean }

  /**
   * Inputs to the `paint` function from CSS.
   * Not supported in any browsers I've tested.
   * Chrome on MacOS will completely break rendering the `paint` function is passed
   * any values in the CSS
   */
  inputArguments?: string[]
}

type PainterOptions = PainterClassRef & PaintCtor

Lastly, for the sake of completeness, the remaining types used by the above definitions are:

/** Not exactly a Canvas but it's pretty similar */
declare type PaintRenderingContext = CanvasRenderingContext2D

declare type PaintSize = { height: number, width: number }

Defining a Worklet

A simple Paint class without any input properties looks something like this:

export class MyCustomPainter implements Paint {
  static get contextOptions() {
    return { alpha: true };
  }
  
  static registerPaint() {
    registerPaint("myCustomPainter", MyCustomPainter);
  }

  paint(ctx: PaintRenderingContext, _size: PaintSize, styleMap: StylePropertyMapReadOnly) { }
}

The registerPaint method isn't strictly necessary but it lets us keep the registration inside of the class which is nice

Registering this as a worklet is done in two steps:

From our main script, we need to run CSS.paintWorklet.addModule. I'm using the Vite url param to get this directly from the path to my worklet file:

import workletUrl from './worklet?url'

CSS.paintWorklet.addModule(workletUrl)

From the worklet, you need to register the MyCustomPainter as a painter:

import { MyCustomPainter } from "./my-painter";

MyCustomPainter.registerPaint()

Taking Inputs

In order for our worklet to do something fun we will probably want to take some inputs. We can specifiy which CSS properties (or custom properties) we want to use as an input - we do this via inputProperties. We can also register a custom property by using CSS.registerProperty

For our example, we'll also define a method in the MyCustomPainter class that does this:

// ... rest of class
static readonly colorProp = '--custom-painter-color'

static registerProperties() {
  CSS.registerProperty({
    name: MyCustomPainter.colorProp,
    syntax: '<color>',
    inherits: false,
    initialValue: 'transparent',
  })
}

static get inputProperties() {
  return [MyCustomPainter.colorProp]
}

// ... rest of class

Then, we need to call the registerProperties method above to register the custom property in the main.ts file (or somewhere in the normal page/JS context)

We can do this along with where we defined the worklet:

import { MyCustomPainter } from './my-painter'
import workletUrl from './worklet?url'

CSS.paintWorklet.addModule(workletUrl)
MyCustomPainter.registerProperties()

While it should be possible to get inputs as arguments to a painter, it seems that is not supported on any browsers as yet

This will make it so that we can provide inputs when painting. Inputs can come from our CSS, so using our worklet now looks like this:

.fancy {
  --custom-painter-color: yellow;
  background-image: paint(myCustomPainter);  
}

Then, we can access this in the paint method to do something like draw a rectangle over the entire canvas

// ... rest of class
paint(ctx: PaintRenderingContext, size: PaintSize, styleMap: StylePropertyMapReadOnly) {
  const color = styleMap.get(MyCustomPainter.colorProp)!.toString()

  ctx.fillStyle = color
  ctx.fillRect(0, 0, size.width, size.height)
}

The paint method receives a rendering context, the size of the element, and the styles that we defined in registerProperties (if they are set on the element)

It's also nice to note that since we've specified that --custom-painter-color has an initialValue it will not be undefined and the browser will provide us with the initialValue if it's not provided

And that's really about it. The API is pretty simple but powerful and makes it possible to do so much

The Complete Worklet

<h1 class="fancy">Hello world</h1>

.fancy {
  --custom-painter-color: yellow;
  background-image: paint(myCustomPainter);  
}

export class MyCustomPainter implements Paint {
  static readonly colorProp = '--custom-painter-color'

  static registerProperties() {
    CSS.registerProperty({
      name: MyCustomPainter.colorProp,
      syntax: '<color>',
      inherits: false,
      initialValue: 'transparent',
    })
  }

  static registerPaint() {
    registerPaint("myCustomPainter", MyCustomPainter);
  }

  static get inputProperties() {
    return [MyCustomPainter.colorProp]
  }

  static get contextOptions() {
    return { alpha: true };
  }

  paint(ctx: PaintRenderingContext, size: PaintSize, styleMap: StylePropertyMapReadOnly) {
    const color = styleMap.get(MyCustomPainter.colorProp)!.toString()

    ctx.fillStyle = color
    ctx.fillRect(0, 0, size.width, size.height)
  }
}

import { MyCustomPainter } from "./my-painter";

MyCustomPainter.registerPaint()

import { MyCustomPainter } from './my-painter'
import workletUrl from './worklet?url'

MyCustomPainter.registerProperties()
CSS.paintWorklet.addModule(workletUrl)

// Types for working with the CSS Paint API

namespace CSS {
  declare const paintWorklet: {
    addModule(url: string): Promise<void>
  }
}

declare function registerPaint(name: string, paintCtor: PainterOptions): Promise<void>

declare interface Paint {
  paint(ctx: PaintRenderingContext, size: PaintSize, styleMap: StylePropertyMapReadOnly): void
}

type PaintCtor = new () => Paint

declare interface PainterClassRef {
  inputProperties?: string[]
  contextOptions?: { alpha: boolean }

  /**
   * Not supported in any browsers I've tested.
   * Chrome on MacOS will completely break rendering the `paint` function is passed
   * any values in the CSS
   */
  inputArguments?: string[]
}

type PainterOptions = PainterClassRef & PaintCtor

declare type PaintRenderingContext = CanvasRenderingContext2D

declare type PaintSize = { height: number, width: number }

References

There are loads of things you can do with the Houdini APIs, some things I recommend reading and taking a look at on this topic are:

Notes

It's kinda annoying how many moving parts this has and that makes it a little challenging to include a live example on this blog. Hopefully the other examples I've linked above will serve this purpose

Some nice next things to look at from here are the other Houdini APIs since they offer very different sets of functionality and can be combined to do some interesting stuff

Accessibility Reference List

Mon, 07 Jul 2025 00:00:00 GMT

Resources

Some websites or content that cover accessibility topics

Courses

WCAG Digital Accessibility Foundations

Bookmarklets

Bookmarklets that make checking accessibility easier

IDK how to save a bookmarklet, look at your browser's documentation I guess

Patching packages with PNPM

Thu, 03 Jul 2025 00:00:00 GMT

Patching packages is a task that's occasionally done to fix bugs or update behavior of 3rd party dependencies within a specific project. This is often done while waiting for a fix from the upstream library or when the library author does not agree with the change that is needed

Creating a Patch

Say we have some package installed that we'd like to patch, for example we'll call this my-package. We currently have my-package version 1.1.1 installed and want to create a patch for that. We can kick this process off with:

pnpm patch my-package@1.1.1

Doing this will result in some output from pnpm with a path to a directory that contains the package's code that we can modify

To modify the code, open the given path in your editor and make the required changes

Once you're satisfied with the changes, you can use the following command to commit the patch:

pnpm patch-commit path/to/folder

This will do the following:

Create a patches/my-package@1.1.1.patch file with the patch
Add a reference to this patch in the pnpm-lock.yaml file
Install dependencies and apply this patch

The patch will be applied in future whenever dependencies are installed/added

Removing a Patch

To remove a patch you can use the pnpm patch-remove command:

pnpm patch-remove my-package@1.1.1

That will remove the patch from the patches directory as well as from the pnpm-lock.yaml file

Updating a Patch

To update a patch you will need to remove it and then re-create it using the info above

Notes

Patching is annoying, avoid it if you can

Resources

PNPM has documentation for all of the above mentioned commands on the Patching Dependencies Docs

Typescript debugging without an IDE

Wed, 18 Jun 2025 00:00:00 GMT

I generally steer clear of IDEs. Their overall slowness and clunkyness makes using them a hassle. There is however one place they execel - debugging.

Recently I've been looking into how to get Node.js to debug some JS and TS code and this proved to be relatively simple

A JS file

Debugging a Javascript app is actually pretty straightforward, you can use node inspect followed by the file to debug. So this just looks like so:

node inspect my-file.js

Runnning the above with a debugger; statement in your code will stop at the breakpoint and you can debug from there

You can then open your browser at chrome://inspect and select inspect on the process you'd like to debug

A TS File

Typescript requires you to have a few things installed. I'm going to assume that TypeScript is already installed in the project you're working in. Additionally, you need to install ts-node into your project - this can be done using whatever package manager you're currently using

You will also need to ensure that you have sourceMap enabled in your tsconfig.json file in order to have a bit of a decent debugging experience:

{
  "compilerOptions": {
	// ... other stuff
    "sourceMap": true
  }
}

Next, you can use node inspect with the ts-loader like so:

node inspect -r ts-node/register my-file.ts

The process then is pretty much the same as for the Javascript file debugging above. You can use debugger; statements to add breakpoints, and you can open the debugger at chrome://inspect

Scan based regex composition

Thu, 27 Mar 2025 00:00:00 GMT

Recently I needed a regex to camel-case some text, this sounds simple enough at first glance but since I had some very specific stylistic requirements it started to get more and more complex and the regex became pretty unwieldy

A colleague suggested the following function to get the job done:

const normalize = (str) => str.replace(/([a-z])([A-Z])/g,'$1-$2')

Now, this works on simple examples like:

normalize("HelloWorld") // Hello-World
normalize("helloThereBob") // hello-There-Bob

But it fails on more complex cases such as when we have numbers or when a word is all caps (abbreviations/initialisms?) such as these

normalize("URLParser") // URLParser, i want URL-Parser
normalize("Text1Parser") // Text1Parser, i want Text-1-Parser
normalize("MyURLExample1ForProgramX") // My-URLExample1For-Program-X, i want My-URL-Example-1-For-Program-X

After a lot of playing around I couldn't find a simple regex that let me do what I wanted. I did however like the idea of scanning across a string to make very specific, usually one or two character replacements. I liked the idea that different parts of the regex could be named and applied, using this idea - I thought we could have small regexes that do really simple things, such as:

const replacement = `$1-$2`
const aB = /([a-z])([A-Z])/g // aB -> a-B
const ABc = /([A-Z])([A-Z][a-z])/g // A-Bc
const wd = /(\w)(\d)/g // a1 -> a-1
const dw = /(\d)(\w)/g // 1a -> 1-a

Now, these small scanners work at the 2-3 letter pattern size and let us perform surgical replacements, scanning across text and applying a replacement on small subsets of a string.

Something like looking for a pattern in URLParser for a set of smaller patterns like: UR, RL,LP, Pa, ar, rs, se, and er.

This takes away the global concern of the regex and lets us think distinctly about the different subproblems. Applying these sub-solutions is also easy:

const normalize = (str) => 
	str.replace(aB, replacement)
	   .replace(ABc, replacement)
	   .replace(wd, replacement)
	   .replace(dw, replacement)

Testing this with the examples that failed earlier we see the following result:

normalize("HelloWorld") // Hello-World
normalize("URLParser") // URL-Parser
normalize("Text1Parser") // Text-1-Parser
normalize("MyURLExample1ForProgramX") // My-URL-Example-1-For-Program-X

And that's basically what I want.

The above solution shows us a general approach that can be used for building complex replacement structures. It is possible that it would be less efficient than a big complicated regex but I think where possible, this solution allows us to incrementally build and add functionality to regex-based solutions in a way that's much easier to understand and modify than one-shot regexes tend to be

Lastly, a quick note - the name of this post can probably use some work, the closest thing to this pattern that I can find is Composed Regex by Martin Fowler which breaks regexes down into smaller parts in order to create a bigger regex

Getting Started with the Language Server Protocol

Wed, 26 Mar 2025 00:00:00 GMT

import Snippet from '@/components/Snippet.astro'

Note that the code here serves as a high level example for the development of a language server and is in no means intended to be a production-ready implementation

The Language Server Protocol

Language Server Protocol Specification

The Language Server Protocol (LSP) is a specification for communication between code editing applications and language servers that provide information about the programming language being worked with

The protocol enables language related functionality such as diagnostics or documentation to be implemented once and reused in all editors that support LSP

The specification uses JSON RPC 2.0 for communicating between the editor and language server

Creating a Basic Language Server

Input and Output

A basic language server can be created using standard in and out as the communication mechanism for the server. This is quite easy to get access to using Node.js as we can get it from process like so:

const { stdout, stdin } = require("process")

stdout and stdin are streams, we can read from the stdin stream using the data event, and we can write to stdout directly

The code editor that's managing the language server will pass data to it using stdin, responses need to be sent using stdout

Each message that's sent/received consists of two parts:

A headers section containing a Content-Length header structured like Content-Length: 123
A content section

The header and content sections are separated by \r\n\r\n

The overall structure looks like this (\r\n shown as well since they are a strictly considered in the message)

Content-Length: 123\r\n
\r\n
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "example/method",
  "params": {
    ...
  }
}

There are two types of messages, namely Requests and Notifications. Request messages require a response. The id of the message is relevant and is sent back with a response, a Response looks a bit like this:

Content-Length: 123\r\n
\r\n
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    ...
  }
}

The other type of message is a Notification. Notification messages don't require a Response. Additionally, they will not have an id and will look something like this:

Content-Length: 123\r\n
\r\n
{
  "jsonrpc": "2.0",
  "params": {
    ...
  }
}

Initializing the Project

I'm assuming you've got some basic idea of how Node.js works and that you're familiar with the overall idea of creating a CLI tool with Node.

Since the language server needs a command, we need to do a little thing with npm to register the server as a command. This can be done by first defining our language server as a bin. To do this we need a JS file with the language server

In this example, I'm creating a folder with the following two files:

cli.js - the language server
package.json

In the cli.js you can simply add something like:

#!/usr/bin/env node

console.log("Hello from Language Server")

Next, you'll need to add the following to a package.json file:

{
  "name": "example-lsp",
  "version": "1.0.0",
  "license": "MIT",
  "bin": "./cli.js",
}

The name and bin fields are needed so that npm knows how to call your command after installing it.

Once the above two files are in place, you can install/register your command using:

npm i -g

This will make the example-lsp command available and it will automatically run the cli.js file

Linking to an Editor

The example below is for Helix, on other editors this process is different and is not the focus of this post

Language servers are started by or connected to from the code editing application. I'm using Helix which uses a config file that specifies the available language servers. We can reference the server we've created in Helix's languages.toml file like so:

## define the language server
[language-server.example]
command = "example-lsp"

## associate a language with the language server
[[language]]
name = "example"
scope = "source.example"
file-types = ["example"]
language-servers = ["example"]

Seeing your editor's logs may vary so I'm not going to get too into that, but if you're trying to debug issues with a language server that's probably a good place to start

Handling Messages

Once we've got our editor setup, we should be able to open a file with a .example extension which will trigger our LSP

When the language server is started it will receive an initialize event which will look something like this:

Content-Length: 2011

{"jsonrpc":"2.0","method":"initialize","params":{"capabilities":{"general":{"positionEncodings":["utf-8","utf-32","utf-16"]},"textDocument":{"codeAction":{"codeActionLiteralSupport":{"codeActionKind":{"valueSet":["","quickfix","refactor","refactor.extract","refactor.inline","refactor.rewrite","source","source.organizeImports"]}},"dataSupport":true,"disabledSupport":true,"isPreferredSupport":true,"resolveSupport":{"properties":["edit","command"]}},"completion":{"completionItem":{"deprecatedSupport":true,"insertReplaceSupport":true,"resolveSupport":{"properties":["documentation","detail","additionalTextEdits"]},"snippetSupport":true,"tagSupport":{"valueSet":[1]}},"completionItemKind":{}},"formatting":{"dynamicRegistration":false},"hover":{"contentFormat":["markdown"]},"inlayHint":{"dynamicRegistration":false},"publishDiagnostics":{"tagSupport":{"valueSet":[1,2]},"versionSupport":true},"rename":{"dynamicRegistration":false,"honorsChangeAnnotations":false,"prepareSupport":true},"signatureHelp":{"signatureInformation":{"activeParameterSupport":true,"documentationFormat":["markdown"],"parameterInformation":{"labelOffsetSupport":true}}}},"window":{"workDoneProgress":true},"workspace":{"applyEdit":true,"configuration":true,"didChangeConfiguration":{"dynamicRegistration":false},"didChangeWatchedFiles":{"dynamicRegistration":true,"relativePatternSupport":false},"executeCommand":{"dynamicRegistration":false},"fileOperations":{"didRename":true,"willRename":true},"inlayHint":{"refreshSupport":false},"symbol":{"dynamicRegistration":false},"workspaceEdit":{"documentChanges":true,"failureHandling":"abort","normalizesLineEndings":false,"resourceOperations":["create","rename","delete"]},"workspaceFolders":true}},"clientInfo":{"name":"helix","version":"25.01.1 (e7ac2fcd)"},"processId":34756,"rootPath":"/example-lsp","rootUri":"file:///example-lsp","workspaceFolders":[{"name":"example-lsp","uri":"file:///example-lsp"}]},"id":0}

We want to try to view and process this message on the language server.The first thing we're going to run into when trying do this is that we can't log things anymore since the stdout is used as a way to send messages to the code editor - so instead, we can output our logs to a file

A little log function will do that:

const { appendFileSync } = require("fs")

function log(contents) {
  const logMessage = typeof contents === 'string' ? contents : JSON.stringify(contents)
  return appendFileSync('./log', "\n" + logMessage + "\n")
}

So this will output the logs to a file called log and can be super useful to debug/inspect any issues that come up

Now, to receive a message we can listen to the data even on stdin and respond by logging on stdout

The initialize message expects a response with some basic information about the language server. Reading this message and responding can be done as follows:

const { stdout, stdin } = require("process")
const { appendFileSync } = require("fs")

function log(contents) {
  const logMessage = typeof contents === 'string' ? contents : JSON.stringify(contents)
  return appendFileSync('./log', "\n" + logMessage + "\n")
}

// listen to the `data` event on `stdin` returns a `Buffer`
stdin.on('data', (buff) => {
  // convert the buffer into lines
  const message = buff.toString().split('\r\n')

  // get the message content
  const content = message[message.length - 1]

  // parse the message content into a request
  const request = JSON.parse(content)
 
  // log the request to a file for later use
  log(request)

  if (message.method !== 'initialize') {
    // currently we only support the initialize message
    throw new Error("Unsupported message " + message.method)
  }

  // respond with a JSON RPC message
  const result = JSON.stringify({
    jsonrpc: "2.0",
    // reference the ID of the request
    id: request.id,
    // the result depends on the type of message being responded to
    result: {
      capabilities: {
        // we can add any functionality we want to support here as per the spec
      },
      serverInfo: {
        name: "example-lsp",
        version: "0.0.1"
      }
    }  
  })
  
  // create the Content-Length header
  const length = Buffer.byteLength(result, 'utf-8')
  const header = `Content-Length: ${length}`
  
  // join the header and message into a response
  const response = `${header}\r\n\r\n${result}`

  // send the response
  stdout.write(response)
})

Note that the above example assumes that the entire message is received completely at once - there is no actual guarantee of this but it's retained here for the sake of simplicity

This is the basic flow for building a language server. Listening to events from some input, often stdin/stdout and responding to it using JSON RPC. The types of messages that can be received along with the expected response or behavior is all outlined in the LSP Specification

Complete Example

A more complete example showing the handling of multiple messages can be seen below:

References

Typescript Workers in NodeJS

Wed, 12 Feb 2025 00:00:00 GMT

Using worker threads with Typescript is pretty straightforward, this probably depends a bit on your setup but the below idea seems to work for me

Single File Worker

In NodeJS we can use a sort of self-referrential worker such that depending on the context in which the file is loaded the behavior will differ, for example:

import { Worker, isMainThread, parentPort, workerData } from "node:worker_threads";

// the worker is the current file
const workerFileName = __filename

// define a type for the data we want to share
type WorkerData = {
  id: number,
  name: string
}

// check if we're on the main or worker thread
if (isMainThread) {
  // launch a worker thread
  const worker = new Promise((resolve, reject) => {
    const worker = new Worker(workerFileName, {
      // pass data to the worker
      workerData: {
        id: "my-worker",
        name: "Bob"
      }
    })

    worker.addListener("message", (result) => resolve(result))
    worker.addListener("error", reject)
  })

  worker.then(console.log)
} else {
  // do stuff for the worker thread
  console.log("Running in worker", workerData as WorkerData)
  const message = `Hello ${workerData.name} from worker`

  // return back to the parent thread
  parentPort?.postMessage({ message })
}

In the above file we can see a few things that are important to using workers:

A check is done using isMainThread to determine if the file is being executed as a worker. the handling for the worker is based on this
A reference to the file which should be run in the worker thread, the __filename variable is defined by NodeJS and holds the name of the curent file - this will be the compiled file name so we don't need to figure that out on our own
A worker is created using new Worker, since the worker is an event based API in the above example it's being wrapped in a Promise
Data can be passed to the worker via the workerData property, this will then be put into the workerData variable from node:worker_threads in the worker thread
Data is sent back to the worker using parentPort.postMessage. This is the same even that's being listened for the Promise to resolve in worker.addListener("message", resolve)

Multi File Worker

Extending the above to work with multiple files is pretty simple, the only additional trick we need is to export the __filename from our worker file to the place where we want to construct the worker

So, the updated code can then be broken into two files as follows:

First, the worker is pretty similar to the one above, except now we export the __filename as workerFileName

import { isMainThread, parentPort, workerData } from "node:worker_threads";

export const workerFileName = __filename

export type WorkerData = {
  id: number,
  name: string
}

if (!isMainThread) {
  console.log("Running in worker", workerData)
  const message = `Hello ${workerData.name} from worker`
  parentPort?.postMessage({ message })
} else {
  console.log("Worker was run in the main thread")
}

The main.ts file then makes use of this exported file name to invoke the worker. Note that this is now just a normal script and will be executed in the main thread as normal

import { Worker } from "node:worker_threads"

// import the worker file name from where we defined the worker
import { workerFileName } from "./worker"

const workers = [1, 2, 3, 4].map(id => new Promise((resolve, reject) => {
  const worker = new Worker(workerFileName, {
    workerData: {
      id,
      name: `Bob ${id}`
    }
  })

  worker.addListener("message", (result) => resolve({ id, result }))
  worker.addListener("error", reject)
}))

async function main() {
  const results = await Promise.all(workers)
  console.log(results)
}

main()

Uncompiled Worker Using `ts-node`

When executing workers it's also possible to execute the Typescript file directly instead of relying on the compilation. This can be useful for cases where the Javascript code is compiled to a single file or in situations where you don't have control over the execution environment

Executing a Typescript worker can be done by using ts-node when running the worker, an example of this is:

const worker = new Worker("my-worker.ts", {
    execArgv: ["-r", "ts-node/register", "--no-warnings"],
    workerData: {
      id
    }
  })

// do stuff with the worker

References

NodeJS Worker Threads

Web Workers and Vite

Mon, 06 Jan 2025 00:00:00 GMT

import Snippet from '@/components/Snippet.astro'

Web workers provide a mechanism for running code outside of the main thread using Javascript. They are currently supported in the browser as well as in Node.js, but for the purpose of this post we'll look at how to use them from within the browser

Vanilla

Web workers can be used directly from the browser using some script tags. For the sake of example, we'll use the following HTML file with it's related script tags:

In the above, we can see three scripts referenced, we'll take a look at these as we talk about the mechanisms for using web workers

Workers

Web workers are simply Javascript modules that are loaded by some other code dynamically as a Worker

If we assume we have some worker called simple-worker.js in the same directory of our page, we can load it using the following code:

const worker = new Worker("./simple-worker.js");

Upon doing this, the simple-worker.js script will be loaded and executed.

Now, the real value of workers comes from the ability to get data from the code that launches it, this is done using worker.postMessage, this allows us to send any data we want to the worker we have defined, so we can do this in our script, an example of this can be seen below:

On the other side of this conversation we have the actual worker script/module. This actually looks really simple:

This file uses self.addEventListener to listen to messsages sent to the worker, when a message is received, the worker can decide what to do with it - what's important to know is that this handler does not block the main thread so it can be as intensive as we want

The worker can also be more complex, for example we can see in the below message that the worker receives a message via self.addEventListener, after doing some intensive processing, it calls self.postMessage to basically respond to the parent

The parent can then listen to these messages with worker.addEventListener

And we can see the worker that sends messages as well:

ServiceWorkers

ServiceWorkers are a special kind of worker that acts as a proxy between the application and network and can interact with network requests even when the device is not connected to the server

Service workers have different events that may be triggered, commonly is the installed, activated, and fetch event among others, we add listners for these events the same as we did in the above

In the above file, we intercept requests to the placeholder url and proxy it by making a new request to https://placeholder.co, we could do anything here though, even responding with our own data if we want

ServiceWorkers are special though since an application can only have a single ServiceWorker. Creating a ServiceWorker is done using navigator.serviceWorker.register and not using new Worker, an example of service worker registration can be seen below:

Vite

In practice we're rarely using some random Javascript files though and often have more complex scenarios with external dependencies or even stuff like Typescript involved, in this case we need to consider things like bundling as well as somehow resolving the name of the file we need to load. Thankfully modern bundlers like Vite have a solution for us

In the case of Vite specifically, it provides us with an API that we can use while importing files that Vite uses to infer that a file should be treated as a worker. Vite does this by using the ?worker at the end of the import path

Assume we have some Vite-based project that uses a worker such as the one below. That worker also imports some code from another file we have and we expect it all to be bundled correctly:

We can see that we have a run function that is called when we receive the message as before, we also have a type specified for the data we expect to receive

When we import something using ?worker Vite creates a constructor for us that will initialize the worker without us having to manually pass the path to the file, so a consumer of this file can now use the worker a bit like this:

import CanvasWorker from './canvas.worker?worker'
import type { InvokeParams } from './canvas.worker'

const worker = new CanvasWorker()

worker.postMessage({ canvas: offscreen } satisfies InvokeParams, [offscreen])

Note that we also use the type import above so we can keep the type definition of the data our worker expects in the same palce as the worker

A more full example of this can be seen below:

Something that's also interesting to note is the second parameter of postMessage, this is an array of TransferrableObjects that are specific objects that the browser allows us to share between the main thread and worker threads. In the above example we're using a OffscreenCanvas but this can be any of the objects that are defined as TransferrableObjects

Conclusion

Overall, web workers are pretty fun to play around with and really increase the options available to us when working to make applications more performant. Modern tools like Vite also make using them pretty easy

There's a lot that you can do with workers and I've simply provided you with a small overview of the functionality, looking at the MDN documentation is always a good place to go to learn more

References

Javascript Proxy Object

Wed, 11 Dec 2024 00:00:00 GMT

Proxies allow us to wrap existing object and modify the behavior when interacting with them in some interesting ways. This post will take a look at a few things we can use them for

Something to work with

For the sake of this example we'll be using Typescript. Let's create a reference object type that we will interact with - we're going to call this MyApi and it's defined simply as an object with a few methods on it:

interface MyApi {
  /**
   * Adds 2 numbers
   * @returns the result of addition
   */
  add: (a: number, b: number) => number;

  /**
   * Difference between 2 numbers
   * @returns the result of subtraction
   */
  subtract: (a: number, b: number) => number;

  /**
   * (secretly does some illegal stuff)
   */
  illegal: (a: number, b: number) => number;
}

Initial implementation

We can implement a simple object that satisifies this APi as below:

const baseApi: MyApi = {
  add(a, b) {
    return a + b;
  },

  subtract(a, b) {
    return a - b;
  },

  illegal(a, b) {
    return a / b;
  },
};

Log accesses

For this example we'll consider the illegal method as special. We'll want to track each time the illegal property is accessed. Using a Proxy we can wrap the baseApi and provide a get method that will handle property access and allow us to see what property of our object is being accessed

When illegal is accessed, we log out a message:

const logAccess = new Proxy<MyApi>(baseApi, {
  get(target, key: keyof MyApi) {
    if (key === "illegal") {
      console.log("Tried to access illegal method");
    }

    // return the property from the original object
    return target[key];
  },
});

logAccess.illegal(1, 2);
// logs out the message before calling the function

Prevent illegal access

We can also do stuff like create a type of WithoutIllegal version of MyApi in which we remove the property from the type definition. Additionally, we can also make this throw an error if someone attempts to access it at runtime as can be seen below. This is very similar to the previous example but now we have a direct impact on the consumer

type WithoutIllegal<T> = Omit<T, "illegal">;

const police = new Proxy<WithoutIllegal<MyApi>>(baseApi, {
  get(target, key: keyof MyApi) {
    if (key === "illegal") {
      throw new Error("accessing illegal properties is a crime");
    }

    return target[key];
  },
});

try {
  // @ts-expect-error this is now an error since we say "illegal is not defined"
  police.illegal;

  // @ts-expect-error if we try to access it, it will throw
  police.illegal(1, 2);
} catch (err) {
  console.error("Got illegal access", err);
}

Interact with other objects

During the proxying process, we can also do things like interact with objects that aren't defined within our base object itself:

const logs: any[][] = [];
const withLogs = new Proxy<MyApi>(baseApi, {
  get<K extends keyof MyApi>(target: MyApi, key: K) {
    const method = target[key];

    return (a: number, b: number) => {
      logs.push(["accessing", key, a, b]);

      const result = method(a, b);

      logs.push(["result", result]);

      return result;
    };
  },
});

withLogs.add(1, 2);
withLogs.subtract(1, 2);
withLogs.illegal(1, 2);

console.log(logs);
// [
//   [ 'accessing', 'add', 1, 2 ],
//   [ 'result', 3 ],
//   [ 'accessing', 'subtract', 1, 2 ],
//   [ 'result', -1 ],
//   [ 'accessing', 'illegal', 1, 2 ],
//   [ 'result', 0.5 ]
// ]

Creating fake objects

We can also create an object in which any properties can exist and have a specific value, for example an object that has this structure for any key given:

{
    myKey: "myKey"
}

This can be done like so:

const fake = new Proxy<Record<string, Record<string, string>>>(
  {},
  {
    get(target, property) {
      return {
        [property]: property,
      };
    },
  }
);

console.log(fake);
// {}

console.log(fake.name);
// { name: 'name' }

console.log(fake.age);
// { age: 'age' }

console.log(fake.somethingelse);
// { somethingelse: 'somethingelse' }

It's also interesting to note that the fake object has no direct properties, and we cxan see that when we log it

Recursive proxies

Proxies can also return other proxies. This allows us to proxy objects recursively. For example, given the following object:

const deepObject = {
  a: {
    b: {
      getC: () => ({
        c: (x: number, y: number) => ({
          answer: x + y,
        }),
      }),
    },
  },
};

We can create a proxy that tracks different actions, such as property acccess. Recursive proxies can be created by returning a new proxy at the levels that we care about. In the below example, we create a proxy for every property that we access as well as for the result of every function call:

const tracker: any[][] = [];
const createTracker = <T extends object>(obj: T, prefix: string = ""): T => {
  return new Proxy<T>(obj, {
    apply(target, thisArg, argArray) {
      tracker.push(["call", prefix, argArray]);

      const bound = (target as (...args: any[]) => any).bind(thisArg);
      const result = bound(...argArray);

      tracker.push(["return", prefix, result]);

      if (typeof result === "undefined") {
        return result;
      }

      // create a new proxy around the object that's returned from a function call
      return createTracker(result, prefix);
    },

    get(_, prop: keyof T & string) {
      const path = `${prefix}/${prop}`;

      tracker.push(["accessed", path]);
      const nxt = obj[prop];

      if (typeof nxt === "undefined") {
        return nxt;
      }

      // create a new proxy around the object that's being accessed
      return createTracker(nxt as object, path);
    },
  });
};

const tracked = createTracker(deepObject);

const result = tracked.a.b.getC().c(1, 2);

console.log({ result });
// { result: { answer: 3 } }

console.log(tracker);
// [
//   [ 'accessed', '/a' ],
//   [ 'accessed', '/a/b' ],
//   [ 'accessed', '/a/b/getC' ],
//   [ 'call', '/a/b/getC', [] ],
//   [ 'return', '/a/b/getC', { c: [Function: c] } ],
//   [ 'accessed', '/a/b/getC/c' ],
//   [ 'call', '/a/b/getC/c', [ 1, 2 ] ],
//   [ 'return', '/a/b/getC/c', { answer: 3 } ]
// ]

References

MDN Proxy Docs

Conditionally Protect Properties in Typescript

Tue, 03 Dec 2024 00:00:00 GMT

Type Guards

So, type guards are really handy in Typescript as they let us check if something meets a certain requirement before moving along, for example, given the following user type:

type User = {
  active: boolean;
  name: string;
  age: number;
};

We can define a type guard that checks if the user is fully is active before allowing certain things. To do this we usually use a type guard, that looks like this:

type ActiveUser = User & { active: true };

const isActive = (base: Partial<User>): base is ActiveUser =>
  !!(base.active && base.age && base.name);

The important thing in a type guard is that it takes something of one type and asserts something about that type, e.g. that it is an ActiveUser in the above. This is done by returning a boolean, if it is true then the assertion applies, otherwise it does not

And normally, we would use it like so:

if (isActive(user)) {
    // do stuff that can only be done with an active user
}

// outside of this scope users are not active

Gaurded Class

The above solution is usually good enough. But it's also possible to couple the types of these checks while not providing direct access to the underlying object. This is useful in cases where we may want to restrict access to some functionality unless a certain set of checks pass

To do this, we can encapsulate the value and check into a class, e.g. the Guarded class below:

class Guarded<Unsafe, Safe extends Unsafe> {
  constructor(
    protected readonly value: Unsafe,
    private readonly safe: (value: Unsafe) => value is Safe,
  ) {}

  /**
   * Type guard that grants access to the wrapped `value`
   */
  isSafe(): this is { value: Safe } {
    return this.safe(this.value);
  }
}

In the above, we define a Guarded class htat has a type of a Safe and Unsafe value. These generics can be inferred by the arguments provided to constructor

Next, we define the isSafe method on the class that is a gaurd that says something about this which is the instance itself.

So, we can use Guarded to wrap something that we want to only make available under certain scenarios:

const guarded = new Guarded(value, isActive);

if (guarded.isSafe()) {
  console.log('Can access value here', guarded.value);
}

// guarded.value; // error: Property 'value' is protected and only accessible within class 'Guarded<Unsafe, Safe>' and its subclasses.ts(2445)
console.log("Can't access value here");

What's great is that this safe check is re-run whenever we want to access the value, this means that we can make certain properties available based on some other state that can be checked in the safe method

This approach means that at compile time the Typescript compiler will check that we are only accessing value within a context that we have defined to be okay by way of our isSafe method. Additionally, since we can specify the behavior of isSafe we end up with a a check that works at the intersection of compile-time and runtime

Note

The first idea of Type Guards is quite often used and should be good enough for normal use

Withr egards to the Guarded idea, use with caution. The idea here it to create a certain safety for consumers of some code, but this is at the expense of complexity and should be used with care. Outside of the "can it be done" discussion I haven't really had much use for something like this since usually a normal type guard is good enough

Type safe URL templates

Thu, 31 Oct 2024 00:00:00 GMT

I don't think the below is a good idea, but I thought it was a fun little Typescript excercise so here you go

Something I see as a sort of recurring pattern is people doing template replacements in URLs, particularly something like this:

const template = '/users/{userId}/projects/{projectId}'

Now, while you can more simply make this template a function like so:

const template = (userId:string, projectId:string) => `/users/${userId}/projects/${projectId}`

For some reason people don't like to do that, I think it's that it looks like a lot of duplication and perhaps they just don't believe in making their own functions (who am I to know)

More often than I'd like, I instead see people doing this:

const template = '/users/{userId}/projects/{projectId}'

// later
const url = template.replace('{userId}', userId).replace('{projectId}', projectId)

I think this is horrible, and while it works, it's quite brittle and unless it's got some tests or other means of verifying that this works it's a really quick source of bugs for when things change, for example if you add or remove a parameter from the URL

Now, I'm a huge fan of using the compiler to derive as much correctness as possible, even before even really thinking about tests, so when looking at something like this all I can think is that the TypeScript compiler can solve this

I'm not saying that you should do what I'm about to present here, I think you're probably better off using the method of a template being a simple function above, but I think it's a fun little mental excercise

The Typescriptssss

Firstly, we need to parse this URL template things, this is simple enough using a little recursive type:

/**
 * Extracts the keys from the URL
 */
type PathKeys<TUrl extends string> = 
    TUrl extends `${string}{${infer Param}}${infer Rest}` 
    ? Param | PathKeys<Rest> 
    : never

Next, since in a URL all keys need to be a string at the end, we can define an object for URLs that uses this to define a record:

/**
 * Uses the keys to define a record in which each key of the URL can be assigned to a string
 */
type PathParams<TUrl extends string> = Record<PathKeys<TUrl>, string>

A little test of the above types shows us:

type Keys = PathKeys<'/users/{userId}/projects/{projectId}'>
//   ^? type Keys = "userId" | "projectId"

type Params = PathParams<'/users/{userId}/projects/{projectId}'>
//   ^? type Params = { userId: string; projectId: string; }

So great, that works and we can use that as the basis for building up more general function for creating these "safe urls"

Before we go there though, let's take a moment to think about the result of this URL. Now we could return the URL as a string but since we're being strict it would be nice to inform downstream consumers of what this string is made up of. Using a very similar method to how the PathKeys type was done we can create a type for the result of a URL where the params have been replaced

/**
 * Represents a URL in which all params have been substituted
 */
type ReplacedUrl<TUrl extends string> = 
    TUrl extends `${infer Base}{${infer _}}${infer Rest}` 
    ? `${Base}${string}${ReplacedUrl<Rest>}` 
    : TUrl

Lastly, we can define the implementation of the URL replacer which is effectively just calling the String.replace method on an input URL while using a strong type for the URL:

const createTypedUrl = <const TUrl extends string>(url: TUrl) => (params: PathParams<TUrl>) => {
    return url.replace(/\{\w+\}/g, (val) => {
        const key = val.slice(1,-1)
        return  (params as Record<string, string>)[key] || val
    }) as ReplacedUrl<TUrl>
}

Using this looks like so:

const template = '/users/{userId}/projects/{projectId}' as const

/**
 * A little factory for making typed URLs from the given template
 */
const typedUrl = createTypedUrl(template)
//    ^? const typedUrl: (params: PathParams<"/users/{userId}/projects/{projectId}">) => string

const url = typedUrl({
    userId: '123',
    projectId: '456'
})
// ^? const url: `/users/${string}/projects/${string}`

console.log(url) // '/users/123/projects/456'

Now, is this better than random string replacements? Yes. Is it better than just using string interpolation? No.

While this isn't something I recommend using I think the idea about thinking of underlying structure of common data types is important and something that's worth thinking about in order to define more complete solutions

Lastly, if you're into this type of thing, take a look at some of my other TS shenanigans:

week 40, year 2024 - paper and pictures

Sun, 06 Oct 2024 00:00:00 GMT

firstly, a small sadness - i missed the past two posts due to my vacation, but on the other hand i guess i had a vacation so that works out i suppose

what i'm working on

spent the past week mostly working on a talk that i'm (hopefully) going to be doing towards the end of the month, i also gave a little intro presentation on nushell that I think went pretty well

what i found

this week i'm going to be sharing a few sites i've come across of people doing interesting things

paper engineering

Julia Yus makes really interesting things out of paper and her portfolio is a really window into the things that you wouldn't have thought were possible with paper

fashion photography

George Morris is a London based photographer and director with a portfolio of images and videos focused on some campaigns that he's done

street photography

Eric Anderson is a photographer and web designer with a particularly great portfolio of street photography

week 37, year 2024 - ui paradigms and comic books

Sun, 15 Sep 2024 00:00:00 GMT

between doomscrolling and visiting FOAM there was barely any time to put something together but here we go

what i found

i spent a decent amount of time comparing different ways of doing the same things. recently i've been thinking about ui state management, but first - some art

mœbius

mœbius is the pseudonym of Jean Henri Gaston Giraud who was a french cartoonist. i've found his work fascinating for quite some time. the level of detail packed into each frame is astonishing

state management

ui state is always something where there are an infinite number of solutions, each with their own drawbacks or drawfronts (of course that's the opposite right?)

reactive programming

something old to get us started. i think reactive programming (using things like RxJS or hand-rolled equivalents) give us a good starting point for managing state and this gist by André Staltz looks like a nice intro to the topic

signals

i came across a library called S.js which is one of the earlier signal implementations around and works with a jsx based framework called surplus. both of which are made by Adam Haile and last updated about 6 years ago (so probably don't pitch this during standup on monday)

if you're looking for a more modern implementation of signals, your favourite framework probably has something: angular - and a bunch of other stuff

state machines

when talking about state management i always like to put xstate on the table since it's a really handy library with some great developer experience and tooling built in. i wrote a little intro a while back if you'd like to get a feel for the library as well

strictness

something more obscure. i find the approach of elm for ui quite interesting and is outlined in the elm architecture. i first came across it in the elmish book which presents this method for building interfaces using f# and fable which is compiler that turns f# into javascript.

week 36, year 2024 - pretty printing

Sun, 08 Sep 2024 00:00:00 GMT

this week has been a little insane so i'm actually amazed i've found 5 mins to put this together

what i found

spent some time playing around with two little projects, one of which are pretty useful and the other is somewhat conceptual

delta

this is a cli tool for better git diffs and provides syntax highlighting for your git diff and some git commands. it's also written in rust if you care about that sort of thing

fancy fonts

i also came across a font which renders markdown as rich text using some fancy font features. it's a little glitchy but really cool that this works without any JS

additionally, there's also this font which does syntax highlighting

overall, fonts are getting pretty cool these days. the chrome devblog has a cool bit about colrv1 fonts which let us do some pretty fancy things too

it's also worth taking at this thread which has some links to font related stuff worth checking out

week 35, year 2024 - datetimes and logic programming

Sun, 01 Sep 2024 00:00:00 GMT

what i'm working on

after many months i spent some time on a project i'm working with a friend to build a sort of "home cloud" platform. it's always weird coming to a project having no clue where you left it and trying to stitch some context from whatever happens when you run pnpm start

i also found a handy little way to open links from the vscode terminal

what i found

i spent a little time learning about logic programming and came across an interesting talk about some of the challenges when writing libraries

prolog

a logic programming language - super weird but this talk on prolog by Michael Hendricks helped shed some light on how this is something that's actually usable

standard libraries

i also came across a great talk about some challenges when maintaining something like the kotlin standard library by Vsevolod Tolstopyatov

An unexpected way to open links in the terminal

Wed, 28 Aug 2024 00:00:00 GMT

I accidentally hit cmd + shift + o while in the terminal on VSCode which popped open a little window to navigate to the links to files that were currently printed out in my terminal - this seems like a more keyboard friendly way to do cmd + click on a link in the terminal

Upon further investigation it looks like the following is a thing:

When in the terminal, after running some command
Find a link (URL or file) that you want to open
Use the command pallette with cmd + shift + p and use the Terminal: Open Detected Link ... command
Browse the links, if it's a file you'll automatically preview the file, click enter to select a file or link to open

Tadaa, that's about it - an accidental shortcut that I'll most definitely be using in the future

A Visual Language for Image Manipulation

Sat, 24 Aug 2024 00:00:00 GMT

I've spent the past few weeks building a concept photo editing application that allows users to visually define and compose shaders to transform an image in ways that are really challenging or impossible in other image manipulation software

For readers of my website it's probably not too much of a surprise that I've been spending a lot of time learning about Parser Combinators as well as Shaders. Armed with this information, it looks like I accidentally created a Visual Programming Language - this post will talk about how that all works

Why

Every time I've shown anyone the app the first question was "Why do you need this?". I think it's fair to say that no one really needs this. This exists to satisfy the artistic urge to do create something new. I've spent so much time feeling limited by the tools I have for working with images are lacking in creative expression. I think there's an intersection between computational/generative art and photography that's underserved and it's a space I would like to play in. I wanted to create a tool that allows creative exploration of images and helps break away from the "filter" based approach that dominates how we edit images

It also seemed like a challenge, it begged to be done.

The App

The high level architecture that makes this all work looks something like this:

I define different types of functions that can be applied to an image using a shader function written in WebGPU
The shader is then parsed into a Node that a user can use in the editing workflow
A user can structure Nodes and Edges into a tree for applying edits
The tree is compiled into shader code and inputs to the shader (Bindings/Uniforms) are identified
The final shader is applied to image and is updated when a user modifies any Bindings

Shaders

Shaders are programs that run on the GPU. Due to this fact there are some interesting limitations on how shaders work and what they can do. For working in our application a shader consists of two parts, a Vertex Shader and a Fragment Shader. The Vertex Shader is related to objects that are being rendered in a scene, since we're rendering an image this isn't of too much consequence, and for our purposes this always looks the same:

struct VertexOutput {
  @builtin(position) position: vec4f,
  @location(0) texcoord: vec2f,
};


@node fn vs(
  @builtin(node_index) nodeIndex : u32
) -> VertexOutput {
  const pos = array(
    vec2( 1.0,  1.0),
    vec2( 1.0, -1.0),
    vec2(-1.0, -1.0),
    vec2( 1.0,  1.0),
    vec2(-1.0, -1.0),
    vec2(-1.0,  1.0),
  );

  var vsOutput: VertexOutput;

  let xy = pos[nodeIndex];
  vsOutput.texcoord = pos[nodeIndex] * vec2f(0.5, 0.5) + vec2f(0.5);
  vsOutput.position = vec4f(pos[nodeIndex], 0, 1);

  return vsOutput;
}

The data output from the Vertex shader is sent to the Fragment shader. We've defined a struct for the data I want to pass called VertexOutput

The Fragment shader is where things get interesting though. This is where I apply transformations to pixel data and it allows us to do stuff to our image. For now I've defined the main part of our Fragment shader as follows:

@group(0) @binding(0) var ourSampler: sampler;
@group(0) @binding(1) var ourTexture: texture_2d<f32>;

// BINDINGS_SECTION_START
// BINDINGS_SECTION_END

@node fn sample(texcoord: vec2f) -> vec4f {
  return textureSample(ourTexture, ourSampler, texcoord);
}

@fragment fn fs(fsInput: VertexOutput) -> @location(0) vec4f {
  var input__result = fsInput;
  // MAIN_SECTION_START
  return textureSample(ourTexture, ourSampler, fsInput__result.texcoord);
  // MAIN_SECTION_END
}

In the above I have also defined a function called sample which has the @node decorator - this is important - our app will use this decorator to identify methods that will be available in the UI that can be applied to images. The sample decorator simply gets the color of a pixel in the input for a given coordinate. In fragment shader I need to output a color as denoted by the return type of the fs function

From the perspective of a shader our goal is to provide a library of functions that apply transformations that users can compose to apply an edit to an image, a simple example of this are the splitChannels and combineChannels functions that we can see below:

struct SplitChannelsOutput {
  r: f32,
  g: f32,
  b: f32,
  a: f32,
};

@node fn splitChannels(color: vec4f) -> SplitChannelsOutput {
  var output: SplitChannelsOutput;

  output.r = color.r;
  output.g = color.g;
  output.b = color.b;
  output.a = color.a;

  return output;
}

@node fn combineChannels(r: f32, g: f32, b: f32, a: f32) -> vec4f {
  return vec4f(r, g, b, a);
}

The above methods can be composed in interesting ways to do things like swap around the green and blue channels of an image:

@fragment fn fs(fsInput: VertexOutput) -> @location(0) vec4f {
    var split = splitChannels(color);
    var result = combineChannels(color.r, color.b, color.g, color.a);
    return result;
}

So this is the basic concept, a bit of complexity comes in when we consider that it's possible that a function may want inputs from multiple other functions but this still doesn't create too much of a hurdle for us as we'll discuss later

Parsing

I wanted to ensure that the application has a single source of truth for the available functions. In order to do this I wanted to use the shader to inform us as to what functions should be available for a user. In order to do this I had to create a parser for the WebGPU langauge. The parser I built is a little simplified since there are only certain things I care about when parsing the shader. It's also optimized for maintenance and probably isn't very fast. My main goal here was to get something working. I wrote the parser using a library called ts-parsec which lets me define parsers using parser combinators.

Defining a parser using this library requires a list of tokens to be defined using regexes. Below are the tokens I currently have defined:

enum Tok {
  Var = 'Var',
  Let = 'Let',
  Const = 'Const',
  Struct = 'Struct',
  Return = 'Return',
  Fn = 'Fn',
  At = 'At',
  Arrow = 'Arrow',
  Integer = 'Integer',
  Comma = 'Comma',
  Dot = 'Dot',
  Plus = 'Plus',
  Semicolon = 'Semicolon',
  Colon = 'Colon',
  Dash = 'Dash',
  Asterisk = 'Asterisk',
  Slash = 'Slash',
  Equal = 'Equal',
  OpenParen = 'OpenParen',
  CloseParen = 'CloseParen',
  OpenBrace = 'OpenBrace',
  CloseBrace = 'CloseBrace',
  OpenBracket = 'OpenBracket',
  CloseBracket = 'CloseBracket',
  OpenAngle = 'OpenAngle',
  CloseAngle = 'CloseAngle',
  Identifier = 'Identifier',
  Space = 'Space',
  Comment = 'Comment',
}

Along with the tokenizer. Each token is configured as [shouldKeepToken, regex, token]

export const tokenizer = ts.buildLexer([
  [true, /^var/g, Tok.Var],
  [true, /^let/g, Tok.Let],
  [true, /^const/g, Tok.Const],
  [true, /^struct /g, Tok.Struct],
  [true, /^fn/g, Tok.Fn],
  [true, /^return/g, Tok.Return],
  [true, /^->/g, Tok.Arrow],
  [true, /^@/g, Tok.At],
  [true, /^,/g, Tok.Comma],
  [true, /^\./g, Tok.Dot],
  [true, /^\+/g, Tok.Plus],
  [true, /^\;/g, Tok.Semicolon],
  [true, /^\:/g, Tok.Colon],
  [true, /^\-/g, Tok.Dash],
  [true, /^\*/g, Tok.Asterisk],
  [true, /^\//g, Tok.Slash],
  [true, /^\=/g, Tok.Equal],
  [true, /^\(/g, Tok.OpenParen],
  [true, /^\)/g, Tok.CloseParen],
  [true, /^\{/g, Tok.OpenBrace],
  [true, /^\}/g, Tok.CloseBrace],
  [true, /^\[/g, Tok.OpenBracket],
  [true, /^\]/g, Tok.CloseBracket],
  [true, /^\</g, Tok.OpenAngle],
  [true, /^\>/g, Tok.CloseAngle],

  [true, /^\d+?/g, Tok.Integer],
  [true, /^[A-Za-z]+([A-Za-z0-9_]*)/g, Tok.Identifier],
  [false, /^\s+/g, Tok.Space],
  [false, /^\/\/ .*/g, Tok.Comment],
])

You can see that I throw away comments and spaces since I don't care about these for now

A relatively simple parser that I have is

All parsers in my application are defined using classes to just help keep things a little tidy. One of the simpler parsers I have is the one that parses integers and is defined as follows:

export class Integer {
  constructor(public value: number) {}

  static parser = ts.apply(ts.tok(Tok.Integer), (t) => new Integer(+t.text))
}

This consists of a constructor for defining the members and a static parser function which defines a parser for the current class. We can compose these parsers to build more complex ones, for example the way I parse Generic uses Integer for an optional length that a generic may have, for example array<f32, 3>:

export class Generic {
  constructor(
    public type: TypeDeclaration,
    public length?: Integer,
  ) {}

  static parser = ts.apply(
    ts.kmid(
      ts.tok(Tok.OpenAngle),
      ts.seq(
        TypeDeclaration.parser,
        ts.opt(ts.kright(ts.tok(Tok.Comma), Integer.parser)),
      ),
      ts.tok(Tok.CloseAngle),
    ),
    (matches) => new Generic(...matches),
  )
}

The composition of these parses results in a tree looking something like this:

TypeDeclaration {
  "generic": Generic {
    "length": Integer {
      "value": 5,
    },
    "type": TypeDeclaration {
      "generic": undefined,
      "identifier": Identifier {
        "name": "i32",
      },
    },
  },
  "identifier": Identifier {
    "name": "array",
  },
}

At a high level, we can apply this to the entire file to get a syntax tree for the whole file, but as you can imagine that starts to get much bigger

The UI

Now, since I'm able to parse the file, I can extract the functions from it and use that to define the different things a user can do in the UI. A simple example of some UI that uses the sample function defined previously can be seen below:

In this we are able to define Nodes that have certain inputs and outputs which can be connected to each other. Each node is a function call and each edge refers to using the result of one function as an input to another function. When defining connections it's important to verify that connections are of the same type and don't create a cycle in any way - while fiddly, the code to do this isn't particularly interesting

This step is actually relatively simple, the complexity comes into converting this back to the WebGPU code.

Compiling

This tree structure effectively represents a Dependency Graph where all functions depend on their inputs, and the final output node depends on any previous edges. The goal in this step is to build a tree out of the nodes and edges and then sort this in a way that defines the order that different operations need to be applied to obtain the output value

The first step to this is compiling a list of nodes and edges into a tree. I've done this using the following:

interface ReadonlyOrderedMap<T> extends ReadonlyMap<Id, T> {
  orderedValues(): ReadonlyArray<T>
  orderedKeys(): ReadonlyArray<Id>
}

/**
 * Wraps the builtin `Map` with an easy way to get keys and values as an array
 * sorted by the original insertion order
 *
 * @see MDN: [Map is insertion-order aware](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map)
 */
class OrderedMap<T> extends Map<Id, T> implements ReadonlyOrderedMap<T> {
  orderedValues() {
    return Array.from(this.values())
  }

  orderedKeys() {
    return Array.from(this.keys())
  }
}

export function buildTree<T extends Node, E extends Edge>(
  nodes: T[],
  edges: E[],
): OrderedMap<Connected<T, E>> {
  const map = new OrderedMap<Connected<T, E>>()

  for (const node of nodes) {
    map.set(node.id, { ...node, connections: [] })
  }

  for (const edge of edges) {
    const from = map.get(edge.from)
    const to = map.get(edge.to)

    if (from && to) {
      from.connections.push([edge, to])
    }
  }

  return map
}

This helps us go from the list of connections to a tree that represents the relationships between the nodes - this looks pretty much the same as the UI from earlier

Next, we need to sort the nodes in an order that ensures that each node comes after any nodes that it depends on, this is called a Topographic Sort and I've loosely implemented it as follows:

/**
 * Does a depth-first search on the input vertices relative to the given ID.
 *
 * @returns a map of the nodes reachable from
 */
export function topologicalSort<T extends Vertex, E extends Edge>(
  vertices: ReadonlyMap<Id, Connected<T, E>>,
  relative: Id,
) {
  const visited = new OrderedMap<Connected<T, E>>()

  const rec = (at: Connected<T, E>) => {
    if (visited.has(at.id)) return

    for (const [_, child] of at.connections) rec(child)

    // Insert self after all children have been inserted to maintain ordering
    visited.set(at.id, at)
  }

  const initial = vertices.get(relative)
  if (!initial) {
    console.error('Relative node not in tree', relative, vertices)
    throw new Error('Relative node not provided')
  }

  rec(initial)
  return visited
}

Using this function, I can transform the earlier example for sampling an image into a list that looks something like [input, sample, output]

What's nice is that this will also work with a more complicated set of connections like so:

Can be compiled into [input, sample, splitChannels, combineChannels, brighten, output]. It's worth taking a moment to let this idea sink in because this is the core idea that allows us to compile the UI based language into a shader

Once we are able to order our nodes, it's a simple matter of iterating through them in order and determining which inputs are needed and which functions these nodes refer to, the output of this looks something like:

@fragment fn fs(fsInput: VertexOutput) -> @location(0) vec4f {
    var value1 = sample(input.texcoord);
    var value2 = splitChannels(value1);
    var value3 = combineChannels(value2.g, value2.r, value2.b, value2.a);
    var value4 = brighten(value3, value2.b);
    return value4;
}

User Input

So, it's all well and good that I can compile a shader, but as I have it right now a user can't really customize what any of these values do. It would be nice if I could allow users to provide some input to the shader. For this we will use a uniform which is basically an input to the GPU. Uniforms are nice because we can change them without having to recompile the shader

For my purpose, I can infer that any input to a node that is not specified is meant to be set by the user, we can depict this using some kind of input field for that value, in the example below we use a range input for a f32 value, this can be set by the user:

Then, when compiling our nodes, we can extract this to a binding, which would look something like this:

@group(1) @binding(0) var<uniform> combineChannelsInputB: f32;

And that can be used in the fragment shader as follows:

@fragment fn fs(fsInput: VertexOutput) -> @location(0) vec4f {
    var value1 = sample(input.texcoord);
    var value2 = splitChannels(value1);
    var value3 = combineChannels(value2.g, value2.r, combineChannelsInputB, value2.a);
    var value4 = brighten(value3, value2.b);
    return value4;
}

Now, actually passing this value to the shader is a bit messy but really comes down to using the WebGPU API. This effectively involves creating a bindGroup adding buffers which hold the value of the data

const dynamicBuffers: Record<BindingId, GPUBuffer> = {}

const entries: GPUBindGroupEntry[] = []
for (const { id, bindings } of shader.compiled.compiledNodes) {
  for (const binding of bindings) {
    const buffer = device.createBuffer({
      size: binding.size,
      usage: GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
      label: 'dynamic binding for: ' + binding.declaration,
    })

    buffer.unmap()

    entries.push({
      binding: binding.binding,
      resource: {
        buffer,
      },
    })

    dynamicBuffers[`${id}.${binding.name}`] = buffer
  }
}

const dynamicBindGroup =
  entries.length &&
  device.createBindGroup({
    label: 'dynamic bind group',
    layout: pipeline.getBindGroupLayout(1),
    entries,
  })

Whenever the user changes some input the buffer values can be updates like so:

for (const [groupId, group] of Object.entries(data || {})) {
  for (const [bindingId, value] of Object.entries(group || {})) {
    const buffer = dynamicBuffers[`${groupId}.${bindingId}`]
    if (buffer) {
      device.queue.writeBuffer(buffer, 0, value)
    }
  }
}

Now, there are some finicky bits involved in assembling all the data here but these are the important bits around using the API

Closing

As you can see there are quite a few moving pieces but I feel like the approach I've taken here fits the problem relatively well. As things are defined now, all app functionality is driven by what is available in the shader and the different nodes can be inferred and combined however the user wants

This isn't the end though. There's a lot more to do, and a lot of art to be made

References

The most important resource in making this work is defintely the WebGPU Fundamentals Site. The UI is inspired by Blender's Node Editor and is implemented using React Flow

For the purpose of implementation, I've also looked at loads of different things, of notable value were the following:

week 34, year 2024 - programming languages of various types

Sat, 24 Aug 2024 00:00:00 GMT

what i'm working on

i've been putting some hours into my image editing app and think i have something that technically works, at the moment it doesn't look like much but proves that the visual programming language idea works:

i wrote a blog post explaining how the application works{rel="nofollow ugc noopener"} which explains the concepts around everything all the way from shaders to the ui

what i found

this week was pretty busy but i managed to get some time to speak to some other devs and we got chatting about configuration languages

rcl

i bumped into Ruud{rel="nofollow ugc noopener"} who works on a reasonable configuration language{rel="nofollow ugc noopener"} which a language for defining configuration that outputs to more common formats like json

pkl

this chat also reminded my about another configuration language made by apple{rel="nofollow ugc noopener"} which was released a while ago and is a language for config as code with support for validation

week 33, year 2024 - compilers, videography and good intentions

Sat, 17 Aug 2024 00:00:00 GMT

what i'm working on

this week i spent some time writing a small compiler to take my node based ui and compile it into a web gpu shader that i can then render images with - and ... that works now. i'm currently in the process of creating a solution for passing user input data between the ui and the gpu which shouldn't be too difficult - that's pretty much the last thing i need to do before i can have something that i can actually edit some photos with

additionally, i spent far too much time fighting with typescript and wrote a bit about generic function overloads and even more generic functions

what i found

crafting interpreters

i've been reading crafting interpreters by Robert Nystrom which takes you through the entire process of implementing an interpreter for a custom programming language

the art of showing up

this vlog by Life of Riza is a little story about just being present, it's a really uplifting watch and just an exciting little journey overall

a phoneless summer

a "TVlog" by Anna Maria Luisa dealing with the awareness of time passing. excellent vibes and great writing

then next comes

if you share my internet you may have seen exurb1a's latest existential crisis which starts off as a look into a potential future of humanity until it takes us full circle into wishing we had any idea what anything is at all

and thanks for reading, till next week!

Parameters, but only sometimes

Fri, 16 Aug 2024 00:00:00 GMT

Ran into this question today and I thought it would be a nice little example to document:

I have the following function doWork that is generic:

function doWork<T>(data?: T): void {
  console.log(data)
}

This function can be called in any of the following ways:

// Works
doWork<string>('hello') //  T is string, data is string
doWork() // T is undefined, data is undefined
doWork(undefined) // T is undefined, data is undefined;

doWork<string>() // T is string, data is undefined

In the above usage, we want to make it so that users of this function need to provide the data parameter when calling the function when the type is provided. Now, a simple solution could be to define our function as follows:

function doWork<T>(data: T): void

The problem is that it's a bit ugly for cases where we want to allow T as undefined, because you now have to do this:

doWork(undefined)

So the issue here is that we only want to make the data parameter required when T is not undefined, there's a lot of funny stuff you can try using generics and other weird typescript notation, but simply defining the necessary overloads for our method works:

/**
 * This specifically handles the case where T is undefined
 */
function doWork<T extends undefined>(): void
/**
 * The data param will be required since T is not undefined here
 */
function doWork<T>(data: T): void
/**
 * This provideds an implementation that is the same as before while providing a
 * better interface for function users
 */
function doWork<T>(data?: T): void {
  console.log(data)
}

Now, having defined those function overloads, we can use the function and it works as expected:

// Works
doWork<string>('hello') //  T is string, data is required
doWork() // T is undefined, data parameter is not required
doWork(undefined) // T is undefined, specifying data is still okay

// Error: Type 'string' does not satisfy the constraint 'undefined'
doWork<string>() // this overload is not valid

The added benefit of this method is that we can also write the doc comments for each implementation separately which can be a nice way for us to give additional context to consumers. It's kind of like having two specialized functions without the overhead of having to implement them independently

More generic than it should be

Thu, 15 Aug 2024 00:00:00 GMT

A method for designing highly generic APIs in Typescript

The Pattern

A recurring point I've seen when designing highly generic API's in Typescript is around giving consumers of an API an interface that requires the least amount of extra information from them. For example, take the function below:

interface Data {
  name: string;
  age: number;
}

declare const data: Data;

function getProperty<D, K extends keyof D>(key: K) {
  return function getProp(data: D): D[K] {
    return data[key];
  };
}

In the above example, we define a function called getProperty that gets the key property of data, we would use this like so:

const getName = getProperty<Data, "name">("name");

The problem here is that the user of this function has to provide both generics when using the function since the object with type D is only provided to the function that is returned. The experience of a consumer here is a little cumbersome and would be nicer if we didn't have to do this.

Some Other Ideas

We can also try the following where we just provide the name, but this won't work:

const getName = getProperty("name"); // Error: Argument of type 'string' is not assignable to parameter of type `never`

This is because the function doesn't know what type it's dealing with. The obvious next thing we can try is just provide a single type argument, but this doesn't work either since typescript needs us to provide both generics since we're not defaulting the second one

const getName = getProperty<Data>("name"); // Error: Expected 2 type arguments, but got 1

We could default the type of K which would take away our error but would also completely remove the benefit of it being a generic parameter in this case which is to narrow down the type the function returns

Now, a general strategy here is to instead swap around our type arguments, what if we defined getProperty instead by defining D in terms of K, something like:

function getProperty<K extends string>(key: K) {
  return function getProp<D extends Record<K, any>>(data: D): D[K] {
    return data[key];
  };
}

Now, this works and may be okay for your usecase, but often we want to constrain the getProperty function with the type of data known in advance. Also it can be more complicated than just a record with the key of the input type and this isn't always something we can extend more generally

This also has the issue that when using the getProperty function we end up with a function that lets anything through that maybe has a name property which may not be what we want

const getName = getProperty("name"); // getter is not constrained by the type of `D`

A Weird Solution

What can be a better solution is to split our function into a part that provides a generic and a part that provides data:

function createGetter<D>() {
  return function getProperty<K extends keyof D>(key: K) {
    return function (data: D): D[K] {
      return data[key];
    };
  };
}

This can now be used in a way where we don't specify any more generics than we need and has the weirdness of us invoking a function twice for no reason other than providing a generic

const getName = createGetter<Data>()("name");

You can also make this a bit easier on the eyes by creating an intermediate variable:

const dataGetter = createGetter<Data>();
const getName = dataGetter("name");

This isn't too bad, and I'd be okay to leave it here, but after speaking to some other developers I end up having a lot of questions around how the mechanics of the generic type need to work as well as the fact that we have multiple levels of functions returning functions that creates confusion when trying to use this code

A Classier Way

Weirdly, thinking of this as a class seemed to help. Instead of using a function to preload a generic, we can use a class:

class Getter<D> {
  property<K extends keyof D>(key: K) {
    return function (data: D): D[K] {
      return data[key];
    };
  }
}

I think this code reads a bit nicer and it's clear where the different generics come from. Ultimately it's doing the exact same thing but we're hiding the complexity of initializing the generic behind some syntax

Using this is quite nice now as well:

const dataGetter = new Getter<Data>();
const getName = dataGetter.property("name");

What's also different with using the class approach is that things have names now, we know that we are calling the .property method which is giving us something to work with. Something about using classes gives people a bit more familiarity when working with generic things and creates a good relationship between the mechanics of the method and the mental model most developers have of class based development

Gross

I'm more and more moving towards the idea that this kind of generic stuff should be kept to a minimum. This is a relatively simple example but these concepts can get pretty complex and perhaps speaks to something more fundementally incorrect about the data structures being worked with. If you can control the data structures, then ideally simplify them so you don't need this level of type magic, but if you do - it's always handy to have a way to express things that makes it easier for other developers to understand and work with

week 32, year 2024 - branded types, nodes, and image segmentation

Sat, 10 Aug 2024 00:00:00 GMT

what i'm working on

i've still been spending a decent amount of time on my photo editing app and progress has been good. this week i put together a basic concept of how i want the ui to work as well as how i can translate configuration from the webGPU backend i'm using to the ui

what i found

`react-flow`

react-flow is a library for building node-based user interfaces and has been pretty great to play around with and pretty much does what it says on the box

branded types

branded types are a way to bring some ideas around primitive types from functional languages to typescript. i find them to be a generally useful thing to know. i think they play very well with the concept of type-first design, as far as it goes this post by Scott Wlaschin is always something i recommend. as far as doing this in typescript i think this thread by matt pocock is a pretty good reference as well as this post on typescript.tv

image segmentation

i came across the segment anything model by meta which is a pretty cool model for image segmentation and is just generally fun to play around with

week 31, year 2024 - webgpu and parsers

Fri, 02 Aug 2024 00:00:00 GMT

welcome

since this is my first post in this format i figured i should just say hey! this series will be a bit of a braindump for project updates and links to things i found this week

what i'm working on

right now i'm the concept of a photo editing app that uses shaders for applying image effects, the initial concept seems to be promising but there's still a fair amount of work to be done before i can have anything even remotely usable

i also had an excuse to use git bisect in a new and interesting way and made some additions to my git notes about using the —first-parent and replay functionality

what i found

`ts-parsec`

i've been pretty invested in parser combinators recently and have been playing around with the ts-parsec library and it's been a pretty fun time. for a general idea of what these things are all about you can take a look at my blog post on parser combinators

`WebGPU`

webgpu is the "new" way to create shaders on the web and comes with a new javsacript API and a new language called wgsl

the api is a bit more complex than the one for webgl but still reasonably manageable if you take a read through the webgpu fundamentals site

additionally, i found a few great resources for using webgpu for image filters, namely work by Alain Galvan and Maxim McNair

Life, Gleam, and Parser Combinators

Sat, 20 Jul 2024 00:00:00 GMT

Time

There's never enough time to do the things we want to do. Recently I find myself struggling with this. There are too few hours in a day, too few days in a week. Between work and trying to simply stay alive there aren't many hours left to do things you care about

For the past year or so I've made a consistent effort to take my camera with me everywhere. It's always either in my bag or in my hand. And I've taken a lot of pictures. The thing is, I feel that my photos have stagnated - they're no better than they were a year ago, and arguably worse than five years ago. I seem to have hit some kind of local maximum, the point where I put just enough effort to passable but not nearly enough time to do anything useful

As far as my photography goes, I'm planning to take fewer pictures, but spend a lot more time looking for ways to be better and create work that means more to me.

The other area of my life that's suffered due to poor time management is my technical learning. I've found programming to be an interesting lens through which to see the world, and different approaches and paradigms make that a constantly moving picture.

Gleam

Gleam is a functional programming language in the Erlang/Elixir ecosystem that compiles to code that can run on the Erlang VM and can also optionally target Javascript/Node

There are a lot of things I still want to learn about and a lot of things I want to do. A few months ago I tried learning the Gleam but between windows being a generally disagreeable operating system and a general lack of direction I never quite got anywhere with it. Throught the documentation the language seemed like a nice little thing to learn and looked like a good balance of the strictness I wanted from a functional language and the ease of learning.

Overall, I found it really easy to pick up Gleam - the syntax is very Javascripty and it's got a very similar type system to F# both in terms of how the syntax looks as well as how it works overall. The standard library is also pretty good and covered most things I had to do and provided some utilities for a lot of common things as well as had testing configured from the get-go which made this overall quite a pleasure to setup

I've also been interested in Parser Combinators and spent some time learning about them from this YouTube Series by Low Byte Productions as well as as some work by Scott Wlaschin on F# for Fun and Profit

The series by Low Byte was pretty great but was done in Javascript and didn't really delve into some of the type complexities and behaviours that crop up when working with static types. It was on my second watch of Scott Wlaschin's talk where the behaviour of generics really clicked and allowed me to related things back to Gleam

As far as my actual experimentation goes I tried to use the kinds of parsers and combinators defined by the LowByte YouTube series while using the approach by Scott Wlaschin

Overall I think I learnt quite a bit, I spent a lot of time looking at the Gleam code and rewriting things to work more consistely with the type of data structure I wanted. One of the things I found a little annoying things I found is that recursive functions seem to be definable only at the top-level of a module which adds some ceremony around doing things that involve recursion

But anyways, all that aside, let's take a look at some parser things

Parsers

What is a Parser

In our context, a parser is simply a function that takes in some input, and tries to convert it into some meaningful pieces of data, further - a combinator is a function that takes one or more parsers and combines them into a new parser - get it? - combine = combinator.

So in our context, we can define a type for a parser as a function that takes a String and returns some result that tells use the part of the string which was matched and what was left over

Additionally, a parser can be either successful or unsuccessful, and we do that using the Result type that's built into Gleam:

pub type ParserState(a) {
  ParserState(matched: a, remaining: String)
}

pub type Err =
  String

pub type Parser(a) =
  fn(String) -> Result(ParserState(a), Err)

A Simple Parser

We can also see that we are using this a thing above, this is a generic type in Gleam. In our implementation we define that a parser just needs to return some kind of data as what was matched, but we don't particularly care what that is

For the sake of example, we can define a simple parser that matches a string exactly:

fn starts_with_xx(input) {
  case string.starts_with(input, "xx") {
    False -> Error("Expected xx" <> " but found " <> input)
    True -> {
      let remaining = string.drop_left(input, string.length("xx"))
      Ok(ParserState("xx", remaining))
    }
  }
}

Above, we define a parser called starts_with_xx which will simply parse the characters xx in a string. Now, overall this isn't super useful and is pretty tedious if we need to do this each time we define a parser, so more generally we can define parsers using a function that takes some configuration and returns a parser

So an example of a function that takes in a String to match and returns a parser to us can be seen below:

pub fn str(start) -> Parser(String) {
  fn(input) {
    case string.starts_with(input, start) {
      False -> Error("Expected " <> start <> " but found " <> input)
      True -> {
        let remaining = string.drop_left(input, string.length(start))
        Ok(ParserState(start, remaining))
      }
    }
  }
}

We can then redefine our above starts_with_x parser in terms of this one, like so:

let starts_with_xx = str("x")

Running a parser we can see a bit about the type of data this returns:

let parser = str("x")

let result = parser("xxY")
// Ok(ParserState("xx", "Y"))


let error = parser("Yxx")
// Error("Expected xx but found Yxx")

We can define other parsers using a similar style but this is the core of it, the above is very specifically a Parser(String) but these can be more generic as we'll see when we get to combinators

A Simple Combinator

Combinators are used to combine parsers in interesing ways. A simple combinator is the left combinator which takes in two parsers and tries to parse them as a sequence, but will only keep the left-side of the parsed result.

We can define this parser as follows:

pub fn left(l: Parser(a), r: Parser(b)) -> Parser(a) {
  fn(input) {
    case l(input) {
      Error(err) -> Error(err)
      Ok(okl) ->
        case r(okl.remaining) {
          Error(err) -> Error(err)
          Ok(okr) -> Ok(ParserState(okl.matched, okr.remaining))
        }
    }
  }
}

It's also interesting to note that we take a Parser(a) and Parser(b) but return a Parser(a)

We can define a parser using the choice combinator which would work as follows:

let parser = choice(str("x"), str("y"))

let result = parser("xyz")
// Ok(ParserState("x", "z"))

The parser above effectively throws away the second match in the choice but moves our inputs to the end of the second parser

So far however we're just working with strings, it would be nice to parse something more complex. In order to facilitate this we're going to define a combinator called map which will allow us to transform the result of the parser for the case where it succeeds (is Ok)

pub fn map(parser: Parser(a), transform) {
  fn(input) {
    case parser(input) {
      Error(err) -> Error(err)
      Ok(ok) -> Ok(ParserState(transform(ok.matched), ok.remaining))
    }
  }
}

In the above definition you can see that the parser is the first input to map. We're defining it like this because Gleam allows us to automatically pass the first argument as a result of a previous computation using the pipe operator which looks like |>

To do something meaningful with map we're going to define a small type called Token which will have a constructor called TokenX which we will just use to represent us matching the character X

type Token {
  TokenX(matched: String)
}

We can then use the type we create along with the map operator to transform the result of the parsing to a more sophisticated data type:

let parser =
  left(
    str("x")
      |> map(TokenX),
    str("y"),
  )

let result = parser("xyz")
// Ok(ParserState(TokenX("x"), "z"))

So that's pretty cool right?

A More Complex Parser

Using the basic idea of a parser and a combinator, we can define a whole bunch of them as I have in my Parz project on GitHub we can compose them to create a parser for a more complex data type, for example:

Say we have a little language that looks like this:

name:string;
age:number;
active:boolean;

We can define a little syntax tree that we want our output to match as follows:

type Ast {
  Ast(List(Node))
}

type Node {
  Node(name: Identifier, kind: Kind)
}

type Kind {
  StringKind
  BooleanKind
  NumberKind
}

type Identifier {
  Identifier(name: String)
}

type NodePart {
  NodeKind(kind: Kind)
  NodeIdentifier(identifier: Identifier)
}

All of the above directly map to something within our syntax other than NodePart which is used as an intermediate type that allows us to search for Kind and Identifier in the same choice due to how Gleam generics work

Finally, we can define a parser using the library I made as follows:

fn parser() {
  // Parse an identifier
  let identifier =
    letters()
    |> map(Identifier)

  // Parse the different node "kinds" that the language has
  let string_kind = str("string") |> map_token(StringKind)
  let number_kind = str("number") |> map_token(NumberKind)
  let boolean_kind = str("boolean") |> map_token(BooleanKind)

  let kind = choice([string_kind, number_kind, boolean_kind])

  // A node is defined as a sequence of (indeitifier, :)(kind, ;)
  let node =
    sequence([
      left(identifier, str(":") |> label_error(custom_error))
        |> map(NodeIdentifier),
      left(kind, str(";")) |> map(NodeKind),
    ])
    // Extract the identifier and kind from our mapping
    |> try_map(fn(ok) {
      case ok {
        [NodeIdentifier(i), NodeKind(k)] -> Ok(Node(i, k))
        _ -> Error("Failed to match identifier:kind")
      }
    })

  let whitespace = regex("\\s*")

  let parser = separator1(node, whitespace) |> map(Ast)

  parser
}

And we can use this to parse our initial input fragment which yields:

let result = parser(input)
// ParserState(
//    Ast([
//      Node(Identifier("name"), StringKind),
//      Node(Identifier("age"), NumberKind),
//      Node(Identifier("active"), BooleanKind),
//    ]),
//    "",
//  )

That's about it. At a high level you should be able to build parsers using the concepts I've mentioned here. Combinators are extremely powerful and I think can work as a great introduction to to functional programming concepts

The Library

Over the course of learning about parser combinators and the gleam language, I put together a little parsing library. It's probably extremely inefficient but was a fun little project and I think it was a useful learning excercise. My libray can be found on GitHub but it's also published as a package that you can use on Hex

References

My time was primarily spent between the following resources when working on this project

Also, if you just wan to learn to program in Gleam you can take a look on Exercism

Additonally, some more generally related information:

My notes on working with the Typescript AST

The Most Awkward Person in the Room

Sat, 13 Jul 2024 00:00:00 GMT

I should preface this by saying that I don't think I'm particularly good at English

It's already challenging enough trying to communicate in a foreign country - add in some loud music and a sketchy location and we've just made that about ten times more difficult

English, or something

A friend of mine mentioned to me the other day that they've been thinking about moving to the UK because of how much gets lost when communicating in a second language

This morning I met up with some developers and we were chatting about software but also politics (since everything is politics) and a few times during the conversation the perfect sentence popped into my head - but I couldn't say it. There was the immediate realization that the meaning would be lost, the nuance would not translate.

I've been experiencing this a lot at work well, while everyone speaks English perfectly, there's just a little bit of style that has to be eliminated to communicate clearly - something I don't find myself doing when speaking to native English speakers. I simplify words, use fewer metaphors, and slow down significantly

Anyways, I went home and spoke to my wife and she had some strong opinions about the churro place she went to. All the signs were in English but when she tried to order in English the person working there didn't seem to understand what she was saying, even though he repeated all the same things back to her in English

Now, if I'm being completely honest here, some of these problems exist when speaking through language boundaries but it's a lot less frequent

The Curse of the Awkward

So, anyways - I'm not a super social person - I don't really know how to speak to people and kind of overthink every interaction

On one of the Photography groups I'm on someone posted an invite to a "Party" which was meant to just be a social event for photographers and creatives. They also added a tiny note saying "no alcohol event, more like networking"

No alcohol you say? Do such things exist? Okay count me in - is what I would have said if I didn't spend two hours contemplating whether or not this would be the worst experience of my life

It was a bit in the middle of nowhere but looked like it wouldn't take me too long to get there - so I was like yeah sure. I dropped them a message and said I'd meet them there - this wasn't someone I'd met before but I know a few people who know them so I figured it should be alright

I got to what Google told me was the destination but this was just a weird abandoned building with automatic doors that wouldn't open automatically

I saw a person with a camera standing in the passage upstairs and figured I was probably in the right place but had no idea what to expect

The dude I was supposed to meet got there about 5 minutes after I did and we went in together (after awkwardly waiting for someone to see us and let us in)

It was super weird. There were photographers everywhere and people posing all over the space, the music was maaad loud and there were some people live-DJing

This was a kind of chaos I wasn't ready for

I think the worst part of going to a place you've never been is not knowing anyone - events are a sort of social fabric and it's always a bit scary being a loose thread

I floated about not too sure what to do, looking around, just trying to see what was happening - I was kind of shocked at how diverse the space was. It's rare in the Netherlands to end up in a room where you aren't the only person of colour in

This is something I noticed the last time I attended a photography meetup which was that it was a lot less white than the professional spaces I tend to be involved in

There may be some social commentary that could be added here but that's not what this post is about so moving on

Later in the evening there was a short poetry performance which was cool - but that's about all I can say about it - the poetry was in Dutch and while I think my rudimentary understanding of the language was enough to grasp the high level idea of it, I feel like a lot of meaning and intent was lost in translation

Thought

I think the way we communicate with people close to us resonates on a level that's more than just the words we're saying. There's an understanding that becomes so difficult to interpret when speaking a second language

Maybe there's something I'm missing here, maybe there's something to be had where language isn't the main factor - where the words we say aren't the foundation of our relationships. But maybe that's a thought for another day

On Automatic Lensess

Sat, 27 Apr 2024 00:00:00 GMT

I enjoy capturing the world around me - places, people, plants - anything the light touches really

I've also spent a fair number of years shooting with manual focus lenses. Every now and then I pick up an auto focus lens - usually my kit lens, usually because I'm experimenting with a focal length or style I don't use often

Between a museum and getting a bite to eat I was strolling the streets of Amsterdam trying to catch some of the Kingsday activities. I used my kit lens most of the time. It's a 15-45mm f3.5 Fujifilm lens that's fully automatic

Recently I've been experimenting with shooting at f11-f22 and aperture priority mode while prefocusing to get some more "classic style" street photos. This setup is really great because it means I don't have to fiddle with anything when trying to get a shot - I can just lift my camera and click. This is especially handy since a scene can change quickly when shooting on a busy street

While shooting today I noticed a few things that annoyed me quite a bit and thought I should put them down mostly as a reminder to myself as to why I don't really shoot with automatic lenses

Firstly, I get the feeling that automatic lenses make the process of turning a camera on and off a lot slower - I also get the feeling that it consumes a lot more battery life when turning on or off since the lens I have, and other lenses I've had previously, have a "collapsed" mode when inactive in which the lens elements are all fitted into the housing - when turned on, the motors in the lens "uncollapse" them which takes a bit of time and seems to eat a lot more battery than when using a fully manual lens

Next, then lens I have uses "fake" focus and zoom rings (it's a fully auto lens as opposed to just auto-focus). I find this extremely annoying when shooting because I can't trust that my camera is going to retain my zoom or focus settings when turning it on or off or that they weren't randomly changed at some time accidentally

Semi related to this is that due to the focus and zoom rings being electronic, I don't have any visible zoom or focus indicators on the lens. This means that I have to look through my camera to focus or zoom. This is a little annoying when shooting quickly because it means I can only change these settings after raising the camera to my eye - by the time I've dialed the settings in my shot is gone

You can see the indicators I'm referring to below when comparing the two lenses I currently use

The fully automatic Fujifilm 15-45mm f3.5 kit lens:

</center>

A fully manual TTArtisan 35mm f1.4 lens:

</center>

Normally when shooting you need to set the following:

Focus
Zoom
ISO
Aperture
Shutter speed

If I have my focus set before even raising the camera, that's about more than half of my time-to-shot saved - it's really handy - particularly with lenses that have slow zoom or focus response like the Fuji lens

Higher end cameras (particularly Fujifilm cameras) also have the ISO and shutter dials at the top of the camera so that you can view these settings on the hardware without having to turn the camera on - this way you're always ready to shoot

Lastly - aperture. Fully manual lenses also have manual aperture rings. This means that on a fully manual lens, even on a lower end camera - at a glance I can view 3 of the 5 things I need to set up in order to take a picture

Okay, so what? Just remember the settings right? Most cameras should retain your settings between restarts so this doesn't really matter too much, and generally you're not changing settings without knowing what your camera and light meter are showing you right? right?

Well ... not exactly

So, when shooting street there's a concept called "shooting from the hip" which is effectively shooting without having to pick up your camera at all. When doing this it can be useful (if not absolutely necessary) to set your camera focus and zoom while not actively shooting so that you can be ready when something comes up - when you're working with a mirrorless camera, this is even more useful since it lets you conserve battery life if your camera is off while doing this

However, there are certain cases where being able to have a lens that is automatic is useful like in video for example, when smooth focusing and zooming can make or break a shot, or for photographers who are just starting out and want to let their camera control their aperture or focus settings so they can focus on just getting the shot

Some Technical Jargon

"Focal Length" not "Zoom"
I don't know what lenses that have electronically controlled focus and zoom rings are called, I've settled for "Automatic Lenses"
The zoom thing obviously doesn't apply to lenses that do not zoom
The aperture thing obviously doesn't apply to lenses that don't have an aperture control
All of this also probably doesn't apply to cameras that are fully manual

On Falling Down

Fri, 19 Apr 2024 00:00:00 GMT

Today was kinda shitty. Actually really shitty.

But also

maybe not

Its always weird sitting down to write about what's going on in your head. I'm currently in a train at night hoping I'll make it home in the next two hours

Today was a bit insane. I has to shuffle between trains and go to places I've never been. Had to go through a storm feeling like I was constantly about to get thrown into a river by winds far stronger than they had any right to be

Well. I guess that's as good a place as any to start

So, we had a little ski trip planned for work to this weird indoor ski slope in the middle of nowhere. The journey there was gonna be long but was straightforward enough - take two trains and two buses then walk for 8 minutes

That was not how it went

So, I got onto the first train - the one I usually take to get to the Utrecht Central station, that was a little busy but nothing too unusual for 4pm on a Friday. When I got to the station I had to catch another train - this one to The Hague - and this is when everything descended into chaos

The train to The Hague was cancelled - but I was already on it - it was going to Leiden, no further - oh, and pretty much any train going anywhere past Leiden was also cancelled

Oh, and there isn't really any app that does "Hi, I'm already on this train but I need to go to this other place, where do I get off?" (Like seriously, this needs to exist)

After going insane for a while, I resigned myself to my fate, put on some headphones - oh, did I mention that mine were dead so I borrowed my wifes? - and put on a good playlist (after briefly attempting to listen to Taylor Swift's newest album). It's kind of weird how much calm the right playlist can bring

But anyways, It looks like there are some buses I can use from Leiden sort of - so let's do that

Now the thing is - since all the trains were stopping in Leiden - the station was absolutely packed. All i could see was a sea of heads

(one moment, i need to switch trains)

So, by the skin of my teeth i managed to get into a bus that was heading in the direction i needed to go. It was full, there were some loud americans, and some teenagers talking about tik tok songs, we drove past lots of sheep

Oh, the bus was also 7 minutes late because of the mess at the station - I wasn't gonna make the next swap - I had to figure out which stop to take

My options were:

Get off at some rando bus stop and walk for 20 minutes
Get off at some other rando bus stop to wait about an hour and then walk for 10 minutes

Eventually I made it to a bus stop in the middle of nowhere

I took option 1 - maybe it's me, maybe i'm the problem - but, just after I got off the bus and oriented my google maps and got my camera out to enjoy my stroll and maybe take some pictures of some lil sheeps - the mother of all storms descended upon lil ol me.

I don't know what happened, but the light drizzle turned into a beast with winds trying to steal my phone and camera and a torrent of rain racing to drown me

Another test of my sanity today. I can't control the weather I suppose, not any more than I can control bus schedules I guess

So I walked, slowly, taking pictures of everything I thought was interesting

Yes, the sheep. And also the little yellow flowers, and the weird giant metal alien space ship thing in the distance, and the poor post person trying their absolute hardest to make it to the next house on their little blue bicycle - the dutch really take their post seriously

After about 10 minutes of walking I seemed to be in a forest, a hiking trail of some sort - I saw a parent and child riding their bicycle and a woman walking a dog - in this weather?? I can only conclude that the forest was haunted

But eventually I figured out where my map was taking me - the weird alien thing - oh, that's the ski slope

With some end in sight, and a sense of where the heck I was, I walked now with a little more purpose

I made it there eventually. Soaking wet, looking like whatever the thing that the cat dragged in would drag in

Then, said hi to some work peeps and got some gear

And so, may I present you with some snowboarding tips based on what some of my coworkers told me and what I learnt from falling a shit ton of times

When trying to get on, dig your board in sideways with your back against the slope, this will keep you stable and help in trying to stand up, then commit when you stand - otherwise you're just gonna spend the entire time on your butt
Lean less than you think you need to and wait for the board to catch the snow
While it's a bit more difficult than the alternatives - take the weird lift thingy because you can keep your boots fully attached and once you get to the top you can just start snowboarding instead of doing the weird crab walk scoot struggle to standup thing
You can always lean back more than you think and you probably won't fall down - it's a lot safer than learning forward and a really good way to quickly slow down if you're bad at snowboarding
When using the weird lift thingy just let it take you, don't fight it - you will fall
Fall. Yeah if you're old like me it's gonna suck tomorrow - but with every weird things on your feet sport, if you don't get over the fear of falling you're not really gonna get anywhere

Oh and the new skiiers will get confused and stuck and not know how to standup - I don't have a solution, this is just an observation - those things are a deathtrap

(also snowboarding pants are cool, I kind of want some just to like wear as clothes)

Also it's cold and I'm waiting for my last train. This has been a really long day, but if this works out I'll have at least managed to beat the plan all the map apps gave me - some consolation knowing you're a bit smarter than some rando pathfinding algorithm i guess

But anyways, what I think im trying to say is, it's easy to get frustrated and angry when faced with challenges but all we can do is to make the best of what we can control and accept what we can't.

Passwords

Fri, 12 Apr 2024 00:00:00 GMT

"Good design is actually a lot harder to notice than poor design, in part because good designs fit our needs so well that the design is invisible, serving us without drawing attention to itself. Bad design, on the other hand, screams out its inadequacies, making itself very noticeable." - Don Norman

Just had a silly experience. I needed to log into an app on my browser and had the following nonsensical series of steps:

Go to website
Click "Login"
Username automatically filled in by my password manager
Click "Next"
Password is automatically filled in by my password manager
Click "Submit"

Maybe there's trickiness I'm not seeing here but from my perspective it doesn't look like I'm required in this flow?

Between my browser and my password manager, and to some extent the application I'm logging into, this problem can be solved without my involvement. You have enough information to know who I am and log me in automatically when visiting a page - there's no real reason I ever need to see a login page when using an application on my device/browser

I'm thinking that since the data is already there at a hardware level this should be a solved problem

Then again, maybe this is what Web Authentication API is meant to solve but I'm not too sure, I think it's still a click too many. That and my experiences with it have been a little inconsistent

I think we're still missing some unifying solution - some means of tying my identity to a set of devices once and having the friction removed at every other point

Generally things get smoother as time goes but I feel that hasn't been then case with auth. Every day there's a new step - OTPs, "Magic Links" or "Your password can't be the same as your past 5 passwords" which create a mess for most people. We like to think that most people just use password managers but that isn't really the case. Many people struggle with this constantly

I'm not sure what the solution is but I think we need to start thinking about security from a bit more of a human angle and less of a computer one

Show Children when Parent is Hidden with CSS

Thu, 04 Apr 2024 00:00:00 GMT

import Example from '@/snippets/css-visibility/CSSVisibility.astro' import HTMLSnippet from '@/components/HTMLSnippet.astro' import CSSSnippet from '@/components/CSSSnippet.astro'

When working with HTML we sometimes have a case where we want to view child items of an HTML element while the parent element itself is hidden

Normally when trying to hide or show some elements, we would apply display:none and move on. However, we may run into a case where we want to make a specific child visible while keeping the remaining content hidden, so, we might try to add the following to the child display:block.

We will quickly see that this solution doesn't work. Another thing we might trying to play with the opacity of the items but this doesn't really solve our problem either

The solution to this is the visibility property

According to the MDN Documentation on Visibility: "The visibility CSS property shows or hides an element without changing the layout of a document."

The MDN definition brings up an important note - the visibility property does not change the layout of the document - this means the hidden elements will still take up space on the screen, which is different to the behaviour of display:none. However, we can mitigate that in other ways if we need to deepending on what we're trying to do

We can use the visibility property to get what we want. For our example let's use a ol with some children, however, we want anything inside of our list to be hidden other than a specific child

The HTML we have is as follows:

It's also important to note that the content before and after the list are not visible but they still take up space

You can play around with the CSS properties below to see how the influence the overall styling:

<CSSSnippet html path="css-visibility/css-visibility.css" variables={{ '--show-visibility': { initial: 'initial', options: { visible: 'visible', hidden: 'hidden', initial: 'initial' }, }, '--show-display': { initial: 'initial', options: { block: 'block', hidden: 'none', initial: 'initial' }, }, '--hide-visibility': { initial: 'initial', options: { visible: 'visible', hidden: 'hidden', initial: 'initial' }, }, '--hide-display': { initial: 'initial', options: { block: 'block', hidden: 'none', initial: 'initial' }, },

}}>

<div class="example"> <div>before hidden</div> <ol class="hide"> <li>Item 1</li> <li class="show">Item 2</li> <li>Item 3</li> </ol> <div>after hidden</div> </div> </CSSSnippet>

CSS @counter-style with Emojis

Thu, 21 Mar 2024 00:00:00 GMT

import Snippet from '@/components/CSSSnippet.astro'

The CSS @counter-style rule is used for defining list-styles in CSS

The syntax is pretty simple:

@counter-style <counter-style-name> {
  system: <cyclic|numeric|alphabetic|symblic|fixed>;
  symbols: <space separated list of symbols>;
  prefix: <content before symbol>;
  suffix: <content after symbol>;
  /* and some more options... */
  /* take a look at the MDN doc for all the customizations you can do, it's pretty cool */
}

Currently symbols just supports strings, but once fully supported it will also be possible to use images as symbols

Cyclic Counter Style

For the sake of example, I'll be using Emojis as the symbols to display. We can define a cycle which uses some emojis like so:

Numeric Counter Style

For this example, we'll use emoji numbers for the counter style

References

Use localStorage for Tab Synchronization

Thu, 07 Mar 2024 00:00:00 GMT

import SynchronizedButton from '@/snippets/localstorage-sync/SynchronizedButton.astro' import SynchronizedButtonRefactor from '@/snippets/localstorage-sync/SynchronizedButtonRefactor.astro' import Snippet from '@/components/Snippet.astro'

localStorage

localStorage is a method for persistent data storage in the browser. The storage itself is a simple key-value store with a synchronous API

We can write data to a given key using the following:

window.localStorage.setItem('my-key', 'my-data')

And we can get that data back using:

const myData = window.localStorage.getItem('my-key')

Pretty neat right?

For the purpose of this post we'll look into how we can use this as a method for sending data between different browser tabs of the same website

Storage Event

Aside from just data storage - localStorage (as well as sessionStorage) provides us with a notification when the data in a storage is changed - this is called a StorageEvent. What's particularly interesting to us is that it's fired in all other tabs of a particular site when any a tab of that site modifies the data

We can listen for this event using window.addEventListener

window.addEventListener('storage', () => {
  console.log('Something has changed in storage')
})

Now, if you think about it for a short while, you may come to an interesting conclusion - we can talk to other tabs

Speaking to Tabs

Open this page in two different windows side-by-side for this example

Since we can talk between tabs, we can synchronize parts of our UI using the messages we recieive

Below we have a button that uses this - each time it is pressed in one tab - it updates the count in all tabs

The code for this can be seen below:

Generic Implementation

So overall, this is a pretty neat method for sharing data between pages, but we end up having to do some repetitive work when setting this up in many places or when working with anything that isn't a string (as you can see with the parseInts required in the above example)

To make this a little more generic we're first going to extract the getting and setting of data into their own functions:

const getValue = <T>(key: string, initial: T) => {
  try {
    const existing = window.localStorage.getItem(key)
    if (!existing) {
      return initial
    }

    return JSON.parse(existing) as T
  } catch {
    return initial
  }
}

const setValue = <T>(key: string, value: T) =>
  window.localStorage.setItem(key, JSON.stringify(value))

These provide relatively simple wrappers around the reading and writing of data and making it possible for us to work with objects

Next, we can provide an easy way to define a data reader and writer - the reason we define these separately is to make the interface a little easier for consumers to use without needing to rearrange all their code:

export type SetValue<T> = (value: T) => void

export const createSyncReader = <T>(
  key: string,
  initial: T,
  onChange: (value: T) => void
) => {
  window.addEventListener('storage', () => {
    const value = getValue(key, initial)
    onChange(value)
  })

  return () => getValue(key, initial)
}

export const createSyncWriter =
  <T>(key: string): SetValue<T> =>
  (value) =>
    setValue(key, value)

Our reader function will call the provided onChange callback that is provided to it when the data is changed with the parsed data. We can implement the same button as above using this new code as can be seen below:

And for the the sake of being complete, we can see the refactored button running below:

View Transitions and an Astro Presentation Framework

Wed, 06 Mar 2024 00:00:00 GMT

import { Slide, Presentation, SlideOnly } from '@/components/slides' import Snippet from '@/components/Snippet.astro'

Well, since this is a post about building a presentation framework within your Astro site, it may be worth mentioning that you can view this page as a presentation using the below button:

The Problem

</Slide>

I'm kind of lazy.

</Slide>

I had to put together a little presentation based on something I've written about previously and wanted a lazy way to reuse my existing content while also making the resulting presentation available on my website

Overall, these are the requirements I had in mind:

<Slide>

My Requirements

Not require any additional build process
Work with Markdown or MDX so I can include it in my website easily
Have a small learning curve
Integrate flexibly with my existing content - pages should be able to be very easily converted to slides

</Slide>

So I investigated a few solutions:

<Slide>

Existing Slide Solutions

Just a markdown doc
Reveal.js
MDX Deck
Spectacle
Plain' ol' HTML
PowerPoint??

</Slide>

The existing solutions just don't work for my case

</Slide>

Now, it's not that they're not good - most of them are pretty great and have some features that I would like to use if this were some once-off throwaway presentation, but since I would like to refer back to and manage the way I want they're not really suitable

I also didn't want to style everything from scratch or write lots of HTML everywhere

The Solution

Build it myself. Obviously

</SlideOnly> </Slide>

Instead of just looking at the existing options, I instead chose to build a library/framework that would work with my existing Astro site while keeping the implementation relatively minimal and just depending on plain CSS, Javascript, MDX, and Astro to get the job done

Use Existing Tooling

HTML
CSS
Javascript (Typescript)
MDX

Facilitated by - not coupled to - Astro

</SlideOnly> </Slide>

Code

So, since I'm building it myself, that means code - and that's what we're going to look at

The API

I wanted to keep the API relatively simple. It should work with existing markdown content and allow me to delineate a slide in a way that can be easily read from the DOM so as to minimize the amount of build-time processing I need to do as well as minimize how much markup I need to write. For this purpose, I decided that I want it to fit into an MDX document like so:

The API

</SlideOnly>

Button to launch the presentation

<Presentation />

... Existing page content

<Slide>
## Heading for Slide

Some content for the slide

```js
console.log('I am a code block')
```

> And literally any other markdown content

</Slide>

</Slide>

Hiven the above, I had a high level idea that I would need two parts to this - firstly I want to use the Slide component to give me something to latch onto in the HTML that I can manipulate, secondly I know I would need some kind of component that would control the overall presentation state, I called that Presentation

The `Slide` Component

The Slide component is simply a wrapper that includes content in an HTML section with a class presentation-slide which will contain the contents of a slide

The `Slide` Component

</SlideOnly>

components/Slide.astro

<section class="presentation-slide">
  <slot />
</section>

</Slide>

The `Presentation` Component

The Presentation component needs to do a few different things:

The `Presentation` Component

</SlideOnly>

Hide presentation until enabled
Allow navigation of slides
Render slide content above existing page
Manage transitions between slide pages

</Slide>

Firstly, we can just take a look at the HTML that we will render contains a few basic elements as well as a script tag that grabs a reference to these elements. The presentation-hidden class is used for hiding or showing the presentation when active/inactive:

Basic Elements

</SlideOnly>

components/Presentation.astro

<button id="presentation-button" class="presentation-hidden" type="button">
  Start Presentation
</button>

<div
  id="presentation-container"
  class="presentation-hidden presentation-overflow-hidden"
>
  <main id="presentation-content">
    <h1>No slides found on page</h1>
  </main>
</div>

<script>
  const button = document.getElementById('presentation-button') as HTMLButtonElement
  const container = document.getElementById('presentation-container') as HTMLDivElement
  const content = document.getElementById('presentation-content') as HTMLDivElement
</script>

</Slide>

Next, we can grab the actual slide content by using the presentation-slide class we defined earlier:

Get Slides

components/Presentation.astro

let slides = Array.from(document.querySelectorAll('.presentation-slide')).map(
  (el) => el.outerHTML,
)

let slide = 0

</Slide>

Once we have the content of the slides and a variable to track which slide we are on, we can define a function that will set the slide content. This will set the innerHTML of the content element to the slide that is active. We can handle this by first defining some utilities for grabbing the next and previous slides as well as mapping a key code to the function that will resolve the next slide

Slide Utilities

</SlideOnly>

components/Presentation.astro

const nextSlide = () => {
  if (slide === slides.length - 1) {
    return slide
  }

  return slide + 1
}

const prevSlide = () => {
  if (slide === 0) {
    return slide
  }

  return slide - 1
}

const keyHandlers: Record<string, () => number> = {
  ArrowRight: nextSlide,
  ArrowLeft: prevSlide,
}

</Slide>

Next, we can define what it means for us to start and end a presentation. For this example, starting a presentation will remove the presentation-hidden class from the main wrapper so we can make the presentation visible on the page as well as set the content to the current slide index (we initialized this to 0 above)

<Slide>

Start and End Presentation

components/Presentation.astro

const startPresentation = () => {
  container.classList.remove('presentation-hidden')
  if (slides.length) {
    content.innerHTML = slides[slide]
  }
}

const endPresentation = () => {
  container.classList.add('presentation-hidden')
}

We set the content to slide instead of 0 so that we can pause and continue the presentation if we wanted to

</Slide>

Next, hook up some event handlers so that we can have a method for controlling our presentation:

<Slide>

Wiring things up

components/Presentation.astro

// If there is no presentation on the page then we don't initialize
if (slides.length) {
  button.addEventListener('click', startPresentation)

  window.addEventListener('keyup', (ev) => {
    const isEscape = ev.key === 'Escape'
    if (isEscape) {
      endPresentation()
      return
    }

    const getSlide = keyHandlers[ev.key]

    if (!getSlide) {
      return
    }

    const nextSlide = getSlide()
    if (slide === nextSlide) {
      return
    }

    slide = nextSlide
    content.innerHTML = slides[slide]
  })
}

</Slide>

In the above, the left and right arrows are used to navigate slides and the escape key is used to end the presentation

Next up, we need to add some CSS to make the slides pin to the root of our application above everything else so that you can actually use this:

<Slide>

Styling

components/Presentation.astro

<style is:global>
  .presentation-overflow-hidden {
    overflow: hidden;
  }

  .presentation-hidden {
    display: none;
  }

  #presentation-container {
    z-index: 10;
    position: fixed;
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
    overflow: auto;

    backdrop-filter: blur(50px);
    background-color: #0000007d;
  }

  #presentation-content {
    display: flex;
    flex-direction: column;

    background-color: black;
    color: white;

    box-sizing: border-box;
    min-height: 100vh;
    width: 100%;
    padding: 4rem;
  }
</style>

</Slide>

And that's pretty much it for the core implementation. One other piece of fanciness that I wanted to add was the ability to make an actual slide transition. To do this I decided to use the View Transitions API and found a few nice references on the Unecesssary View Transitions API List

In order to use this you need to have the feature enabled in your browser at the moment but it should be stable soon (I hope)

For this implementation, we will need to have different animations for the case where we are moving forwards or backwards. In order to do this, we will define some classes as part of our keyboard handler resolution that we will append to the presentation-container:

<Slide>

View Transitions

components/Presentation.astro

const nextClass = 'presentation-next'
const prevClass = 'presentation-prev'

const transitionClasses = [nextClass, prevClass]

const keyHandlers: Record<string, [string, () => number]> = {
  ArrowRight: [nextClass, nextSlide],
  ArrowLeft: [prevClass, prevSlide],
}

</Slide>

Then, we will update our event handling logic to set these classes on the contaienr

<Slide>

Setting the Classes

components/Presentation.astro

const [transitionClass, getSlide] = handler

content.classList.remove(...transitionClasses)
content.classList.add(transitionClass)

const nextSlide = getSlide()
if (slide === nextSlide) {
  return
}

</Slide>

Then, instead just setting the content.innerHTML directly, we do it within the document.startViewTransition callback which will be what handles the state transition between the content leaving the DOM and the new content that is entering the DOM

Note that the startViewTransition API is experimental and typescript may complain, you will need to install @types/dom-view-transitions which will provide the type definition you need to use this API

<Slide>

Starting the Transition

components/Presentation.astro

document.startViewTransition(() => {
  slide = nextSlide
  content.innerHTML = slides[slide]
})

</Slide>

The last thing we need to do is define the view transitions for when the content enters and exists the DOM. The transitions are defined in the style tag of our component as follows:

Firstly we need to

<Slide>

Animations

components/Presentation.astro

@keyframes slide-out-right {
  0% {
    transform: translateX(0) scale(1);
  }
  15% {
    transform: translateX(0) scale(0.8) translateY(0%);
  }
  85% {
    transform: translateX(100%) scale(0.8) translateY(0%);
  }
  100% {
    transform: translateX(100%) scale(1);
  }
}

@keyframes slide-out-left {
  0% {
    transform: translateX(0) scale(1);
  }
  15% {
    transform: translateX(0) scale(0.8) translateY(0%);
  }
  85% {
    transform: translateX(-100%) scale(0.8) translateY(0%);
  }
  100% {
    transform: translateX(-100%) scale(1);
  }
}

</Slide>

The above defines two basic animations that we will use for our transitions. We define an animation that moves an element off the screen to the right called slide-out-right and another to move it to the left called slide-out-left. These animations can also be reversed to slide content in from the right or left respectively

<Slide>

For the "Next" animation we need to do the following:

Slide the old content to the left
Slide the new content from the right

</Slide>

In the below we set that the presentation-next class defines a view-transition-name called next. Then, we define the transitions for the old and new content that applies to the next transition name as an Animation. We are referencing a slide-out-right and slide-out-left animations which we defined previously:

<Slide>

CSS Transitions

Below is the transition for moving to the next slide

components/Presentation.astro

.presentation-next {
  view-transition-name: next;
}

::view-transition-old(next) {
  animation: slide-out-left 0.5s linear;
}
::view-transition-new(next) {
  animation: slide-out-right 0.5s linear reverse;
}

view-transition-old refers to content that is being removed from the DOM, view-transition-new refers to the content that is being added to the DOM

</Slide>

Lastly, the implementation for the presentation-prev we can reuse the same animations for moving left or right as we defined previously, but change the directions as needed for the relevant section

components/Presentation.astro

.presentation-prev {
  view-transition-name: prev;
}

::view-transition-old(prev) {
  animation: slide-out-right 0.5s linear;
}
::view-transition-new(prev) {
  animation: slide-out-left 0.5s linear reverse;
}

And yes, that's a fair amount of code. All-in it's about 200 lines - most of which is the CSS for the transition though. Generally the implementation is pretty straightforward and should be relatively easy to tweak to match the vibe of your website without adding any dependency bloat.

As it stands right now the implementation is pretty simple but leaves a lot of space to be extended

<Slide>

Added since this post was written

[x] Presenter mode with some kind of synchronized state for multiple monitors (LocalStorage?)
[x] Progress tracking
[x] Support for non-static components

Future Ideas

[ ] Presenter notes and preview of next slide
[ ] Make this a library so other people can use it with less copy pasta
[ ] More transitions and styling possibilities
[ ] Dynamic code blocks/customizable transitions (Code Surfer)
[ ] Automatic zooming

</Slide>

<Slide>

Conclusion

We used some interesting CSS here and overall we can see that it's not alays a huge amount of work to write your own implementation of something.

</Slide>

Additionally, for the sake of completeness - since this component is alive and ever changing within this website - you can view the current state of the code (sans commentary) below

References

Interacting with Kafka with Kotlin Coroutines

Thu, 09 Nov 2023 00:00:00 GMT

Overview

The purpose of this post is to illustrate a method of interacting with Kafka using Kotlin in a functional programming style while using Kotlin coroutines for a multi-threading. We will be interacting with the Kafka Client for Java and will be building a small library on top of this for the purpose of simplifying communication and handling tasks like JSON Serialization

If you would like to view the completed source code, you can take a look at the kotlin-kafka GitHub repository

Kafka

According to then Kafka Website:

"Apache Kafka is an open-source distributed event streaming platform:

Generally we can think of Kafka as a platform that enables us to connect data producers to data.

Kafka is an event platform that provides us with a few core functions:

Publishing and subscribing event data
Processing of events in real-time or retrospectively
Storage of event streams

From a more detailed perspective, Kafka internally handles storage of event streams, but we are given control over the means of data production, consumption, and processing via the Kafka API, namely:

The Producer API for production
The Consumer API for subscription
The Streams API for processing stream data

Kotlin

Kotlin is a statically typed programming language built on the Java Virtual Machine that provides interop with Java code

The Code

Config

To get some admin stuff out of the way, before you can really do any of this you will to have a .env file in the project that you can load which contains some application configuration, for the purpose of our application we require the following config in this file - below is some example content

.env

BOOTSTRAP_SERVERS=my-server-url:9092
SASL_JAAS_CONFIG=org.apache.kafka.common.security.scram.ScramLoginModule required username="someUsername" password="somePassword";

Additionally, we have some non-sensitive config in our application.properties file in our application resources folder which contains the following:

resources/application.properties

sasl.mechanism=SCRAM-SHA-256
security.protocol=SASL_SSL
key.serializer=org.apache.kafka.common.serialization.StringSerializer
key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
auto.offset.reset=earliest
group.id=$GROUP_NAME
application.id=example-app

Next, we need to load this in our application to create a Properties object along with all the other application config we require. We can create the Properties object using the application.properties and .env files as follows:

App.kt

package example

import io.github.cdimascio.dotenv.Dotenv
import java.io.FileInputStream
import java.util.*

fun loadProperties(): Properties {
    val props = Properties()
    val resource = ClassLoader.getSystemResource("application.properties")
    println("File  path: ${resource.path}")
    FileInputStream(resource.path).use { stream ->
        props.load(stream)
    }

    val dotenv = Dotenv.load()
    props["bootstrap.servers"] = dotenv["BOOTSTRAP_SERVERS"]
    props["sasl.jaas.config"] = dotenv["SASL_JAAS_CONFIG"]

    return props
}

The above example uses the io.github.cdimascio:dotenv-java:3.0.0 package for loading the environment variables and some builtin Java utilities for loading the application properties file

Next, for the purpose of using it with our library we will create a Config class that wraps the properties file we defined so that we can use this a little more elegantly in our consumers. Realistically we probably should do some validation on the resulting Properties that we load in but we'll just keep it simple and define Config as a class that contains the properties as a property:

Config.kt

package za.co.nabeelvalley.kafka

import java.util.Properties

open class Config(internal val properties: Properties) {}

Working with JSON Data

An important part of what we want our client to handle is the JSON serialization and deserialization when sending data to Kafka. Sending JSON data is not a requirement of Kafka as a platform, but it's the usecase that we're building our library around and so is something we need to consider

Serialization

Serialization in this context refers to the process of converting our Kotlin classes into a string and back to a Kotlin class. For this discussion we will refer to a class that is able to do this bidirectional conversion as a Serializer.

We can define generic representation of a serializer as a class that contains a method callsed serialize that takes in data of type T and returns a string, and contains a method called deserialize that takes in a string and returns an object of type T

Not that at this point we're not considering that the serializer needs to return JSON. In our context a JSON serializer is just a specific implementation of the serialization concept that we have defined

An interface that describes the Serializer we mentioned above can be seen as follows:

Serializer.kt

package za.co.nabeelvalley.kafka

interface ISerializer<T : Any> {
    fun serialize(data: T): String
    fun deserialize(data: String): T
}

JSON Serialization

Given the definition of a serializer we can define a JSON serializer that uses the kotlinx.serialization library and implements our ISerializer as follows:

JsonSerializer.kt

package za.co.nabeelvalley.kafka

import kotlinx.serialization.KSerializer
import kotlinx.serialization.json.Json
import kotlinx.serialization.serializer
import kotlin.reflect.KClass

class JsonSerializer<T : Any>(type: KClass<T>) : ISerializer<T> {
    private val serializer: KSerializer<T> = serializer(type.java) as KSerializer<T>

    override fun serialize(data: T): String = Json.encodeToString(serializer, data)

    override fun deserialize(data: String): T = Json.decodeFromString(serializer, data)
}

The above code is a little funky since we're using reflection on the actual class of the input data to define our serializer, other than we're just using the kotlinx serializer to handle the data transformation. The thing that matters in this context is that we are able abstract the reflection aspect of the serializer, this will help make the final interface we provide to the user for working with Kafka simpler

Serde Serializer

Now that we have defined a simple representation of a serializer that provides some interop with the Kotlin data types, we need to implement the other side of this which is a SerdeSerializer which is what the Kafka Clients need to work with. The requirements of this serializer are a little different to the one we defined above. This serializer needs to:

Have a separate Serializer and Deserializer interfaces that need to be implemented
Return a ByteArray instead of String

We can define these serializers such that they can be constructed from and ISerializer interface that we defined previously. This will make it possible for consumers of our library to swap our their serialization strategy to enable other usecases than the simple JSON communication we are considering

As mentioned above, we need to implement a separate Serializer and Deserializer respecively as:

SerdeSerializer.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.common.serialization.Serializer

class SerdeSerializer<T : Any>(private val serializer: ISerializer<T>) : Serializer<T> {
    override fun serialize(topic: String?, data: T): ByteArray {
        val result = serializer.serialize(data)
        return result.toByteArray()
    }
}

And

SerdeDeserializer.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.common.serialization.Deserializer

class SerdeDeserializer<T : Any>(private val serializer: ISerializer<T>) : Deserializer<T?> {
    override fun deserialize(topic: String?, data: ByteArray?): T? {
        try {
            val string = String(data!!)
            return serializer.deserialize(string)
        } catch (error: Error) {
            println("Error Deserializing Data: $error")
            return null
        }
    }
}

Our implementation is a little basic and will just ignore any data that we can't serialize, however depending on our usecase we may need to handle this differently

Lastly, we define the actual Serde Serializer implementation using the above implementations:

Serializer.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.common.serialization.Serde

class Serializer<T : Any>(private val serializer: ISerializer<T>) : Serde<T> {
    override fun serializer() = SerdeSerializer<T>(serializer)

    override fun deserializer() = SerdeDeserializer<T>(serializer)
}

As far as serialization and deserialization goes, this should be everything we need for working with JSON data

Producing Data

Producing data is a method by which a client sends data to a Kafka topic. We can define this as a type as follows:

Producer.kt

typealias Send<T> = (topic: String, message: T) -> Unit

Now, to provide a functional library interface we will want to provider application code a space in which they will be able to work with the producer that we populate without needing to create a new producer for each message we want to send

We'll codify this intent as a type as follows:

Producer.kt

typealias Produce<T> = suspend (send: Send<T>) -> Unit

Note that we define this as a suspend function that will enable users to send messages from within a coroutine context

Next, we define the type of our producer as method with a way to create a producer instance for users who may want to manage the lifecycle of the KafkaProducer on their own. This however also means they lose access to the automatic serialization and deserialization that we will provide via our producer method

This interface is defined as follows:

Producer.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.clients.producer.KafkaProducer
import java.util.*

interface IProducer<T> {
    /**
     * Returns the raw producer used by Kafka
     */
    fun createProducer(): KafkaProducer<String, String>
    fun produce(callback: Produce<T>)
}

For the purpose of our implementation we can define some functions ourside of our class that will provde the implementation we require

For the createProducer function, we simply provide a wrapper around the KafkaProducer provided to us by the Java Kafka Client Library:

Producer.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.clients.producer.KafkaProducer
import org.apache.kafka.clients.producer.ProducerRecord
import java.util.*
import kotlinx.coroutines.runBlocking

fun createProducer(properties: Properties) =
    KafkaProducer<String, String>(properties)

For the sake of consistency, we will do the same for the concept of a ProducerRecord which will be used by the produce function:

Producer.kt

fun createRecord(topic: String, message: String) =
    ProducerRecord<String, String>(topic, message)

Next, the produce function can be defined. The role of this function is to handle serialization of data and provide a means for a user to send data to a topic

The producer will take a callback which is the context in which any usage of the send function should be used before the producer is disposed:

Producer.kt

fun <T : Any> produce(properties: Properties, serializer: ISerializer<T>, callback: Produce<T>) {
    createProducer(properties).use { producer ->
        val send = fun(topic: String, message: T) {
            val payload = serializer.serialize(message)
            val record = createRecord(topic, payload)
            producer.send(record)
        }

        runBlocking {
            callback(send)
        }
    }
}

We have also added the properties and serializer values as an input to the producer as this is needed by Kafka, lastly, we will define our actual Producer implementation which builds on the functions we defined above

Note that our Producer class implements IProducer and extends Config, this is because we use the Config class as the source of truth of the configuration to be used for our Kafka instance and we want to able to access this config

Producer.kt

class Producer<T : Any>(
    properties: Properties,
    private val serializer: ISerializer<T>
) : Config(properties),
    IProducer<T> {
    override fun createProducer() = createProducer(properties)

    override fun produce(callback: Produce<T>) = produce<T>(properties, serializer, callback)
}

At this point we have a complete implementation of a producer

Using the Producer

In our application code we can instantiate and use the producer as follows:

Firstly, we need to define the type of data we are goind to send with the @Serializable annotation

App.kt

import kotlinx.coroutines.*
import kotlinx.serialization.Serializable
import za.co.nabeelvalley.kafka.*
import java.util.*


@Serializable
data class ProducerData(val message: String, val key: Int)

Next, we can define a function for producing data, this will require the properties we loaded previously:

App.kt

fun instantiateAndProduce(properties: Properties): Unit {
    val serializer = JsonSerializer(ProducerData::class)
    val producer = Producer(properties, serializer)

    runBlocking {
        producer.produce { send ->
            val data = ProducerData("Hello world", 1)
            send("my-topic", data)
        }
    }
}

We use runBlocking since our producer needs a coroutine scope in which to send data. Sending data us used within the produce method in which we create some data and call the send method provide by the produce function

An interesting to note is that we are passing the class of our data to the serializer to create an instance - this is the usage of the funky reflection thing we saw previously

Consuming Data

Our code for consuming data will follow a similar pattern to what we use to consume the data in the previous section

For consuming data, Kafka relies on the concept of polling for records from the part of the consumer, for our client, we will expose using the following type which defines a poll as a method that takes nothing and returns a list of data of type T

Consumer.kt

typealias Poll<T> = () -> List<T>

Next, we can define the type that defines how we want our data to be consumed. For our sake, this is a suspend function that will receive a poll method that it can call to get data

Consumer.kt

typealias Consume<T> = suspend (poll: Poll<T>) -> Unit

Next, as before, we can define an interface for a Consumer in which we have a method to create a KafkaConsumer and a method for actually consuming the data. In the case of consuming we need a list of topics to read from as well as the polling frequency duration.

Consumer.kt

package za.co.nabeelvalley.kafka

import kotlinx.coroutines.runBlocking
import org.apache.kafka.clients.consumer.ConsumerRecord
import org.apache.kafka.clients.consumer.KafkaConsumer
import java.time.Duration
import java.util.*

interface IConsumer<T> {
    /**
     * Returns the raw consumer used by Kafka
     */
    fun createConsumer(): KafkaConsumer<String, String>
    fun consume(topics: List<String>, duration: Long, callback: Consume<T>)
}

Next, we can define our createConsumer method quite simply as:

Consumer.kt

fun createConsumer(properties: Properties) =
    KafkaConsumer<String, String>(properties)

And we can define our consume method such that it takes in the properties and serializer as with the producer, but will also take som patterns to be used for subscribing to and the duration above, and finally the callback Consume function:

Consumer.kt

fun <T : Any> consume(
   properties: Properties,
   serializer: ISerializer<T>,
   patterns: List<String>,
   duration: Long,
   callback: Consume<T>
) {
   createConsumer(properties).use { consumer ->
      consumer.subscribe(patterns)
      val poll = fun(): List<T> {
         val records = consumer.poll(Duration.ofMillis(duration))
         val data = records.toList()
               .map(ConsumerRecord<String, String>::value)
               .map(serializer::deserialize)

         return data
      }

      runBlocking {
         callback(poll)
      }
   }
}

The consume function is very similar to the produce function we defined previously, however now instead of being provided a function to send data we now have a function that will return that data

Lastly, we can finish off the definition of our Consumer using what we have above:

Consumer.kt

class Consumer<T : Any>(
    properties: Properties,
    private val serializer: ISerializer<T>
) : Config(properties), IConsumer<T> {
    override fun createConsumer() = createConsumer(properties)

    override fun consume(topics: List<String>, duration: Long, callback: Consume<T>) =
        consume(properties, serializer, topics, duration, callback)
}

Using the Consumer

Using the Consumer follows a very similar pattern to the producer, however we need to create a loop that will poll for data and handle as necessary when data is received:

App.kt

@Serializable
data class ConsumerData(val message: String, val key: Int)

fun instantiateAndConsume(properties: Properties): Unit {
   val serializer = JsonSerializer(ConsumerData::class)
   val consumer = Consumer(properties, serializer)

   runBlocking {
       consumer.consume(listOf("my-topic"), 1000) { poll ->
         while (true) {
            val messages = poll()
            println("Received ${messages.size} messages")
            messages.forEach(fun(message) {
               println("Received: $message")
            })
         }
      }
   }
}

In the above, we use a while(true) loop to re-poll continuously but this can freely change on the implementation, similar to with the producer code

Stream Processing

In Kafka, we can think of a stream process as a combination of a consumer and producer such that data comes in from a topic and is sent to a different topic

The thing that makes streams interesting is the builder API that the Kafka Java Library provides to us for defining the operations to be done on the stream data. For our implementation we'll be referring to this as a TransformProcessor, this processor needs to take in some data of type TConsume and return data of type TProduce, however, since we want to provide users complete flexibility in working with this data, we will instead more generally allow a user to convert a stream between the predefined data types, using the underlying library this is called a KStream

From a type perspective, we can define a TransformProcessor as follows:

SerializedStream.kt

typealias TransformProcessor<TConsume, TProduce> = (stream: KStream<String, TConsume>) -> KStream<String, TProduce>

Now, we're going to be starting this implementation from what we want, assuming that it is possible for us to in some way define a KStream that is instantiated to work with our connection and the respective TConsume and TProduce data.

We will also be using a type called Produced which is what the Kafka Client uses to represent the data that the stream will return since this is what we need in order to send data to a processor

Our implementation will be called a SerializedStream and this looks like the following:

SerializedStream.kt

package za.co.nabeelvalley.kafka

import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.Job
import kotlinx.coroutines.launch
import org.apache.kafka.streams.KafkaStreams
import org.apache.kafka.streams.StreamsBuilder
import org.apache.kafka.streams.kstream.KStream
import org.apache.kafka.streams.kstream.Produced
import java.util.*

typealias TransformProcessor<TConsume, TProduce> = (stream: KStream<String, TConsume>) -> KStream<String, TProduce>
typealias Close = () -> Unit
typealias Process = suspend (close: Close) -> Unit

class SerializedStream<TConsume : Any, TProduce : Any>(
    private val properties: Properties,
    private val builder: StreamsBuilder,
    private val producer: Produced<String, TProduce>,
    private val stream: KStream<String, TConsume>
) {
   fun startStreaming(
      topic: String,
      processor: KStream<String, TProduce>,
      process: Process
   ): Job {
      processor.to(topic, producer)

      val streams = KafkaStreams(
         builder.build(),
         properties
      )

      val scope = CoroutineScope(Dispatchers.IO)
      return scope.launch {
         streams.start()
         process()
         streams.close()
      }
   }

   fun getProcessor(
       processor: TransformProcessor<TConsume, TProduce>
   ): KStream<String, TProduce> = processor(stream)
}

In the above implementation we have an input to our startStreaming function called process, the process function is a callback that needs to call close once it is done running. When the process function returns the processing will stop, the scope of this function also defines lifecycle of the stream processor and is used for that purpose

So we have defined the processing methodology using a KStream but have not provided a way to create a KStream. Since the stream can be defined in many different ways, we can define this using a builder class called StreamBuilder. This class will be instantiated with the Kafka connection properties and input/output serializers, thereafter it can produce methods for instantiating the SerializedStream instance that we can use for data processing

For the sake of our example we will provide a method called fromTopic which returns a SerializedStream that is configured to work on a single topic, and a fromTopics method which will return a SerializedStream that listens to multiple topics:

StreamBuilder.kt

package za.co.nabeelvalley.kafka

import org.apache.kafka.common.serialization.Serdes
import org.apache.kafka.streams.StreamsBuilder
import org.apache.kafka.streams.kstream.Consumed
import org.apache.kafka.streams.kstream.Produced
import java.util.*

interface IStreamBuilder<TConsume : Any, TProduce : Any> {
    fun fromTopic(topic: String): SerializedStream<TConsume, TProduce>
    fun fromTopics(topics: List<String>): SerializedStream<TConsume, TProduce>
}

An implementation of this interface is as follows:

StreamBuilder.kt

class StreamBuilder<TConsume : Any, TProduce : Any>(
    properties: Properties,
    consumeSerializer: ISerializer<TConsume>,
    producerSerializer: ISerializer<TProduce>,
) : Config(properties), IStreamBuilder<TConsume, TProduce> {
    private val inputSerde = Serializer<TConsume>(consumeSerializer)
    private val consumed = Consumed.with(Serdes.String(), inputSerde)

    private val outputSerde = Serializer<TProduce>(producerSerializer)
    private val produced = Produced.with(Serdes.String(), outputSerde)

    override fun fromTopic(topic: String): SerializedStream<TConsume, TProduce> {
        val builder = StreamsBuilder()
        val stream = builder.stream(mutableListOf(topic), consumed)

        return SerializedStream(properties, builder, produced, stream)
    }

    override fun fromTopics(topics: List<String>): SerializedStream<TConsume, TProduce> {
        val builder = StreamsBuilder()
        val stream = builder.stream(topics.toMutableList(), consumed)

        return SerializedStream(properties, builder, produced, stream)
    }
}

The above class makes use of the produced and consumed properties which are what Kafka will use for serializing and deserializing data in the stream

And that's about it as far as our implementation for streaming goes

Using the Stream Processor

We can use the stream processor code:

App.kt

fun initializeAndProcess(properties: Properties): Job {
    val producedSerializer = JsonSerializer(ProducerData::class)
    val consumedSerializer = JsonSerializer(ConsumerData::class)
    val streamBuilder = StreamBuilder(properties, consumedSerializer, producedSerializer)
    val stream = streamBuilder.fromTopic("input-topic")

    val processor = stream.getProcessor { kStream ->
        kStream.mapValues { key, value ->
            ProducerData("Message processed: $key", value.key)
        }
    }

    val scope = CoroutineScope(Dispatchers.IO)
    return scope.launch {
        stream.startStreaming("output-topic", processor) { close ->
            coroutineScope {
                println("Processor starting")
                // Non-blocking loop as long as the coroutine is active
                while (isActive) {
                    delay(10_000)
                }

                // close when no longer active
                close()
                println("Processor closed")
            }
        }
    }
}

Most of this is just the normal construction that you will have for any instance of the stream client, what is interesting is the part where we define the processor:

App.kt

val processor = stream.getProcessor { kStream ->
    kStream.mapValues { key, value ->
        ProducerData("Message processed: $key", value.key)
    }
}

In the above example we are simply mapping a single record using mapValues, this is very similar to the Collection methods available in Kotlin but is instead used to define how data will be transformed in the stream

The processor we define is what will be executed on records or groups of records depending on how we want to handle the resulting data

Conclusion

In this post we've covered the basic implementation of how we can interact with Kafka using the Kotlin programming language and built a small library that takes us through the basic use cases of Serializing, Producing, Consuming, and Processing stream data

References

JPA Queries without the Magic

Fri, 20 Oct 2023 00:00:00 GMT

If you're just looking for the solution you can just skip ahead

The go-to way to build web applications with Java or Kotlin is using Spring. Spring is a pretty decent web framework that provides a lot of utilities for handling common web application related tasks

However I find that Spring does a fair amount of compiler magic to make it work the way it does - of particular concern to this post is how Spring generates implementations for database queries using Query kewords as part of your function names which spring will resolve to a query on the underlying database

Below is a short example of how we would use Spring to query our database to illustrate the problem:

The Example

Let us consider an application in which we have some entity in our system that we would like to store in a database and be able to search over

In order to do this, we need to define a couple of things things:

An Entity which represents that data structure for our User

In Spring we can define entities, like a User as a class with some annotations

import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;

@Entity
@Table(name = "User")
public class User {

    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private long id;

    @Column(unique = true)
    private String email;
}

A Repository which is the class through which we will interact with the database for the specific entity

import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.stereotype.Repository;

import java.util.List;
import java.util.Optional;

@Repository
public interface UserRepository extends JpaRepository<User, Long> { }

For the repository all we really need is to use the @Repository annotation which will register the repository with the Spring dependency injection, and we need to extend JpaRepository which provides some base methods for our repository such as findAll, save, or delete among others

Up until this point, we haven't seen anything too weird and the code thus far is fairly easy to reason about - however, this becomes interesting when we want to add a method like findByEmail which we define in the interface as follows:

@Repository
public interface UserRepository extends JpaRepository<User, Long> { 
    List<User> findByEmail(String email);
}

That's it, we don't implement this anywhere - Spring will generate this implementation for us - weird - but okay. We can use this function anywhere we have a repsitory instance and it will behave as expected

Growing Pains

But now, what if we want the findByEmail function to be case incensitive? Well seemingly we can just name this however so let's try:

@Repository
public interface UserRepository extends JpaRepository<User, Long> { 
    List<User> findByEmailCaseInsensitive(String email);
}

You can try using that, but it won't work - turns out the magic is documented at least, and Spring tells us that we actually need to use IgnoreCase in the name - this makes use of the Query Keywords I mentioned above

So sure, we can change this:

@Repository
public interface UserRepository extends JpaRepository<User, Long> { 
    List<User> findByEmailIgnoreCase(String email);
}

And also go and change everwhere we're using that method, since it's name now dictates its implementation we will need to keep in mind that if we ever want to change the implementation we will need to update all our clients

But wait - that's very UN-Java like isn't it? This seems to break the entire purpose of using functions - if my function's behaviour is directly dependant on it's name then how is this any different to just repeating the implementation everywhere need this? We're typing the same code either way

Now, I'm not all complaints - using a half competent IDE we should be able to do this refactor fairly painlessly and move on with our day - this isn't the main issue yet

Where we start running into problems is when our queries need to get more complex, for example when we decide that we want the search to extend to something like a secondary email filed, for example if we add a new field to our entity, like companyEmail:

@Entity
@Table(name = "User")
public class User {
    // ... existing code

    // new field
    @Column()
    private String companyEmail;
}

The Limit Does Exist

And we now need to update our email function to:

@Repository
public interface UserRepository extends JpaRepository<User, Long> { 
    List<User> findByEmailIgnoreCaseOrCompanyEmailIgnoreCase(String email);
}

That's a bit rough right? There's really no reason our method name should be that long - oh, and it won't work either - even though this follows the JPA naming convention it's just a little too complicated for the code generator to get right

Okay, so now we're stuck right? No - Why would I go on this rant if I don't have a solution in my head

Well, time to share it I guess

The Solution(s)

Query Annotations

Spring gives us two escape hatches for handling the problem we just ran into - we can use the @Query annotation which allows us to define a custom JQL query (Not SQL) that is defined inline and will do the appropriate value substitutions as needed:

@Repository
public interface UserRepository extends JpaRepository<User, Long> { 
    @Query("SELECT u FROM User u WHERE LOWER(u.email) LIKE LOWER(:query) OR LOWER(u.comanyEmail) LIKE LOWER(:query)")
    List<User> findByEmail(String email);
}

Now, if this were the only option, I could live with it, it takes away some of the magic and gives me a decent amount of control over what I'm doing

... but ... how can i say ... It's ugly

Even if we assume that we don't somehow have any errors in the above string, it's just a little odd - like other ORMs have lovely fluent interfaces like LINQ in C# or knex or prisma in Javascript/TypeScript that understand SQL and fit naturally into our programming languge

Specifications

Specifications provide us with a way to define our query within our programming language, and I think that's nice, so here's how they work

In order to use specificiations we need to do a few things:

Define the specification method

It doesn't matter too much where we define the specification, but for our example we'll put it in a static class so we can use it wherever:

import org.springframework.data.jpa.domain.Specification;

public class UserSpecification {

    /**
     * Search for an User using a free text search on the email and name fields
     *
     * @param search text to be searched
     * @return a specification that can be used with the `repository.findAll()` on a `JpaSpecificationExecutor`
     */
    public static Specification<User> searchForUserByEmail(String search) {
        var likeSearch = search.toLowerCase();

        return (root, query, criteriaBuilder) -> {
            return criteriaBuilder.or(
                    criteriaBuilder.like(criteriaBuilder.lower(root.get("email")), likeSearch),
                    criteriaBuilder.like(criteriaBuilder.lower(root.get("companyEmail")), likeSearch)
            );
        };
    }
}

The structure of the specification is a little verbose but you could refactor this to be a bit more reusable if you wanted

Extend the JpaSpeficationExecutor on our repository

We can delete all the methods in the repository and add the JpaSpecificationExecutor<User> to the list of things we're extending

// ... existing imports

import org.springframework.data.jpa.repository.JpaSpecificationExecutor;

@Repository
public interface UserRepository extends JpaRepository<User, Long>, JpaSpecificationExecutor<User> { }

The above definition will provide us with two nice method on the repository, namely findAll and findOne which both take a Speficication

Use the speficication with our repository

Now that we have everything defined, we can use this in some part of our application as such:

var spec = UserSpecification.searchForUserByEmail("bob@email.com");
var result = userRepository.findOne(spec);

// or for findAll
var results = userRepository.findAll(spec)

And that's it, In my opinion the Specification solution is a bit easier to manage within the context of the greater codebase without having to worry about too much compile time magic and possible typos in the JQL query

Conclusion

I think either of the the provided solutions are fair and give us with a good way to manage more complex queries more flexibly in a way that isolates our implementation from where we intend to use the code

Oh, and since you made it this far - I'm SO sorry you're writing Java and I wish your sanity all the best

Using Type Guards and Unions to prevent bugs and increase maintainability

Mon, 26 Jun 2023 00:00:00 GMT

When working with a dynamic language, like Javascript, a problem that we can often run into is one where a variable may be of multiple possible types in a given place in our code. Due to this, we often run into a need to check the type of an object

The Scene

For the sake of introducing the need for type guards, we're going to consider an imaginary news site in which users can read articles posted to the site. For our site, we want to enable users to add comments and interact with articles. For this purpose, we require that each user is logged in and exists in our database

The object we use for representing a user who just reads our site is as follows:

const reader = {
  username: 'john',
  email: 'john@email.com',
}

After some time, we decide to make it such that certain users can create articles on the website. To do this, we add a value on the user that states that the user is a writer:

const writer = {
  username: 'smith',
  email: 'smith@email.com',
  isWriter: true,
}

This is great, our data now reflects that we have a user type that can post articles to the website.

The site is growing well and the content available is flourishing, by we seem to be getting a frequent complaints from users that some articles are not appropriate for the site or may be incorrect or misleading

So, to mitigate this issue, we decide that for our website we want to add a new type of permission that defines users that are responsible for moderating content. However, to prevent a conflict of interest - moderators are not allowed to write articles for the site

So we choose to add a field for that indicates a user is a moderator:

const moderator = {
  username: 'bob',
  email: 'bob@email.com',
  isWriter: false,
  isModerator: true,
}

Based on this, we update the application types to account for the new field, and we have the following:

interface User {
  username: string
  email: string
  isWriter: boolean
  isModerator: boolean
}

We update the application to handle the above solution. We even add a few helper methods in our code that allow us to determine if a user is a writer or moderator

const isWriter = (user: User): boolean => user.isWriter

const isModerator = (user: User): boolean => user.isModerator

Fantastic! We're happy with our implementation and we release the code into production

Everything is going great, our moderators are keeping the content quality up and our writers are writing great content

The Bug

After a few weeks though, readers start to notice a problem - some articles that are taken down by moderators seem to be reappearing - it looks as if some malicious moderator is re-enabling bad articles

Upon some further investigation, the team notices that all these articles are ones in which the moderator is the writer of the article - but this shouldn't have happened right? We said that each moderator should not be able to also function as a writer

Eventually, we track down the bug that made this possible, it was in the function called convertUserToModerator:

const convertUserToModerator = (user: User): User => {
  user.isModerator = true

  return user
}

The issue was caused by us not setting the isWriter field to false in the above function, cool - so we just fix that:

const convertUserToModerator = (user: User): User => {
  user.isModerator = true
  user.isWriter = false

  return user
}

Great - we deploy the fix and migrate to ensure that isWriter is set to false for all existing moderators. The issue seems to be fixed, but after a while users start reporting that the problem is happening again

The cause seems to be the same - it looks like users are somehow still having isWriter and isModerator simultaneously set to true and ending up with a permission that shouldn't be possible in our system

Impossible States

So we decide to investigate the problem and determine that the reason we are running into this bug seems to be due to some fundamental way that the data has been defined

Since each user can have true or false for any combination of isWriter and isModerator we end up representing something unintended which can be seen in the table of possible combinations below:

User Type	`isWriter === true`	`isWriter === false`
`isModerator === true`	Unknown	Moderator
`isModerator === false`	Writer	Reader

From the above, we can see that we have a state that is not handled in the current code - this "Unknown" state. In this state, the isModerator and isWriter functions return true

We want this state to be impossible in our code. This is not a state that our application should ever have to consider. From a type-design perspective, it's not enough to handle these impossible states, we need to prevent them from existing in the first place

The Solution

The problem above stems from the fact that we're using two different states to represent something that should be a single state. From the above diagram, we can resolve the following valid types of users in our system

Reader - reads and adds comments to articles
Writer - writes articles
Moderator - moderates articles, does not write articles

Using these user types, we can update the User definition from above:

interface User {
  type: 'reader' | 'writer' | 'moderator'

  username: string
  email: string
}

The type field allows us to determine the type of a user. We can then use this value to check the type of a user and use this to inform how we use the object down the line

Extending the Moderator

Our solution is running well and we have managed to fully eliminate the bug.

But now, the moderators are a little bored with their duties and would like some way that they can compete with each other on the quality of their content moderation - so they decide to create a leaderboard in which they look at how many articles they have read and approved or denied - as interest goes however, the moderators reach out to the development team and ask if it would be possible to add the number of approvals and denials so that the moderators don't need to calculate this manually every month for their leaderboard

The development team has the idea to add these scores into the User definition type:

interface User {
  type: 'reader' | 'writer' | 'moderator'

  username: string
  email: string

  approvedArticles: number
  deniedArticles: number
}

During a PR review session, the team decides that this is a bit weird since it implies that any user can have some approved or denied articles - this doesn't make sense, so someone suggests to use a Discriminated Union in which only users of the moderator type can have these properties - the developer also says that this will make it easier later to add additional fields to any role without having to worry about the implication of this on other roles

The team decides that they want to have a Base user which has some shared properties and a union which adds additional fields

The BaseUser user has a generic parameter called Type which is the type for the user, since we want to ensure that each possible user type has this:

interface BaseUser<Type extends string> {
  type: Type

  username: string
  email: string
}

Each specialized user type can inherit from BaseUser with a specific type applied, for the writer and reader types, these can just be aliases to the base interface

type ReaderUser = BaseUser<'reader'>
type WriterUser = BaseUser<'writer'>

The moderator however, will need to extend this to add the additional properties:

interface ModeratorUser extends BaseUser<'moderator'> {
  approvedArticles: number
  deniedArticles: number
}

Lastly, the three types above can be used to define a single type called User which has the same functionality as the User interface we've been using until now:

type User = ReaderUser | WriterUser | ModeratorUser

The code above is the Typescript syntax for defining a Union which is a type that can be one of multiple types. Each type in the Union is separated by the |. The union we have above is a special type of union called a Discriminated Union which is a special type of Union that has a field called a discriminator which uniquely identifies each element of the union - in our case this is the type field of BaseUser which used the generic Type parameter

In our backend code, we can now use the type property of our user to conditionally fetch the data needed to populate the moderator object

Checks, Checks

As the codebase grows, the team writes some functions to check the type of the user so that specific computations can be done, for example checking the total number of articles a moderator has been reviewed. An example of one of these checks can be seen below:

const isModerator = (user: User): boolean => user.type === 'moderator'

The above check is used in quite a few places, like when counting the total number of articles reviewed:

const getTotalArticlesReviewed = (user: User): number | undefined => {
  if (isModerator(user)) {
    const moderatorUser = user as ModeratorUser

    return moderatorUser.approvedArticles + moderatorUser.deniedArticles
  }

  return undefined
}

Similar casting patterns as above seem to be cropping up in a lot of places, another example is when listing the moderators that a user follows:

const moderatorUsers = users.filter(isModerator) as ModeratorUser[]

Type Guards

The above solutions use Type Assertions which allow a user to override the Typescript Compiler's type system to manually state the type of an object - this is dangerous because it essentially means that we give up type checking at that point in the code

The type assertions around user types are leading to lots of bugs, the team has a discussion about how this problem can be solved and someone suggests using a Type Guard. Type Guards are functions that can be used to check if a given variable meets a specific criterion under which it can be considered a narrower type that the input type. The difference between a Type Guard in this context and a regular function is that a Type Guard returns type information that can be used to inform the Typescript compiler about the state of a variable

The proposed example is to convert the isModerator function into an Assertion Function by converting the return type from boolean to user is ModeratorUser - this assertion tells the compiler that within the scope that the check is true, the value is of the stated type and not the original type

For example, isModerator can be updated as such:

const isModerator = (user: User): user is ModeratorUser =>
  user.type === 'moderator'

This means that when using the code above, we no longer need to cast since Typescript can infer the variable appropriately

const getTotalArticlesReviewed = (user: User): number | undefined => {
  if (isModerator(user)) {
    // within the scope of this condition the `user` type is known to be `ModeratorUser`
    return user.approvedArticles + user.deniedArticles
  }

  return undefined
}

// the `.filter` function accepts the type guard and can now infer that moderatorUsers is `ModeratorUser[]`
const moderatorUsers = users.filter(isModerator)

Conclusion

Discriminated Unions and type Guards are a powerful methods that can be used for working with complex object types and encoding application rules into your type system to minimize bugs and make code easier to extend

Nvim Error when using Plugins

Tue, 13 Jun 2023 00:00:00 GMT

When trying to use an NVim plugin you may run into the following error message:

Error executing lua ...path/to/some/vim/file.lua

This can happen due to a plugin update that was run or another plugin being installed that caused the working tree of the plugin to be out of date or in an invalid state

To fix this, you may want to try updating and cleaning your plugins. In my case, I am using VimPlug so the commands are:

:PlugUpdate
:PlugClean

Thereafter, quitting and reopening vim seems to usually fix the issue, if not then you may need to investigate the plugin itself

Generic Object Property Formatter and Validator using Typescript

Tue, 09 May 2023 00:00:00 GMT

At times there's a need for transforming specific object properties from an input format to some specific output, this can be done using the following code in some generic means

The Concept of a Schema

The concept of a schema to be used for formatting some data can be defined as follows

export type Primitive = string | number | boolean | Symbol | Date

/** * Define a formatter that can take in a generic object type and format each entry of that object */
export type FormatterSchema<T> = {
  [K in keyof T]?: T[K] extends Primitive
    ? // if the value is primitive then we can just transform it using a simple function
      (val: T[K]) => T[K]
    : // if it's an object then it should either be a formatter schema or a transformer function
      FormatterSchema<T[K]> | ((val: T[K]) => T[K])
}

An Interpreter

The implementation of an interpreter that satisfies the above can done as follows:

type Formatter<T> = (val: T) => T
const isFormatterFn = <T>(
  formatter: FormatterSchema<T>[keyof T]
): formatter is Formatter<T[keyof T]> => typeof formatter === 'function'

const isFormatterObj = <T>(
  formatter: FormatterSchema<T>[keyof T]
): formatter is Exclude<Formatter<T[keyof T]>, Function> =>
  typeof formatter === 'object'

export const interpret =
  <T>(schema: FormatterSchema<T>): Formatter<T> =>
  (val) => {
    const keys = Object.keys(schema) as Array<keyof T>
    return keys.reduce<T>((prev, key) => {
      const keySchema = schema[key]
      const keyVal = val[key]

      const isSchemaFn = isFormatterFn(keySchema)
      if (isSchemaFn) {
        return {
          ...prev,
          [key]: keySchema(keyVal),
        }
      }

      const isSchemaObj = isFormatterObj(keySchema)
      if (isSchemaObj) {
        return {
          ...prev,
          [key]: interpret(keySchema)(keyVal),
        }
      }

      return prev
    }, val)
  }

Transformers

We can define some general transformers, for example:

const trunc = Math.trunc

const round = Math.round

const length = (len: number) => (val: string) => val.slice(0, len)

const trim = (val: string) => val.trim()

const constant =
  <T>(val: T) =>
  () =>
    val

const optional =
  <T>(fn: (val: T) => T) =>
  (val?: T) =>
    val === undefined ? undefined : fn(val)

export const formatter = {
  trunc,
  round,
  length,
  trim,
  constant,
  optional,
}

Usage

For the sake of example we can define a data type to use

type User = {
  name: string
  location: {
    address: string
    city: string
    gps?: [number, number]
  }
}

const user: User = {
  name: 'Bob Smithysmithson',
  location: {
    city: 'Somewhere secret',
    address: '123 Secret street',
    gps: [123, 456],
  },
}

Using as a Transformer

We can use the above by defining the transformer and using it to transform the input data according to the defined schema

const format = interpret<User>({
  name: formatter.length(10),
  location: {
    city: (val) => (['Home', 'Away'].includes(val) ? val : 'Other'),
    gps: (val) => (isValidGPS(val) ? val : undefined),
  },
})

const result = format(user)

console.log(result)
// {
//   name: 'Bob Smithy',
//   location: {
//     city: 'Other',
//     adddtess: '123 Secret street',
//     gps: undefined,
//   }
// }

Using as a Validator

Using the same principal as above we can use it for validaing data. Validation can be done by throwing in the transformer function

const validate = interpret<User>({
  name: (val) => {
    if (val.length > 5) {
      throw 'Length too long'
    }

    return val
  },
})

// throws with Length too long
validate({
  name: 'Jack Smith',
})

Exploration

It may be worth taking the concept and expanding it into a more fully fledged library with a way to handle validations more simply or to provide more builtin transformers and validators but for the moment it can be stated that there are enough javascript libraries for validation and the formatter alone is likely of limited value but may be worth exploring further if gaps in the existing technology are found

Regardless of the utility of the particular type above I think the concept and types give some insight into how one can build out a library as the ones discussed above

Working Example

A working playground with the code from here can be found at The Typescript Playground

Postman Flows

Fri, 03 Mar 2023 00:00:00 GMT

Postman is a tool used for making HTTP requests during development that enables easy testing and experimentation with HTTP/REST APIs

Recently Postman released something called Flows which is a visual editor for building API flows between different applications and stitching them together into some kind of output view

Flows can be done to DO things but can also be used to SHOW information based on the data that passes through them

Below is an example of a Postman Flow that reads the posts from my website and lists/counts them when run from postman

Another example of a Postman Flow is of their Stock dashboard example

References

For more information on how to build an use a flow you can take a look at the Postman Flows Overview

Postman flows seems targeted at more technical users. In addition to this I think IFTTT is worth taking a look at if you're interested in doing some small flow-style automations

For a bit of a more self-hosted alernative, you can also take a look at Node-RED

Update or Append to DynamoDB Attributes

Tue, 28 Feb 2023 00:00:00 GMT

DynamoDB Overview

DynamoDB is AWS's No SQL Database Service. Dynamo uses partition keys and sort keys to uniquely identify and partion item in the database which allows for high scalability and throughput

When working with traditional SQL databases a common operation is to update the value of a specific column without having to first fetch the entire row. DynamoDB offers us similar functionality using the AWS SDK

The Update Command

DynamoDB commands are simple objects that consist of a few different parts. When looking to update an item the following are relavant:

Key - which is an object that uniquely identifies an item in the database
UpdateExpression - which is an expression that describes the upadate operation to be done using placeholders for attribute names and values (AWS Documentation - Update Expressions)
ExpressionAttributeNames - generic placehodlders for attribute names
ExpresssionAtributeValues - generic placeholders for values

When defining UpdateExpression we use placeholders for the attribute names and values to prevent any escaping/unsupported character related issues. A common convention for this is to use # at the start of attribute names and : at the start of attribute values to make expressions a bit easier to negotiate, for example, if I wanted to set name to bob I would do '#name': 'name' in the ExpressionAttributeNames and '#name': 'bob' in the ExpressionAttributeValues

DynamoDB uses an object representation that is a bit inconvenient to work with, we can use the marshall and unmarshall functions from @aws-sdk/util-dynamodb to simplify things a bit but if you'd like to know more about the data format used you can look towards the end of this post

Our Example

For our example, imagine we have a table of user check-ins with data structured as follows:

type Item = {
  // partition key
  group: number
  // sort key
  username: string

  status: string
  checkIns: {
    place: string
    time: number
  }[]
}

Update an Attribute

We can use a bit of a generic structure to outline our data that we plan to update a single attribute. To do this, we can use the SET command

The update expression for setting a value will look like so:

SET #attr = :value

Yup, that's pretty much it as far as the expression goes, the actual names of the fields we want to update aren't relevant here, instead they're mentioned in the ExpressionAtrributeNames and ExpressionAttributeValues

The overall command with all of that is as follows:

import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
import { marshall, unmarshall } from '@aws-sdk/util-dynamodb'

const client = new DynamoDBClient({})

const pk = 'bob'
const sk = 12
const updateValue = 'available'

const command = new UpdateItemCommand({
  TableName: 'my-table-name',
  Key: marshall({
    username: pk,
    group: sk,
  }),
  UpdateExpression: 'SET #attr = :value',
  ExpressionAttributeNames: {
    '#attr': 'status',
  },
  ExpressionAtrributeValues: marshall({
    ':value': updateValue,
  }),
})

await client.send()

And that's pretty much the process for updating an attribute witha specific value

Append to a List Attribute That Exists

In the above data structure we have the checkIns field which is a list of objects

We can use the same method as above to define the UpdateExpression, however this time we can use the list_append function in our expression to state that we would like to append a value to

The list_append function appends one list to another - this means that it needs two lists to operate on. To update a list we can just use itself as the first input and the new list as the second. The UpdateExpression looks like this:

SET #attr = list_append(#attr, :value)

For our sake we only want to append a single item, so we can just wrap it in an array when we pass it on in the command. The command for the above update looks like so:

import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
import { marshall, unmarshall } from '@aws-sdk/util-dynamodb'

const client = new DynamoDBClient({})

const pk = 'bob'
const sk = 12
const updateValue = {
  place: 'home',
  time: Date.now(),
}

const command = new UpdateItemCommand({
  TableName: 'my-table-name',
  Key: marshall({
    username: pk,
    group: sk,
  }),
  UpdateExpression: 'SET #attr = list_append(#attr, :value)',
  ExpressionAttributeNames: {
    '#attr': 'checkIns',
  },
  ExpressionAtrributeValues: marshall({
    // array since the update expression works on two lists
    ':value': [updateValue],
  }),
})

await client.send()

Append to a List Attribute That May Not Exist

In some cases, we can end up trying to append to an attribute that may not exist, under these circumstances we can use the is_not_exists function that takes an attribute name and a fallback value and will return the fallback if the attribute does not exist in the item

We can change the UpdateExpression to make use of this as follows:

SET #attr = list_append(if_not_exists(#attr, :fallback), :value)

And the full command looks like so:

import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
import { marshall, unmarshall } from '@aws-sdk/util-dynamodb'

const client = new DynamoDBClient({})

const pk = 'bob'
const sk = 12
const updateValue = {
  place: 'home',
  time: Date.now(),
}

const command = new UpdateItemCommand({
  TableName: 'my-table-name',
  Key: marshall({
    username: pk,
    group: sk,
  }),
  UpdateExpression:
    'SET #attr = list_append(if_not_exists(#attr, :fallback), :value)',
  ExpressionAttributeNames: {
    '#attr': 'checkIns',
  },
  ExpressionAtrributeValues: marshall({
    ':value': [updateValue],
    // fallback to an empty array if the value does not exist before appending
    ':fallback': [],
  }),
})

await client.send()

The above expresson helps us append an item to the list while also providing a fallback for the case where the list item may not exist

A Note on Marshalled/Unmarshalled data

DynamoDB works with data in the "marshalled" form, which is an object representation for primitive data types (AWS Documentation - Attribute Value). Some examples of marshalled and unmarshalled data can be seen below:

// STRING
// Unmarshalled
"hello"

// Marshalled
{
  "S": "Hello"
}

// NUMBER
// Unmarshalled
25

// Marshalled
{
  "N": "25"
}


// MAP
// Unmarshalled
{
  name: "bob"
}

/// Marshalled
{
  "M":{
    "name":{
      "S":"hello"
    }
  }
}

Additional Resources

Speaking of DynamoDB updates, it looks like there's a library for building queries which seems promising and may bwe worth taking a look at called ElectroDB

Draggable Div with XState and React

Tue, 31 Jan 2023 00:00:00 GMT

Why State Management

Recently I've been interested in understanding non-standard user interactions and about how different applications develop this functionality. A particularly good example for me has been looking into the codebase for TLDraw where I ran into state machines

Now, I've played around with a few state machine libraries and I'm fairly familiar with them and have implemented some fairly simple examples for working with multi-step forms and the like, but I haven't really used them in the specific context of designing state

The State Designer Library mentions the idea of Designing State which suggests that the design of how the user interface should work and the implementation of it should be treated as independent, in order to do so, we should design the states that we would like to work with separately from the UI that implements the state

This provides us with a decent abstraction which should make things about state a lot less tangled and easier to reason about

XState

State management libraries provide us with a set of abstractions and tools for designing state. There are a few that are commonly used, of which I'll be using XState in this post. XState has pretty good TypeScript support as well as some great tooling for visualizing and designing state on Stately

The Problem

For this post I've chosen to build a simple draggable div component that makes use of XState. The idea here was to get a feel for the library and see how it can be used to tackle problems around UI interaction

Visualizing the State

Defining the state can be done using the XState visual editor that can be used in VSCode or Stately, this is useful because it lets you visualize different ways that the state can be represented in a relatively low-friction setting

The structure I've decided on for the representing my component's state can be seen below:

State Machine Code

The code for the above state machine, with some added type information, can be seen below:

Draggable.machine.ts

import { createMachine, assign } from 'xstate'

type Position = {
  x: number
  y: number
}

type Delta = {
  dx: number
  dy: number
}

type Focus = {
  focused: boolean
}

export const machine = createMachine(
  {
    schema: {
      context: {} as Position,
      events: {} as
        | { type: 'MOVE'; position: Delta }
        | { type: 'FOCUS' }
        | { type: 'BLUR' },
    },

    initial: 'inactive',
    states: {
      inactive: {
        on: {
          FOCUS: 'active',
        },
      },

      active: {
        on: {
          MOVE: {
            target: 'active',
            internal: true,
            actions: 'updatePosition',
          },
          BLUR: 'inactive',
        },
      },
    },
  },
  {
    actions: {
      updatePosition: assign((context, event) => ({
        x: event.position.dx + (context.x || 0),
        y: event.position.dy + (context.y || 0),
      })),
    },
  }
)

In the above, we can also see the following:

An initial state that is inactive with an event of FOCUS which sets the state to active
An active state that has an internal event of MOVE which will trigger the updatePosition action and an event of BLUR which sets the state to inactive
There is a context that will store the coordinates of the dragged element
An actions object which has an action called updatePosition which assigns the context to the new position

We can also see that in the schema the type of events is specified. This makes it so that XState can infer the type of event passed to the updatePosition function

Attach the State to the UI

In order to move from one state to another we use the send method from XState

To use a state machine in React we use the useMachine hook from XState:

Draggable.tsx

import { useMachine } from "@xstate/react";
import React from "react";
import { machine } from "./Draggable.machine";

interface DraggableProps {
  children: React.ReactNode;
}

export const Draggable = ({ children }: DraggableProps) => {
  const [current, send] = useMachine(machine);


  // rest of component

Next, we can use the current to figure out if we are in an active state so that we can do some styling based on that:

Draggable.tsx

const isActive = current.matches('active')

And lastly we can hook up the UI Events to the state machine:

Draggable.tsx

export const Draggable = ({ children }: DraggableProps) => {
  const [current, send] = useMachine(machine)

  const isActive = current.matches('active')

  return (
    <div style={{ position: 'relative' }}>
      <div
        onMouseDown={() => send('FOCUS')}
        onMouseUp={() => send('BLUR')}
        onMouseLeave={() => send('BLUR')}
        onMouseMove={(ev) => {
          send({
            type: 'MOVE',
            position: {
              dx: ev.movementX,
              dy: ev.movementY,
            },
          })
        }}
        style={{
          position: 'absolute',
          backgroundColor: isActive ? 'skyblue' : 'red',
          userSelect: 'none',
          top: current.context.y,
          left: current.context.x,
          padding: 20,
        }}
      >
        {children}
      </div>
    </div>
  )
}

The above will result in a draggable div like so:

Repl.it link

Possible Improvements

If you play around with the above example you'll probably notice that it's not perfect. Though the states we've defined are correct,the complexity in mapping the UI events becomes apparent, as well as the various edge cases that may arise around how DOM events fire in response to user interaction

Though I don't aim to solve all of these points for the sake of the example, it's worth pointing out as a source of exploration. Some possible solutions that may help are things like changing where we listen to specific events, for example listening to the mouseUp events on only the wrapper and not the draggable component, or changing which specific events we're listening to

The idea is that the handling of the UI events is now separated from the actual state management which should make fixing interaction bugs simpler while also decoupling our state from any specific implementation of the UI

Structuring HTML Content

Thu, 26 Jan 2023 00:00:00 GMT

Isn't HTML a structure?

HTML content is structured as a tree - while this is useful for the medium, this structure isn't very convenient for transforming into data that can be used outside of the web or with libraries that use content in a more flat structure

While building Articly I wanted to use a library called EditorJS for displaying and making text content interactive. An immediate problem I ran into was importing content into the editor since it requires the data in a specific format - which is not HTML but rather an array of objects with simplified content

StackOverflow is helpful sometimes

In order to get the content into the EditorJS format, I needed to find a way to transform the HTML that I had from scraping the web and reading RSS feeds into something I could use as some kind of base

After a little bit of searching, I found this handy function on StackOverflow:

//Recursively loop through DOM elements and assign properties to object
function treeHTML(element, object) {
  object['type'] = element.nodeName
  var nodeList = element.childNodes
  if (nodeList != null) {
    if (nodeList.length) {
      object['content'] = []
      for (var i = 0; i < nodeList.length; i++) {
        if (nodeList[i].nodeType == 3) {
          object['content'].push(nodeList[i].nodeValue)
        } else {
          object['content'].push({})
          treeHTML(nodeList[i], object['content'][object['content'].length - 1])
        }
      }
    }
  }
  if (element.attributes != null) {
    if (element.attributes.length) {
      object['attributes'] = {}
      for (var i = 0; i < element.attributes.length; i++) {
        object['attributes'][element.attributes[i].nodeName] =
          element.attributes[i].nodeValue
      }
    }
  }
}

Using the above as a guideline, I concluded that the main thing that I need was to iterate over the HTML content in a way that would allow me to build a content array which is recursive - the idea at this point is not to remove the tree structure, but rather to transform it into something that's a bit easier to work with

A different tree

Now that I had an idea of how to approach the problem, the next step was to define the structure I wanted, below is the final structure I decided on:

The source HTML element object needs to be stored as this may useful to downstream processors
The tagName of the element will be needed to determine how a specific element needs to be handled
The textContent and innerHTML of the element to be stored - this is the core data of the element
The attributes of the HTML element should be unwrapped to make it easier for downstream code to read
The children of the element need to be converted into the the same type by recursively following steps 1-5

Based on the above, the type definition can be seen below along with the implementation:

type TagName = Uppercase<keyof HTMLElementTagNameMap>

type HTMLStringContent = string

type TransformResult = {
  element: Element
  tagName: TagName
  textContent?: string
  htmlContent?: HTMLStringContent
  attrs: Record<string, string>
  children: TransformResult[]
}

const transform = (el: Element): TransformResult => ({
  element: el,
  // tag names are strings internally but that's not very informative downstream
  tagName: el.tagName as TagName,
  textContent: el.textContent || undefined,
  htmlContent: el.innerHTML,
  children: Array.from(el.children).map(transform),
  attrs: Array.from(el.attributes).reduce(
    (acc, { name, value }) => ({ ...acc, [name]: value }),
    {}
  ),
})

Now that I have the data as a tree structure, we can pass in some simple HTML to see what pops out the other end.

Given the following HTML content:

<div>
  <p>Hello World</p>

  <section>
    <img src="hello.jpg" alt="this is an image" />
  </section>
</div>

We can transform it like so:

// use the DOM to parse it from a string
const el = new DOMParser().parseFromString(html, 'text/html').body

// the convertHtmlToBlocks function takes an HTML ELement
const transformed = transform(el)

The transformed data looks something like this:

{
  "element": {},
  "tagName": "BODY",
  "textContent": "\n  Hello World\n\n  \n    \n  \n",
  "htmlContent": "<div>\n  <p>Hello World</p>\n\n  <section>\n    <img src=\"hello.jpg\" alt=\"this is an image\">\n  </section>\n</div>",
  "children": [
    {
      "element": {},
      "tagName": "DIV",
      "textContent": "\n  Hello World\n\n  \n    \n  \n",
      "htmlContent": "\n  <p>Hello World</p>\n\n  <section>\n    <img src=\"hello.jpg\" alt=\"this is an image\">\n  </section>\n",
      "children": [
        {
          "element": {},
          "tagName": "P",
          "textContent": "Hello World",
          "htmlContent": "Hello World",
          "children": [],
          "attrs": {}
        },
        {
          "element": {},
          "tagName": "SECTION",
          "textContent": "\n    \n  ",
          "htmlContent": "\n    <img src=\"hello.jpg\" alt=\"this is an image\">\n  ",
          "children": [
            {
              "element": {},
              "tagName": "IMG",
              "htmlContent": "",
              "children": [],
              "attrs": { "src": "hello.jpg", "alt": "this is an image" }
            }
          ],
          "attrs": {}
        }
      ],
      "attrs": {}
    }
  ],
  "attrs": {}
}

Not too dissimilar to the structure raw HTML we would have had if we just used DOMParser directly, however it now has a lot less noise

Deforestation

As I mentioned earlier, we need to transform the data into a flat array of items - so the question that comes up now is - how do we do that?

Looking at the input HTML we used, we can break things up into two types of elements - containers and content

Containers are pretty much useless to the content structure we're trying to build - the content is all we really care about

This is an important distinction because it tells us what we can throw away

Secondly, we can think of the content inside of a container as an array of content, once we remove all the containers, this content will become flat

So for example, this HTML:

<div>
  <p>Hello World</p>

  <section>
    <img src="hello.jpg" alt="this is an image" />
  </section>
</div>

When we remove the containers can be thought of as:

<p>Hello World</p>

<img src="hello.jpg" alt="this is an image" />

Which can be thought of as an array like so:

;[paragraph, image]

This is our end goal. In order to get here we still have to figure out two things:

How can we separate out the containers from the content
How can we transform the content into the structure that's useful to us

Separating the leaves from the wood

If we think as containers as having no meaningful data, and just being containers for content, then we can conclude that a way to view the data structure is as an array of content - the container is the array, and the content is the items in the array

So, we can write a transformer for a container as something that just returns an array of content

const removeContainer = (data: TransformResult) => data.children

The above is pretty useful, since this means that the following HTML:

<div>
  <p>Hello World</p>

  <section>
    <img src="hello.jpg" alt="this is an image" />
  </section>
</div>

Will be essentially be converted to this:

<p>Hello World</p>

<section>
  <img src="hello.jpg" alt="this is an image" />
</section>

Now, we still see that there's a section leftover since we only returned one layer of children. Since we already have a way to get rid of the wrapper, we can just apply that to the child that's a section,

So we could have something like this:

const removeContainer = (data: TransformResult) =>
  data.children.map((child) =>
    isContainer(child) ? removeContainer(child) : child
  )

Now the function is a bit weird because the inner map is either returning an array if the child is a container, or a single child if it's not - for consistency, let's just always return an array:

const removeContainer = (data: TransformResult) =>
  data.children.map((child) =>
    isContainer(child) ? removeContainer(child) : [child]
  )

Much better, but now we've introduced something weird - instead of just returning a TransformResult[] we're now returning a TransformResult[][] - let's just leave this here for now, we can always unwrap the arrays later - importantly we now know that we've eliminated the wrappers, so the content we're left with now represents:

<p>Hello World</p>

<img src="hello.jpg" alt="this is an image" />

So this is pretty great, and is the general idea of how we can unwrap things - next up we can talk about transforming the specific elements into useful data blocks

Building blocks

EditorJS has different sections - blocks as it calls them - of content. These are simple Javascript objects that represent the data for the block

For the sake of our discussion, we're going to consider two blocks - ParagraphBlock and ImageBlock

The simplified types that represent their data can be seen below:

export type ParagraphBlock = {
  type: 'paragraph'
  data: {
    text: string
  }
}

export type SimpleImageBlock = {
  type: 'image'
  data: {
    url: string
    caption: string
  }
}

We can look at the the TransformResult for each of the above elements from when we passed the HTML into our transform function previously, we can see

For the paragraph:

{
  "element": {},
  "tagName": "P",
  "textContent": "Hello World",
  "htmlContent": "Hello World",
  "children": [],
  "attrs": {}
}

Which can be translated to the ParagraphBlock data as:

{
  type: "paragraph",
  data: {
    text: "Hello World"
  }
}

A function for doing this could look something like:

const convertParagraph = (data: TransformResult): ParagraphBlock | undefined =>
  data.textContent
    ? {
        type: 'paragraph',
        data: {
          text: data.textContent,
        },
      }
    : undefined

Cool, this lets us transform a paragraph into some structured data.

We can do something similar for images:

const convertImage = (data: TransformResult): ImageBlock | undefined =>
  data.attrs.src
    ? {
        type: 'image',
        data: {
          url: data.attrs.src,
          caption: data.attrs.alt || '',
        },
      }
    : undefined

Putting it all together

So now that we know how to transform the HTML into something useful, remove the wrappers, and represent individual HTML sections as content blocks, we can put it all together into something that lets us convert a section of HTML fully:

First, we can update the removeContainer function to call the converter on the type of tag that it finds:

// note that we need a handler for BODY since the `DOMParser` will always add a body element when parsing
const isContainer = (data: TransformResult) =>
  data.tagName === 'BODY' ||
  data.tagName === 'DIV' ||
  data.tagName === 'SECTION'

const removeContainer = (data: TransformResult) =>
  data.children.map((child) => {
    if (isContainer(child)) {
      return removeContainer(child)
    } else {
      if (child.tagName === 'IMG') {
        const block = convertImage(child)

        return block ? [block] : []
      } else if (child.tagName === 'P') {
        const block = convertParagraph(child)

        return block ? [block] : []
      }
    }
  })

Now, you can probably see a pattern that's going to arise as we add more and more elements that we want to handle - so it may be better to create a list of handlers for different tag types:

type Block = ParagraphBlock | ImageBlock

const handlers: Partial<Record<TagName, (data: TransformResult) => Block[]>> = {
  // content blocks
  IMG: convertImage,
  P: convertParagraph,

  // container blocks - we will always want to remove these
  DIV: removeContainer,
  SECTION: removeContainer,
  BODY: removeContainer,
}

Using the above structure, we can tweak the convertImage and convertParagraph functions a bit so that they return the Block[] consistently:

const convertParagraph = (data: TransformResult): ParagraphBlock[] =>
  data.textContent
    ? [
        {
          type: 'paragraph',
          data: {
            text: data.textContent,
          },
        },
      ]
    : []

const convertImage = (data: TransformResult): ImageBlock[] =>
  data.attrs.src
    ? [
        {
          type: 'image',
          data: {
            url: data.attrs.src,
            caption: data.attrs.alt || '',
          },
        },
      ]
    : []

And we can update the removeContainer function to handle things a bit more genericallly:

const removeContainer = (data: TransformResult): Block[] => {
  const contentArr = data.children.map((child) => {
    const handler = handlers[child.tagName]

    if (!handler) {
      return []
    }

    return handler(child)
  })

  return contentArr.flat(1)
}

If you're really attentive, you'll notice the contentArr.flat(1) that was added in the above snippet, this flattens the Block[][] into a Block[]

Once we've got that, we can define a convert function that will take the HTML and output the structured blocks like so:

const convert = (el: Element): Block[] => {
  const transformed = transform(el)

  const initialHandler = handlers[transformed.tagName]

  if (!initialHandler) {
    throw new Error('No handler found for top-level wrapper')
  }

  return initialHandler(transformed)
}

Adding more content types

That's about it, to handle more specific types of content or other HTML elements is just a matter of following the recipe that we did above:

Define if an element is a container or content
If it's a container, just use the removeContainer handler
If it's content then define a handler for the specific kind of content

If you roll this out for loads of elements you'll eventually have a pretty robust converter

Conclusion

That's it! We've covered the basics for building a transformer like this, and once you have a good feel for how this works, the concepts can be applied to loads of different usecases

If you'd like to see the completed version of my converter, you can take a look at the html-editorjs GitHub repo and if you'd like to look at it in action in an application then take a look at Articly

Let's talk about Feeds

Tue, 24 Jan 2023 00:00:00 GMT

Order Up

Life's busy and disastrous and trying to pop in and talk to myself for a few hours is unattainable most of the time

Recently I've gotten really into RSS. RSS is short for Really Simple Syndication and it's basically just a format/standard that enables websites to share their content in a way that's easy for other websites or applications to consume

Starter

An RSS Feed is really just an XML file that contains information on a website's content, and depending on the nature of the website this is usually something like a blog or podcast feed, but I've also seen RSS used for things like project statuses or website uptime information

Main Course

Here's a sample RSS Feed that I found on RSS Tools that I modified a bit to represent something you might see when working with websites or blog content:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
  <channel>
    <title>RSS Example</title>
    <description>This is an example of an RSS feed</description>
    <link>http://www.domain.com/link.htm</link>
    <lastBuildDate>Mon, 28 Aug 2006 11:12:55 -0400</lastBuildDate>
    <pubDate>Tue, 29 Aug 2006 09:00:00 -0400</pubDate>
    <item>
      <title>Item Example</title>
      <description>This is an example of an Item</description>
      <link>http://www.domain.com/link.htm</link>
      <guid isPermaLink="false">1102345</guid>
      <pubDate>Tue, 29 Aug 2006 09:00:00 -0400</pubDate>
      <content:encoded>This is some content</content:encoded>
    </item>
  </channel>
</rss>

In the above example we can see a channel which defines a source for content

Next up we've got the title, description, link, lastBuildDate, and pubDate which are the metadata for the overall feed and are required for readers that may be accessing the feed

Now, all of that is really cool, but we're not into the juicy bits yet - diving into the content itself we see in the item section.

A feed can define multiple item tags - each of these would have some metadata as well as the actual content associated with the post

In the example, each item has the following:

title
description
link
guid
content:encoded

The above fields are pretty so-so, but the part I find intriguing is the guid which is any unique identifier for an item in a feed, this doesn't have a specific format but just needs to be unique to the feed.

Next, is the content:encoded section which is what the feed uses to share it's content - this can either be plain text or HTML, and it's up to the client to figure it out - based on this I suppose you could really share any content or data which is a pretty cool concept

Dessert

Overall, RSS is pretty simple as a format and has some issues and limitation - versions prior to RSS 2.0 can also be a bit difficult to work with as it's not totally standardised

Another issue I've encountered when working with RSS is the size of the file itself. Some feeds post frequently and may have very long posts. This can lead to very long posts

I've noticed that some blogs with large RSS feeds choose to only store their latest items on the RSS feed, this means that consumer interested in reading older items need to have some cache of their own which was made when the items were initially posted - you could use the guid for this or some kind of diffing method but pretty much anything outside of the XML file delivered is the job of a consumer to work with

The Bill

I've seen a lot of revived interest in the RSS format but there are also competitors to the format like ATOM which is a more complete standard and supports features like paged feeds and explicit content type definitions

ATOM seems to solve the few issues I have with RSS but RSS is still literally everywhere which is tough to fight

My latest side project is an RSS reader called Articly which is built with Svelte/SvelteKit since I really wanted to get a feeling for the framework to build a full application and this seemed like a fun opportunity to do that

Virtual Machine vs Containers

Tue, 20 Dec 2022 00:00:00 GMT

Virtual Machines

The below diagram illustrates a VM as it would traditionally be run along with it's relevant applications and dependencies:

Containers

The below diagram shows conainers running on a Container runtime in which they share dependencies as well as shows the relationship between an image and container instances

Smooth Bottom Navigator with Secondary Actions

Thu, 15 Dec 2022 00:00:00 GMT

Not dissimilar from my previous post on the expanding bottom nav this expands on the idea of animating between states in the navigator in a more global way, this version makes use of a similar animation/transition pattern but does so by modifying the heights as well as the absolute positioning of different components

The implementation below takes the overall concept to the simplest possible state, however some refinements that can still be made include being responsive to the size of the additional content as well as doing a more accurate calculation on how large the sliding tab should be and how it's placed

The values for the padding/positions are very hardcoded, but in practice you'd likely want to make these respond to data provided and size appropriately in regards to rest of the component

Here's the svelte code below:

<script>
  import {
    InboxIcon,
    HomeIcon,
    DatabaseIcon,
    MessageCircleIcon,
    MicIcon,
    MusicIcon,
  } from "svelte-feather-icons";

  let selected = "home";

  const icons = [
    {
      id: "home",
      icon: HomeIcon,
    },
    {
      id: "database",
      icon: DatabaseIcon,
    },
    {
      id: "message",
      icon: MessageCircleIcon,
    },
    {
      id: "mic",
      icon: MicIcon,
    },
  ];

  $: selectedIndex = icons.findIndex((item) => item.id === selected);
  $: expanded = selected === "mic";
</script>

<h1>{selected}</h1>

<div class="wrapper">
  <div class="background" class:expanded class:collapsed={!expanded}>
    <div
      class="slider"
      style="--left: {(selectedIndex / icons.length) * 100}%;--right: {((selectedIndex+1) / icons.length) * 100}%;"
    />
    <div class="content" class:expanded class:collapsed={!expanded}>
      <MusicIcon />
      <p>Recording: 00:01:23</p>
    </div>
  </div>

  <div class="items">
    {#each icons as icon}
      <div
        class="item"
        on:click={() => (selected = icon.id)}
        on:keypress={console.log}
      >
        <svelte:component this={icon.icon} />
      </div>
    {/each}
  </div>
</div>

<style global>
  /* uses fixed postion in order to lock it to lock the component to the bototom of the screen */
  .wrapper {
    position: fixed;
    width: 100vw;
    bottom: 0px;
  }

  /* ensure the background and items are all placed the same since they need to overlap	 */
  .items,
  .background {
    height: 0px;
    position: absolute;
    left: 0px;
    bottom: 20px;
    right: 0px;
    max-width: 80vw;
    margin-left: auto;
    margin-right: auto;
  }

  .background {
    border: solid 1px black;
    border-radius: 16px;
    background-color: white;
    height: 52px;

    transition: height 300ms ease-in-out;
  }

  .background.collapsed {
    transition-delay: 150ms;
  }

  .background.expanded {
    height: 100px;
    transition-delay: 0ms;
  }

  .content {
    overflow: hidden;
    display: flex;
    flex-direction: row;
    gap: 16px;
    height: 24px;
    opacity: 1;
    padding: 20px 40px;
    transition: all 300ms ease-in-out;
  }

  .content.collapsed {
    height: 0;
    opacity: 0;
    padding: 0px 40px;
    transition-delay: 0ms;
  }

  .content.expanded {
    height: 24px;
    opacity: 1;
    transition-delay: 150ms;
  }

  .content p {
    margin: 0;
  }

  .slider {
    position: absolute;
    left: calc(var(--left) + 6px);
    right: va(--left);
    bottom: 6px;
    height: 40px;
    width: calc(var(--right) - var(--left) - 12px);
    border-radius: 12px;
    background-color: #87b5eb70;
    transition: left 300ms ease-in-out;
  }

  .items {
    display: flex;
    flex-direction: row;
    justify-content: space-around;
    height: 40px;
  }

  /* set the icon color	 */
  .item {
    color: black;
  }

  /* select a better default font */
  * {
    font-family: Arial, Helvetica, sans-serif;
    margin: 0;
    padding: 0;
  }
</style>

And the current version of the component can be seen here:

Podcast and RSS Reader App Design

Wed, 14 Dec 2022 00:00:00 GMT

I've been developing an app for reading RSS feeds and listening to podcasts for personal use recently, and I finally had some inspiration as far as the design goes. Though it's ever-changing, here's a preview of the design at this point

The app is meant to provide a reading space for RSS feeds as well as allow a user to bookmark articles and tweets for later reading. In addition, it provides a way to listen to podcasts while browsing other content that's been added in a way that feels consistent and easy to use across all platforms

Mobile

I've tried to remove anything that's not absolutely essential from the overall app with the goal of preventing clutter on the mobile app. In order to do this I've made the bottom navigation something that adapts and shows the user relevant actions based on the screen they're on as opposed to providing a combination of local and global menus as is normally seen in mobile apps

For the navigation I'm liking the idea of a static bottom navigation that transforms and adapts as the user moves through the app - this is a decision that's persisted with the desktop design as well because of how it aids in delivering a cohesive cross-platform experience

Light

For the light theme I've settled on a warm-ish background to link up with the idea of "highlighting" when reading as one might do on paper

Dark

For the dark color scheme I wanted to bring something interesting to the color palette since the typical "black" style was feeling boring. I figured color was a way to add some interest to the relatively text-heavy UI and the green was chosen for the feeling of comfort it brings which should bring a softer feel for reading

Desktop

For the desktop view my focus was to keep things pretty minimal and reduce the visual noise, focusing on the reader experience while being consistent with the mobile view, offering some simple style and layout modifications to better make use of the available space

As far as the inbox view goes, there are some simple tabs at the top that allow switching between the different article types as in the mobile case but aside from some spacing and font size tweaks everything else is pretty much aligned to the mobile style

For the article view I'm taking advantage of the added space in the bottom to add some of the actions that are at the top of the screen in the mobile view - this helps make better use of the space and keep the reading canvas clean as well as isolate actions into a single place which should make it easier to navigate around the app using a mouse

Light

Dark

Typescript Utilities

Mon, 12 Dec 2022 00:00:00 GMT

import Snippet from '@/components/Snippet.astro'

Here's a collection of some utility types I've put together:

Functions

Arrays

Objects

Strings

JSON Schema

Type for inferring a simple JSON Schema into a TypeScript type. I think this will become super useful once Typescript adds support for importing JSON values as const. This will mean that using the schema would be something like:

// currently the `const` part doesn't exist, but would be super cool if we could get here
import schema from './my-schema.json' with { type: 'json', const: true };

export type MySchema = FromJSONSchema<typeof schema>;

The open issue that would make the above possible is this: https://github.com/microsoft/TypeScript/issues/32063

It's relatively trivial to implement some kind of pre-build script that could do this for all JSON schemas but I think it's worth waiting till the TS part of this solution exists because it would be great to be able to infer this statically

Un-editable sections inside of a content editable

Wed, 16 Nov 2022 00:00:00 GMT

I was working on an EditorJS plugin and needed to have a way of creating a block that would restrict certain user interactions

Initially, I tried to make use of event listeners to block the undesirable events, but the behavior on mobile was inconsistent and I wanted to avoid the mobile keyboard popping up as well when needed

In addition to the above, I also noticed that even when blocking the events using JavaScript, though the debugger didn't register the event, the user was still able to make changes to the sections being blocked

The solution proved to be fairly simple, though only tested on chrome so it may not be robust.

In my case, what worked was the following:

.demo-uneditable { padding: 10px; background-color: salmon; } </style>

<div contenteditable="true">
  <div contenteditable="false">
    I should be uneditable, and on mobile won't even trigger a keyboard popup
  </div>
</div>

<div class="demo-editable" contenteditable="true"> <div contenteditable="false"> I should be uneditable, and on mobile won't even trigger a keyboard popup </div> </div>

Now, more generally, we can have other editable content while still having uneditable sections

<div contenteditable="true">
  I continue to be editable

  <div contenteditable="false">
    But I should be uneditable, and on mobile won't even trigger a keyboard
    popup
  </div>
</div>

<div class="demo-editable" contenteditable="true"> I continue to be editable

<div class="demo-uneditable" contenteditable="false"> But I should be uneditable, and on mobile won't even trigger a keyboard popup </div> </div>

Expanding Bottom Navigation with CSS Transitions

Sun, 13 Nov 2022 00:00:00 GMT

So I was looking around on YouTube and came across a video about this library called google_nav_bar for flutter and I really liked the idea of how it works and wanted to implement something similar in an app I've been working on

The basic functionality of the of the library can be seen on google_nav_bar package page

I thought the main challenge of this would be implementing the fade-in of the text with the expanding content section, I took a first shot with the result below:

<script>

	let selected = 'home';

	let items = ['home', 'search', 'archive', 'settings'];
	let colors = {
		home: 'lightpink',
		search: 'violet',
		archive: 'skyblue',
		settings: 'lightgrey'
	};
</script>

<svelte:head>
	<link rel="stylesheet" href="https://unpkg.com/mono-icons@1.0.5/iconfont/icons.css" />
</svelte:head>

<nav class="wrapper">
	<ul class="list">
		{#each items as item (item)}
			<li class="item" class:selected={selected === item} on:click={() => (selected = item)}>
				<div class="icon" style={`--bg: ${colors[item]}`}>
					<i class={`mi mi-${item}`} />
				</div>
				<div class="text">
					{item}
				</div>
			</li>
		{/each}
	</ul>
</nav>

<style>
	.mi {
		font-size: 24px;
	}

	.wrapper {
		display: flex;
		width: 100%;
	}

	.list {
		flex: 1;
		display: flex;
		flex-direction: row;
		align-items: center;
		justify-content: space-between;
		padding: 10px 20px;
	}

	.item {
		display: flex;
		flex-direction: row;
		align-items: center;
		gap: 8px;
		justify-content: flex-start;
	}

	.item .text {
		width: 0px;
		opacity: 0;
		overflow: hidden;
		transition: width 0.15s 0s, opacity 0.15s 0s;
	}

	.item.selected .text {
		width: 100px;
		opacity: 1;
		transition: width 0.15s 0s, opacity 0.15s 0.075s;
	}

	.icon {
		display: flex;
		align-items: center;
		justify-content: center;
		color: var(--bg);
		line-height: 0;
	}

	/* resets	 */
	ul {
		padding: 0;
		margin: 0;
	}

	li {
		margin: 0;
		list-style: none;
	}
</style>

I like the overall feel and I think the finicky bits of using the transition are done, but I'd still like to play around a bit more to see how close to the original library I can get it

For now though, here's the REPL with the current working state of the component:

Education App Design Ideas

Tue, 27 Sep 2022 00:00:00 GMT

General App Overview

Content Creation and Editing Experience

Alternate Themes

Read Metadata from Images using Rust

Thu, 25 Aug 2022 00:00:00 GMT

The complete Rust code discussed in this post can be found in the exiflib GitHub repo

Introduction

Image files, such as JPEG, PNG, and RAW formats from digital cameras and software, contain metadata about the image. This metadata can contain information ranging from the make and model of the camera used to the specific shooting conditions under which a picture was taken

Reading this data depends on the image format used. This post looks at specifically reading metadata from images that use the Exchangeable Image File Format (EXIF) for storing metadata

The Rust Programming Language

The Rust programming language is used to read and process the image files. Rust is a general-purpose programming language with an emphasis on performance and type safety

While this post doesn't cover the specifics of programming in Rust, any code samples are accompanied by a description of what the code does but it's useful to have a basic understanding of programming for understanding exactly what the code is doing

It's also worth noting that this post covers a lot of bit-level processing of image files, to get a basic understanding of binary data works take a look at the previous post on Understanding Binary File Formats

The Exchangeable Image File Format (EXIF)

The Exchangeable Image File Format (EXIF) is based on the Tag Image File Format (TIFF) specification for storing metadata. This data is organised into Image File Directories (IFDs) within an image file

The EXIF section in an image file is structured as follows:

Section	Subsection	Number of Bytes
Header
	EXIF Marker (Exif00)	6 bytes
IFD
	Byte Order (II or MM)	2 bytes
	Magic Number (42)	2 bytes
	Data Start Location	4 bytes
	Data Count	2 bytes
	Data Entries	Data Count x 12 bytes/entry
Additional Data Section

Reading EXIF Data

Reading EXIF data is done by reading the bytes in the file. The following examples will use a JPEG from a Fujifilm X-T200 as a reference, though the same concepts can be applied to understanding data from any file format that stores metadata using the EXIF structure

Under the Hood of an Image File

Below is a snippet of the Hex data for a JPEG file alongside the bytes decoded as text:

Hex Data                                           Decoded Text                       Approximate EXIF Subsections

FF D8 FF E1 57 FE 45 78 69 66 00 00 49 49 2A 00    . . . . W . E x i f . . I I * .    Header, Byte Order, Magic Number
08 00 00 00 0C 00 0F 01 02 00 09 00 00 00 9E 00    . . . . . . . . . . . . . . . .    Data Count, Data Start Location
00 00 10 01 02 00 07 00 00 00 A8 00 00 00 12 01    . . . . . . . . . . . . . . . .    |
03 00 01 00 00 00 01 00 00 00 1A 01 05 00 01 00    . . . . . . . . . . . . . . . .    |
00 00 B0 00 00 00 1B 01 05 00 01 00 00 00 B8 00    . . . . . . . . . . . . . . . .    | Data Entries
00 00 28 01 03 00 01 00 00 00 02 00 00 00 31 01    . . ( . . . . . . . . . . . 1 .    |
02 00 1E 00 00 00 C0 00 00 00 32 01 02 00 14 00    . . . . . . . . . . 2 . . . . .    |_
00 00 DE 00 00 00 13 02 03 00 01 00 00 00 02 00    . . . . . . . . . . . . . . . .    |
00 00 98 82 02 00 05 00 00 00 F2 00 00 00 69 87    . . . . . . . . . . . . . . i .    |
04 00 01 00 00 00 14 01 00 00 A5 C4 07 00 1C 00    . . . . . . . . . . . . . . . .    | Additional Data Section
00 00 F8 00 00 00 EC 29 00 00 46 55 4A 49 46 49    . . . . . . . ) . . F U J I F I    |
4C 4D 00 00 58 2D 54 32 30 30 00 00 48 00 00 00    L M . . X - T 2 0 0 . . H . . .    |
01 00 00 00 48 00 00 00 01 00 00 00 44 69 67 69    . . . . H . . . . . . . D i g i    |

The above snippet shows the hex data, in here the EXIF marker can be found on the first line - 45 78 69 66 00 00 which decodes to EXIF\0\, followed by the byte order 49 49 - II which means that the byte order for the file is Little Endian - which means that the smallest value in a sequence is the first byte - this can be used to decode 2A 00 to 42 if the byte order was Big endian the bytes representing 42 would be flipped

The byte order section is the most important thing to note on this first line as it tells an application how to read the data in the IFD as well as it is what any byte offsets should be calculated relative to

Additionally, the data entries section and the additional data section are broadly marked off, to understand where data is located in this file

Reading a File as Bytes

Rust provides a method for reading a file in the standard library is fs::read which can be used by providing it with a path to the file to read, the code for this looks like so:

let file = fs::read("./sample.jpg").unwrap();

The .unwrap() at the end tells rust to either get the file data or exit the program with an error if it could not read the file

The result of this is a Vec[u8] which means a vector (or list) of bytes - the bytes in the list are represented as integer values between 0 and 255, these are equivalent to the hex values in the snippet above

The file is what is used to read the bytes from and will be the data source for reading the EXIF data

Finding the EXIF Starting Point

To find the starting point of the EXIF data we can scan through the file until we find the Exif\0\0 pattern, a function can be defined for searching for a pattern in a list of bytes:

pub fn get_sequence_range(bytes: &[u8], pattern: &[u8]) -> Option<Range<usize>> {
    let start = bytes
        .windows(pattern.len())
        .position(|window| window == pattern)?;

    let end = start + pattern.len();

    Some(start..end)
}

The function above called get_sequence_range searches the bytes for a pattern. This uses the windows function in rust which creates a bunch of smaller lists and finds the position where the window, which is a section of bytes that's the same length as the search pattern and checks if the value is equal to the pattern

If the pattern can be found, then the function will return a range (basically, a start and end point) that goes from the start of the found pattern until the end of the pattern, which is simply the start value plus the length of the pattern

The Option indicates that the function returns either a value if it finds one (denoted by Some) or will return nothing if no value is found, denoted by None. The above function uses the shorthand for the None case which is done by placing a ? at the end of the check for the pattern - which will cause the function to end early if it could not find the pattern

The above function for finding the starting point can be used by passing it the file's bytes like so:

const EXIF_MARKER: &[u8] = "Exif\0\0".as_bytes();

let exif_range = get_sequence_range(file, EXIF_MARKER)?;

Note that in the above function the EXIF_MARKER is defined as the Exif\0\0 text converted to bytes, this is passed as the search pattern to the get_sequence_range function. This gets the EXIF header location which is used to find the Byte order (Endian) marker

Getting the Byte Order

Once we know the location of the EXIF marker, the byte order values go from the 6th and 7th byte after the start of the marker. Since this is done using a range, this means that the range goes from 6 to 8, since the end value is not included in the range, this can be see defined below:

const ENDIAN_RANGE_FROM_EXIF_MARKER: Range<usize> = 6..8;

The bytes for the endian value can be found relative to the exif_range like so:

let start = exif_range.start + ENDIAN_RANGE_FROM_EXIF_MARKER.start;

let bytes = file.get(start..)?;
let endian_bytes = bytes.get(0..2)?;
let endian = get_endian(endian_bytes)?;

The above makes use of a get_endian function to get the byte order which can be defined as follows:

fn get_endian(endian_bytes: &[u8]) -> Option<Endian> {
    let endian = parsing::full_bytes_string(endian_bytes)?;

    match endian.as_str() {
        "MM" => Some(Endian::Big),
        "II" => Some(Endian::Little),
        _ => None,
    }
}

The above function takes the bytes which start at the endian location and converts them to a string (text) value

These values are then compared using a match. If it is II or MM the function returns Big Endian (Endian::Big) or Little Endian (Endian::Little) respectively. If no matching value is found, then None is returned instead

The code above also finds the bytes, which defines the file's bytes but trims off all the bytes that are before the endian marker - this is important since any data in the IFD needs to be read relative to the this location

Following the Endian bytes are 2 bytes which specify the Magic Number (42) as mentioned above - this can also be checked to verify the byte order of the file but is not covered in this post

Getting the IFD Data Start Location and Count

Immediately after the Magic Number is four bytes that specify where the IFD data starts, in the snippet above, these bytes are 08 00 00 00 which convert to the value of 8, this informs us that the IFD data starts from 8 bytes from the Endian marker

By following the offset value, the number of entries in the IFD can be found at the 8 bytes from the Endian marker, in this case, bytes 0C 00 which convert to the value of 12, which indicates that there are 12 entries in the IFD

The code applying the above logic can be seen below:

let ifd0_offset = get_ifd_offset(&endian, ifd)? as usize;
let ifd0_entry_offset = ifd0_offset + 2;

let ifd0_count = u16::from_offset_endian_bytes(&endian, ifd, ifd0_offset)?;

The function which gets the ifd0_offset does the lookup of bytes from the range 4 to 8, relative to the Endian marker

Reading Entries in the IFD

As a reference example, the bytes for the first entry in the IFD above will be used to understand the data and how it's stored

After the bytes indicate the count, the next section consists of the entries. Each entry consists of 12 bytes and is structured like so:

Tag	Data Format	Component Length	Data
2 bytes	2 Bytes	4 Bytes	4 Bytes
`0F 01`	`02 00`	`09 00 00 00`	`9E 00 00 00`

The Tag is an identifier that specifies what the value of the entry represents
The Data format states how the data should be read
The Component Length states how many bytes the data for the entry consists of
The data can either be the actual data, or a value that gives the offset to the data, depending on the Component Lenght

Tag ID

Reading the Tag is done by parsing the first two bytes of an entry - This converts the value into a 16-bit unsigned integer (a positive integer)

The TagID is read like so:

let tag = u16::from_endian_bytes(endian, entry)?;

The value of the tag is a 16-bit unsigned integer, but it's more commonly represented as Hex value in the tag lookup tables, a lookup table for these can be found at the EXIF Tool Tag Names Doc

The value of the tag above 0F 01 can be converted to hex for the Little Endian notation resulting in 0x010F, the lookup table states that this tag identifies the Make property in the Exif data

Data Format

The data stored in an entry can be of 12 different formats, each of these associated with a format value - the format value can be read by reading from byte index 2 in the entry, like so:

let format_value = u16::from_offset_endian_bytes(endian, entry, 2)?;

The format value is a 16-bit unsigned integer, the same as the Tag, though this is used as an integer value and not hex. Each integer value maps to a specific format type, as seen in the below table:

Format Value	Format	Bytes per Component	Data Type	Description
1	Unsigned Byte	1	u8	8-bit positive integer
2	ASCII String	1	String	Text/String value
3	Unsigned Short	2	u16	16-bit positive integer
4	Unsigned Long	4	u32	32-bit positive integer
5	Unsigned Rational	8	u32, u32	positive fraction - numerator and denominator
6	Signed Byte	1	i8	8-bit integer
7	Undefined	1	[u8]	list of bytes
8	Signed Short	2	i16	16-bit integer
9	Signed Long	4	i32	32-bit integer
10	Signed Rational	8	i32, i32	fraction value - numerator and denominator
11	Single Float	4	f32	floating point/decimal
12	Double Float	8	f64	double precision floating point

The above table is implemented in code by first defining a type that states all the possible format types:

pub enum TagFormat {
    UnsignedByte,
    AsciiString,
    UnsignedShort,
    UnsignedLong,
    UnsignedRational,
    SignedByte,
    Undefined,
    SignedShort,
    SignedLong,
    SignedRational,
    SingleFloat,
    DoubleFloat,
}

Each type of value can also be described in terms of the data type it stores as follows:

pub enum ExifValue<'a> {
    UnsignedByte(u8),
    AsciiString(String),
    UnsignedShort(u16),
    UnsignedLong(u32),
    UnsignedRational(u32, u32),
    SignedByte(i8),
    Undefined(&'a [u8]),
    SignedShort(i16),
    SignedLong(i32),
    SignedRational(i32, i32),
    SingleFloat(f32),
    DoubleFloat(f64),
}

Thereafter, a function to go from the given Format Value to the type of the tag being used:

fn get_tag_format(value: &u16) -> Option<TagFormat> {
    match value {
        1 => Some(TagFormat::UnsignedByte),
        2 => Some(TagFormat::AsciiString),
        3 => Some(TagFormat::UnsignedShort),
        4 => Some(TagFormat::UnsignedLong),
        5 => Some(TagFormat::UnsignedRational),
        6 => Some(TagFormat::SignedByte),
        7 => Some(TagFormat::Undefined),
        8 => Some(TagFormat::SignedShort),
        9 => Some(TagFormat::SignedLong),
        10 => Some(TagFormat::SignedRational),
        11 => Some(TagFormat::SingleFloat),
        12 => Some(TagFormat::DoubleFloat),
        _ => None,
    }
}

As done previously, if the correct value can't be found, the function returns None

So, adding to the above code, the code for reading the tag value is:

let format_value = u16::from_offset_endian_bytes(endian, entry, 2)?;
let format = get_tag_format(&format_value)?;

Component Length

The Component length specifies the number of components for the tag format being read - for most tag formats this will be 1, however, for specific values like AsciiString or Undefined, this may be different in which case it specifies the length of the string or how many bytes are required respectively

The value for the component length can be found by reading the relevant bytes in the entry and converting them to a 32-bit unsigned integer, starting from byte index 4, like so:

let component_length = u32::from_offset_endian_bytes(endian, entry, 4)?;

Data

Once the component length is known, getting the total length of the data to be read is done by multiplying the component length by the bytes per component - since different components need different amounts of data

A function that gets the bytes per component for a given tag format can be seen below:

fn get_bytes_per_component(format: &TagFormat) -> u32 {
    match format {
        TagFormat::UnsignedByte => 1,
        TagFormat::AsciiString => 1,
        TagFormat::UnsignedShort => 2,
        TagFormat::UnsignedLong => 4,
        TagFormat::UnsignedRational => 8,
        TagFormat::SignedByte => 1,
        TagFormat::Undefined => 1,
        TagFormat::SignedShort => 2,
        TagFormat::SignedLong => 4,
        TagFormat::SignedRational => 8,
        TagFormat::SingleFloat => 4,
        TagFormat::DoubleFloat => 8,
    }
}

This is based on the table shown previously on component formats

Next, the total length can be defined as the component length multiplied by the bytes per component which can be seen in the code below:

let component_length = u32::from_offset_endian_bytes(endian, entry, 4)?;

let bytes_per_component = get_bytes_per_component(&format);

let total_length = component_length * bytes_per_component;

The data can be read from the data bytes, which start at index 8 of the entry

let data = entry.get(8..12)?;

The data value must be the raw bytes because depending on the resulting length it needs to be processed differently

If the total_length is less than or equal to 4, the data can be read directly from the data bytes, this can be done using a function that converts a TagFormat and data to the relevant value as defined in the table:

fn parse_tag_value<'a>(
    format: &TagFormat,
    endian: &'a Endian,
    bytes: &'a [u8],
) -> Option<ExifValue<'a>> {
    match format {
        TagFormat::UnsignedByte => parsing::bytes_to_unsigned_byte(endian, bytes),
        TagFormat::AsciiString => parsing::bytes_to_ascii_string(bytes),
        TagFormat::UnsignedShort => parsing::bytes_to_unsigned_short(endian, bytes),
        TagFormat::UnsignedLong => parsing::bytes_to_unsigned_long(endian, bytes),
        TagFormat::UnsignedRational => parsing::bytes_to_unsigned_rational(endian, bytes),
        TagFormat::SignedByte => parsing::bytes_to_signed_byte(endian, bytes),
        TagFormat::Undefined => parsing::bytes_to_undefined(bytes),
        TagFormat::SignedShort => parsing::bytes_to_signed_short(endian, bytes),
        TagFormat::SignedLong => parsing::bytes_to_signed_long(endian, bytes),
        TagFormat::SignedRational => parsing::bytes_to_signed_rational(endian, bytes),
        TagFormat::SingleFloat => parsing::bytes_to_single_float(endian, bytes),
        TagFormat::DoubleFloat => parsing::bytes_to_double_float(endian, bytes),
    }
}

And the data value can be obtained using the function above like so:

let value = parse_tag_value(&format, endian, data)

However, if the total_length is greater than 4, the data value needs to be read as an offset from the IFD which is then converted, again, using the parse_tag_value function above

// the value needs to be checked at the offset and used from there
let offset = u32::from_endian_bytes(endian, data)?;

let start = (offset) as usize;
let end = start + (length) as usize;

let range = start..end;

let value_bytes = bytes.get(range)?;

let result = parse_tag_value(&format, endian, value_bytes)

Putting all the above together, reading the tag above will give:

Tag	Data Format	Component Length	Data
2 bytes	2 Bytes	4 Bytes	4 Bytes
`0F 01`	`02 00`	`09 00 00 00`	`9E 00 00 00`
`0x010f`	ASCII String	9	FUJIFILM\0

Reading Additional Entries

Once a single entry can be read - reading additional entries follows the same pattern. Since the bytes per entry is fixed - always 12 - and the number of entries is known from the IFD count, each entry can be iterated over by going 12 bytes at a time and reading their data individually. A more detailed implementation of this as well as the rest of the code can be found on the exiflib GitHub repo

Conclusion

This post provides a basic outline on reading EXIF data from an image, as well as covers the byte structure for reading EXIF entries from an image file. There's a lot more to reading EXIF data from images, but at a high level the parsing covered here should form a basic grounding in how reading this data works

For further reference and inspiration take a look a the reference list at the end of this post as well as the exiflib GitHub repo mentioned previously

References

Implementation details and guidance for reading metadata from:

Some reference implementations and libraries:

Binary Data and File Formats

Sat, 20 Aug 2022 00:00:00 GMT

Introduction

Computers work with data stored in binary formats. Reading and interpreting binary data is an important part of understanding file formats so their data can be read and interpreted

Bits and Bytes, and Binary

Binary files store data using bits

A bit can be either a 1 or 0. When working with bit-data, it's useful to group them into sets of 8, known as bytes

A byte is a sequence of 8-bits which can contain a value ranging from 0 to 255 - these are referred to as the decimal representation

Bytes consist of 8-bit, with each position representing a power of 2 from 2^0 to 2^7, as seen below:

0 0 0 0 0 0 0 0
| | | | | | | |
| | | | | | | |__ 2^0 - 0 or 1
| | | | | | |____ 2^1 - 0 or 2
| | | | | |______ 2^2 - 0 or 4
| | | | |________ 2^3 - 0 or 8
| | | |__________ 2^4 - 0 or 16
| | |____________ 2^5 - 0 or 32
| |______________ 2^6 - 0 or 64
|________________ 2^7 - 0 or 128

total:                  0 to 255

The byte above represents the value for 0, this is because all the bits have a value of 0

Using the above explanation, the number 1 is represented using the following:

0 0 0 0 0 0 0 1
| | | | | | | |
| | | | | | | |__ 2^0 - 1
| | | | | | |____ 2^1 - 0
| | | | | |______ 2^2 - 0
| | | | |________ 2^3 - 0
| | | |__________ 2^4 - 0
| | |____________ 2^5 - 0
| |______________ 2^6 - 0
|________________ 2^7 - 0

total:                  1

Where the position for 2^0 is the only bit with a value (1)

Similarly, 2 is represented as:

0 0 0 0 0 0 2 0
| | | | | | | |
| | | | | | | |__ 2^0 - 0
| | | | | | |____ 2^1 - 2
| | | | | |______ 2^2 - 0
| | | | |________ 2^3 - 0
| | | |__________ 2^4 - 0
| | |____________ 2^5 - 0
| |______________ 2^6 - 0
|________________ 2^7 - 0

total:                  2

Where the bit for 2^1 has a value

Or the number 5 with bits 2^0 and 2^2 having a value:

0 0 0 0 0 1 0 1
| | | | | | | |
| | | | | | | |__ 2^0 - 1
| | | | | | |____ 2^1 - 0
| | | | | |______ 2^2 - 4
| | | | |________ 2^3 - 0
| | | |__________ 2^4 - 0
| | |____________ 2^5 - 0
| |______________ 2^6 - 0
|________________ 2^7 - 0

total:                  5

Which is calculated by adding 2^0 + 2^2 = 1 + 4 = 5

A larger number, like 234 is:

1 1 1 0 1 0 1 0
| | | | | | | |
| | | | | | | |__ 2^0 - 0
| | | | | | |____ 2^1 - 2
| | | | | |______ 2^2 - 0
| | | | |________ 2^3 - 8
| | | |__________ 2^4 - 0
| | |____________ 2^5 - 32
| |______________ 2^6 - 64
|________________ 2^7 - 128

total:                  234

The calculation for the above value is:

2^1 + 2^3 + 2^5 + 2^6 + 2^7 = total = 234

When substituting the powers of 2:

2 + 8 + 32 + 64 + 128 = total = 234

The numbers discussed above are all 1-byte (8-bit) numbers, which have a range between 0 and 255, adding bits to the value will allow the representation of bigger numbers, for example, a 2-byte (16-bit) number can have a value from 0 to 65,535

Hexadecimal (Hex)

In the above example, numbers are represented in binary format (e.g. 000000020), or decimal format (e.g. 2)

When looking at binary data, it can be a bit easier to navigate around by representing data in hexadecimal (hex) format - which represents every 4 bits as a value ranging from 0-15, so, similar to the byte example above, but instead using 4-bits:

0 0 0 0
| | | |
| | | |__ 2^0 - 0 or 1
| | |____ 2^1 - 0 or 2
| |______ 2^2 - 0 or 4
|________ 2^3 - 0 or 8

Using what's already been discussed, the number 12 can be represented in bits as 1100,

Hex numbers additionally convert each of these values into a value from 0-9 or A-F, as seen in the following table:

Decimal	Bits	Hex
0	`0000`	`0`
1	`0001`	`1`
2	`0010`	`2`
3	`0011`	`3`
4	`0100`	`4`
5	`0101`	`5`
6	`0110`	`6`
7	`0111`	`7`
8	`1000`	`8`
9	`1001`	`9`
10	`1010`	`A`
11	`1011`	`B`
12	`1100`	`C`
13	`1101`	`D`
14	`1110`	`E`
15	`1111`	`F`

Using the binary representation, a byte can be represented using 2 Hex values which are taken by using the first 4-bits as the first hex value, and the second 4-bits as the second hex value

For example, the value for 234, represented as bits: 11101010 can be split into 2 sets of 4-bits 1110 1010, using the table above, this becomes EA in hex

Binary Files

Binary files encode data using bits, when viewing them, it's convenient to view the data in them using bytes represented as hex values as was shown above

Binary files usually require some knowledge of how their data is structured to correctly interpret the information. This is usually described in the specification for the file format

Text Files

Plain text files are usually in the UTF-8 or UTF-16 format - this means that they use 8-bits or 16-bits to represent each character, but there are lots of other formats that a text file can use

UTF-8 data can be read by converting the binary data to text data using a table which maps the byte/hex value to the character -for the UTF-8 format, the character "A" is encoded in hex as 41 and "z" is 7a

The Hexadecimal representation for a text file that contains the following UTF-8 data:

Hello World!

Would be stored as a binary file which contains:

48 65 6C 6C 6F 20 57 6F 72 6C 64 21

Putting the hex below the text content, the hex to character mapping can be seen:

H  e  l  l  o     W  o  r  l  d  !
48 65 6C 6C 6F 20 57 6F 72 6C 64 21

Binary files can be viewed in a hex editor to see the raw binary data, but interpreting these files depends on the format used and will differ between file formats

Conclusion

Computers store data using bits. Bits can be structured into sets of 8-bits, called a byte

Data can be represented using bits, bytes, decimal values, or hexadecimal values

Files store data using binary. Binary data can be represented as decimal or hex depending and can be viewed as whichever is appropriate

References

Using React.memo for Controlling Component Rendering

Tue, 16 Aug 2022 00:00:00 GMT

React Top Level API

The React library contains some functions at it's top level scope. Amongst these are the built-in hooks (like useState, useCallback, etc.) as well as some other functions for manipulating React Elements directly - which I've covered in a previous post on The React Top Level API

Component Rendering

By default, React will trigger a component render whenever there is a change to its state or props. React.memo allows us to take control of the props triggered render by giving us a way to look into the prop-change process

React.memo is a higher order component (HOC) that allows us to wrap a component and control whether or not it is updated/re-rendered by defining a function that tells react whether or not it's props are different - and effectively whether this should trigger a new render

Doing the above is useful for complex components that don't necessarily need to be rendered every time their props are changed

API Definition

The React Docs give us the following example for the React.memo HOC:

const MyComponent = (props) => {
  /* render using props */
}

const areEqual = (prevProps, nextProps) => {
  /*
  return true if passing nextProps to render would return
  the same result as passing prevProps to render,
  otherwise return false
  */
}

const MyComponentMemo = React.memo(MyComponent, areEqual)

The component MyComponent will be rendered whenever props are changed, however, using React.memo lets us define a function called areEqual that we can can use to tell React.memo whether or not the new props would render a different result to the old props

We can then use MyComponentMemo in place of MyComponent to take control of when the component is rendered

Rendering On a Specific Type of Change

Say we have the specific component TimeDisplay which shows the time that's being passed into it from App:

import './App.css'
import React, { useState, useEffect } from 'react'

interface TimeDisplayProps {
  time: number
}

const TimeDisplay: React.FC<TimeDisplayProps> = ({ time }) => {
  const display = new Date(time).toString()

  return <h1>{display}</h1>
}

export default function App() {
  const [time, setTime] = useState(Date.now())

  useEffect(() => {
    const handle = setInterval(() => {
      setTime(Date.now())
    }, 100)

    return () => {
      clearInterval(handle)
    }
  }, [])

  return (
    <main>
      <TimeDisplay time={time} />
    </main>
  )
}

The TimeDisplay component in our case only displays time to the second, so any millisecond-level changes don't matter to the component and so we can save on those renders by checking if the difference in time is similar to the previous render's time

Let's assume for our purpose that it's acceptable for the time to be delayed by about 5 seconds, we then can define a function called areTimesWithinOneSecond which compares the next render's props with the previous and returns if they are within 5 seconds of each other:

const areTimesWithinFiveSeconds = (
  prev: TimeDisplayProps,
  next: TimeDisplayProps
): boolean => {
  const diff = next.time - prev.time

  return diff < 5000
}

We can use the above function in a React.memo to define a version of the TimeDisplay component which will prevent unnecessary renders:

const TimeDisplayMemo = React.memo(TimeDisplay, areTimesWithinFiveSeconds)

And it can then be used as a drop-in replacement for the TimeDisplay component:

export default function App() {
  const [time, setTime] = useState(Date.now())

  useEffect(() => {
    const handle = setInterval(() => {
      setTime(Date.now())
    }, 100)

    return () => {
      clearInterval(handle)
    }
  }, [])

  return (
    <main>
      <TimeDisplayMemo time={time} />
    </main>
  )
}

Conclusion

From the above implementation we can see that it's possible to delay rendering of a component using React.memo if the component doesn't need to be re-rendered, hence improving performance by decreasing the number of renders react needs to carry out

REPL

The REPL with the example above can seen below:

References

The React Docs

Dev Tools Update

Wed, 03 Aug 2022 00:00:00 GMT

Programming Languages

I've primarily switched over to Typescript as my language for building applications, but I've also been learning Rust for the purpose of building some low-level libraries

For Typescript, something I've been finding to be really informative is the Type Challenges Repository which has a lot of challenges along with solutions for modeling and solving complex problems using Typescript's type system

For Rust, the Rust Book along with the No Boilerplate YouTube Channel have been really informative

Frameworks

The current version of my website is built using 11ty which is a static site generator which uses Javascript and templates for generating HTML from a variety of data sources - in my case just JSON and Markdown files

A lot of the professional work I do revolves around the React and React-Native ecosystem, but I've made some changes to my stack for personal projects due to the overhead that comes with using React

For now I'm using 11ty for my website as well as Hugo for some smaller projects

I've moved on from Gatsby for my site due to the slow build times as a static site builder - and the unnecessary complexity of graphql for this purpose along with the overall burden of node modules that comes with it

For other projects I'd like to give the Remix Framework a shot for full stack apps. Remix is a full stack framework for building React applications with a pretty cool looking data fetching and project structure

I've also been playing around with Tauri for building desktop applications. Tauri uses a Rust backend with a bring-your-own client approach and pretty much uses any client side javascript framework - I'm looking towards Svelte for this since it keeps things a lot more vanilla which is a good break from working with the other major frameworks

Code Editor

I'm back to VSCode after a short stint with NeoVim.

My reasons for leaving NeoVim were mostly due to the amount of setup when switching devices as well as the issues I experienced on Windows where configuration management was a bit inconsistent as well as the dependency management for some plugins became a lot of admin. Even preconfigured solutions like LunarVim and AstroNvim didn't work well due to lots of compat issues on Windows

As far as VSCode is concerned, some things I've been using are:

The GitHub Online Editor
The GitHub Colorblind Theme (GitHub.github-vscode-theme)
Code Spell Checker Extension (streetsidesoftware.code-spell-checker)
GitLens Extension (eamodio.gitlens)
Hex Editor (ms-vscode.hexeditor)
Rust Analyzer Extension (rust-lang.rust-analyzer)
Fix Extension (withfig.fig)

Terminal

Some terminal tools I've been enjoying recently are:

Hyper - An alternative terminal that's so far been less sluggish and more customizable than the Windows Terminal application
Fig - Provides autocomplete for commands and integrates with loads of CLI tools to provide detailed suggestions and documentations
NuShell - A shell that works with the idea of command output as data that can be manipulated and transformed which adds a lot of cool data processing functionality on top of your normal shell comands

A Simple JSON Backed Database in Typescript

Thu, 07 Jul 2022 00:00:00 GMT

Define the database

The database is defined as a class which has an in-memory store data and a persist method that allows for persisting the database to a file as well as content when an instance is created from an existing JSON file

The code for the database can be seen below:

import { existsSync, readFileSync, writeFileSync } from 'fs'

export class Database<TData> {
  public data: TData

  constructor(private readonly dbPath: string, initial: TData) {
    this.data = this.load(initial)
  }

  public update = (data: Partial<TData>) =>
    (this.data = { ...this.data, ...data })

  public commit = () => this.persist(this.data)

  private persist = (data: TData) =>
    writeFileSync(this.dbPath, JSON.stringify(data))

  private read = (): TData =>
    JSON.parse(readFileSync(this.dbPath).toString()) as TData

  private load = (initial: TData): TData => {
    if (!existsSync(this.dbPath)) {
      this.persist(initial)
    }

    const current = this.read()

    return {
      ...initial,
      ...current,
    }
  }
}

Usage

The database can be used by creating an instance and modifying the data in it by using the update method, and can be written to a file using the commit method:

import { Database } from './db'

interface User {
  name: string
  age: number
}

interface DB {
  users: User[]
}

const initial: DB = {
  users: [],
}

const db = new Database<DB>('./data.json', initial)

db.update({
  users: [
    {
      name: 'Test',
      age: 20,
    },
  ],
})

db.commit()

Functional Example

The above code is found in the below REPL as a runnable example:

A type for getting properties common in all objects from a union

Wed, 08 Jun 2022 00:00:00 GMT

Overview

Something that may come up in practice is a use for a type that allows us to enforce that an object has only the common properties for a given object

For example, given the two objects below:

type AThing = {
  name: string
  age: number
  email: string
  phone: number
}

type BThing = {
  businessName: string[]
  email: string
  phone: string
}

I would like a type that contains only the things that these objects have in common, namely phone and email

A Type for Common Object Keys

This isn't something that typescript has out-of-the-box, however it can be implemented by using some type juggling

First, we define a type called CommonKeys which gets us all the keys which are common in the two objects

type CommonKeys<T, R = {}> = R extends T
  ? keyof T & CommonKeys<Exclude<T, R>>
  : keyof T

The CommonKeys type makes use of a condition to check if R which is the recursive type extends T which is the input type. Based on this, we cut down T one type at a time until there is no more object that can extend R, then for an input type T which is the same as R (an empty object) the result of CommonKeys<{}> will be never since {} has no keys, and will end the recursion

Applying this to the above types, we get:

type ABCommonKeys = CommonKeys<AThing | BThing>
// type ABCommonKeys = "email" | "phone"

And as a sanity check, we can also apply this to {}:

type Basic = CommonKeys<{}>
// type Basic = never

A Type for Common Object

Next, we can use the CommonKeys type defined above to create a Common type which wne used with the intersection will result in a type that has all the keys common in all types from the intersection

type Common<T> = Pick<T, CommonKeys<T>>

We can apply this now to a type of AThing | BThing like so:

type ABCommonObject = Common<AThing | BThing>
// type ABCommonObject = {
//   email: string;
//   phone: string | number;
// }

And we can see that we have the desired result which is an object with the properties that are common between both input object types

Final Solution

We can put the code from above together into the final solution which is just the two above types:

/** Gets the keys common to any type/union of `T` */
type CommonKeys<T, R = {}> = R extends T
  ? keyof T & CommonKeys<Exclude<T, R>>
  : keyof T

/** Gets an object type containing all keys that exist within a type/union `T` */
type Common<T> = Pick<T, CommonKeys<T>>

Type Narrowing in Typescript

Tue, 31 May 2022 00:00:00 GMT

Type Narrowing allows us create conditions under which an object of one type can be used as if it is of another type. We usually use this in conjunction with union types to allow us to specify different handling of the types based on the resulting value

Using `typeof`

We can use the typeof keyword in javascript to find out whether what type an object is. This is useful if we have an object that can take on different structures, for example

type Data = string[]
type GetData = Data | (() => Data)

In the above example, we have a type called GetData which can be either some data or a function to get data. Using this, we can can create a function which fetches data like so:

const fetchData = (getData: GetData): Data => {
  if (typeof getData === 'function') {
    return getData()
  }

  return getData
}

Using `in`

Javascript also has the in operator which can be used to infer types by us checking a property of an object

type SimpleData = {
  name: string
}

type ComplexData = {
  name: {
    first: string
    last: string
  }
  isComplex: true
}

type AnyData = SimpleData | ComplexData

We can then use the in operator to check the existence of a property of an object by using it along with a key that we expect to be in one object but not another

const getComplexName = (data: AnyData): string => {
  // isComplex is the name of the key that we expect in `ComplexData` but not `SimpleData`
  if ('isComplex' in data) {
    return [data.name.first, data.name.last].join(' ')
  }

  return data.name
}

Using `is`

We can use the typescript is keyword to specify that the return of a boolean means that a variable satisfies a specific condition

For example, we can create a function that basically does what the in operator in the above function does:

const isComplex = (data: AnyData): data is ComplexData => {
  return (data as ComplexData).isComplex
}

This can be used in place of the in check in the above example like so:

const getComplexName2 = (data: AnyData): string => {
  // isComplex is the name of the key that we expect in `ComplexData` but not `SimpleData`
  if (isComplex(data)) {
    return [data.name.first, data.name.last].join(' ')
  }

  return data.name
}

References

Localhost HTTP Proxy with Node.js

Tue, 08 Mar 2022 00:00:00 GMT

An localhost HTTP proxy is useful for debugging and can be easily defined using Node.js by installing http-proxy

yarn add http-proxy

And then adding the following to an index.js file:

index.js

const httpProxy = require('http-proxy')

const target = 'http://my-target-website.com:1234'

httpProxy.createProxyServer({ target }).listen(8080)

Which will create a server that listens on 8080 and will proxy requests to the target

React Top Level API

Tue, 01 Mar 2022 00:00:00 GMT

For a reference on the React Top-Level API you can take a look at the React Docs

Introduction

React allows us to do lots of different things using concepts like composition and higher order components. Most of the time these methods are good enough for us to do what we want, however there are some cases where these methods can prove to be insufficient such as when building complex library components or components that need to allow for dynamic composition or do runtime modification of things like child component props, etc.

For the above purposes we can make use of the React Top-Level API. For the purpose of this writeup I'll be making use of the parts of this API that allow us to modify a component's children and modify their props as well as how they're rendered

Our Goal

For the purpose of this doc I'll be using the React API to get to do the following:

Show the count of children (Items) passed to a component (Wrapper)
Render each child in a sub-wrapper ItemWrapper
Modify the props of the children by adding a position prop

When this is done, we want to render a component that results in the following markup:

<Wrapper>
  <Count />
  <ItemWrapper>
    <Item name="" position="" />
  </ItemWrapper>
  <ItemWrapper>
    <Item name="" position="" />
  </ItemWrapper>
  <ItemWrapper>
    <Item name="" position="" />
  </ItemWrapper>
</Wrapper>

But a consumer can be used like:

<Wrapper>
  <Item name="">
  <Item name="">
  <Item name="">
</Wrapper>

Using `React.Children` to work with a component's `children`

The React.Children API (see docs) provides us with some utilities for traversing the children passed to a component

Before we can do any of the following, we need to define the structure of an item. Our Item component is defined as follows:

interface ItemProps {
  name: string
  position?: number
}

const Item: React.FC<ItemProps> = ({ name, position }) => (
  <div>
    {name}, {position}
  </div>
)

Use `React.Children.count` to get the count

The React.Children.count function counts the number of child nodes passed to a React component, we can use it like so:

const count = React.Children.count(children)

For our example, let's start off by creating a Count component that simply takes a count prop and displays some text:

interface CountProps {
  count: number
}

const Count: React.FC<CountProps> = ({ count }) => <p>Total: {count}</p>

Next, we can define our Wrapper which will take children and pass the count to our Count component:

const Wrapper: React.FC = ({ children }) => {
  const count = React.Children.count(children)

  return (
    <div>
      <Count count={count} />
    </div>
  )
}

Use `React.Children.map` to wrap each child

Next, the React.Children.map function allows us to map over the children of an element and do stuff with it, for example:

const items = React.Children.map(children, (child) => {
  return <ItemWrapper>{child}</ItemWrapper>
})

Based on the above, we can define an ItemWrapper as so:

const ItemWrapper: React.FC = ({ children }) => (
  <li
    style={{
      backgroundColor: 'lightgrey',
      padding: '10px',
      margin: '20px',
    }}
  >
    {children}
  </li>
)

And we can update the Wrapper to make use of React.children.map:

const Wrapper: React.FC = ({ children }) => {
  const count = React.Children.count(children)

  const items = React.Children.map(children, (child) => {
    return <ItemWrapper>{child}</ItemWrapper>
  })

  return (
    <div>
      <Count count={count} />
      <ul>{items}</ul>
    </div>
  )
}

Use `React.cloneElement` to change child props

Lastly, we want to append a position prop to the Item. To do this we can make use of the React.cloneElement function which allows us to clone an element and modify the props of it. Using this function looks like so:

const childProps = child.props
const newProps = { ...child.props, position: index }

const newChild = React.cloneElement(child, newProps)

Integrating this into the React.Children.map function above will result in our Wrapper looking like so:

const Wrapper: React.FC = ({ children }) => {
  const count = React.Children.count(children)

  const items = React.Children.map(children, (child, index) => {
    const childProps = child.props
    const newProps = { ...child.props, position: index }

    const newChild = React.cloneElement(child, newProps)

    return <ItemWrapper>{newChild}</ItemWrapper>
  })

  return (
    <div>
      <Count count={count} />
      <ul>{items}</ul>
    </div>
  )
}

Use `React.isValidElement`

We've completed most of what's needed, however if for some reason our child is not a valid react element our component may still crash. To get around this we can use the React.isValidElement function

We can update our map function above to return null if the element is not value:

const items = React.Children.map(children, (child, index) => {
  if (!React.isValidElement(child)) return null

  const childProps = child.props
  const newProps = { ...child.props, position: index }

  const newChild = React.cloneElement(child, newProps)

  return <ItemWrapper>{newChild}</ItemWrapper>
})

Which results in our Wrapper now being:

const Wrapper: React.FC = ({ children }) => {
  const count = React.Children.count(children)
  const items = React.Children.map(children, (child, index) => {
    if (!React.isValidElement(child)) return null

    const childProps = child.props
    const newProps = { ...child.props, position: index }

    const newChild = React.cloneElement(child, newProps)

    return <ItemWrapper>{newChild}</ItemWrapper>
  })

  return (
    <div>
      <Count count={count} />
      <ul>{items}</ul>
    </div>
  )
}

The Result

Lastly, we'll render the above using the App component, the API for the above components should be composable as we outlined initially. The App component will now look like so:

const App: React.FC = () => {
  return (
    <Wrapper>
      <Item name="Apple" />
      <Item name="Banana" />
      <Item name="Chocolate" />
    </Wrapper>
  )
}

And the rendered HTML:

<div>
  <p>Total: 3</p>
  <ul>
    <li style="background-color: lightgrey; padding: 10px; margin: 20px;">
      <div>Apple, 0</div>
    </li>
    <li style="background-color: lightgrey; padding: 10px; margin: 20px;">
      <div>Banana, 1</div>
    </li>
    <li style="background-color: lightgrey; padding: 10px; margin: 20px;">
      <div>Chocolate, 2</div>
    </li>
  </ul>
</div>

<div> <p>Total: 3</p> <ul> <li style="background-color: lightgrey; padding: 10px; margin: 20px;"> <div>Apple, 0</div> </li> <li style="background-color: lightgrey; padding: 10px; margin: 20px;"> <div>Banana, 1</div> </li> <li style="background-color: lightgrey; padding: 10px; margin: 20px;"> <div>Chocolate, 2</div> </li> </ul> </div>

And lastly, if you'd like to interact with the code from this sample you can see it in this Repl

Visualizations with React

Fri, 11 Feb 2022 00:00:00 GMT

Data Visualization with D3 and React

React is a library for building reactive user interfaces using JavaScript (or Typescript) and D3 (short for Data-Driven Documents) is a set of libraries for working with visualizations based on data

Before getting started, I would recommend familiarity with SVG, React, and D3

Some good references for SVG are on the MDN SVG Docs

A good place to start for React would be the React Docs or my React Notes

And lastly, the D3 Docs

Getting Stared

To follow along, you will need to install Node.js and be comfortable using the terminal

I'm going to be using a React App with TypeScript initialized with Vite as follows:

yarn create vite

And then selecting the react-ts option when prompted. Next, install d3 from the project root with:

yarn add d3
yarn add --dev @types/d3

Now that we've got a basic project setup, we can start talking about D3

Scales (`d3-scale`)

d3-scale Documentation

Broadly, scales allow us to map from one set of values to another set of values,

Scales in D3 are a set of tools which map a dimension of data to a visual variable. They help us go from something like count in our data to something like width in our rendered SVG

We can create scales for a sample dataset like so:

type Datum = {
  name: string
  count: number
}

export const data: Datum[] = [
  { name: '🍊', count: 21 },
  { name: '🍇', count: 13 },
  { name: '🍏', count: 8 },
  { name: '🍌', count: 5 },
  { name: '🍐', count: 3 },
  { name: '🍋', count: 2 },
  { name: '🍎', count: 1 },
  { name: '🍉', count: 1 },
]

Also, a common thing to do when working with scales is to define margins around out image, this is done simply as an object like so:
const margin = {
  top: 20,
  right: 20,
  bottom: 20,
  left: 35,
}
This just helps us simplify some position/layout things down the line

Scales work by taking a value from the domain (data space) and returning a value from range (visual space):

const width = 600
const height = 400

const x = d3
  .scaleLinear()
  .domain([0, 10]) // values of the data space
  .range([0, width]) // values of the visual space

const position = x(3) // position = scale(value)

Additionally, there's also the invert method which goes the other way - from range to domain

const position = x(3) // position === 30
const value = x.invert(30) // value === 3

The invert method is useful for things like calculating a value from a mouse position

D3 has different Scale types:

Continuous (Linear, Power, Log, Identity, Time, Radial)
Sequential
Diverging
Quantize
Quantile
Threshold
Ordinal (Band, Point)

Continuous Scales

These scales map continuous data to other continuous data

D3 has a few different continuous scale types:

Linear
Power
Log
Identity
Radial
Time
Sequential Color

For my purposes at the moment I'm going to be looking at the methods for Linear and Sequential Color scales, but the documentation explains all of the above very thoroughly and is worth a read for additional information on their usage

Linear

We can use a linear scale in the fruit example for mapping count to an x width:

const maxX = d3.max(data, (d) => d.count) as number

const x = d3
  .scaleLinear<number>()
  .domain([0, maxX])
  .range([margin.left, width - margin.right])

If we don't want the custom domain to range interpolation we can create a custom interpolator. An interpolator is a function that takes a value from the domain and returns the resulting range value

D3 has a few different interpolators included for tasks such as interpolating colors or rounding values

We can create a custom color domain to interpolate over and use the interpolateHsl or interpolateRgb functions:

const color = d3
  .scaleLinear<string>()
  .domain([0, maxX])
  .range(['pink', 'lightgreen'])
  .interpolate(d3.interpolateHsl)

Sequential Color

If for some reason we want to use the pre-included color scales

The scaleSequential scale is a method that allows us to map to a color range using an interpolator.

D3 has a few different interpolators we can use with this function like d3.interpolatePurples, d3.interpolateRainbow or d3.interpolateCool among others which look quite nice

We can create a color scale using the d3.interpolatePurples which will map the data to a scale of purples:

const color = d3
  .scaleSequential()
  .domain([0, maxX])
  .interpolator(d3.interpolatePurples)

These can be used instead of the scaleLinear with interpolateHsl for example above but to provide a pre-calibrated color scale

Ordinal Scales

Ordinal scales have a discrete domain and range and are used for the mapping of discrete data. These are a good fit for mapping a scale with categorical data. D3 offers us the following scales:

Band Scale
Point Scale

Band Scale

A Band Scale is a type of Ordinal Scale where the output range is continuous and numeric

We can create a mapping for where each of our labels should be positioned with scaleBand:

const names = data.map((d) => d.name)

const y = d3
  .scaleBand()
  .domain(names)
  .range([margin.top, height - margin.bottom])
  .padding(0.1)

The domain can be any size array, unlike in the case of continuous scales where the are usually start and end values

Building a Bar Graph

When creating visuals with D3 there are a few different ways we can output to SVG data. D3 provides us with some methods for creating shapes and elements programmatically via a builder pattern - similar to how we create scales.

However, there are also cases where we would want to define out SVG elements manually, such as when working with React so that the react renderer can handle the rendering of the SVG elements and we can manage our DOM structure in a way that's a bit more representative of the way we work in React

The SVG Root

Every SVG image has to have an svg root element. To help ensure that this root scales correctly we also use it with a viewBox attribute which specifies which portion of the SVG is visible since the contents can go outside of the bounds of the View Box and we may not want to display this overflow content by default

Using the definitions for margin, width and height from before we can get the viewBox for the SVG we're trying to render like so:

const viewBox = `0 ${margin.top} ${width} ${height - margin.top}`

And then, using that value in the svg element:

return <svg viewBox={viewBox}>{/* we will render the graph in here */}</svg>

At this point we don't really have anything in the SVG, next up we'll do the following:

Add Bars to the SVG
Add Y Labels to the SVG
Add X Labels to the SVG

Bars

We can create Bars using the following:

const bars = data.map((d) => (
  <rect
    key={y(d.name)}
    fill={color(d.count)}
    y={y(d.name)}
    x={x(0)}
    width={x(d.count) - x(0)}
    height={y.bandwidth()}
  />
))

We make use of the x and y functions which help us get the positions for the rect as well as y.bandWidth() and x(d.count) to height and width for the element

We can then add that into the SVG using:

return (
  <svg viewBox={viewBox}>
    <g>{bars}</g>
  </svg>
)

At this point, the resulting SVG will look like this:

Y Labels

Next, using similar concepts as above, we can add the Y Labels:

const yLabels = data.map((d) => (
  <text key={y(d.name)} y={y(d.name)} x={0} dy="0.35em">
    {d.name}
  </text>
))

Next, we can add this into the SVG, and also wrapping the element in a g with a some basic text alignment and translation for positioning it correctly:

return (
  <svg viewBox={viewBox}>
    <g
      fill="steelblue"
      textAnchor="end"
      transform={`translate(${margin.left - 5}, ${y.bandwidth() / 2})`}
    >
      {yLabels}
    </g>
    <g>{bars}</g>
  </svg>
)

The state of the SVG at this point is:

X Labels

Next, we can add the X Labels over each rect using:

const xLabels = data.map((d) => (
  <text key={y(d.name)} y={y(d.name)} x={x(d.count)} dy="0.35em">
    {d.count}
  </text>
))

And the resulting code looks like this:

return (
  <svg viewBox={viewBox}>
    <g
      fill="steelblue"
      textAnchor="end"
      transform={`translate(${margin.left - 5}, ${y.bandwidth() / 2})`}
    >
      {yLabels}
    </g>
    <g>{bars}</g>
    <g
      fill="white"
      textAnchor="end"
      transform={`translate(-6, ${y.bandwidth() / 2})`}
    >
      {xLabels}
    </g>
  </svg>
)

And the final SVG:

Final Result

The code for the entire file/graph can be seen below:

<details> <summary>Fruit.tsx</summary>

import React from 'react'
import * as d3 from 'd3'
import { data } from '../data/fruit'

const width = 600
const height = 400

const margin = {
  top: 20,
  right: 20,
  bottom: 20,
  left: 35,
}

const maxX = d3.max(data, (d) => d.count) as number

const x = d3
  .scaleLinear<number>()
  .domain([0, maxX])
  .range([margin.left, width - margin.right])
  .interpolate(d3.interpolateRound)

const names = data.map((d) => d.name)

const y = d3
  .scaleBand()
  .domain(names)
  .range([margin.top, height - margin.bottom])
  .padding(0.1)
  .round(true)

const color = d3
  .scaleSequential()
  .domain([0, maxX])
  .interpolator(d3.interpolateCool)

export const Fruit: React.FC = ({}) => {
  const viewBox = `0 ${margin.top} ${width} ${height - margin.top}`

  const yLabels = data.map((d) => (
    <text key={y(d.name)} y={y(d.name)} x={0} dy="0.35em">
      {d.name}
    </text>
  ))

  const bars = data.map((d) => (
    <rect
      key={y(d.name)}
      fill={color(d.count)}
      y={y(d.name)}
      x={x(0)}
      width={x(d.count) - x(0)}
      height={y.bandwidth()}
    />
  ))

  const xLabels = data.map((d) => (
    <text key={y(d.name)} y={y(d.name)} x={x(d.count)} dy="0.35em">
      {d.count}
    </text>
  ))

  return (
    <svg viewBox={viewBox}>
      <g
        fill="steelblue"
        textAnchor="end"
        transform={`translate(${margin.left - 5}, ${y.bandwidth() / 2})`}
      >
        {yLabels}
      </g>
      <g>{bars}</g>
      <g
        fill="white"
        textAnchor="end"
        transform={`translate(-6, ${y.bandwidth() / 2})`}
      >
        {xLabels}
      </g>
    </svg>
  )
}

</details>

Ticks and Grid Lines

Note that D3 includes a d3-axis package but that doesn't quite work given that we're manually creating the SVG using React and not D3's string-based rendering

We may want to add Ticks and Grid Lines on the X-Axis, we can do this using the scale's ticks method like so:

const xGrid = x.ticks().map((t) => (
  <g key={t}>
    <line
      stroke="lightgrey"
      x1={x(t)}
      y1={margin.top}
      x2={x(t)}
      y2={height - margin.bottom}
    />
    <text fill="darkgrey" textAnchor="middle" x={x(t)} y={height}>
      {t}
    </text>
  </g>
))

And then render this in the svg as:

return (
  <svg viewBox={viewBox}>
    <g>{xGrid}</g>
    {/* previous graph content */}
  </svg>
)

The result will look like this:

Building a Line Graph

We can apply all the same as in the Bar Graph before to draw a Line Graph. The example I'll be using consists of a Datum as follows:

export type Datum = {
  date: Date
  temp: number
}

Given that the X-Axis is a DateTime we will need to do some additional conversions as well as formatting

Working with Domains

In the context of this graph it would also be useful to have an automatically calculated domain instead of a hardcoded one as in the previous example

We can use the d3.extent function to calculate a domain:

const dateDomain = d3.extent(data, (d) => d.date) as [Date, Date]
const tempDomain = d3.extent(data, (d) => d.temp).reverse() as [number, number]

We can then use this domain definitions in a scale:

const tempScale = d3
  .scaleLinear<number>()
  .domain(tempDomain)
  .range([margin.top, height - margin.bottom])
  .interpolate(d3.interpolateRound)

const dateScale = d3
  .scaleTime()
  .domain(dateDomain)
  .range([margin.left, width - margin.right])

Create a Line

The d3.line function is useful for creating a d attribute for an SVG path element which defines the line segments

The line function requires x and y mappings. The line for the graph path can be seen as follows:

const line = d3
  .line<Datum>()
  .x((d) => dateScale(d.date))
  .y((d) => tempScale(d.temp))(data) as string

We also include the Datum type in the above to scope down the type of data allowed in the resulting function

Formatting

D3 includes functions for formatting DateTimes. We can create a formatter for a DateTime as follows:

const formatter = d3.timeFormat('%Y-%m')

We can then use the formatter like so:

formatter(dateTime)

Grid Lines

We can define the X Axis and grid lines similar to how we did it previously:

const xGrid = dateTicks.map((t) => (
  <g key={t.toString()}>
    <line
      stroke="lightgrey"
      x1={dateScale(t)}
      y1={margin.top}
      x2={dateScale(t)}
      y2={height - margin.bottom}
      strokeDasharray={4}
    />
    <text fill="darkgrey" textAnchor="middle" x={dateScale(t)} y={height}>
      {formatter(t)}
    </text>
  </g>
))

And the Y Axis grid lines:

const yGrid = tempTicks.map((t) => (
  <g key={t.toString()}>
    <line
      stroke="lightgrey"
      y1={tempScale(t)}
      x1={margin.left}
      y2={tempScale(t)}
      x2={width - margin.right}
      strokeDasharray={4}
    />
    <text fill="darkgrey" textAnchor="end" y={tempScale(t)} x={margin.left - 5}>
      {t}
    </text>
  </g>
))

Final result

Using all the values that have been defined above, we can create the overall graph and grid lines like so:

return (
  <svg viewBox={viewBox}>
    <g>{xGrid}</g>
    <g>{yGrid}</g>
    <path d={line} stroke="steelblue" fill="none" />
  </svg>
)

The final code can be seen below:

<details> <summary>Temperature.tsx</summary>

import React from 'react'
import * as d3 from 'd3'
import { data, Datum } from '../data/temperature'

const width = 600
const height = 400

const margin = {
  top: 20,
  right: 50,
  bottom: 20,
  left: 50,
}

const tempDomain = d3.extent(data, (d) => d.temp).reverse() as [number, number]

const tempScale = d3
  .scaleLinear<number>()
  .domain(tempDomain)
  .range([margin.top, height - margin.bottom])
  .interpolate(d3.interpolateRound)

const tempTicks = tempScale.ticks()

const dateDomain = d3.extent(data, (d) => d.date) as [Date, Date]

const dateScale = d3
  .scaleTime()
  .domain(dateDomain)
  .range([margin.left, width - margin.right])

const dateTicks = dateScale.ticks(5).concat(dateScale.domain())

const line = d3
  .line<Datum>()
  .x((d) => dateScale(d.date))
  .y((d) => tempScale(d.temp))(data) as string

const formatter = d3.timeFormat('%Y-%m')

export const Temperature: React.FC = ({}) => {
  const viewBox = `0 0 ${width} ${height}`

  const xGrid = dateTicks.map((t) => (
    <g key={t.toString()}>
      <line
        stroke="lightgrey"
        x1={dateScale(t)}
        y1={margin.top}
        x2={dateScale(t)}
        y2={height - margin.bottom}
        strokeDasharray={4}
      />
      <text fill="darkgrey" textAnchor="middle" x={dateScale(t)} y={height}>
        {formatter(t)}
      </text>
    </g>
  ))

  const yGrid = tempTicks.map((t) => (
    <g key={t.toString()}>
      <line
        stroke="lightgrey"
        y1={tempScale(t)}
        x1={margin.left}
        y2={tempScale(t)}
        x2={width - margin.right}
        strokeDasharray={4}
      />
      <text
        fill="darkgrey"
        textAnchor="end"
        y={tempScale(t)}
        x={margin.left - 5}
      >
        {t}
      </text>
    </g>
  ))

  return (
    <svg viewBox={viewBox}>
      <g>{xGrid}</g>
      <g>{yGrid}</g>
      <g>
        <path d={line} stroke="steelblue" fill="none" />
      </g>
    </svg>
  )
}

</details>

And the resulting SVG will then look something like this:

</path> </svg>

Logging Aliases for Javascript

Tue, 02 Nov 2021 00:00:00 GMT

I often find myself writing a function to JSON.stringify some data to log in either a pretty or flat structure

It's just more of a convenience method, and it's pretty much the same as doing console.log(JSON.stringify(data)) and looks like this:

const _ = (data: any, pretty: boolean = false) => {
  return console.log(JSON.stringify(data, null, pretty ? 2 : 0))
}

And then, when I need to log something:

_(myData)

Or, if I want to pretty print the JSON

_(myData, true)

Javascript Range Function

Fri, 29 Oct 2021 00:00:00 GMT

Something i often find myself needing is a way to create a range in javascript, similar to what python has

Here's a basic implementation of something that works like that:

const range = (start, end, count, includeEnd = false) => {
  const space = (end - start) / (count - includeEnd)
  return new Array(count).fill(0).map((_, i) => start + i * space)
}

The above will output something like:

const withoutEnd = range(20, 21, 5)
// withoutEnd === [ 20, 20.2, 20.4, 20.6, 20.8 ]

const withEnd = range(20, 21, 5, true)
// withEnd === [ 20, 20.25, 20.5, 20.75, 21 ]

Template Literal Types with Typescript

Mon, 16 Aug 2021 00:00:00 GMT

Template literal types provide us a way to combine string types or enums in Typescript.

In the below example, we can make use of a string type called Action and an enum called Resource to define a type which is a combination of an action combined with a resource

type Action = 'UPDATE' | 'CREATE' | 'DELETE'

enum Resource {
  Person = 'Person',
  Product = 'Product',
  Sale = 'Sale',
}

// the `Lowercase` type concerts all string type options to lowercase
type ResourceAction = `${Resource}.${Action}`

Such that the ResourceAction type is now defined as:

type ResourceAction =
  | 'Person.UPDATE'
  | 'Person.CREATE'
  | 'Person.DELETE'
  | 'Product.UPDATE'
  | 'Product.CREATE'
  | 'Product.DELETE'
  | 'Sale.UPDATE'
  | 'Sale.CREATE'
  | 'Sale.DELETE'

Now, if you're like me - you probably want your types to be consistent in some way. For this purpose, we can use the Lowercase type as follows:

type ResourceActionLowercase = Lowercase<`${Resource}.${Action}`>

Which results in the following types:

type ResourceActionLowercase =
  | 'person.update'
  | 'person.create'
  | 'person.delete'
  | 'product.update'
  | 'product.create'
  | 'product.delete'
  | 'sale.update'
  | 'sale.create'
  | 'sale.delete'

Building Serverless Apps using the Serverless Stack Framework

Thu, 17 Jun 2021 00:00:00 GMT

Prior to doing any of the below you will require your ~/.aws/credentials file to be configured with the credentials for your AWS account

Serverless Stack Framework

SST Framework is a framework built on top of CDK for working with Lambdas and other CDK constructs

It provides easy CDK setups and a streamlined debug and deploy process and even has integration with the VSCode debugger to debug stacks on AWS

Init Project

To init a new project use the following command:

npx create-serverless-stack@latest my-sst-app --language typescript

Which will create a Serverless Stack applocation using TypeScript

Run the App

You can run the created project in using the config defined in the sst.json file:

{
  "name": "my-sst-app",
  "stage": "dev",
  "region": "us-east-1",
  "lint": true,
  "typeCheck": true
}

Using the following commands command will build then deploy a dev stack and allow you to interact with it via AWS/browser/Postman/etc.

npm run start

Additionally, running using the above command will also start the application with hot reloading enabled so when you save files the corresponding AWS resources will be redeployed so you can continue testing

The Files

The application is structured like a relatively normal Lambda/CDK app with lib which contains the following CDK code:

Stack

lib/index.ts

import MyStack from './MyStack'
import * as sst from '@serverless-stack/resources'

export default function main(app: sst.App): void {
  // Set default runtime for all functions
  app.setDefaultFunctionProps({
    runtime: 'nodejs12.x',
  })

  new MyStack(app, 'my-stack')

  // Add more stacks
}

lib/MyStack.ts

import * as sst from '@serverless-stack/resources'

export default class MyStack extends sst.Stack {
  constructor(scope: sst.App, id: string, props?: sst.StackProps) {
    super(scope, id, props)

    // Create the HTTP API
    const api = new sst.Api(this, 'Api', {
      routes: {
        'GET /': 'src/lambda.handler',
      },
    })

    // Show API endpoint in output
    this.addOutputs({
      ApiEndpoint: api.httpApi.apiEndpoint,
    })
  }
}

And src which contains the lambda code:

src/lambda.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  return {
    statusCode: 200,
    headers: { 'Content-Type': 'text/plain' },
    body: `Hello, World! Your request was received at ${event.requestContext.time}.`,
  }
}

Add a new Endpoint

Using the defined constructs it's really easy for us to add an additional endpoint:

src/hello.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  const response = {
    data: 'Hello, World! This is another lambda but with JSON',
  }

  return {
    statusCode: 200,
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(response),
  }
}

And then in the stack we just update the routes:

lib/MyStack.ts

const api = new sst.Api(this, 'Api', {
  routes: {
    'GET /': 'src/lambda.handler',
    'GET /hello': 'src/hello.handler', // new endpoint handler
  },
})

So that the full stack looks like this:

lib/MyStack.ts

import * as sst from '@serverless-stack/resources'

export default class MyStack extends sst.Stack {
  constructor(scope: sst.App, id: string, props?: sst.StackProps) {
    super(scope, id, props)

    // Create the HTTP API
    const api = new sst.Api(this, 'Api', {
      routes: {
        'GET /': 'src/lambda.handler',
        'GET /hello': 'src/hello.handler',
      },
    })

    // Show API endpoint in output
    this.addOutputs({
      ApiEndpoint: api.httpApi.apiEndpoint,
    })
  }
}

VSCode Debugging

SST supports VSCode Debugging, all that's required is for you to create a .vscode/launch.json filw with the following content:

.vscode/launch.json

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug SST Start",
      "type": "node",
      "request": "launch",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["start"],
      "port": 9229,
      "skipFiles": ["<node_internals>/**"]
    },
    {
      "name": "Debug SST Tests",
      "type": "node",
      "request": "launch",
      "runtimeExecutable": "${workspaceRoot}/node_modules/.bin/sst",
      "args": ["test", "--runInBand", "--no-cache", "--watchAll=false"],
      "cwd": "${workspaceRoot}",
      "protocol": "inspector",
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen",
      "env": { "CI": "true" },
      "disableOptimisticBPs": true
    }
  ]
}

This will then allow you to run Debug SST Start which will configure the AWS resources using the npm start command and connect the debugger to the instance so you can debug your functions locally as well as make use of the automated function deployment

Add a DB

From these docs

We can define our table using the sst.Table class:

const table = new sst.Table(this, 'Notes', {
  fields: {
    userId: sst.TableFieldType.STRING,
    noteId: sst.TableFieldType.NUMBER,
  },
  primaryIndex: {
    partitionKey: 'userId',
    sortKey: 'noteId',
  },
})

Next, we can add some endpoint definitions for the functions we'll create as well as access to the table name via the environment:

const api = new sst.Api(this, 'Api', {
  defaultFunctionProps: {
    timeout: 60, // increase timeout so we can debug
    environment: {
      tableName: table.dynamodbTable.tableName,
    },
  },
  routes: {
    // .. other routes
    'GET  /notes': 'src/notes/getAll.handler', // userId in query
    'GET  /notes/{noteId}': 'src/notes/get.handler', // userId in query
    'POST /notes': 'src/notes/create.handler',
  },
})

And lastly we can grant the permissions to our api to access the table

api.attachPermissions([table])

Adding the above to the MyStack.ts file results in the following:

import * as sst from '@serverless-stack/resources'

export default class MyStack extends sst.Stack {
  constructor(scope: sst.App, id: string, props?: sst.StackProps) {
    super(scope, id, props)

    const table = new sst.Table(this, 'Notes', {
      fields: {
        userId: sst.TableFieldType.STRING,
        noteId: sst.TableFieldType.STRING,
      },
      primaryIndex: {
        partitionKey: 'userId',
        sortKey: 'noteId',
      },
    })

    // Create the HTTP API
    const api = new sst.Api(this, 'Api', {
      defaultFunctionProps: {
        timeout: 60, // increase timeout so we can debug
        environment: {
          tableName: table.dynamodbTable.tableName,
        },
      },
      routes: {
        // .. other routes
        'GET  /notes': 'src/notes/getAll.handler', // userId in query
        'GET  /notes/{noteId}': 'src/notes/get.handler', // userId in query
        'POST /notes': 'src/notes/create.handler',
      },
    })

    api.attachPermissions([table])

    // Show API endpoint in output
    this.addOutputs({
      ApiEndpoint: api.httpApi.apiEndpoint,
    })
  }
}

Before we go any further, we need to install some dependencies in our app, particularly uuid for generating unique id's for notes, we can install a dependency with:

npm install uuid
npm install aws-sdk

Define Common Structures

We'll also create some general helper functions for returning responses of different types, you can view the details for their files below but these just wrap the response in a status and header as well as stringify the body

src/responses/successResponse.ts

const successResponse = <T>(item: T) => {
  return {
    statusCode: 200,
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(item),
  }
}

export default successResponse

src/responses/badResuestsResponse.ts

const badRequestResponse = (msg: string) => {
  return {
    statusCode: 400,
    headers: { 'Content-Type': 'text/plain' },
    body: msg,
  }
}

export default badRequestResponse

src/responses/internalErrorResponse.ts

const internalErrorResponse = (msg: string) => {
  console.error(msg)
  return {
    statusCode: 500,
    headers: { 'Content-Type': 'text/plain' },
    body: 'internal error',
  }
}

export default internalErrorResponse

And we've also got a Note type which will be the data that gets stored/retreived:

src/notes/Note.ts

type Note = {
  userId: string
  noteId: string
  content?: string
  createdAt: number
}

export default Note

Access DB

Once we've got a DB table defined as above, we can then access the table to execute different queries

We would create a DB object instance using:

const db = new DynamoDB.DocumentClient()

Create

A create is the simplest one of the database functions for us to implement, this uses the db.put function with the Item to save which is of type Note:

const create = async (tableName: string, item: Note) => {
  await db.put({ TableName: tableName, Item: item }).promise()
}

Get

We can implement a getOne function by using db.get and providing the full Key consisting of the userId and noteId

const getOne = async (tableName: string, noteId: string, userId: string) => {
  const result = await db
    .get({
      TableName: tableName,
      Key: {
        userId: userId,
        noteId: noteId,
      },
    })
    .promise()

  return result.Item
}

GetAll

We can implement a getByUserId function which will make use of db.query and use the ExpressionAttributeValues to populate the KeyConditionExpression as seen below:

const getByUserId = async (tableName: string, userId: string) => {
  const result = await db
    .query({
      TableName: tableName,
      KeyConditionExpression: 'userId = :userId',
      ExpressionAttributeValues: {
        ':userId': userId,
      },
    })
    .promise()

  return result.Items
}

Define Lambdas

Now that we know how to write data to Dynamo, we can implement the following files for the endpoints we defined above:

Create

src/notes/create.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'
import { DynamoDB } from 'aws-sdk'
import { v1 } from 'uuid'
import internalErrorResponse from '../responses/internalErrorResponse'
import successResponse from '../responses/successResponse'
import badRequestResponse from '../responses/badRequestResponse'
import Note from './Note'

const db = new DynamoDB.DocumentClient()

const toItem = (data: string, content: string): Note => {
  return {
    userId: data,
    noteId: v1(),
    content: content,
    createdAt: Date.now(),
  }
}

const parseBody = (event: APIGatewayProxyEventV2) => {
  const data = JSON.parse(event.body || '{}')

  return {
    userId: data.userId,
    content: data.content,
  }
}

const isValid = (data: Partial<Note>) =>
  typeof data.userId !== 'undefined' && typeof data.content !== 'undefined'

const create = async (tableName: string, item: Note) => {
  await db.put({ TableName: tableName, Item: item }).promise()
}

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  if (typeof process.env.tableName === 'undefined')
    return internalErrorResponse('tableName is undefined')

  const tableName = process.env.tableName
  const data = parseBody(event)

  if (!isValid(data))
    return badRequestResponse('userId and content are required')

  const item = toItem(data.userId, data.content)
  await create(tableName, item)

  return successResponse(item)
}

Get

src/notes/get.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'
import { DynamoDB } from 'aws-sdk'
import badRequestResponse from '../responses/badRequestResponse'
import internalErrorResponse from '../responses/internalErrorResponse'
import successResponse from '../responses/successResponse'

type RequestParams = {
  noteId?: string
  userId?: string
}

const db = new DynamoDB.DocumentClient()

const parseBody = (event: APIGatewayProxyEventV2): RequestParams => {
  const pathData = event.pathParameters
  const queryData = event.queryStringParameters

  return {
    noteId: pathData?.noteId,
    userId: queryData?.userId,
  }
}

const isValid = (data: RequestParams) =>
  typeof data.noteId !== 'undefined' && typeof data.userId !== 'undefined'

const getOne = async (tableName: string, noteId: string, userId: string) => {
  const result = await db
    .get({
      TableName: tableName,
      Key: {
        userId: userId,
        noteId: noteId,
      },
    })
    .promise()

  return result.Item
}

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  const data = parseBody(event)

  if (typeof process.env.tableName === 'undefined')
    return internalErrorResponse('tableName is undefined')

  const tableName = process.env.tableName

  if (!isValid(data))
    return badRequestResponse(
      'noteId is required in path, userId is required in query'
    )

  const items = await getOne(
    tableName,
    data.noteId as string,
    data.userId as string
  )

  return successResponse(items)
}
import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'
import { DynamoDB } from 'aws-sdk'
import badRequestResponse from '../responses/badRequestResponse'
import internalErrorResponse from '../responses/internalErrorResponse'
import successResponse from '../responses/successResponse'

type RequestParams = {
  noteId?: string
  userId?: string
}

const db = new DynamoDB.DocumentClient()

const parseBody = (event: APIGatewayProxyEventV2): RequestParams => {
  const pathData = event.pathParameters
  const queryData = event.queryStringParameters

  return {
    noteId: pathData?.noteId,
    userId: queryData?.userId,
  }
}

const isValid = (data: RequestParams) =>
  typeof data.noteId !== 'undefined' && typeof data.userId !== 'undefined'

const getOne = async (tableName: string, noteId: string, userId: string) => {
  const result = await db
    .get({
      TableName: tableName,
      Key: {
        userId: userId,
        noteId: noteId,
      },
    })
    .promise()

  return result.Item
}

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  const data = parseBody(event)

  if (typeof process.env.tableName === 'undefined')
    return internalErrorResponse('tableName is undefined')

  const tableName = process.env.tableName

  if (!isValid(data))
    return badRequestResponse(
      'noteId is required in path, userId is required in query'
    )

  const items = await getOne(
    tableName,
    data.noteId as string,
    data.userId as string
  )

  return successResponse(items)
}

GetAll

src/notes/getAll.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'
import { DynamoDB } from 'aws-sdk'
import badRequestResponse from '../responses/badRequestResponse'
import internalErrorResponse from '../responses/internalErrorResponse'
import successResponse from '../responses/successResponse'

type PathParams = {
  userId?: string
}

const db = new DynamoDB.DocumentClient()

const parseBody = (event: APIGatewayProxyEventV2): PathParams => {
  const data = event.queryStringParameters

  return {
    userId: data?.userId,
  }
}

const isValid = (data: PathParams) => typeof data.userId !== 'undefined'

const getByUserId = async (tableName: string, userId: string) => {
  const result = await db
    .query({
      TableName: tableName,
      KeyConditionExpression: 'userId = :userId',
      ExpressionAttributeValues: {
        ':userId': userId,
      },
    })
    .promise()

  return result.Items
}

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  const data = parseBody(event)

  if (typeof process.env.tableName === 'undefined')
    return internalErrorResponse('tableName is undefined')

  const tableName = process.env.tableName

  if (!isValid(data)) return badRequestResponse('userId is required in query')

  const items = await getByUserId(tableName, data.userId as string)

  return successResponse(items)
}

Testing

Once we've got all the above completed, we can actually test our endpoints and create and read back data

create:

POST https://AWS_ENDPOINT_HERE/notes

{
  "userId": "USER_ID",
  "content": "Hello world"
}

Which responds with:

200

{
  "content": "Hello world",
  "createdAt": 1619177078298,
  "noteId": "NOTE_ID_UUID",
  "userId": "USER_ID"
}

get:

GET https://AWS_ENDPOINT_HERE/notes/NOTE_ID_UUID?userId=USER_ID

200

{
  "content": "Hello world",
  "createdAt": 1619177078298,
  "noteId": "NOTE_ID_UUID",
  "userId": "USER_ID"
}

getAll

GET htttps://AWS_ENDPOINT_HERE/notes?userId=USER_ID

200

[
  {
    "content": "Hello world",
    "createdAt": 1619177078298,
    "noteId": "NOTE_ID_UUID",
    "userId": "USER_ID"
  }
]

Creating Notes Using a Queue

When working with microservices a common pattern is to use a message queue for any operations that can happen in an asynchronous fashion, we can create an SQS queue which we can use to stage messages and then separately save them at a rate that we're able to process them

In order to make this kind of logic we're going to break up our create data flow - a the moment it's this:

lambda -> dynamo
return <-

We're going to turn it into this:

lambda1 -> sqs
 return <-

          sqs -> lambda2 -> dynamo

This kind of pattern becomes especially useful if we're doing a lot more stuff with the data other than just the single DB operation and also allows us to retry things like saving to the DB if we have errors, etc.

A more complex data flow could look something like this (not what we're implementing):

lambda1 -> sqs
 return <-

           sqs -> lambda2 -> dynamo // save to db
               -> lambda3 -> s3     // generate a report
           sqs <-

           sqs -> lambda4           // send an email

Create Queue

SST provides us with the sst.Queue class that we can use for this purpose

To create a Queue you can use the following in stack:

const queue = new sst.Queue(this, 'NotesQueue', {
  consumer: 'src/consumers/createNote.handler',
})

queue.attachPermissions([table])
queue.consumerFunction?.addEnvironment(
  'tableName',
  table.dynamodbTable.tableName
)

The above code does the following:

Create a queue
Give the queue permission to access the table
Add the tableName environment variable to the queue's consumerFunction

We will also need to grant permissions to the API to access the queue so that our create handler is able to add messages to the queue

api.attachPermissions([table, queue])

Which means our Stack now looks like this:

lib/MyStack.ts

import * as sst from '@serverless-stack/resources'

export default class MyStack extends sst.Stack {
  constructor(scope: sst.App, id: string, props?: sst.StackProps) {
    super(scope, id, props)

    const table = new sst.Table(this, 'Notes', {
      fields: {
        userId: sst.TableFieldType.STRING,
        noteId: sst.TableFieldType.STRING,
      },
      primaryIndex: {
        partitionKey: 'userId',
        sortKey: 'noteId',
      },
    })

    const queue = new sst.Queue(this, 'NotesQueue', {
      consumer: 'src/consumers/createNote.handler',
    })

    queue.attachPermissions([table])
    queue.consumerFunction?.addEnvironment(
      'tableName',
      table.dynamodbTable.tableName
    )

    // Create the HTTP API
    const api = new sst.Api(this, 'Api', {
      defaultFunctionProps: {
        timeout: 60, // increase timeout so we can debug
        environment: {
          tableName: table.dynamodbTable.tableName,
          queueUrl: queue.sqsQueue.queueUrl,
        },
      },
      routes: {
        'GET  /': 'src/lambda.handler',
        'GET  /hello': 'src/hello.handler',
        'GET  /notes': 'src/notes/getAll.handler',
        'POST /notes': 'src/notes/create.handler',
        'GET  /notes/{noteId}': 'src/notes/get.handler',
      },
    })

    api.attachPermissions([table, queue])

    // Show API endpoint in output
    this.addOutputs({
      ApiEndpoint: api.httpApi.apiEndpoint,
    })
  }
}

Update the Create Handler

Since we plan to create notes via a queue we will update our create function in the handler to create a new message in the queue, this is done using the SQS class from aws-sdk:

src/notes/create.ts

import { SQS } from 'aws-sdk'

const queue = new SQS()

Once we've got our instance, the create function is done by means of the queue.sendMessage function:

src/notes/create.ts

const create = async (queueUrl: string, item: Note) => {
  return await queue
    .sendMessage({
      QueueUrl: queueUrl,
      DelaySeconds: 0,
      MessageBody: JSON.stringify(item),
    })
    .promise()
}

Lastly, our handler remains mostly the same with the exception of some additional validation to check that we have the queue connection information in the environment:

src/notes/create.ts

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  // pre-save validation
  if (typeof process.env.queueUrl === 'undefined')
    return internalErrorResponse('queueUrl is undefined')

  const queueUrl = process.env.queueUrl

  const data = parseBody(event)

  if (!isValid(data))
    return badRequestResponse('userId and content are required')

  // save process
  const item = toItem(data.userId, data.content)
  const creatresult = await create(queueUrl, item)

  if (!creatresult.MessageId) internalErrorResponse('MessageId is undefined')

  return successResponse(item)
}

Implementing the above into the create handler means that our create.ts file now looks like this:

src/notes/create.ts

import { APIGatewayProxyEventV2, APIGatewayProxyHandlerV2 } from 'aws-lambda'
import { v1 } from 'uuid'
import internalErrorResponse from '../responses/internalErrorResponse'
import successResponse from '../responses/successResponse'
import badRequestResponse from '../responses/badRequestResponse'
import Note from './Note'
import { SQS } from 'aws-sdk'

const queue = new SQS()

// helper functions start

const toItem = (data: string, content: string): Note => {
  return {
    userId: data,
    noteId: v1(),
    content: content,
    createdAt: Date.now(),
  }
}

const parseBody = (event: APIGatewayProxyEventV2) => {
  const data = JSON.parse(event.body || '{}')

  return {
    userId: data.userId,
    content: data.content,
  }
}

const isValid = (data: Partial<Note>) =>
  typeof data.userId !== 'undefined' && typeof data.content !== 'undefined'

// helper functions end

const create = async (queueUrl: string, item: Note) => {
  return await queue
    .sendMessage({
      QueueUrl: queueUrl,
      DelaySeconds: 0,
      MessageBody: JSON.stringify(item),
    })
    .promise()
}

export const handler: APIGatewayProxyHandlerV2 = async (
  event: APIGatewayProxyEventV2
) => {
  // pre-save validation
  if (typeof process.env.queueUrl === 'undefined')
    return internalErrorResponse('queueUrl is undefined')

  const queueUrl = process.env.queueUrl

  const data = parseBody(event)

  if (!isValid(data))
    return badRequestResponse('userId and content are required')

  // save process
  const item = toItem(data.userId, data.content)
  const creatresult = await create(queueUrl, item)

  if (!creatresult.MessageId) internalErrorResponse('MessageId is undefined')

  return successResponse(item)
}

Add Queue-Based Create Handler

Now that we've updated our logic to save the notes into the queue, we need to add the logic for the src/consumers/createNote.handler consumer function as we specified above, this handler will be sent an SQSEvent and will make use of the DynamoDB Table we gave it permissions to use

First, we take the create function that was previously on the create.ts file for saving to the DB:

src/consumers/createNote.ts

import { DynamoDB } from 'aws-sdk'

const db = new DynamoDB.DocumentClient()

const create = async (tableName: string, item: Note) => {
  const createResult = await db
    .put({ TableName: tableName, Item: item })
    .promise()
  if (!createResult) throw new Error('create failed')

  return createResult
}

We'll also need a function for parsing the SQSRecord object into a Note:

src/consumers/createNote.ts

const parseBody = (record: SQSRecord): Note => {
  const { noteId, userId, content, createdAt } = JSON.parse(record.body) as Note

  // do this to ensure we only extract information we need
  return {
    noteId,
    userId,
    content,
    createdAt,
  }
}

And finally we consume the above through the handler, you can see in the below code that we are iterating over the event.Records object, this is because the SQSEvent adds each new event into this array, the reason for this is because we can also specify batching into our Queue so that the handler is only triggered after n events instead of each time, and though this isn't happening in our case, we still should handle this for our handler:

src/consumers/createNote.ts

export const handler: SQSHandler = async (event) => {
  // pre-save environment check
  if (typeof process.env.tableName === 'undefined')
    throw new Error('tableName is undefined')

  const tableName = process.env.tableName

  for (let i = 0; i < event.Records.length; i++) {
    const r = event.Records[i]
    const item = parseBody(r)
    console.log(item)

    const result = await create(tableName, item)
    console.log(result)
  }
}

Putting all the above together our createNote.ts file now has the following code:

import { SQSHandler, SQSRecord } from 'aws-lambda'
import Note from '../notes/Note'
import { DynamoDB } from 'aws-sdk'

const db = new DynamoDB.DocumentClient()

const create = async (tableName: string, item: Note) => {
  const createResult = await db
    .put({ TableName: tableName, Item: item })
    .promise()
  if (!createResult) throw new Error('create failed')

  return createResult
}

const parseBody = (record: SQSRecord): Note => {
  const { noteId, userId, content, createdAt } = JSON.parse(record.body) as Note

  // do this to ensure we only extract information we need
  return {
    noteId,
    userId,
    content,
    createdAt,
  }
}

export const handler: SQSHandler = async (event) => {
  if (typeof process.env.tableName === 'undefined')
    throw new Error('tableName is undefined')

  const tableName = process.env.tableName

  for (let i = 0; i < event.Records.length; i++) {
    const r = event.Records[i]
    const item = parseBody(r)
    console.log(item)

    const result = await create(tableName, item)
    console.log(result)
  }
}

This completes the implementation of the asynchronous saving mechanism for notes. As far as a consumer of our API is concerned, nothing has changed and they will still be able to use the API exactly as we had in the Testing section above

Deploy

Thus far, we've just been running our API in debug mode via the npm run start command, while useful for testing this adds a lot of code to make debugging possible, and isn't something we'd want in our final deployed code

Deploying using sst is still very easy, all we need to do is run the npm run deploy command and this will update our lambda to use a production build of the code instead:

npm run deploy

Teardown

Lastly, the sst CLI also provides us with a function to teardown our start/deploy code. So once you're done playing around you can use this to teardown all your deployed services:

npm run remove

Note that running the remove command will not delete the DB tables, you will need to do this manually

Multi-module Python project using an init.py file

Thu, 06 May 2021 00:00:00 GMT

When creating a multi-module project for Python, especially when coming from another language ecosystem, you may encounter issues importing code from other files, let's take a look at this

The Modules

In our project, we've got the following files:

package1
  |- module1.py
package2
  |- namespace2
       |- module2.py
main.py

Let's say each of our modules export a function, here's the code for them:

package1/module1.py

def hello_p1_m1():
  return "hello"

package2/subpackage1/module1.py

def hello_p2_s1_m1():
  return "hello"

The Main

Now that we've got our files with their functions separated, we try using them from the main.py file:

main.py

import package1.module1 as p1m1
import package2.namespace1.module1 as p2s1m1

print(p1m1.hello_p1_m1())
print(p2n1m1.hello_p2_s1_m1())

The Error

Now, we try to run this using python:

python main.py

And we get the following error:

Traceback (most recent call last):
  File "main.py", line 1, in <module>
    import package1.module1 as p1m1
ImportError: No module named package1.module1

The Solution

Now, this can be pretty annoying, but the solution is very simple. Python identifies a module as a single file, and the assumption that leads to the issue above is that we assume a package has the same convention of just being a directory

However, a packgage requires an __init__.py file to be defined so that they are recognized correctly

So, we need to add two files:

package1/__init__.py
package2/subpackage1/__init__.py

Additionally, these files are completely empty, and only serve as information. Note that we don't need to include a file in package2 directly as this directory does not contain any modules within it so is not really a package in itself and is simply a wrapper subpackage1

XUnit with F#

Sat, 10 Apr 2021 00:00:00 GMT

An important part of writing any software is testing. Unit testing is an automated testing method in which we test individual components of our software to verify that their behaviour aligns with our expectations

This post will take a look at the process of setting up a new F# library and two methods of configuring XUnit to test your project's code

Create a Project

Before we can start testing we need a project that we can run tests on

First, we're going to create a folder that we can work in:

mkdir MyProject
cd MyProject

We can use the following command to create a new project in our MyProject directory:

dotnet new classlib -lang=f# -o MyProject.Lib

The project that we created will contain the following Library.ts file, this is the file that we'll write tests for. First, we want to update the hello function so that it returns a formatted string:

MyProject.Lib/Library.fs

namespace MyProject.Lib

module Say =
    let hello name =
        sprintf "Hello %s" name

Adding Tests

Depending on our preferred project structure we can either:

Add tests in a separate project
Add test files alongside lib files

Method 1: Create Tests in Separate Project

The standard method of .NET unit testing with XUnit is to make use of separate Project and Test solutions, so a normal test setup would look something like:

MyProject.Lib
MyProject.Lib.Tests

To add an new XUnit test project you can run:

dotnet new xunit -lang=f# -o MyProject.Tests

Then, so we're able to test the code from MyProject.Lib, we need to add a reference to it from the Test project we just created:

dotnet add MyProject.Tests reference MyProject.Lib

Then we can create a file in our test project called LibraryTests.fs which will contain our test code which we will cover in the last section, as well as adding a reference to this file in the MyProject.Tests.fsproj

MyProject.Tests.fsproj

...
  <ItemGroup>
    <Compile Include="Tests.fs" />
    <!-- Add the next line -->
    <Compile Include="LibraryTests.fs" />
    <Compile Include="Program.fs" />
  </ItemGroup>
...

Method 2: Create Tests Alongside Lib Files

The second method I'm going to discuss is keeping our test.fs files alongside the code that the file is testing. The structure of our project is something more like this:

MyProject.Lib
|- Library.fs
|- Library.test.fs

Overall I find this more manageable is the way I keep my test code in other languages and frameworks as well

To implement this method we need to add some dependencies to our project

cd MyProject.Lib
dotnet add package Microsoft.NET.Test.Sdk
dotnet add package xunit
dotnet add package xunit.runner.visualstudio

Then we can create a file in our project called Library.test.fs which will contain our test code which we will cover next, as well as a reference to this file in the MyProject.Lib.fsproj

MyProject.Lib.fsproj

...
  <ItemGroup>
    <Compile Include="Library.fs" />
    <!-- Add the next line -->
    <Compile Include="Library.test.fs" />
  </ItemGroup>
...

It's important to note that this file must be added below the Library.fs file as it will reference it for tests to run

Test Files

More detailed information on XUnit can be found in Unit Testing notes

Since we've configured XUnit it may be useful to understand how these tests work in the context of F#. XUnit tests are organized into modules. Regardless of which of the two methods above you're using the test files work the same

Generally, a test file will contain:

A top-level module definition
open statements to import XUnit
Test functions annotated with Fact or Theory

XUnit tests can be broken into 2 types:

Single-case tests without input parameters inputs are labelled Fact
Multi-case tests which make use of input parameters are labelled Theory and use InlineData

Fact

Let's add the following content into our test file into one that tests the hello function from our Lib code with the input "Name"

LibraryTests.fs/Library.test.fs

module LibraryTests

open Xunit
open MyProject.Lib.Say

[<Fact>]
let ``Say.hello -> "Hello name" `` () =
    let name = "name"
    let expected = "Hello name"

    let result = hello name

    Assert.Equal(expected, result)

F# allows us to name our functions using special characters provided they're enclosed in backticks as seen above. Naming test functions this way allows them to be more discriptive than more traditional variable names

Additionally, there's the normal XUnit test setup which includes calling our test function with some input and asserting something about it using Assert.Equal from XUnit

Theory

We can add a Theory to test our function with multiple different inputs:

LibraryTests.fs/Library.test.fs

[<Theory>]
[<InlineData("name", "Hello name")>]
[<InlineData("World", "Hello World")>]
let ``Say.hello -> concantenated string`` (name:string, expected: string) =
    let result = hello name

    Assert.Equal(expected, result)

In the above we add the InlineData attribute which allows us to provide inputs to our test, as well as specifying a name and expected argument for our function. The test framework will then call our test using the arguments as specified in InlineData

When we're done our test file should have the following content:

module LibraryTests

open Xunit
open MyProject.Lib.Say

[<Fact>]
let ``Say.hello -> "Hello name" `` () =
    let name = "name"
    let expected = "Hello name"

    let result = hello name

    Assert.Equal(expected, result)

[<Theory>]
[<InlineData("name", "Hello name")>]
[<InlineData("World", "Hello World")>]
let ``Say.hello -> concantenated string`` (name:string, expected: string) =
    let result = hello name

    Assert.Equal(expected, result)

Running Tests

In order to run tests we can use the dotnet-cli. Depending on the method used you can run your test from the project's root directory using the following command:

Method 1 - dotnet test MyProject.Tests
Method 2 - dotnet test MyProject.Lib

Alternatively, tests can also be run from your IDE or Visual Studio Code with the Ionide and .NET Core Test Explorer extensions installed

Additional Resources

If you'd like a deeper look into F# or XUnit here are some of my other posts which cover those:

Nabeel Valley

Control a Raspberry Pi GPIO with Python

Sun, 28 Mar 2021 00:00:00 GMT

Circuit Diagram

This script will make use of the GPIO Pins to flicker an LED

The circuit diagram is below:

I've used a $330\Omega$ resistor for the resistor connected in series $R_LED$, however the resistance for a given LED can be calculated with this equation (From Circuit Specialists):

$$ R_{LED} = \frac{V_{source} - V_{LED}}{I_{LED}} $$

It's important to connect the LED in the correct direction on the circuit

Code

Below is a simple python script which will handle turning the GPIO pins on and off using the RPi.GPIO library

import RPi.GPIO as GPIO
import time

pin = 21
dur = 1

GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)
GPIO.setup(pin, GPIO.OUT)

while True:
  GPIO.output(pin, GPIO.HIGH)
  print "LED ON"
  time.sleep(dur)

  GPIO.output(pin, GPIO.LOW)
  print "LED OFF"
  time.sleep(dur)

Custom Styles in Markdown

Tue, 23 Mar 2021 00:00:00 GMT

When working with markdown it can often be useful to be able to style elements using custom CSS

We can accomplish this by including a style tag. An example to illustrate this is changing the way a table renders in a specific document by changing the style of table rows to be striped, you would include the following CSS:

<style>
tr:nth-child(even) {
  background-color: #b2b2b2;
  color: #f4f4f4;
}
</style>

If the platform you're displaying the HTML in already has the styles included then you may need to add !important to override in the CSS:

<style>
tr:nth-child(even) {
  background-color: #b2b2b2!important;
  color: #f4f4f4!important;
}
</style>

my-sample-doc.md

# Custom CSS in Markdown Example

This is a document where tables are rendered with every second table row with a specific background and foreground colour

<style>
tr:nth-child(even) {
  background-color: #b2b2b2!important;
  color: #f4f4f4!important;
}
</style>

The above CSS will lead to the following table being rendered with alternating row colours:

| Col 1 | Col 2 | Col 3 |
| ----- | ----- | ----- |
| A     | B     | C     |
| 1     | 2     | 3     |
| A1    | B2    | C3    |

Note that depending on your markdown renderer the embeded style tags may be removed, you'll have to look at your specific renderer/converter/platform to see whether this works for you

With the converter I'm using, showdown.js, this is how the table above looks (with !important included to override my current table styles):

Col 1	Col 2	Col 3
A	B	C
1	2	3
A1	B2	C3

Render Element by Tag Name in React

Fri, 19 Mar 2021 00:00:00 GMT

When using React it can sometimes be useful to render a standard HTML element given the element name dynamically as a prop

React allows us to do this provided we store the element name in a variable that starts with a capital letter, as JSX requires this to render a custom element

We can do something like this by defining a generic element in React which just takes in the name of the tag in addition to it's usual props and children, something like this:

const GenericElement = ({ tagName: Tag, children, ...innerProps }) => (
  <Tag {...innerProps}>{children}</Tag>
)

We can then render this element by calling the GenericElement with the tagName prop and then any children we'd like:

<GenericElement tagName="h1">I am an h1</GenericElement>

Here's a short example using Replit for reference, all the important stuff is in the src/App.jsx file:

Setup HomeAssistant RaspberryPi with WiFi configured

Tue, 09 Mar 2021 00:00:00 GMT

A short lil guide to setting up Home Assistant on a RaspberryPi using WiFi instead of Ethernet

Download Balena Etcher and install from here
Go to the HomeAssistant website and copy the relevant release URL from here
Connect your RaspberryPi's SD card to your computer
Open Balena Etcher and select Flash from URL and select the RaspberryPi's SD to flash to
Once flashing is complete, open the RaspberryPi's SD in File Explorer, and create the following file:

RASPBERRYPIDRIVE/CONFIG/network/my-network

!cat .\my-network
[connection]
id=my-network
uuid=72111c67-4a5d-4d5c-925e-f8ee26efb3c3
type=802-11-wireless
[802-11-wireless]
ssid=YOUR_WIFI_NETWORK_NAME
#hidden=true

[802-11-wireless-security]
auth-alg=open
key-mgmt=wpa-psk
psk=YOUR_WIFI_PASSWORD

[ipv4]
method=auto

[ipv6]
addr-gen-mode=stable-privacy
method=auto

Be sure to use LF line endings, additional information on this setup can be found on GitHub

Serialize a JsonValue Array using F# and FSharp.Data's JsonProvider

Wed, 03 Feb 2021 00:00:00 GMT

When working with the FSharp.Data.JsonProvider type provider you may encounter a need to serialize a JsonValue array

Let's take a look at an example where this may be necessary:

First, we'll define the type of our data using a type provider, in this case it's connected to the following:

data/api-response.json

{
  "data": [
    {
      "id": "id",
      "caption": "caption",
      "media_type": "IMAGE",
      "media_url": "url_to_image"
    }
  ],
  "paging": {
    "cursors": {
      "before": "hash",
      "after": "hash"
    },
    "next": "full_request_url"
  }
}

And the F# file that's using this as the basis for the type definition is as follows:

open FSharp.Data

type ApiResponse = JsonProvider<"./data/api-response.json", SampleIsList=true>

We can also create some sample data using the ApiResponse.Parse method that's now defined on our type thanks to the JsonProvider

// get some sample data
let sampleData = ApiResponse.Parse("""{
  "data": [
      {
          "id": "id",
          "caption": "caption",
          "media_type": "IMAGE",
          "media_url": "url_to_image"
      }
  ],
  "paging": {
      "cursors": {
          "before": "hash",
          "after": "hash"
      },
      "next": "full_request_url"
  }
}""")

If you were to use the sampleData object and try to parse it to JSON you could directly call the sampleData.JsonValue.ToString() method. As designed, this immediately returns the JSON representation of the object

However, when we take a look at the sampleData.Data property, we will notice this is a Datum array, the problem here is that the array type doesn't have JsonValue property

However, if we look at how JsonValue is defined we will see the following:

union JsonValue =
  | String of string
  | Number of decimal
  | Float of float
  | Record of properties : (string * JsonValue) array
  | Array of elements : JsonValue array
  | Boolean of bool
  | Null

Based on this, we can see that an Array of elements can be seen as a JsonValue array, using this information, we can make use of the JsonValue.Array constructor and the JsonValue property of each of the elements of the Data property to convert the JsonValue array to a JsonValue

let jsonValue =
    // get the property we want
    sampleData.Data
    // extract the JsonValue property from each element
    |> Array.map (fun p -> p.JsonValue)
    // pass into the JsonValue.Array constructor
    |> JsonValue.Array

let json = jsonValue.ToString()

Jenkins: A Git process may have crashed in the repository

Wed, 03 Feb 2021 00:00:00 GMT

Taken from this StackOverflow answer

Another git process seems to be running in this repository, e.g.
an edirot opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue

Sometimes when using Jenkins a Git process may crash or timeout resulting in an error message like the above one. This can result in issues when running later Git steps or running other Git steps in the repository in the Jenkins workspace

The simplest way I've found to fix this is to simply delete the .git/index.lock or .git/shallow.lock file from the workspace that's giving the issue and run again

Custom Attributes in C# Web Controllers

Thu, 17 Dec 2020 00:00:00 GMT

Implementing an attribute for a WebAPI or class in C# can help to reduce duplication and centralize parts of the application logic. This could be used for a variety of tasks such as logging information when methods are called as well as managinng authorization

In this post I'm going to cover the following:

Attribute Types and Execution Order

There are a few different attribute types that we can handle on a WebAPI that provide us with the ability to wrap some functionality around our endpoints, below are some of the common attributes that we can implement and the order in which they execute (StackOverflow)

Authorization - IAuthorizationFilter
Action - IActionFilter
Result - IResultFilter
Exception - IExceptionFilter

IActionFilter

The IActionFilter executes before and after a method is executed and contains two different methods for doing this, namely the OnActionExecuting and OnActionExecuted methods respectively. A basic implemtation of IActionFilter would look like this:

namespace CSharpAttributes.Attributes
{
  public class LogStatusAttribute : Attribute, IActionFilter
  {
    public LogStatusAttribute()
    {
      Console.WriteLine("Attribute Initialized");
    }

    public void OnActionExecuting(ActionExecutingContext context)
    {
      Console.WriteLine("OnActionExecuting");
    }

    public void OnActionExecuted(ActionExecutedContext context)
    {
      Console.WriteLine("OnActionExecuted");
    }
  }
}

This can then be implemented on a controller method with a [LogStatus] attribute:

[LogStatus]
[HttpGet]
public IEnumerable<WeatherForecast> Get()
{
  Console.WriteLine("Executing Get");
  return data;
}

The order of logging which we see will be as follows:

Attribute Initialized when the controller is instantiated
OnActionExecuting when the controller is called
Executing Get when the controller is executed
OnActionExecuted when the controller is done executing

IAuthorizationFilter

The IAuthorizationFilter executes as the first filter on a controller's method call

namespace CSharpAttributes.Attributes
{
  public class CustomAuthorizeAttribute : Attribute, IAuthorizationFilter
  {
    public CustomAuthorizeAttribute()
    {
      Console.WriteLine("Attribute Initialized");
    }

    public void OnAuthorization(AuthorizationFilterContext context)
    {
      Console.WriteLine("OnAuthorization");
    }
  }
}

This can then be implemented on a controller method with a [CustomAuthorize] attribute:

[CustomAuthorize]
[HttpGet]
public IEnumerable<WeatherForecast> Get()
{
  Console.WriteLine("Executing Get");
  return data;
}

The order of logging which we see will be as follows:

Attribute Initialized when the controller is instantiated
OnAuthorization when the controller is called
Executing Get when the controller is executed

Modify Response Data

An attribute's context parameter gives us ways by which we can access the HttpContext as well as set the result of a method call so that it can be handled down the line. For example, we can implement our CustomAuthorize attribute with the following:

public void OnAuthorization(AuthorizationFilterContext context)
{
  if (!context.HttpContext.Request.Headers.ContainsKey("X-Custom-Auth"))
  {
    context.Result = new UnauthorizedResult();
  }

  Console.WriteLine("Attribute Called");
}

This will mean that if we set the context.Result in our method then the controller will not be executed and the endpoint will return the UnauthorizedResult early. You can also see that we're able to access things like the HttpContext which makes it easy for us to view the request/response data and do things based on that

Attribute on a Class

Note that it's also possible to apply the above to each method in a class by adding the attribute at the top of the class declaration:

[ApiController]
[LogStatus]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
  ...

Attributes with Input Parameters

We are also able to create attributes that enable the consumer to modify their behaviour by taking input parameters to the constructor, we can update our LogStatus attribute to do something like add a prefix before all logs:

namespace CSharpAttributes.Attributes
{
  public class LogStatusAttribute : Attribute, IActionFilter
  {
    private readonly string _prefix;

    public LogStatusAttribute(string prefix = "")
    {
      _prefix = prefix;
      Console.WriteLine("Attribute Initialized");
    }

    public void OnActionExecuted(ActionExecutedContext context)
    {
      Console.WriteLine(_prefix + "OnActionExecuted");
    }

    public void OnActionExecuting(ActionExecutingContext context)
    {
      Console.WriteLine(_prefix + "OnActionExecuting");
    }
  }
}

Then, applying to our controller method like so:

[LogStatus("WeatherController-Get:")]
[HttpGet]
public IEnumerable<WeatherForecast> Get()
{
  Console.WriteLine("Executing Get");
  return data;
}

So the new output will look like so:

Attribute Initialized when the controller is instantiated
WeatherForecast-Get:OnActionExecuting when the controller is called
Executing Get when the controller is executed
WeatherForecast-Get:OnActionExecuted when the controller is done executing

Attribute Setting at Class and Method Level

Since an attribute can be implemented at a class and method level it's useful for us to be able to implement it at a class and the override the behaviour or add behaviour for a specific method

We can do this by setting the attribute inheritence to false

Updating out LogStatusAttribute we can add the AttributeUsage Attribute as follows:

namespace CSharpAttributes.Attributes
{
  [AttributeUsage(AttributeTargets.All, Inherited = false)]
  public class LogStatusAttribute : Attribute, IActionFilter
  {
    ...

This means that we can independently apply the attribute at class and method levels, so now our controller can look something like this:

namespace CSharpAttributes.Controllers
{
  [ApiController]
  [Route("[controller]")]
  [LogStatus("WeatherForecast:")]
  public class WeatherForecastController : ControllerBase
  {
    [LogStatus("Get:")]
    [HttpGet]
    public IEnumerable<WeatherForecast> Get()
    {
      Console.WriteLine("Executing Get");
      return data;
    }
  }
}

Which will output the logs as follows:

Attribute Initialized when the controller is instantiated
WeatherForecast:OnActionExecuting when the class is called
Get:OnActionExecuting when the controller is called
Executing Get when the controller is executed
Get:OnActionExecuted when the controller is done executing
WeatherForecast:OnActionExecuted when the class is done executing

Debug POSTs using an Express App

Fri, 04 Dec 2020 00:00:00 GMT

Sometimes it's useful to have an endpoint that you can use to debug data that's being POSTed to an application

You can make use of the following express.js app to log your application's POST requests:

const express = require('express')
const app = express()

// parse json
app.use(express.json())

// GET endpoint to check uptime
app.get('/', (req, res) => {
  res.json({ data: 'hello' })
})

// POST endpointthat logs request body
app.post('/', (req, res) => {
  console.log(req.body)
  res.json(req.body)
})

// listen for requests
const listener = app.listen(process.env.PORT, () => {
  console.log('listening on port ' + listener.address().port)
})

Backup SQL Server Database as Script

Thu, 26 Nov 2020 00:00:00 GMT

To create a scripted backup of a database on SQL Server do the following:

Right click on the database name > Tasks > Generate Scripts...
On the Choose Objects screen select Scriot entire database and all database objects
On the Set Scripting Options screen click on Advanced and select:
1. Script DROP and CREATE = Script CREATE
2. Types of data to script = Schema and data
Set the location to save the script
On the Summary page you can click Next which will start the generation
Thereafter you can view the generated database script from the location it was generated to

Generate data for a Postman request

Tue, 24 Nov 2020 00:00:00 GMT

When making requests with Postman to test an API it is often useful to have generated data, for example when creating users in a backend system you may want the ability to get custom user data

The method we will use to do this is by setting Postman environment variables in the Pre-request Script of a Postman request

Firstly, we can write some basic logging in the Pre-request script section on Postman and view the result in the Postman console:

const name = 'Nabeel'

console.log(`Hello ${name}`)

Additionally, postman gives us some functions to set and get environment variables, so we can set the name as an environment variable like so:

const name = 'Nabeel'

pm.environment.set('name', name)

And log it like so:

console.log(pm.environment.get('name'))

Aditionally, we can make HTTP Requests using pm.sendRequest which then takes a callback for what we want to do after the request, we can use the randomuser api to get a user:

pm.sendRequest('https://randomuser.me/api/', (err, res) => {
  const apiResponse = res.json()

  // do more stuff
})

Using this method, we can get some random user data and set it in the environment variables as follows:

pm.sendRequest('https://randomuser.me/api/', (err, res) => {
  const apiResponse = res.json()

  const user = apiResponse.results[0]

  const email = user.email
  const firstName = user.name.first
  const lastName = user.name.last

  pm.environment.set('email', email)
  pm.environment.set('firstName', firstName)
  pm.environment.set('lastName', lastName)
})

We can then use this in our request body using Postman's {\{}\} syntax. If we were making a JSON request, our body would look something like this:

POST: https://my-api.com/users

{
  "email": "{{email}}",
  "firstName": "{{firstName}}",
  "lastName": "{{lastName}}"
}

Postman will then populate the slots from the environment variables that we set automatically when making the request

Capture Fetch with Cypress

Tue, 10 Nov 2020 00:00:00 GMT

To capture the result of a fetch request with Cypress you will need to make use of cy.route2

Setup

The cy.route2 command needs to be enabled in your cypress.json file before usage, to do so add the following:

cypress.json

{
  "experimentalNetworkStubbing": true
}

Usage

Next, using the command to capture requests to a route looks a bit like this in a test:

cy.route2('POST', '/do-stuff').as('data')

In the above you need to use the cy.route2 function. In the above we're capturing POST requests to the /do-stuff endpoint. The object which contains the request and response data is stored in the @data which can be retreived and worked with like so:

cy.wait('@data').then((data) => {
  // do stuff using the data object
  console.log(data)
})

The cy.wait function retreives the data object, we use the .then function with a callback function to do stuff with the fully resolved data object

Furthermore, the HTTP response from the data object can be parsed using JSON.parse, you can then also add assertions based on the response object. Adding this in, the above code would look more like this:

cy.wait('@data').then((data) => {
  console.log(data)

  // parse the response body
  const res = JSON.parse(data.response.body)

  // assertions on the response
  assert.isTrue(res.success)
  assert.isNotNull(res.message)
})

Scripting with FSharp

Tue, 13 Oct 2020 00:00:00 GMT

Before you can use the following, you will need the .NET Core SDK installed

Open the F# Interactive Console

To open an F# interactive console using the dotnet CLI. You can run the following command:

dotnet fsi

Note that to end statements/execute in the F# Interactive console use ;; at the end of a line or section of code

Once in the console, running #help;; from the fsi console to view the help menu, and #quit;; to quit the interactive session

Additionally, you can write multi-line F# code as well as just single line expressions. Each expression should be terminated with ;;. For example, you can write a function that will print some data into the console:

let printer s =
    printfn s
;;

Next, you can call the function with:

printer "Hello World";;

Which will execute the above code and output Hello World

Run an F# Script

F# scripts can be run using the dotnet fsi command as well, followed by the path to an F# script file:

dotnet fsi ./myscript.fsx

Package Management with Poetry

Wed, 07 Oct 2020 00:00:00 GMT

Poetry Docs

Python package management is typically quite a mess. Managing packages with pip often requires additional management of things like virtual environments and python version management

Poetry is a package manager that abstracts a lot of the typical Python dependency and environment management away from the user

Install Poetry

Before you can install poetry you need to have Python installed

To install poetry you can do one of the following depending on your OS:

Windows Powershell

(Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -UseBasicParsing).Content | python -

Bash

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

Initialize Poetry

Create a New Project

If you're starting a new project, you can run the following:

poetry new my-project

The above will generate a pyproject.toml file with the project settings. Alternatively you can use the following to add poetry to an existing project

Add to Existing Project

To add poetry to an existing project, run the following:

poetry init

Using Poetry

Add Dependency

To manage dependencies you can use the poetry add command. For example, if we would like to install flask

poetry add flask

Run Application

To run an application using the virtual environment created by poetry you can use the poetry run command, followed by the command you want to run:

poetry run python app.py

Running a flask app would look something like this:

poetry run flask run

Create a Shell

To create a shell in the poetry virtual environment run:

poetry shell

The above will open a poetry shell with the virtual environment. You can then do something like run the python command to open a python shell in the environment

Trust .NET Core Dev Certificates

Wed, 07 Oct 2020 00:00:00 GMT

When developing .NET Core web applications with HTTPS, to simplifiy your experience and avoid browser warnings you can trust the .NET Core dev certs on your local device with the followning commands:

dotnet dev-certs https --clean
dotnet dev-certs https --trust

Highlight Specific Elements

Tue, 29 Sep 2020 00:00:00 GMT

You can highlight all HTML elements which respond to a specific CSS selector with the following. This can be useful for debugging purposes:

document
  .querySelectorAll(selector)
  .forEach((el) => (el.style.border = 'solid 2px red'))

For example, the following [id] selector will find all elements with an id attribute:

document
  .querySelectorAll('[id]')
  .forEach((el) => (el.style.border = 'solid 2px red'))

URL Text Fragments

Tue, 29 Sep 2020 00:00:00 GMT

When linking to a webpage it can be useful to link to a specific part of a page. Using a hash in the URl you can link to the id attribute in the HTML of a page

For example, https://my.website.com#overview would link to the element with id="overview" in the HTML

However, it may be useful for us to link to a general piece of text on a page, we can use text fragments to do so. These work similar to the way a hash does, but we can link to any text on a page even if it does not have an id. Additionally, the user's browser will highlight the text set to draw the user's attention to it

Note that at the moment only Chromium based browsers support this

To do this, we need to identifiy the text on the page we want to link to. Say we'd like to link to the first paragraph of this page, we can use the text directive (#:~:text=) param in the URL to link to specific text instead of just the # like we use for an ID

We would need to take the part of the text we want to focus, URL encode it, then add it to the text directive the final URL may look something like this:

https://my.website.com/#:~:text=When%20linking%20to%20a%20webpage

If you need to highlight a larger piece of text, you can use the start and end of your text segment, separated by a comma, for example selecting the entire first paragraph of this post:

https://my.website.com/#:~:text=when%20linking,HTML%20of%20a%20page

The link below should link to the first paragraph of this page if opened in a new tab (provided your browser supports text fragments):

ctrl + click me

You can read more about text fragments in URLs on web.dev

Node.js Dirname vs Resolve

Fri, 25 Sep 2020 00:00:00 GMT

Node.js has a few methods by which we can get the directory in which we are currently executing, and get paths relative to it

Get the Current Directory

Process Directory

We can get the working directory from where we started the node script with:

const processDir = process.cwd()

File/Module Directory

And we can get the directory in which the currently executing file is in with:

const fileDir = __dirname

Get Path to a Target Location

To get an absolute path to a specific target file/directory we have a few methods

Joining Paths

We can use the path module's join method to get a path given any path pieces you can go up or down a directory using the ../ notation

const { join } = require('path')

const path1 = join(basePath, './downdir/myfile.txt')
const path2 = join(basePath, '../updir')

Absolute Path from Process Directory

To get an absolute path from the process directory, you can use the path module's resolve function:

const { resolve } = require('path')

const absPath = resolve('./downdir/myfile.txt')

Like the join method you can also use the ../ notation to move up a directory

Using resolve is basically shorthand for using join with process.cwd()

const absPath = join(process.cwd(), './downdir/myfile.txt')

Absolute Path from File/Module Directory

If it makes more sense to get the path relative to the executing file, you can use a combination of __dirname and join like so:

const { join } = require('path')

const absPath = join(__dirname, './downdir/myfile.txt')

Split an Array into Segments

Wed, 23 Sep 2020 00:00:00 GMT

It can sometimes be useful to split an array into a separated set of rows or columns, such as when trying to separate elements into buckets while maintaing order.

For example, given the following input:

const input = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

With the desired output array as follows, in which we split elements into different rows, while maintaing a top-down/left-right ordering:

const output = [
  [1, 4, 7, 10],
  [2, 5, 8],
  [3, 6, 9],
]

We can accomplish this using a function like so:

const transform = (arr, segments) =>
  [...new Array(segments)].map((_, outerPos) =>
    input.filter((el, innerPos) => innerPos % segments === outerPos)
  )

Which can then be used with:

const rows = 3
const transformedData = transform(input, rows)

Mongo DB in Docker

Tue, 22 Sep 2020 00:00:00 GMT

Considerations

To run MongDB in a Docker Container there are a few things to take note of:

You may need to configure auth
Volume storage to be set up

From Terminal

To run a Docker container using a single command in the terminal you can run the following command:

docker run -d -p 27017:27017

You can also specify additional information like the volumes you would like to use using flags when running:

docker run -d -p 27017:27017

From Compose

More info can be found here

You can also run a MongoDB Container with Compose which may be a bit easier:

docker-compose.yml

version: '3.3'

services:
  # mongonode0:27017
  mongo0:
    image: mongo
    hostname: mongo0
    container_name: mongo0
    restart: always
    ports:
      - '37000:27017'
    volumes:
      - ./mongo0-data:/data/db
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: password

Web APIs with AdonisJS and PostgreSQL

Sun, 06 Sep 2020 00:00:00 GMT

In this post, we'll take a look at building an API using the AdonisJS framework. We'll specifically be looking at the AdonisJS v5 Release Preview and how we can create a simple REST API that handles some CRUD operations on a Database

Because we're going to be creating a REST API, you should have some experience making HTTP Requests, for our purposes we can use a website called Postwoman to interact with our API, but you can also use any other tool you prefer

About AdonisJS

AdonisJS is an opinionated, Typescript-first web framework that provides a lot more "out of the box" functionality than the traditional framework or library in the Node.js ecosystem and is more comparable to something like .NET's WebAPI or MVC Framework than things like Express or Next in Node.js

Some of the built-in features and decisions that stand out for me are:

First-class Typescript support
Class-based controllers
Class-Based SQL ORM
Dependency Injection
Request Body Validation
Authentication
Server-side View Rendering

There are a lot more features, and it would be impractical for me to talk about all of them in a single post. Overall, the framework seems to be very complete at this point

Prerequisites

We'll be using a Node.js and a SQL Database for this post, so if you're going to follow along with this post you will need to have a couple of things installed:

Node.js and NPM
Any Supported SQL Database:
- MySQL
- SQLite
- Microsoft SQL Server
- PostgreSQL
- MariaDB
- OracleDB

Alternatively, you can run a Docker image for the database which may be easier, I'll be using Postgres via Docker with VSCode

To learn more about VSCode Dev Containers you can look at my previous post on how to use Dev Containers

Initialize an Application

Now that we've got all our necessary dependencies installed, we can initialize an application using the npm init command:

During the initialization the CLI will ask you what type of project to initialize, be sure to select API Server

npm init adonis-ts-app my-api

The above command is made up of the following parts:

npm init which is the npm command that will run an initialization script from the provided package
adonis-ts-app which is the package to be used for initialization
my-api is the name of the folder/project we want to create

Once the project has been configured, navigate into the project directory:

cd my-api

And start the app:

npm start

The Ace CLI

AdonisJS makes use of a command-line application called ace, ace can be used to do common tasks as well as custom tasks that we define. By default, it can scaffold controllers, commands, and a bunch of other things as well as run and build an AdonisJS application

Environment Variables

AdonisJS Makes use of Environment Variables do set the application configuration, in your generated files you should see a file named .env, this file contains the environment variables used by the application. Looking at this file will give us an idea of our current configuration:

.env

PORT=3333
HOST=0.0.0.0
NODE_ENV=development
APP_KEY=mE8zN8V_7PzazKfv9_ds-8CGjVRLA2wo

At the moment, we can see that our application will be listening on host 0.0.0.0 and port 3333, this means that we can access our application from our browser at http://localhost:3333

If we open this page on our browser we will see the adonis logs kicked off in our command line, something like this:

ℹ  info      cleaning up build directory build
ℹ  info      copy .env,ace build
☐  pending   compiling typescript source files
✔  success   built successfully
ℹ  info      copy .adonisrc.json build
…  watch     watching file system for changes
ℹ  info      starting http server
✔  create    ace-manifest.json
[1595755199986] INFO  (my-app/1531 on 47f057d8722c): started server on 0.0.0.0:3333

The first request may take some time, this is because the server is still starting itself up. In the meantime, however, let's discuss how we'll be accessing our API

Making Requests

You will need to use Postwoman or something similar when making requests

Now that our application is running, we can start making requests to our API. By default, AdonisJS sets up a hello-world route at the base of our application (the / route`)

We can reach this endpoint by simply making a GET request to http://localhost:3333 which will return the following:

{
  "hello": "world"
}

AdonisJS can define routes using a method similar to libraries like Express.js, the "Hello World" route is defined in the start/routes.ts file and has the following:

routes.ts

import Route from '@ioc:Adonis/Core/Route'

Route.get('/', async () => {
  return { hello: 'world' }
})

The Route.get portion states that this is a GET route on the / path with an async handler function that returns an object

However, for this post, we won't be defining our route's handler functions like this. We're going to make use of controllers

Controllers

AdonisJS uses controllers to structure our API. This makes use of a class which contains functions intended to work as an interface between the HTTP request and the work we want to do via a provider

We will simply state which controller and method we want to use in our routes.ts file instead of containing all the handler logic in that file

To generate a controller we can use ace. We will use ace to create a UsersContoller.

First, stop your running application with ctrl + c in the terminal, then use the following ace command to generate a User controller:

node ace make:controller User

This will generate an app/Controllers/Http/UsersController.ts file with the following contents:

UsersController.ts

export default class UsersController {}

Which exports a class called UsersController with no default functionality

We can create any functions inside of here that we want to, and then map these functions to routes. For now, let's add a get function that will put some placeholder data into. We'll update this later to use our database

A few things to note about the function we're going to create:

The function is asynchronous, so if there are any long-running operations it won't block other things from running
The name of the function is get and it has no parameters
The function returns an array of objects that represents a User

UsersController.ts

export default class UsersController {
  public async get() {
    return [
      {
        id: 1,
        name: 'Bob Smith',
        email: 'bob@smithmail.com',
      },
    ]
  }
}

Now that we have defined our function, we can add a route that will cause this function to be called. We do this by adding the following in the routes.ts file:

Route.get('users', 'UsersController.get')

We can then run npm start from the command line and make a GET request to http://localhost:3333/users which should return our user:

[
  {
    "id": 1,
    "name": "Bob Smith",
    "email": "bob@smithmail.com"
  }
]

Database

Now that we've got some basic understanding of how AdonisJS maps routes to functionality, we can connect our application to a Database

To add database functionality, we first want to install the adonisjs/lucid package to our application. Behind the scenes, AdonisJS makes use of Lucid for connecting to and working with databases

npm i @adonisjs/lucid@alpha

And then run the following command to initialize it:

node ace invoke @adonisjs/lucid

Which should give the following output:

   create    config/database.ts
   update    .env
   update    tsconfig.json { types += @adonisjs/lucid }
   update    .adonisrc.json { commands += @adonisjs/lucid/build/commands }
   update    .adonisrc.json { providers += @adonisjs/lucid }
✔  create    ace-manifest.json

If you open your .env file you will see the configuration for a sqlite database

.env

## ... other config
DB_CONNECTION=sqlite
DB_HOST=127.0.0.1
DB_USER=lucid
DB_PASSWORD=lucid
DB_NAME=lucid

You can follow the general database setup information on the AdonisJS Docs to set yours up, but I'll be using Postgres as I've mentioned before

To use Postgress, you need to install the pg package from npm:

npm i pg

Then configure your application to use Postgres. We will first update our .env file to have our database credentials:

.env

DB_CONNECTION=pg
DB_HOST=db
DB_USER=user
DB_PASSWORD=pass
DB_NAME=data
DB_PORT=5432

Once you've configured your database connection information in the .env file, our database connection information will be taken care of by AdonisJS

These environment variables are used in the config/database.ts file, we can see them in the pg part of the file

database.ts

  ...
    pg: {
      client: 'pg',
      connection: {
        host: Env.get('DB_HOST', '127.0.0.1') as string,
        port: Number(Env.get('DB_PORT', 5432)),
        user: Env.get('DB_USER', 'lucid') as string,
        password: Env.get('DB_PASSWORD', 'lucid') as string,
        database: Env.get('DB_NAME', 'lucid') as string,
      },
      healthCheck: true,
    },
  ...

In the database.ts file above, look for the section for your relevant database, and set the healthCheck property to true

Now that we've set things up, we can create a health-check route that will enable us to view the current status of our database connection. We will add a handler for this in the routes.ts file:

import HealthCheck from '@ioc:Adonis/Core/HealthCheck'
// ... existing file content

Route.get('health', async ({ response }) => {
  const report = await HealthCheck.getReport()
  return report.healthy ? response.ok(report) : response.badRequest(report)
})

Once you've done all the above, start the development server again with npm start

Then make a GET request to http://localhost:3333/health to view your health-check information

{
  "healthy": true,
  "report": {
    "env": {
      "displayName": "Node Env Check",
      "health": {
        "healthy": true
      }
    },
    "appKey": {
      "displayName": "App Key Check",
      "health": {
        "healthy": true
      }
    },
    "lucid": {
      "displayName": "Database",
      "health": {
        "healthy": true,
        "message": "All connections are healthy"
      },
      "meta": [
        {
          "connection": "pg",
          "message": "Connection is healthy",
          "error": null
        }
      ]
    }
  }
}

In the lucid section we will see if our database connection is working or any applicable error information.

Defining Models

Once we've configured our database, we will want to interact with the data in it. Lucid makes use of models essentially as proxies for database tables. We can generate a model for our User with ace as follows:

Stop your development server with ctrl + c before running this:

node ace make:model User

The above script will have generated a User model class in the app/Models/User.ts file that extends BaseModel. All models must do this. The generated User.ts file looks like this:

User.ts

import { DateTime } from 'luxon'
import { BaseModel, column } from '@ioc:Adonis/Lucid/Orm'

export default class User extends BaseModel {
  @column({ isPrimary: true })
  public id: number

  @column.dateTime({ autoCreate: true })
  public createdAt: DateTime

  @column.dateTime({ autoCreate: true, autoUpdate: true })
  public updatedAt: DateTime
}

When viewing the above file your code editor may give you a warning saying that decorators are not supported, if you see this then set the experimentalDecorators property to true in your tsconfig.json file:

...
  "compilerOptions": {
    "experimentalDecorators": true,
...

Note the @column decorators, these are used to map columns in our database to our model fields. We'll add a field for our user's name and email as follows:

User.ts

export default class User extends BaseModel {
  // ... other stuff in class
  @column()
  public name: String

  @column()
  public email: String
}

Any properties or methods defined in a class without the @column decorator will not be mapped to the database, we can just use these as normal functions in the class and implement utilities from them

Migrating the Database

At this point, our database does not contain a user table, which will be used to store our data for the User model. We need to create a Database Migration which will add the required table and fields.

ace provides us with a method to scaffold a migration file. To create this file we will run the following command:

node ace make:migration users

Which will have created a database/migrations/SOME_ID_users.ts file with the following:

*_users.ts

import BaseSchema from '@ioc:Adonis/Lucid/Schema'

export default class Users extends BaseSchema {
  protected tableName = 'users'

  public async up() {
    this.schema.createTable(this.tableName, (table) => {
      table.increments('id')
      table.timestamps(true)
    })
  }

  public async down() {
    this.schema.dropTable(this.tableName)
  }
}

The generated file (above) contains an up function which will create a users table with an id as well as createdAt and updatedAt fields. We will need to modify the up function to add our new fields as well:

export default class Users extends BaseSchema {
  ...

  public async up () {
    this.schema.createTable(this.tableName, (table) => {
      table.increments('id')
      table.timestamps(true)

      // our added fields
      table.string('name').notNullable()
      table.string('email').unique().notNullable()
    })
  }

  ...

Once we've defined our migration script we need to build the application with:

node ace build

And then we can run the migration using ace as follows:

node ace migration:run

If there is an error in a migration script, we can rollback the migration with node ace migration:rollback which will run the down function in your migration

Interact with the Database

Now that we've got our database, we can interact with it using the User model we defined earlier

The first change we'll make is to modify the get function to return all users. To do this we need to import App/Models/User and use the User.all method:

import User from 'App/Models/User'

export default class UsersController {
  public async get() {
    // get all users
    return await User.all()
  }
}

We use await because the User.all function is asynchronous

Next, we'll add a function to create a User. We will call it post. To make this function work we need to do a couple of things:

Import HttpContextContract from @ioc:Adonis/Core/HttpContext
Retrieve the request from the HttpContextContract
Create a User
Return the created user

import { HttpContextContract } from '@ioc:Adonis/Core/HttpContext'
// ... other imports

export default class UsersController {
  // ... get user code above
  public async post({ request }: HttpContextContract) {
    // get the user from the request body
    const newUser = request.all() as Partial<User>

    // create a user using the object we received
    const user = await User.create(newUser)

    // return the created user object
    return user
  }
}

The request.all function combines the data from the request body and query string into a single object

When we've added that, our completed UsersController.ts file should look like this:

UsersController.ts

import { HttpContextContract } from '@ioc:Adonis/Core/HttpContext'
import User from 'App/Models/User'

export default class UsersController {
  public async get() {
    return await User.all()
  }

  public async post({ request }: HttpContextContract) {
    const newUser = request.all() as Partial<User>
    const user = await User.create(newUser)
    return user
  }
}

Now that we've added a new function to our controller, we need to expose a route to it in the routes.ts file:

routes.ts

...
Route.post('users', 'UsersController.post')
...

Consume the API

Now that we've created API endpoints for listing all users and creating a user we can restart our development server with npm start and consume our API from wherever we want to make some HTTP requests

Create a User

To create a User we need to make aPOSTrequest tohttp://localhost:3333/usersand a Content-Type ofapplication/jsonwith the following as thebody` for our request:

{
  "name": "Bob Smith",
  "email": "bob@smithmail.com"
}

Which should return a created user like:

{
  "name": "Bob Smith",
  "email": "bob@smithmail.com",
  "created_at": "2020-07-26T14:35:25.987-00:00",
  "updated_at": "2020-07-26T14:35:25.988-00:00",
  "id": 4
}

We can also try to create a User with the same information but we will see that this fails due to the database constraints we added in our migration script

Next, we can get a list of all Users by making a GET request to http://localhost:333/users which should give us back our created users:

[
  {
    "id": 4,
    "created_at": "2020-07-26T14:35:25.987-00:00",
    "updated_at": "2020-07-26T14:35:25.988-00:00",
    "name": "Bob Smith",
    "email": "bob@smithmail.com"
  }
]

Summary

My overall impression of AdonisJS is pretty good. The framework feels very stable and I had much fewer issues in the process of learning it and writing this post than I have had using other more popular frameworks.

AdonisJS is very well documented with code samples for most traditional tasks and is backed by some solid libraries for things like database integration

Working with SQL using the framework has been fairly straightforward, and the ability to write database migrations using functionality provided by the framework instead of just tossing some wild SQL together makes it more approachable

Personally, however, I prefer no-SQL databases like MongoDB and tend to use them more often when using JavaScript or TypeScript, but I feel like if the need arises for a SQL database then AdonisJS is a really good option, especially if you're a JavaScript developer and don't want to have to learn Java or C# for this type of functionality

There is also a lot more functionality than what I've gone through in this post, so I'd recommend browsing the AdonisJS docs to get a broader sense of what the framework entails

If you feel like playing around with the code I've gone through in this post without having to write it all then I've got it all on GitHub and it should just be plug-and-play using Visual Studio Code Remote Containers. If you want to learn more about that, then you can check out my previous blog post on developing within a container

Automate Anything with GitHub Actions

Tue, 11 Aug 2020 00:00:00 GMT

You can take a look at the Action we build in this post here on GitHub. I've also included an action that's responsible for building and updating the action dist which may also be of interest. It's all in this Repository

A few weeks ago I was playing around with GitHub actions and the recently introduced GitHub Account README functionality and wanted a way to make this "static" file a bit more dynamic

Enter GitHub Actions. GitHub actions are a way for you to define and run programmatic tasks. If you can put it in code, then you can run it as an action

Usually, you'll be using actions that have been defined by GitHub or another developer for common tasks, such as running your application CI or deployment processes. However, since actions are extremely generic, we're going to do something a little different - update our Twitter bio?

In this post, we're going create a GitHub action that uses the Twitter API to update our bio, as well as make the action we created run automatically on GitHub

Create a GitHub Repository for us to work in
Set up Twitter Developer Credentials
Write a Node.js script that uses the Twitter API
Configure our GitHub Action Metadata
Add our Twitter Secrets in GitHub
Configure a GitHub Action that will run our script (an action, within an action)

Prerequisites

Git
Node.js
Visual Studio Code or any other Code Editor

Create a GitHub Repo

For us to run our action we'll need a GitHub Repository to use, to create a Repo go to GitHub and sign in, thereafter go the 'Create a new repository page' and fill in the details, be sure to select Initialize this repository with a README, pick Node as the .gitignore file, and select a license if you'd like to

Once you've done that, click Create repository and you should see the initial files we added to the Repo. Next, click on the Code button and copy the URL in the text box

Now, from a terminal, you will need to clone the repository. Run the following command and make sure to paste the link you just copied in place of <YOUR URL> below:

git clone <YOUR URL>

Next, open the folder you just cloned in your code editor and create a new file called .env in the folder root directory. We'll keep our Twitter credentials in this file for testing. For now, just add the following to the file:

.env

TWITTER_CONSUMER_KEY=
TWITTER_CONSUMER_SECRET=
TWITTER_ACCESS_KEY=
TWITTER_ACCESS_SECRET=

In the next step, we're going to get the credentials from Twitter, we'll add them into the file above when we're done

Get Some Twitter Cred.

Twitter exposes the Twitter Developer API that allows us to do all kinds of useful and pointless things by interacting with Twitter's data

For us to consume the Twitter API we require credentials for the API. To set these up we'll need to do a few things

First, open your browser on the Twitter Developer Portal and click the sign-in button. Then, once you've signed in, you should see your name on the top right of the page, click on the dropdown arrow and select Apps to go to the App Dashboard

On the App Dashboard, click Create an app and fill in the required information. For the Website URL you can put the URL of your GitHub repository that you copied previously into this field. Once you've filled in all the details click Create at the bottom of the page

You should now see the Details Page for the app we just created. Next, click on the Keys and Tokens tab and click Generate then copy and paste each token after the = in the .env file we created without any spaces before or after the key

When you've pasted your tokens into their places, your .env file should look something like this:

Note that using the template for .gitignore that we chose, the .env file will be automatically ignored. Don't publically upload this data just anywhere as it can potentially give someone access to your Twitter account

.env

TWITTER_CONSUMER_KEY=xxxxxxxxxxxxxxxxxxxxxx
TWITTER_CONSUMER_SECRET=xxxxxxxxxxxxxxxxxxx
TWITTER_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxx
TWITTER_ACCESS_SECRET=xxxxxxxxxxxxxxxxxxxxx

(With the actual keys, and not just xxxx)

Now that we've got our credentials set up, we can start to work on the application

Using the Twitter API

We're going to be writing a script that runs on Node.js (JavaScript) and makes use of the Twitter API using the twit library for Node.js and the GitHub libraries for working with GitHub actions

To get started, run the following command from a terminal within your repository's directory to initialize a new Node.js project and select the defaults for all the questions:

npm init -y

Next, we'll add the dependencies that our application is going to need to run:

@actions/core allows us to interact with the data provided to our action by GitHub
twit is used to interact with the Twitter API
dotenv enables us to load in our .env file so our application can use it

npm install @actions/core twit dotenv

Once the application is done installing we'll create an index.js file inside of our repo folder and we can get started on our code

Inside of the index.js the first thing we'll want to do is import our environment variables from the .env file we configured, we'll do that using the dotenv NPM Package we installed previously:

index.js

require('dotenv').config()

Next, we'll configure a new Twitter API Client using our environment variables. The twitter Package exports a Twitter class that we can create an instance of, do to this we will use the Twitter constructor and get our environment variables that are all stored in the process.env variable:

index.js

const Twitter = require('twitter')

const client = new Twitter({
  consumer_key: process.env.TWITTER_CONSUMER_KEY,
  consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
  access_token_key: process.env.TWITTER_ACCESS_KEY,
  access_token_secret: process.env.TWITTER_ACCESS_SECRET,
})

Now that we've got an instance of a Twitter client that we can use to interact with the API we'll want to use it

The Twitter API Client runs asynchronously, for us to work with it correctly we have two options:

Using the callback method, which means that we give the client a function to run when it's done sending the data to Twitter. While this is easier in our specific circumstance it can become difficult to keep track of when we have many different callback functions
The second option is to use async and await syntax, which allows us to write our code more sequentially and makes it easier for us to control our sequence and flow

To use the async/await functionality, we need the code that we need to await to be inside of an async function. In this one function, we can await as many tasks as we want sequentially without worrying about callbacks

We'll create a main function that is async in which we will do any asynchronous interactions, in our case - interact with Twitter

After the code we've already got in our index.js file, define the main function as follows:

index.js

const main = async () => {
  // this is where we'll work with the twitter client
}

Next, we'll add the code to interact with the Twitter API inside of this function. To update our Profile Bio (or description as it's called in the Twitter API) we'll make use of the account/update_profile endpoint

The data we send the Twitter API will need to be an object with a description property for what we want to set our Twitter Bio as. We'll await this function call by adding it within our main function:

index.js

const main = async () => {
  const response = await client.post('account/update_profile', {
    description: 'Hello, World!',
  })

  console.log(response)
}

The above code will set our Twitter description to Hello, World! and then print our the response we get back from Twitter

The way we've written the above code is a little bit dangerous as we aren't handling any potential errors/failures that may happen when interacting with the Twitter API

When working with an API via an HTTP Request there's always the chance that there could be an error. Errors can be caused by anything ranging from poor network connections, incorrect credentials, or a system outage on the API host itself

For our application to give us a bit more information about what happened, we may want to handle the error before passing it on to the process that kicked off our script. We'll make use of a try/catch to handle this error:

index.js

const main = async () => {
  try {
    const response = await client.post('account/update_profile', {
      description: 'Hello, World!',
    })

    console.log(response)
  } catch (error) {
    console.error(error)
    throw new Error(
      `The Twitter API Responded with an Error: ${error[0].code}, ${error[0].message}`
    )
  }
}

You can see above that we're just doing some cleanup of the error message before throwing the exception to the process

Now that we've fully defined our client and main functions we can call the function as the last line of the script:

index.js

main()

This should make a request to the Twitter API and print our the response or error if there is one

So, the above function will always set our Twitter Bio to the same value, this isn't interesting. Next, we'd like to get some data from the workflow that's going to be running our action. The way we would do this is with an input

Our action is going to take an input called bio. We can use the @actions/core library to get the value of the input. We can do this with the following:

index.js

const core = require('@actions/core')

const bio = core.getInput('bio') || 'Hello, World!'

The above code also sets a default value of Hello, World! to the description, this will allow us to also run the action without throwing an exception on our local machine as well as if the bio is not provided to the action

Furthermore, instead of just throwing errors, we can rather set the error-status using the core.setFailed function. Updating our main function to do this we now have:

index.js

const main = async () => {
  try {
    const response = await client.post('account/update_profile', {
      description: bio,
    })

    console.log(response)
  } catch (error) {
    console.error(error)
    core.setFailed(
      `The Twitter API Responded with an Error: ${error[0].code}, ${error[0].message}`
    )
  }
}

Finally, our full index.js file should look like this:

index.js

re('dotenv').config()

const Twitter = require('twitter')
const core = require('@actions/core')

const bio = core.getInput('bio') || 'Hello, World!'

const client = new Twitter({
  consumer_key: process.env.TWITTER_CONSUMER_KEY,
  consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
  access_token_key: process.env.TWITTER_ACCESS_KEY,
  access_token_secret: process.env.TWITTER_ACCESS_SECRET,
})

const main = async () => {
  try {
    const response = await client.post('account/update_profile', {
      description: bio,
    })

    console.log(response)
  } catch (error) {
    console.error(error)
    core.setFailed(
      `The Twitter API Responded with an Error: ${error[0].code}, ${error[0].message}`
    )
  }
}

main()

Now that we've written the functionality for our action we'll want to turn it into an action

Configure the Action Metadata

For GitHub to recognise our code as an action, we need to create an action.yml file that contains a description of our action. Our action.yml file needs to have the following information:

name for our action
description of the action itself
an input parameter of bio that will be used by our application
runs which states the script to run

The action.yml file for our action looks like the following:

action.yml

name: 'Twitter Bio Update'
description: 'Update your Twitter Account Bio'
inputs:
  bio:
    description: 'Text that you would like to set as your Twitter Bio'
    required: true
    default: 'Hello, World!'
runs:
  using: 'node12'
  main: 'index.js'

The fields we've got above are all pretty much required. The above fields are the simplest configuration for a Node.js action. The GitHub Docs have a lot more information on more complex configurations and actions

Now that we've defined our action, we will want to configure it to run. But before we can do that, we'll want to set up our secrets

Setting Up Secrets in GitHub

Now that we've got our action defined, we're almost ready to write a Workflow that will use this action. However, our action requires environment variables (that we've got saved in our .env file) but we don't want these to be pushed to GitHub as part of our source code. The way to set up these environment variables in GitHub is called a secret

To add our environment variables in GitHub you'll need to open your Repo on GitHub and navigate to Settings > Secrets then click New secret and add your first secret. If we use our .env file as a reference we'll want to create a secret for each line in the file. To do this look at the name of the environment variable (everything before the =) and set this as the Name for the secret, then look at the value (everything after the =) and set this as the Value for the secret then click Add secret. Do this for every line in your .env file (every environment variable`

Create a Workflow

Workflows are GitHub's way of tying together a bunch of actions to run. Often, we will want to run multiple actions. We place these into what's called a step in a job. Each Workflow can have multiple steps and jobs and a repository can have multiple workflows

The structure of a workflow is something like this:

workflow
|-- Job 1
|   |-- Step 1
|   |-- Step 2
|-- Job 2
    |-- Step 1

A Workflow can have any number of jobs and steps

We're going to create a Workflow with a single job with the goal of running our script. For us to do this, the job will need to do the following:

Checkout our code
Configure Node.js and NPM
Install Dependencies
Run our Action

To do Steps 1 and 2 we're going to use actions that are defined by GitHub. We're going to go through our Workflow file working from the top-down as this is the order in which everything will be run

Firstly, we'll need to create a new directory in our Repository named .github and inside of this another called workflows. Inside of the .github/workflows directory create a file named main.yml (this can be any name so long as it's a yml file)

Next, on the first line of this file we will define a name for our workflow like so:

main.yml

name: Run Twitter Bio Action

This is the name that will be displayed when the workflow runs. Actions are run when a specific GitHub event happens (more about that in the Docs). We're going to configure our action to run manually only, so we will use the workflow_dispatch event. After the name in the main.yml file, add the following:

main.yml

on:
  workflow_dispatch:
    inputs:
      bio:
        description: 'Twitter Bio'
        required: true

In the above, we set an on event for workflow_dispatch with an input for bio that we're going to pass on to our action

Now that we've got the workflow metadata defined, we're going to add a job. To do this, we'll add a jobs object with a name of update-bio, a specification on what OS it needs to run on, and the steps that will be a part of it:

main.yml

jobs:
  update-bio:
    runs-on: ubuntu-latest
    name: Update Twitter Bio
    steps:
      # we'll add the steps here in a moment

We can see above that we're running on the ubuntu-latest OS. Next, we'll add the first step for checking out our code in the action. This will use actions/checkout:

main.yml

steps:
  - name: Checkout
    uses: actions/checkout@v2

In the above, we give our step a name which is for display purposes, and a uses which says what action this step should use. In our case, we're checking out our code using the actions/checkout@v2 action

Next, we'll want to configure Node.js and NPM because our action needs these to run. We can use the actions/setup-node@v1 for this:

main.yml

- name: Setup Node.js
  uses: actions/setup-node@v1
  with:
    node-version: 12.x

In this action, we use the with to state an input that the action needs. In this case, we specify that we want to use a node-version of 12.x

Now we've got Node.js and NPM, we need to install the dependencies for our action to run. We do this with the npm install command like so:

main.yml

- name: Install Dependencies
  run: npm install

And lastly, we will configure our action to run with the following:

main.yml

- name: Run Action
  uses: ./
  env:
    TWITTER_CONSUMER_KEY: ${{ secrets.TWITTER_CONSUMER_KEY }}
    TWITTER_CONSUMER_SECRET: ${{ secrets.TWITTER_CONSUMER_SECRET }}
    TWITTER_ACCESS_KEY: ${{ secrets.TWITTER_ACCESS_KEY }}
    TWITTER_ACCESS_SECRET: ${{ secrets.TWITTER_ACCESS_SECRET }}
  with:
    bio: ${{ github.event.inputs.bio }}

In this last step we're doing quite a few things:

We're setting a name for our action
We set a uses to be ./ which means we want to run the action at the root of our Repo (our action.yml)
We set the env, these are the different environment variables we want to pass to the application. We use ${\{ ... }\} to mean that it's a variable, and we use secrets.variable to access the variable from the secrets we configured in the repository
We use the with to set the bio from the input we give to the workflow on our workflow_dispatch event. GitHub exposes this in the github.event.inputs.bio variable

With the Run Action step added, we've got a full workflow. The overall main.yml file should have the following:

main.yml

name: Run Twitter Bio Action
on:
  workflow_dispatch:
    inputs:
      bio:
        description: 'Twitter Bio'
        required: true

jobs:
  update-bio:
    runs-on: ubuntu-latest
    name: Update Twitter Bio
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Node.js
        uses: actions/setup-node@v1
        with:
          node-version: 12.x
      - name: Install Dependencies
        run: npm install
      - name: Run Action
        uses: ./
        env:
          TWITTER_CONSUMER_KEY: ${{ secrets.TWITTER_CONSUMER_KEY }}
          TWITTER_CONSUMER_SECRET: ${{ secrets.TWITTER_CONSUMER_SECRET }}
          TWITTER_ACCESS_KEY: ${{ secrets.TWITTER_ACCESS_KEY }}
          TWITTER_ACCESS_SECRET: ${{ secrets.TWITTER_ACCESS_SECRET }}
        with:
          bio: ${{ github.event.inputs.bio }}

Run the Action

To run the action, we first need to get everything to GitHub. From Repo directory, in the terminal, run the following commands:

git add .
git commit -m "I made a GitHub Action!"
git push

Then, go to your repository on GitHub and click on the Actions tab. You should then see your Workflow listed. Click on your workflow name, and then the Run workflow dropdown. Fill in your Twitter Bio, wait for the workflow to complete and look at your Twitter profile!

From GitHub, you are also able to inspect and view any logs or errors from a Workflow run. If the workflow fails you can also take a look at the output that was thrown by the step that resulted in the failure

Summary

And that's about it. We've taken a look at how you can use GitHub actions to automate a pretty silly task, but there's a lot more to using GitHub Actions, and the sky's the limit in terms of what you can use them for

Overall, GitHub actions aren't too different from similar options like Azure Pipelines, Jenkins, or any other CI service. What makes them so useful is how easily they hook into the source control system and how well they work and are supported within the GitHub ecosystem

That all being said, they're not the easiest things to configure and it can take a while to get them right if you're doing something especially complex. Overall they're pretty cool and are a pretty good place to get started with automating tasks and working with things like continuous integration and deployments and I'd definitely recommend giving them a shot

Code in a Container

Sat, 25 Jul 2020 00:00:00 GMT

Recently I'd started using Visual Studio Code's Remote Containers functionality for development and it's been really useful

The Remote Containers extension allows us to write code and develop applications within a virtualized environment that makes it easier for us to manage our development environment as well as more closely resemble our target deployment environment (if we're deploying to Docker or Kubernetes)

In this post, I'll take a look at what a Docker container is, why we would want to use one as a development environment, and how we can go about setting one up for VSCode

Prerequisites

If you intend to follow along with this post you'll need to have the following installed:

A Windows or Mac OS version capable of running Docker Desktop
Docker Desktop
Visual Studio Code
Visual Studio Code's Remote Containers Extension
Visual Studio Code's Docker Extension
Some familiarity with using the command line

Docker Containers

A Container, in this context, is a simple virtual machine that contains the code required to run an application with all its dependencies

A Docker container is built from a docker image and run by the docker command. I'll explain these as we go along

To check that Docker is installed correctly on your machine run the following command:

docker run hello-world

If your install is working correctly you should see something like this:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
0e03bdcc26d7: Pull complete

Digest: sha256:49a1c8800c94df04e9658809b006fd8a686cab8028d33cfba2cc049724254202
Status: Downloaded newer image for hello-world:latest

Hello from Docker

...

Docker Images

Docker images are typically used to run applications in a production-type environment, every Docker container we run needs to be based on an image, every running container is like an instance of an image - similar to how objects are an instance of a class

An image states what our container will need to be made of, what it depends on, and how it runs. We define how docker should build our image in a Dockerfile

We're going to go through some of the basics of Docker Images and Docker as would typically be done when creating a container to be run in production before we get into development containers so you've got an understanding of how this all works

To get started create a new folder and open it from Visual Studio Code and do the following:

Create an Application

We'll need a simple "hello-world" web server using Node.js, for the sake of example. You can, however, use any language (or Languages) you want when creating an application to run within Docker. You do not need to have any dependencies for the specific application or language installed on your computer, we will handle this using Docker

For our purpose, create a file called index.js with the following:

const http = require('http')

const requestListener = function (req, res) {
  res.writeHead(200)
  res.end('Hello, World!')
}

const serverListeningCallback = function () {
  console.log('Server started')
}

const server = http.createServer(requestListener)
server.listen(8080, serverListeningCallback)

You can see in the above on the last line that the application will listen on port 8080, just keep this in mind

We don't need to run this file as yet, but if we want, we can run this with the following command from our working directory:

node app.js

At this point our working directory should look like this:

working-directory
|__ index.js

Create a Dockerfile

There are a few steps that are the same for most Dockerfiles you'll be building:

A Base Image that your container/image should use, in our case node:12, which has node and npm preinstalled
Copy all the code in the current (.) directory
Define your runtime port/ports (in the case of a web application)
The command that will be run to start the application

Any line starting with a # is a comment, Docker will ignore these

Dockerfile

## step 1 - FROM baseImage
FROM node:12

## step 2 - COPY source destination
COPY . .

## step 3 - EXPOSE port
EXPOSE 8080

## step 4 - CMD stratupCommandArray
CMD ["node", "app.js"]

At this point our working directory should look like this:

working-directory
|__ index.js
|__ Dockerfile

We can build our image, based on the Dockerfile using the following docker command:

Note the . at the end of the command

docker build -t my-docker-app .

The above command can be broken down as follows:

docker build the command from the Docker CLI to build an image
-t my-docker-app says what we want our image to be called, in the above my-docker-app
. which is the directory in which the Dockerfile is located, in our case our current directory

We can then run the image we just built like so:

docker run -p 8080:8080 my-docker-app

docker run is the command from the Docker CLI to run a container
-p 8080:8080 is our port mapping, it is ordered as HOST_PORT:CONTAINER_PORT and allows us to say which port on our host we want to map to our container, the container port is the same port that our app listens on and is EXPOSEd in the Dockerfile
my-docker-app is the image tag we would like to run

Each time we change the app files for a container like above we need to rebuild the container before running, and that normally making changes to files during the image build or container run will not modify the original files on our computer

Now that the application is running on port 8080 you can open http://localhost:8080 in your browser and you should see your Hello World app running

When you're done with that you can go back to the terminal where the container was started and use ctrl + c to stop the container

If you've never used Docker before and have got everything running this far, congratulations! If you've got any questions you can comment below or hit me up on Twitter @not_nabeel

Moving swiftly along

Development Containers

So now that we understand a bit about containers and how we can go about using them in production, we'll look at why we may want to use them as a development environment

Why Develop in a Container

As developers, we are far too familiar with the "it runs on my machine" dilemma. Development environments can be wildly inconsistent between different developers or different operating systems, and ensuring that our development code runs easily on everyone's computer can be challenging

Containers can help us to explicitly define our development environment, our application dependencies, what networking relationships, and (potentially) what other sibling applications need to be running in development, like databases, or other application tiers

Visual Studio Code can help transport us into a container so that we work on our application in a well-defined environment, not just run our application within one while reducing the overall number of things we need to have installed on our computer

How to Develop in a Container

To develop in a Container using Visual Studio Code we will need to have:

What's important to note is that we don't need any of our application's runtime or development dependencies installed, like Node.js, these will all be handled by VSCode within our container

To configure our project for running in a container we need to first open the project folder (the folder we used previously) in Visual Studio Code

Use an Existing Dockerfile

Once open use the keyboard shortcut ctrl + shift + p to open the Command Palette and search for Remote-Containers: Add Development Container Configuration Files and click enter, you will then have an option to use the existing Dockerfile from Dockerfile which will generate a .devcontainer/devcontainer.json file

At this point our working directory should look like this:

working-directory
|__ .devcontainer
|   |__ devcontainer.json
|
|__ index.js
|__ Dockerfile

The .devcontainer/devcontainer.json file that was created will contain the following:

devcontainer.json

// For format details, see https://aka.ms/vscode-remote/devcontainer.json or this file's README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.128.0/containers/docker-existing-dockerfile
{
    "name": "Existing Dockerfile",

    // Sets the run context to one level up instead of the .devcontainer folder.
    "context": "..",

    // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
    "dockerFile": "..\\Dockerfile",

    // Set *default* container specific settings.json values on container create.
    "settings": {
        "terminal.integrated.shell.linux": null
    },

    // Add the IDs of extensions you want installed when the container is created.
    "extensions": []

    ...
}

The above file is the configuration for our development container, we can also allow VSCode to generate a Dockerfile which we'll look at later in the post

We'll stick to our simple Dockerfile for this post, but if you've got a different Dockerfile when running your application in Production and Development then you may need a different file in the dockerFile property below

Now that we've got a starting point we can add a little to our configuration so that everything is just right:

Change the name property to name our workspace (purely aesthetic)
Add a forwardPorts property to expose our application port to our localhost network, be sure to add the , after "extensions":[]

Once we make the above changes we should have this:

devcontainer.json

{
    "name": "My Workspace",

    // Sets the run context to one level up instead of the .devcontainer folder.
    "context": "..",

    // Update the 'dockerFile' property if you aren't using the standard 'Dockerfile' filename.
    "dockerFile": "..\\Dockerfile",

    // Set *default* container specific settings.json values on container create.
    "settings": {
        "terminal.integrated.shell.linux": null
    },

    // Add the IDs of extensions you want installed when the container is created.
    "extensions": [],

    // Use 'forwardPorts' to make a list of ports inside the container available locally.
    "forwardPorts": [
        8080
    ],
    ...
}

Now that we've configured our build container, use ctrl + shift + p to open the Command Palette again and search for Remote-Containers: Reopen in Container and click enter which will build the container and set up an image with the following setup for us:

Linked ports as defined in the forwardPorts property
Configure a VSCode development server inside the container so our editor can link to it
Mount our system's file directory into the container so we can edit our files
Does not run the CMD command from our Dockerfile
Open a VSCode window linked to the container so we can start working with our code

Now that you're in the container you can edit your files and run it by doing the following:

Use ctrl + shift + p and then search for Terminal: Create new Integrated Terminal and click enter
Type node app.js into the new Terminal window and click enter to run our app.js file
Navigate to http://localhost:8080 in your browser to view your running app

At this point we've created a container to use as a development file and run our application, you can stop the application with ctrl + c

You can switch from developing in a container back to your local environment with ctrl + shift + p and searching for Remote-Containers: Reopen locally and clicking enter

Now that we're back on our local environment (and not docker) we can look at the other way we can set up our project for VSCode

Using a Preconfigured Dockerfile

Visual Studio Code's Remote Containers Extension provides some pre-configured Dockerfiles for common application or application framework types. One of the available preconfigured Dockerfiles is for working on Node.js applications

The preconfigured files usually just provide a starting point for applications and often you will need to modify these to suit your application, we don't need this for the application we're working on however

To redefine our Docker development config, let's delete the .devcontainer directory in our application and regenerate this

We can regenerate the files needed with ctrl + shift + p, and searching for Remote-Containers: Add Development Container Configuration Files again, clicking enter and then selecting the From a predefined configuration definition option, and then selecting Node.js 12, this should now create a .devcontainer/devcontainer.json file as well as a new .devcontainer/Dockerfile that we did not have previously, our working directory will now look like so:

working-directory
|__ .devcontainer
|   |__ devcontainer.json
|   |__ Dockerfile         # predefined dev container Dockerfile
|
|__ index.js
|__ Dockerfile             # our self-defined Dockerfile

If we look at the devcontainer.json file we will see something similar to what we had before:

devcontainer.json

{
    "name": "Node.js 12",
    "dockerFile": "Dockerfile",

    // Set *default* container specific settings.json values on container create.
    "settings": {
        "terminal.integrated.shell.linux": "/bin/bash"
    },

    // Add the IDs of extensions you want installed when the container is created.
    "extensions": [
        "dbaeumer.vscode-eslint"
    ]

    ...
}

You may, however, note that the dockerFile property is missing, this just means that VSCode will use the default Dockerfile which has been created in the .devcontainer directory

We can go ahead and change the name if we want, we should also add the forwardPorts option as we did previously:

devcontainer.json

{
    ...

    "forwardPorts": [
        8080
    ],

    ...
}

Now looking at the Dockerfile which defines the base development container:

Dockerfile

FROM mcr.microsoft.com/vscode/devcontainers/javascript-node:0-12

This is a bit different to ours because Visual Studio Code will handle the file copying and port exposing on its own for the development container. Note that this configuration can only be used for development and can't really be deployed as a production container. This type of setup is necessary if our development image and production image will be different (which they usually are)

Now that the development container has been set-up, we can use ctrl + shift + p and Remote-Containers: Reopen in Container to open our development container, from here we can work on our application and run the application the same as we did before

Which Method to Use

We've looked at two different methods for configuring our development container, either of which can be used in any project. Below are my recommendations:

If you've got an existing Dockerfile and your development container can be the same as your production container, for things like simple node.js or python apps, and you don't want to maintain another Dockerfile then this may be a quick solution to opt for

Otherwise, if your development container needs to be different from your production one then it's probably easier to start with a predefined VSCode Container as a base and add in any development configuration you need to the .devcontainer/Dockerfile

Lastly, if you don't have an existing Dockerfile at all then I'd suggest using a predefined one so that even if it's not fully configured you've got a relatively good starting point, especially when working with more complex languages and frameworks as a custom Dockerfile for these can be some work to configure

Summary

In this post, we've covered the basics of using Docker to run your applications in a container as well as how to define and build your images. We also looked at why we may want to use a container for development and how we can do this using Visual Studio Code

Code for Noobs

Sun, 17 May 2020 00:00:00 GMT

So I've been meaning to write an introductory post about the basics of JavaScript for quite some time. A few weeks ago I managed to finally get something going on this Twitter thread (@not_nabeel)

Let's get into it

0. What is Programming

Programming is our way of telling a computer to do things. This can be anything from checking our spelling to making trippy digital art

The basics of programming are the same in most languages and doesn't involve much math or binary (0010101) like a lot of people seem to think. Programming languages usually make use of a very small subset of words and concepts of a normal human language like English

The language I'm going to be using is called JavaScript but many of these concepts are the same in most other programming languages

1. What is JavaScript?

JavaScript is the language used to add interactivity to a website but it can also be used to do pretty much anything you like. JavaScript runs in all modern web browsers and can be accessed through your browser's developer tools, but it can also be used on other devices using something like @nodejs

For our purposes though, we don't need to worry too much about how that all works right now. To get started we'll be using tools like @CodePen, @replit and @glitch which give you text editor and a place to run your code

You'll see some code samples embeded in the

I'll be posting links to the code on these sites as I go along

2. Text Data

In a program, we can store different types of data (known as "data types")

When we store text we use a data type called string. In JavaScript a string is just text surrounded in either double quotes, single quotes, or `these things` (which are known as backticks):

'i am text'

'i am also text'

;`me too.`

;`unlike the others,
  i can b
multiple lines
lllooonnnggg
    and have random spaces`

3. Variables

If we want to keep our data for use at a later stage we need to give it a name, otherwise how do we know what data we're trying to use right? To give a piece of data (or text, in our case) a name we create a variable

We create a variable using let or const along with a variable name and the data that we want to give the name to :

// this code shouldn't do anything if you run it
// we are just creating some variables

let myText = 'Hello World'

const myOtherText = 'Bye World'

</details>

We use let for data whose data may change and const for data whose data won't you can remember this like: const = constant

Note that it is also possible to use the keyword var to create a variable. This is something that is left over from older versions of JavaScript and can have some effects that are better to just avoid (if you're interested you can read this article about how it impacts variable scope but it is a bit of a more challenging concept to understand)

When creating a variable, there are a few rules we need to follow:

Variable names must not contain spaces or stuff like !@#%%^&\*(), underscores and dollar signs are okay though
Variable names must not be a word the language has set aside for something else. e.g const or let
Variable names must not start with a number
Variable names should be descriptive. while names like "x" and "blah" are allowed, if they have no meaning in the context it's better to opt for something people will understand

4. Printing Data

Languages have ways we can show data to a user (or programmer). In JavaScript this is the "console.log" function. We'll discuss functions later on but for now know that when we give them data, and they do stuff

// print the data
console.log('Bob Smith') //prints -> Bob Smith

// print a variable
const jenny = 'Jenny Smith'
console.log(jenny) // prints -> Jenny Smith

</details>

5. Numbers

Numbers are another type of data in JavaScript, to store a number we can just write the number. using numbers we can also do things like getting fat or running from the po-lice

- is plus
- is minus
* is multiply
/ is divide

const myWeight = 100
const food = 100 + 50

const myNewWeight = myWeight + food

console.log(myNewWeight) // print -> 250

const myWeightAfterRunnin = myWeight - myWeight * 0.2

console.log(myWeightAfterRunnin) // print -> 80

</details>

But, JavaScript lets us add do maths with anything. so maybe like - don't do this:

const dontDoThis = 'apple' * 'pineapple'

console.log(dontDoThis) // print -> NaN (Not a Number)

</details>

6. Arrays

Arrays are how we store a set of data. an array can have different data types in it, but usually, we want to be storing the same stuff in an array. we make an array by wrapping our items in [ ] and separating each item with a comma (,)

const myFriends = ["i", "don't", "have", "any"]

const thingsToRemember = ["uhmm", 42, 12, "idk, i guess i forgot"]

const multipleLines = [
"arrays don't have to be",
"on a single",
"line"
]

</details>

If we want to get a specific element in an array we can use the variable name with the index (position) of the element. the index starts at 0. we will get undefined (I'll explain this in a bit) if we try to get a value at an index that does not exist

  const myData = [
  "zeroeth index",
  "first index",
  "second index",
  "third index"
]

console.log(myData[0]) // print -> zeroeth index
console.log(myData[1]) // print -> first index
console.log(myData[2]) // print -> second index
console.log(myData[3]) // print -> third index

// indexes that do not exist in our array
console.log(myData[-1]) // print -> undefined
console.log(myData[100]) // print -> undefined

</details>

7. Booleans

So far we've looked at strings and numbers, we have an even more basic data type called a boolean. a boolean is a value that can either be true or false. when creating a variable for a boolean we do not wrap the value in quotes

var myTrueBool = true

var myFalseBool = false

console.log(myTrueBool)
console.log(myFalseBool)

</details>

8. Complex Data

If we want to store data that are more complex than the ones we've seen above, we can make use of "objects". use { } around our data to group more basic data. these use keys as names for values in the object

const jenny = {
  id: 123,
  name: 'Jenny',
  surname: 'Smith',
  age: "idk, can't ask a woman that",
  favouriteBooks: [
    'To not kill a Mockingbird',
    'Why are books so voilent, yoh',
  ],
}

// we can get specific values using their keys
const jennyName = jenny.name
const jennyBooks = jenny.favouriteBooks

</details>

9. Undefined and Null

the values undefined and null have a special meaning.

undefined is a representation for a variable that does not have a value
null is a value of "nothing"

For example:

What's wrong with chocolate? null
What's a flobuir? undefined

10. Functions

Functions are a way we can group code for a specific set of instructions, usually with some end purpose. In JavaScript we define a function in one of two ways: using the word function or with the fat arrow syntax

// using the "function" keyword
// this creates a variable called "myFunc1"
function myFunc() {
  const stuff = 'I am Function'
  console.log(stuff)
}

// we can also create the variable using
// "let" or "const" like we do for other variables
const myLetFunc = function () {
  const stuff = 'I am another Function'
  console.log(stuff)
}

// fat arrow function
const myFatFunc = () => {
  console.log('Phat Funk')
}

// single line fat-arrow functions don't need the {}
const mySingleLineFunct = () => console.log('single line')

</details>

To use a function we simply write the name of the function and then add () at the end, provided the function doesn't take any data (also known as parameters)

const sayHello = function () {
  console.log('Hello')
}

// "call" the function
sayHello() // prints -> Hello

const sayBye = () => console.log('BYEE')

// "call" the function
sayBye() // prints -> BYEE

</details>

Remember earlier I said that functions take data and do stuff? For us to give a function data we need to tell it what variables to store that data as when we call it. We do this by having the data in the () when we create (or 'declare') our function

function sayHello(name) {
  console.log('Hello ' + name)
}

// use the function
sayHello('Bob') // print -> Hello Bob

const sayBye = (name) => console.log('Bye ' + name)

//use the function
const name = 'Jenny'
sayBye(name) // print -> Bye Jenny

</details>

When computers are following our code they do so from the top down, so the first line is handled by the computer before the second. the second before the third, etc. functions are a way for us to reuse code that was written on a line higher up in our code. You may notice that we use the variable name in a lot of different places above. this uses scope and broadly means any variable name created in a function (or pretty much anywhere between { }) is only available within that section

Scope complicated, you can read more about it here

Functions can also take multiple parameters, this is done by listing the parameters between the () and separating them by commas. the order that we list them when we create our function is the same as the order we need to put them when using the function

const sayBye = (firstName, lastName) => {
  console.log('Bye ' + firstName + ' ' + lastName)
}

// use the function
const jennyName = 'Jenny'
const jennySurname = 'Smith'
sayBye(jennyName, jennySurname) // print -> Bye Jenny Smith

</details>

Functions are also able to give us back a value after doing some stuff. They do this by using the return keyword which tells it what to give back. A function stops processing when it sees the keyword return - anything after it in a function is ignored

const addNumbers = (num1, num2) => {
  const result = num1 + num2
  return result
}

const mySum = addNumbers(1, 2)

console.log(mySum) // print -> 3

</details>

Summary

That's it for now, I'll definitely be updating this post/series as things are added

Nabeel Valley

Colour in Black and White Photography

Tue, 07 Apr 2020 00:00:00 GMT

Let's talk about black and white shall we?. As a first-time monochromer there are a couple of aspects of shooting and editing a black and white picture that isn't immediately apparent

When shooting in Black and White (henceforth B&W) there are some things we need to be aware especially aware of, namely:

Brightness
Texture
Colour

Now, of course, other considerations can be made but most of these boil down to how you use these three elements. For the most part, I'll be discussing how colour comes into play from an editing perspective, but this should give you an awareness of how this applies when shooting too

Brightness

When I refer to brightness in this context I am not referring to the exposure of the image but rather the distribution of brightness across the image itself, where are your shadows and highlights, how bright is your subject in relation to your background, etc.

Typically when we are working with a B&W image we try to have our subject be a little brighter (or much brighter) than everything else, this helps us to draw the viewer's attention more easily, of course, this is only one of the factors

Texture

When shooting B&W we also look for texture, I would argue that texture is to a black and white picture what colour is to a colour one. Texture helps us to differentiate between different elements in a photo and is very evident when used a lot of strong contrast. Sometimes you want a lot of texture, sometimes you don't - it is, however, important that you acknowledge how it contributes to an image

Colour

My focus in this post is primarily colour because I think it's the easiest to play around with and can make the biggest impact to a picture when editing (how else does anyone get B&W these days?)

At a simplified level, the way we work with colour in a B&W picture is by manipulating the brightness with which different colours are interpreted which in turn allows us to emphasize (or de-emphasize) specific elements in an image

Over the remainder of this post I'll be discussing a lot of this in the context of the following image:

</center>

For our discussion there are a few important aspects of this image to note:

The image is darkest in the background (top left) and transitions to being brighter in the foreground (bottom right)
We primarily have the subject in red and the background element (leaves) in green
The brightness of our red and green elements in their respective sections of the image are approximately the same

If we were to simply convert this image to a B&W one we get the following:

</center>

From the above we see that the distinction between our red and green elements isn't really visible anymore, this is because of the impact that colour mixing has on our image. Mixing allows us to control the way the different colours are seen in the image and most photo editing tools have some version of a Black and White Mixer that allows us to apply a colour filter to a Black and White image

Neutral Filtering

This is the "natural" way that a B&W conversion works, using this method for any two colours at approximately the same brightness we should see the same brightness in the B&W version

If we look at the neutral version of our image we can observe that the brightness across our image is fairly consistent with that of the colour version with a darker background and brighter foreground. We can also see that there is very little differentiation between the red and green elements in our image

</center>

In photography, there are few well-known colour filters which can be placed in front of your lens to achieve what we are doing here digitally. Commonly we have red, green, blue, and yellow filters. We'll just be looking at the red and green filters that were used to edit this image

Red Filter

In a picture like this where our subject is red, we can use a red filter to increase the brightness of the red portions of our image. To do this we would bump up the red and orange sections of our colours while trying to maintain a smooth change from one colour to the next to prevent any sharp changes in the brightness of our image

Additionally, I've also made the green slightly darker to add some additional contrast. We can see the result of doing so along with the changes made to the brightness of the respective colours below:

</center>

Using a method like this we can draw our focus back to the red elements a little more than if we had left the brightnesses as they were in the Neutral image while pushing down the brightness of our green elements so we don't have the two competing for focus

Green Filter

We can also do something like the above using a filter that brightens the green and darkens the red

</center>

In the resulting image above we can see that the greens are now much brighter and the reds are almost black - however we still maintain a good level of contrast between our too resulting elements. We can see a comparison of the impact the different colour filters have on our image below. These are ordered from left to right as red, neutral and green

</center>

Using what we've looked at in the above we can see that the way we choose to filter a black and white image has a huge impact on how we direct the viewer's attention in our image as well as how clearly we can differentiate between different elements

Note that while we do have different common colour filters, when editing a B&W image it usually makes sense to play around with your Black and White Mixer Tool to see what works best for your image

Nabeel Valley

The Gatsby Migration, pt.3 - Smart Pages

Sun, 15 Mar 2020 00:00:00 GMT

Introduction

So far we've created the initial react application as with a few routes for our Home, Blog, and 404 pages. In this post we'll look at how we can set up our Post component to render our pages dynamically based on the JSON data we have. We'll also extend this so that we can have some more content in a markdown file that we'll parse and add to our Gatsby data

Creating the initial React App
Rendering the "Dumb" pages with Gatsby
Rendering the "Smart" page with Gatsby (This post)

Setting Up

We're going to make our data a little more complex by creating two additional markdown files in our static/posts directory to enable us to have more content with each post

Create the following markdown files in the application and align the names with our post-1.json and post-2.json files:

static/posts/post-1.md
static/posts/post-2.md

Gatsby Plugins

To read the data from our files we're going to do the following:

Use the gatsby-source-filesystem to read our files into the Gatsby Data Layer
Define our own plugin that can read the file content, parse the markdown, and add it into the data layer

Reading the File Metadata

To read our file data we will need to first install the gatsby-source-filesystem plugin. Plugins in Gatsby enable us to ingest or transform data in our application. We then make use of GraphQL to query the data from the relevant component

Install the gatsby-source-filesystem plugin with:

yarn add gatsby-source-filesystem

And then add the plugin configuration to the gatsby-node.js file into the plugins array:

gatsby-node.js

{
    resolve: `gatsby-source-filesystem`,
    options: {
        name: `content`,
        path: `${__dirname}/static/posts`,
    },
}

This will read all the data from our posts directory into the filesystem. We can now start the application back up with yarn start and navigate to http://localhost:8000/__graphql in our browser to view the GraphQL data. We should be able to see the GraphiQL interface

From the GraphiQL interface run the following query to see the data from the files in our directory:

query PostData {
  allFile {
    nodes {
      name
      extension
      absolutePath
    }
  }
}

This should yield the following JSON with our file meta data in it:

{
  "data": {
    "allFile": {
      "nodes": [
        {
          "name": "post-1",
          "extension": "json",
          "absolutePath": "C:/repos/cra-to-gatsby/static/posts/post-1.json"
        },
        {
          "name": "1",
          "extension": "jpg",
          "absolutePath": "C:/repos/cra-to-gatsby/static/posts/1.jpg"
        },
        {
          "name": "post-1",
          "extension": "md",
          "absolutePath": "C:/repos/cra-to-gatsby/static/posts/post-1.md"
        }
        // file 2 data
      ]
    }
  }
}

Processing the Files

Now that we have our metadata for each file in the file system, we're going to create a plugin that will allow us to read the file data and add it the GraphQL data layer

In order to do this, create a plugins directory in the root folder. Inside of the plugins directory create a folder and folder for our plugin.

Create a new folder in the plugins directory with another folder called gatsby-transformer-postdata

From this directory run the following commands to initialize and link the yarn package:

plugins/gatsby-transformer-postdata

yarn init -y
yarn link

We'll also add the showdown package which will allow us to convert the markdown into the HTML so we can render it with our Post component

yarn add showdown

And then create an gatsby-node.js file in this directory with the following content:

/plugins/gatsby-transformer-postdata/gatsby-node.js

const fs = require('fs')
const crypto = require('crypto')
const showdown = require('showdown')

exports.onCreateNode = async ({ node, getNode, actions }) => {
  const { createNodeField, createNode } = actions

  // we'll process the node data here
}

This exposes the onCreateNode Gatsby API for our plugin. This is what Gatsby calls when creating nodes and we will be able to hook into this to create new nodes with all the data for each respective post based on the created file nodes

From the onCreateNode function we'll do the following to create the new nodes:

Check if it is a markdown node
Check if the JSON file exists
Read file content
Parse the metadata into an object
Convert the markdown to HTML
Get the name of the node
Define the data for our node
Create the new node using the createNode function

gatsby-transformer-postdata/gatsby-node.js

const fs = require('fs')
const crypto = require('crypto')
const showdown = require('showdown')

exports.onCreateNode = async ({ node, actions, loadNodeContent }) => {
  const { createNodeField, createNode } = actions

  // 1. Check if it is a markdown node
  if (node.internal.mediaType == 'text/markdown') {
    const jsonFilePath = `${node.absolutePath.slice(0, -3)}.json`

    console.log(jsonFilePath)

    // 2. Check if the JSON file exists
    if (fs.existsSync(jsonFilePath)) {
      // 3. Read file content
      const markdownFilePath = node.absolutePath
      const markdownContent = fs.readFileSync(markdownFilePath, 'utf8')
      const jsonContent = fs.readFileSync(jsonFilePath, 'utf8')

      // 4. Parse the metadata into an object
      const metaData = JSON.parse(jsonContent)

      // 5. Convert the markdown to HTML
      const converter = new showdown.Converter()
      const html = converter.makeHtml(markdownContent)

      // 6. Get the name of the node
      const name = node.name

      // 7. Define the data for our node
      const nodeData = {
        name,
        html,
        metaData,
        slug: `/blog/${name}`,
      }

      // 8. Create the new node using the `createNode` function
      const newNode = {
        // Node data
        ...nodeData,

        // Required fields.
        id: `RenderedMarkdownPost-${name}`,
        children: [],
        internal: {
          type: `RenderedMarkdownPost`,
          contentDigest: crypto
            .createHash(`md5`)
            .update(JSON.stringify(nodeData))
            .digest(`hex`),
        },
      }

      createNode(newNode)
    }
  }
}

From the root directory you can clean and rerun the application:

yarn clean
yarn start

Now, reload the GraphiQL at http://localhost:8000/__graphql and run the following query to extract the data we just pushed into the node:

query AllPostData {
  allRenderedMarkdownPost {
    nodes {
      html
      name
      slug
      metaData {
        body
        image
        title
      }
    }
  }
}

This should give us the relevant post data:

{
  "data": {
    "allRenderedMarkdownPost": {
      "nodes": [
        {
          "html": "<p>Hello here is some content for Post 1</p>\n<ol>\n<li>Hello</li>\n<li>World</li>\n</ol>",
          "name": "post-1",
          "slug": "/blog/post-1",
          "metaData": {
            "body": "Hello world, how are you",
            "image": "/posts/1.jpg",
            "title": "Post 1"
          }
        },
        {
          "html": "<p>Hello here is some content for Post 2</p>\n<ol>\n<li>Hello</li>\n<li>World</li>\n</ol>",
          "name": "post-2",
          "slug": "/blog/post-2",
          "metaData": {
            "body": "Hello world, I am fine",
            "image": "/posts/2.jpg",
            "title": "Post 2"
          }
        }
      ]
    }
  }
}

Create Pages

Now that we've got all our data for the pages in one place we can use the onCreatePages API to create our posts, and the Post component to render the pages

Setting Up

Before we really do anything we need to rename the Blog.js file to blog.js as well as create the src/components directory and move the Post.js file into it, you may need to restart your application again using yarn start

Create Pages Dynamically

In our site root create a gatsby-node.js file which exposes an onCreatePages function:

gatsby-node.js

const path = require('path')

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions
}

From this function we need to do the following:

Query for the PostData using the graphql function
Create a page for each renderedMarkdownPost

gatsby-node.js

const path = require('path')

exports.createPages = async ({ graphql, actions }) => {
  const { createPage } = actions

  // 1. Query for the PostData using the `graphql` function
  const result = await graphql(`
    query AllPostData {
      allRenderedMarkdownPost {
        nodes {
          html
          name
          slug
          metaData {
            body
            image
            title
          }
        }
      }
    }
  `)

  result.data.allRenderedMarkdownPost.nodes.forEach((node) => {
    // 2. Create a page for each `renderedMarkdownPost`
    createPage({
      path: node.slug,
      component: path.resolve(`./src/components/Post.js`),
      context: node,
    })
  })
}

Render the Page Data

From the Post component we need to:

Export the query for the data
Get the data for the Post
Render the data

components/post.js

import { graphql } from 'gatsby'
import App from '../App'

const Post = ({ data }) => {
  // 2. Get the data for the Post
  const postData = data.renderedMarkdownPost

  // 3. Render the data
  return (
    <App>
      <div className="Post">
        <p>
          This is the <code>{postData.slug}</code> page
        </p>
        <h1>{postData.metaData.title}</h1>
        <div
          className="markdown"
          dangerouslySetInnerHTML={{ __html: postData.html }}
        ></div>
      </div>
    </App>
  )
}

export default Post

// 1. Export the `query` for the data
export const query = graphql`
  query PostData($slug: String!) {
    renderedMarkdownPost(slug: { eq: $slug }) {
      html
      name
      slug
      metaData {
        body
        image
        title
      }
    }
  }
`

From the Post component above we use the slug to determine which page to render, we also set the HTML content for the markdown element above using the HTML we generated. We also now have our pages dynamically created based on the data in our static directory

You can also see that we have significantly reduced the complexity in the Post component now that we don't need to handle the data fetching from the component

If you look at the site now you should be able to navigate through all the pages as you'd expect to be able to

Summary

By now we have completed the the entire migration process - converting our static and dynamic pages to use Gatsby. In order to bring the dynamic page generation functionality to our site we've done the following:

Used the gatsby-source-filesystem plugin to read our file data
Created a local plugin to get the data for each post and convert the markdown to HTML
Use the onCreatePages API to dynamically create pages based on the post data
Update the Post component to render from the data supplied by the graphql query

And that's about it, through this series we've covered most of the basics on building a Gatsby site and handling a few scenarios for processing data using plugins and rendering content using Gatsby's available APIs

Nabeel Valley

The Gatsby Migration, pt.2 - Dumb Pages

Sat, 01 Feb 2020 00:00:00 GMT

Introduction

In the last post we looked setting up an application with a few basic routes. These routes were all assigned to Components in the src/pages directory.

This post will be going throught the Gatsby Setup necessary in order to migrate our current site to Gatsby, we will be looking at the second step in the process that was outlined in the last post:

Creating the initial React App
Rendering the "Dumb" pages with Gatsby (This post)
Rendering the "Smart" page with Gatsby

Getting Ready

In order to Gatsbyify the application, there are three steps that we will need to take before we can start updating our pages to work with the new system

Update folder structure
Install Gatsby
Create the necessary Gatsby Config files

Update Folder Structure

Gatsby Builds are placed into the public directory. This currently is the output directory of a React build if we are using the standard React configuration

Before we do anything more we should rename our public directory to static as this is what Gatsby uses for static files, and then make a git commit before we add the public directory to our .gitignore

Now we need to add the following lines to the end of our .gitignore file so that we do not track the gatsby build files:

.gitignore

## gatsby
public
.cache

Install Gatsby

To add Gataby to our project we need to add the gatsby package from npm as a dependency to our project. From the project's root directory run:

yarn add gatsby

Next we'll add/update the following commands in our package.json file so that we can start the Gatsby Dev Server as well as build the application

package.json

"scripts": {
    "start": "gatsby develop",
    "build": "gatsby build",
    "clean": "gatsby clean",
    ...
},

Create Config Files

In order to enable gatsby we need to add the gatsby-config.js file to our root directory, we can use the starter file with the following content, as it currently stands this doesn't do anything

gatsby-config.js

module.exports = {
  plugins: [],
}

Next we'll create an html.js file in the src directory with any relevant content from your index.html file if any of it has been updated. Also be sure to remove the %PUBLIC_URL% stuff from the file content

The html.js file needs to be a React Component with the following basic structure for a standard CRA app

src/html.js

import React from 'react'
import PropTypes from 'prop-types'

export default function HTML(props) {
  return (
    <html {...props.htmlAttributes}>
      <head>
        <meta charSet="utf-8" />
        <link rel="shortcut icon" href="/favicon.png" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
        <meta name="theme-color" content="#ffffff" />

        <link rel="manifest" href="/manifest.json" />

        <title>React App</title>

        <script
          dangerouslySetInnerHTML={{
            __html: `
                        // Any scripts that need to be included in the HTML itself
                        // Like tracking code, etc.
                        console.log("Inline Javascript")
                    `,
          }}
        ></script>

        {props.headComponents}
      </head>
      <body {...props.bodyAttributes}>
        {props.preBodyComponents}
        <div
          key={`body`}
          id="___gatsby"
          dangerouslySetInnerHTML={{ __html: props.body }}
        />
        {props.postBodyComponents}
      </body>
    </html>
  )
}

HTML.propTypes = {
  htmlAttributes: PropTypes.object,
  headComponents: PropTypes.array,
  bodyAttributes: PropTypes.object,
  preBodyComponents: PropTypes.array,
  body: PropTypes.string,
  postBodyComponents: PropTypes.array,
}

You can add or remove any elements in that file as per your specific requirements but most of the time the above should be fine

Now that we have updated the index.js file delete the static/index.html file you can run the Gatsby Dev Server

yarn clean
yarn start

Your application should now be running on http://localhost:8000/, if started correctly you should see the Gatsby Development 404 Page:

<details> <summary>Development 404 Page</summary>

<h1>Gatsby.js development 404 page</h1>

There's not a page yet at /

<button>Preview custom 404 page</button>

Create a React.js component in your site directory at src/pages/index.js and this page will automatically refresh to show the new page component you created.

If you were trying to reach another page, perhaps you can find it below.

<h2>Pages (4)</h2>

Search:

<label>Search:<input type="text" id="search" placeholder="Search pages..." value=""></label><input type="submit" value="Submit">

/Blog/
/Home/
/NotFound/
/Post/

</details>

Page Setup

From the Development 404 Page we can see that Gatsby has found our previously created pages - this is because Gatsby looks for pages in the pages directory, however when we click on a the links we will notice the following:

Home and 404 render correctly
Blog results in an error message
Post results in an error message (we will address this page in the next post)
Routes aren't aligned with our initial setup
Components no longer have the layout we setup in the App.js file

These are, for the most part, easy problems to solve

Fix the Blog Error Message

If we look at the Blog page we will see that there is an issue with the Page render, this is because we need to change the Link components to be imported from gatsby instead of react-router-dom because with Gatsby we are no longer using the React Router

Blog.js

import { Link } from 'gatsby'

Fix the Routes

We can also see that our routes are capitalized which is not what we want. Gatsby uses the file organization to do routing, so in order to correct our routes to align what we defined previously in our App.js we will first need to rename our files:

Component	Route	Old Name	New Name
Home	`/`	`Home.js`	`index.js`
Blog	`/blog`	`Blog.js`	`blog.js`
NotFound	`/*`	`NotFound.js`	`404.js`

We're not going to handle the Post component for now because this uses a dynamic route which is something I'll cover in the next part of this series

You should stop and restart the Gatsby Dev server with:

yarn clean
yarn start

When the page loads up we should now see our Home content rendered on the / route

Fix the Layout

When we were using the standard React Routing we made use of the App component to wrap our page routes as well as our navigation. We'll convert our App.js file into a component that we can use in our other pages

Remove the Router and Switch
Update the Links to use gatsby instead of react-router-dom
Add the children input parameter
Render the children in the main section
Remove unused imports

App.js

import React from 'react'
import './App.css'
import { Link } from 'gatsby'

const App = ({ children }) => (
  <div className="App">
    <header>
      <nav>
        <h1>My Website Title</h1>
        <Link to="/">Home</Link>
        <Link to="/blog">Blog</Link>
      </nav>
    </header>
    <main>{children}</main>
  </div>
)

export default App

Use the Layout

Now that we've essentially created a Layout component in the form of the App component we can use it to wrap the pages that we've exported. We can do this for the Home page like so:

index.js

import React from 'react'
import App from '../App'

const Home = () => (
  <App>
    <div className="Home">
      <h1>Home</h1>
      <p>This is the Home page</p>
    </div>
  </App>
)

export default Home

The same applies for the blog and 404 pages

blog.js

import React from 'react'
import { Link } from 'gatsby'
import App from '../App'

const Blog = () => (
  <App>
    <div className="Blog">
      <h1>Blog</h1>
      <p>This is the Blog page</p>
      <div>
        <Link to="/blog/post-1">Post 1</Link>
      </div>
      <div>
        <Link to="/blog/post-2">Post 2</Link>
      </div>
    </div>
  </App>
)

export default Blog

</details>

404.js

import React from 'react'
import App from '../App'

const NotFound = () => (
  <App>
    <div className="NotFound">
      <h1>404</h1>
      <p>Page Not Found</p>
    </div>
  </App>
)

export default NotFound

</details>

Fix the CSS

Before we can use the App component we need to add the gatsby-plugin-postcss and postcss-preset-env plugins so that Gatsby knows how to interpret the default create-react-app method of importing our CSS into a component

Stop the Dev Server and install the required packages:

yarn add gatsby-plugin-postcss postcss-preset-env

We then need to update the gatsby-config.js file to use the plugin we just added:

gatsby-config.js

module.exports = {
  plugins: [
    {
      resolve: `gatsby-plugin-postcss`,
      options: {
        postCssPlugins: [require(`postcss-preset-env`)({ stage: 0 })],
      },
    },
  ],
}

If you start the application now you'll notice that the font is not correct, this is because we specify the font styles in the index.css file. In order to fix this we need to import the index.css file into our application, in the default React application this is done via the src/index.js file, but since we don't use that to load up the application anymore we need to create another file to do that with Gatsby

In the root directory create a file called gatsby-browser.js, in this we just need to import the index.css file:

gatsby-browser.js

import './src/index.css'

You can now run the following commands and start up the application:

yarn clean
yarn start

Assuming everything went as planned the application should start up correctly and the styling from our index.css file will be correctly applied

Summary

By now we have completed the first part of the migration process - converting our static pages to use Gatsby - by taking the following steps:

Updating our file structure to better align with Gatsby
Create the relevant Gatsby config files
Update our Link usage
Update our Routing
Use a shared layout component
Fixing the CSS to work with Gatsby

In the next part we'll look at how we can go about providing the data needed for our Post component using GraphQL and show our posts. We'll also look at how we can pre-process the data that we provide to our component so that we can enhance our content while improving our content creation process

Nabeel Valley

The Gatsby Migration, pt.1 - Setting the Scene

Tue, 21 Jan 2020 00:00:00 GMT

Introduction

Lately I've been a little concerned with my current SPA approach on my personal site as well as a few others. More specifically the high initial load time due to the calls to the backend to retrieve content

With the aim of solving this problem I've spent a lot of time looking at and playing with Static Site Generators.Foreword: they're all a lot more complicated than one would think

So for a static site the "static" content only changes so often, based on this we can generate page content with the data we're planning to load in - that's what we're going to try to do

Now we'll be starting off with a React app generated with create-react-app so that we can have a starting point for our Gatsby site as well as understanding how we can approach some of the challenges when switching over to a site generator like Gatsby

For the sake of completeness, this series will be broken into four posts covering the following:

Creating the initial React App (This post)
Rendering the "Dumb" pages with Gatsby
Rendering the "Smart" page with Gatsby

The React App

First we're going to be starting off with a new React app that we will work on changing into a Gatsby one parts 2 and 3, we'll then focus on using plugins to enhance our content in 4

For this part we'll build a basic React app that has the following:

Two "hard-coded" pages and a 404 page
A dynamic page with an API call to retrieve data
Overall app layout with child routes for 1 - 3

To get started, we'll create a fresh React App:

npx create-react-app gatsby-to-be

And add the react-router

yarn add react-router-dom

Running this command should set up the application, if you don't know much about React I'd suggest taking a look at the documentation

Next cd gatsby-to-be and run yarn start, you should be able to visit the application in your browser at http://localhost:3000/

Looking at the generated files we have a public directory with some icons, an index.html file into which our React application will run once built, and a src directory that has the application code. The index.js file is what loads the application into the DOM and the App.js file which is the main component for our application

Hard Coded Pages

We will create the following three hard-coded pages in the src/pages directory

These pages are just React components that we will assign Routes to.

The pages we are using are known as functional components because they are javascript functions that return JSX

If we intend to use JSX in a file we need to ensure that we import React. The other component we are importing is the Link component which is a lot like a normal HTML a tag but with some special functionality to make the client-side navigation work

Blog.js

import React from 'react'
import { Link } from 'react-router-dom'

const Blog = () => (
  <div className="Blog">
    <h1>Blog</h1>
    <p>This is the Blog page</p>
    <div>
      <Link to="/blog/post-1">Post 1</Link>
    </div>
    <div>
      <Link to="/blog/post-2">Post 2</Link>
    </div>
  </div>
)

export default Blog

Additionally we have the Home.js and NotFound.js files which are similar to the Blog.js file we created

Home.js

import React from 'react'

const Home = () => (
  <div className="Home">
    <h1>Home</h1>
    <p>This is the Home page</p>
  </div>
)

export default Home

</details>

<details> <summary>Not Found</summary>

NotFound.js

import React from 'react'

const NotFound = () => (
  <div className="NotFound">
    <h1>404</h1>
    <p>Page Not Found</p>
  </div>
)

export default NotFound

</details>

Dynamic Post Page

Next up we'll create a component that can render out content for a blog post. This will consist of a few hooks which are react functions that we can use to sort of control the data in a function

The Post component will:

Display a loading indicator initially
Figure out what post we're trying to render based on the URL
Retrieve a JSON file from the public directory based
Set the component state after reading the file
Display the content from the file in a JSX template

The useState hook is used to initialize the state the component, in this case using the hasError and data variables, as well as providing the functions necessary for updating those in the form of setHasError and setData respectively

We use fetch in the useEffect hook to retrieve the data from the public directory. The useEffect hook allows us to pass a function that will be called to update side effects. The second input, in our case [] is the array of objects that, when are changed, we want the hook to run - since we only want it to run once and don't care about any other state changes we pass in an empty array for this value

Post.js

import React, { useEffect, useState } from 'react'

const Post = ({ match }) => {
  const slug = match.params.slug

  const [hasError, setHasError] = useState(false)
  const [data, setData] = useState(null)

  useEffect(() => {
    const fetchData = async () => {
      try {
        const res = await fetch(`/posts/${slug}.json`)
        const json = await res.json()
        setData(json)
      } catch (error) {
        console.log(error.toString())
        setHasError(true)
      }
    }

    fetchData()
  }, [])

  return (
    <div className="Post">
      <p>
        This is the <code>{slug}</code> page
      </p>
      {data ? (
        <div className="content">
          <h1>{data.title}</h1>
          <p>{data.body}</p>
          <img src={data.image} alt="" />
        </div>
      ) : hasError ? (
        <div className="error">
          <h1>Error</h1>
          <p>{hasError}</p>
        </div>
      ) : (
        <p>Loading .. </p>
      )}
    </div>
  )
}

export default Post

In the above component we're making use of the following pattern to decide what to render conditionally:

If the data is loaded then show the data
Else if there is an error then show the error data
Otherwise show a loading message

The way we are rendering the Post component with a parameter for match. The match parameter will be passed in as a prop from the Router that we will configure next, this allows us to use the slug from the URL to retrieve the content for the page

The two data files we have to pull content from in the public/posts directory are post-1.json and post-2.json

post-1.json

{
  "title": "Post 1",
  "body": "Hello world, how are you",
  "image": "/posts/1.jpg"
}

</details>

post-2.json

{
  "title": "Post 2",
  "body": "Hello world, I am fine",
  "image": "/posts/2.jpg"
}

The images posts/1.jpg and posts/2.jpg can also just be any images in that directory

</details>

App Layout and Routes

Lastly, we'll specify our application layout with the relevant routes in the App.js file, referencing the components we have created, we do this using the BrowserRouter. When we switch the project over to a Gatsby one, the App.js file will be converted into our Layout component to wrap our different pages

We use the Router component and the page inside of it, this essentially handles Routing via the Link components. Next we have a div as a wrapper for our component as well as a header, nav, and main tags to organise the page

The Route component takes in the component that we would like to display for a given route, and the Switch helps us to ensure that only route is actively being fisplayed at a time. The Switch will navigate from the first to last Route and render the first one that matches the given path

Fow now, our App.js file is as follows:

import React from 'react'
import './App.css'
import { BrowserRouter as Router, Link, Switch, Route } from 'react-router-dom'
import Home from './pages/Home'
import Blog from './pages/Blog'
import Post from './pages/Post'
import NotFound from './pages/NotFound'

const App = () => (
  <Router>
    <div className="App">
      <header>
        <nav>
          <h1>My Website Title</h1>
          <Link to="/">Home</Link>
          <Link to="/blog">Blog</Link>
        </nav>
      </header>
      <main>
        <Switch>
          <Route exact path="/" component={Home}></Route>

          <Route exact path="/blog" component={Blog}></Route>

          <Route exact path="/blog/:slug" component={Post}></Route>

          <Route path="/*" component={NotFound}></Route>
        </Switch>
      </main>
    </div>
  </Router>
)

export default App

In the above component we can see that where we are rendering the Post component we have a parameter in the route called slug, this parameter will be passed in to the Post component as part of the match object

Summary

We have built a fairly simple application that makes use of both static and data-based pages, we have also tied all of these together using the App component and a Router with the following routes:

/ which will render the Home component
/blog which will render the Blog component
/blog/:slug which will render the Post component and retrieve the data based on the given slug
/* which will match any other routes and render the NotFound component

In the next post we're going to look at how to take what we have so far and transform this application into a Gatsby one, but for now it may be useful to think about what kind of steps we may need to take if we'd like to make this a static website based on the way our current routes work

Nabeel Valley

Real-time Communication with MQTT

Tue, 12 Nov 2019 00:00:00 GMT

Overview

MQTT is an open standard for communication and is especially useful for communication between IoT devices due to its low bandwidth requirements

Today we're going to be taking a look at using MQTT to easily communicate between multiple applications. We'll look at using a static website and Eclipse Mosquitto as a message broker to enable communication between two instances of the application, but the same principles can be extended to other programming languages and MQTT clients. For our clients we'll be using MQTT.js which runs in the browser via CDN and we interact with it using JavaScript, but can also be added from npm

If you'd like to follow along with the completed code you can get it <a href="https://github.com/nabeelvalley/RTCWithMQTT" target="_blank">here</a>

Broker

Eclipse Mosquitto is an open-source (EPL/EDL licensed) message broker that implements the MQTT protocol versions 5.0, 3.1.1 and 3.1. Mosquitto is lightweight and is suitable for use on all devices from low power single board computers to full servers. (<a href="https://mosquitto.org/" target="_blank">Eclipse Mosquitto</a> )

Message brokers are a method of intra-application communication, MQTT makes use of a publish/subscribe model for application communication. In this model different applications, or clients, can publish messages to a topic which can then be picked up and operated on by any other applications that are subscribed to the topic

Setting Up the Broker

You will first need to download Mosquitto from <a href="https://mosquitto.org/download/" target="_blank">here</a> , all 1.5mb of it, and run through the installer and then ensure that mosquitto is in your path. If you need to know how to add something to your path take a look at <a href="https://gist.github.com/nex3/c395b2f8fd4b02068be37c961301caa7" target="_blank">this Gist</a> . For a Windows 64 bit installation of Mosquitto you should add the following to your path C:\Program Files\mosquitto

The MQTT installation should include the following three commands

mosquitto - This runs the MQTT server/broker
mosquitto_pub - A simple message publisher
mosquitto_sub - A simple message subscriber

We can get more information by running either of the above commands with the --help flag, e.g. mosquitto --help

Basic Messaging

Let's start the message broker in verbose mode by opening a new shell and running the mosquitto -v command, you should see some output like the following:

1573494107: mosquitto version 1.6.7 starting
1573494107: Using default config.
1573494107: Opening ipv6 listen socket on port 1883.
1573494107: Opening ipv4 listen socket on port 1883.

We can see from the above that the message broker is running on our local port 1883

In a new shell, we can create a client which is subscribed to a topic. We'll name this topic messages but it can be pretty much anything you want, to start the subscriber client we can run mosquitto_sub -t "messages" -v, you won't see any output in the subscriber shell as yet, but looking at the broker shell you should logging which says that a client was connected

1573494642: New connection from ::1 on port 1883.
1573494642: New client connected from ::1 as mosq-fwbsJxdXOnQW0LOaIn (p2, c1, k60).
1573494642: No will message specified.
1573494642: Sending CONNACK to mosq-fwbsJxdXOnQW0LOaIn (0, 0)
1573494642: Received SUBSCRIBE from mosq-fwbsJxdXOnQW0LOaIn
1573494642:     messages (QoS 0)
1573494642: mosq-fwbsJxdXOnQW0LOaIn 0 messages
1573494642: Sending SUBACK to mosq-fwbsJxdXOnQW0LOaIn
1573494702: Received PINGREQ from mosq-fwbsJxdXOnQW0LOaIn
1573494702: Sending PINGRESP to mosq-fwbsJxdXOnQW0LOaIn

To publish a message open another shell and run mosquitto_pub -t "messages" -m "This is my message!", in our subscriber shell we should see the following output:

messages This is my message!

And in the broker we'll see the following:

1573494844: New connection from ::1 on port 1883.
1573494844: New client connected from ::1 as mosq-MWbaa2TpZGV0FrTmcF (p2, c1, k60).
1573494844: No will message specified.
1573494844: Sending CONNACK to mosq-MWbaa2TpZGV0FrTmcF (0, 0)
1573494844: Received PUBLISH from mosq-MWbaa2TpZGV0FrTmcF (d0, q0, r0, m0, 'messages', ... (19 bytes))
1573494844: Sending PUBLISH to mosq-fwbsJxdXOnQW0LOaIn (d0, q0, r0, m0, 'messages', ... (19 bytes))
1573494844: Received DISCONNECT from mosq-MWbaa2TpZGV0FrTmcF
1573494844: Client mosq-MWbaa2TpZGV0FrTmcF disconnected.

When you're done with this you can close the open shell windows

Broker with WebSockets

For our usecase, we will make use of WebSockets so we can communicate directly from the browser. When starting up the message broker we have the option to pass in a configuration file. Let's create one that states the port and the protocol we would like to use. Create a new directory called mqtt, and in it create a new file called mosquitto.conf and place the following contents which simply tell it to listen on port 9001 and use the websockets protocol

mosquitto.conf

listener 9001
protocol websockets

If you'd like to know what else can be done in the configuration file you can <a href="https://mosquitto.org/man/mosquitto-conf-5.html" target="_blank">read the documentation</a>

We can start the broker again from the mqtt directory using mosquitto -c mosquitto.conf -v, this time we should see that it is running on 9001 with websockets

1573499281: mosquitto version 1.6.7 starting
1573499281: Config loaded from mosquitto.conf.
1573499281: Opening websockets listen socket on port 9001.

Now that we have configured the broker to use Web Sockets we can start connecting to it from a web page and publish some messages

Client

We'll be using <a href="https://github.com/mqttjs/MQTT.js" target="_blank">MQTT.js</a> to connect to our message broker. MQTT.js is a simple MQTT Client Library for connecting to message brokers, we'll be using it via CDN for the sake of simplicity and will also be using <a href="https://materializecss.com/" target="_blank">Materialize</a> for our CSS because I'm not going to write a bunch of CSS and I don't want this to look completely terrible

HTML

Let's first setup the index.html. It consists of a two-column layout with the following:

Main Heading with a badge to indicate if we successfully connected to the broker
Inputs to Publish Message
Table of all received messages
Materialize and MQTT.js files via CDN
Script tag for all our JavaScript, because we don't need two files for this

index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <meta http-equiv="X-UA-Compatible" content="ie=edge" />
    <title>Real-time Communication with MQTT</title>

    <!-- MQTT.js -->
    <script src="https://unpkg.com/mqtt@3.0.0/dist/mqtt.min.js"></script>

    <!-- Materialize -->
    <!-- Compiled and minified CSS -->
    <link
      rel="stylesheet"
      href="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/css/materialize.min.css"
    />
  </head>

  <body>
    <div class="container">
      <!-- Page Content goes here -->
      <div class="row">
        <div class="col s12">
          <h1>
            Messages
            <span
              id="connect-badge"
              class="new badge orange"
              data-badge-caption=""
              >connecting</span
            >
          </h1>
        </div>
      </div>
      <div class="row">
        <div class="col s12 m6">
          <h2>Input</h2>
          <div class="input-field">
            <input id="name" type="text" class="validate" />
            <label class="active" for="name">First Name</label>
          </div>

          <div class="input-field">
            <textarea
              id="message"
              class="materialize-textarea"
              data-length="120"
            ></textarea>
            <label for="message">Message</label>
          </div>

          <button class="waves-effect waves-light btn" onclick="handleSubmit()">
            Publish Message
          </button>
        </div>
        <div class="col s12 m6">
          <h2>Output</h2>
          <table>
            <thead>
              <tr>
                <th>Name</th>
                <th>Message</th>
              </tr>
            </thead>
            <tbody id="messages"></tbody>
          </table>
        </div>
      </div>
    </div>

    <script>
      // Our Javascript will go here
    </script>
  </body>

  <!-- Materialize -->
  <!-- Compiled and minified JavaScript -->
  <script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/js/materialize.min.js"></script>
</html>

Running the Website

Since we are including resources from other domains your browser may have some issues with you using the 'double click' method of running the files, so we're going to need to use a web server

A list of quick one-line web servers can be found on this <a href="https://gist.github.com/willurd/5720255" target="_blank">Gist</a> and if you're using Visual Studio Code you can use the Live Server Extension

Connecting to the Message Broker

Now that we're up and running we can connect to the broker that we set up previously. The Mosquitto Broker should be running on 127.0.0.1:9001, using the mqtt object that is exposed in the global space by MQTT.js we can simply create an instance of the client in our <script> section with:

const client = mqtt.connect('mqtt://127.0.0.1:9001')

This gives us a client object that is the MQTT Client connected to our broker. Next, we need to set a handler for the connect event which is when the client successfully connects to the message broker, we do this with the following code:

client.on('connect', function () {
  client.subscribe('messages', function (err) {
    if (!err) {
      const badge = document.getElementById('connect-badge')
      badge.innerText = 'connected'
      badge.classList.remove('orange')
      badge.classList.add('green')
    }
  })
})

You can see that the client.on function takes a callback, in which we tell our client to subscribe to the messages topic as we did previously via the mosquitto_sub, and then once it has successfully subscribed we change the badge status to indicate that we are connected

Since we have successfully subscribed the client to the broker we should be able to publish messages. We'll do that using the data from the input fields in on the page and the Publish Message button's handler

Publishing Messages

Create a handler for the Publish Message button called handleSubmit, we've already referenced this in our button's HTML. The handler simply needs to read the values from the input fields and call client.publish with a stringified form of the data

function handleSubmit() {
  const nameInput = document.getElementById('name')
  const messageInput = document.getElementById('message')

  const data = {
    name: nameInput.value,
    message: messageInput.value,
  }

  const payload = JSON.stringify(data)

  client.publish('messages', payload)
}

The client.publish function above takes in a topic and message as its inputs, in our case we want to publish to the messages topic and we are publishing the JSON object as a string

Refreshing the page, entering some text into the fields and clicking the Publish Message button should publish the message to the broker, we should see this in the message broker's output

1573506416: Received PUBLISH from mqttjs_d5cdbc8e (d0, q0, r0, m0, 'messages', ... (34 bytes))
1573506416: Sending PUBLISH to mqttjs_d5cdbc8e (d0, q0, r0, m0, 'messages', ... (34 bytes))

Now that we can publish messages to the topic, we will need to create a handler for the message event which is triggered every time one of our subscribed topics has something published to it. We can do this again with the client.on function:

client.on('message', function (topic, message) {
  const data = JSON.parse(message)

  document.getElementById('messages').innerHTML += `<tr><td>${
    data.name
  }</td><td>${data.message.replace(/\n/g, '<br>')}</td></tr>`
})

Our message handler receives the topic, which in our case will always be messages and the message which is a buffer of the data published to the topic, this will be the same data that we published using the client.publish function, this time we read the JSON back into an object and add it to the messages table as HTML

You should be able to open a few different windows of the web page and they should all be able to Publish Messages to one another

What Next?

If you're not familiar with message brokers this could be a lot to take in and that's okay. I think playing around with the mosquitto_pub and mosquitto_sub tools can be helpful to get a more interactive feel for the way everything works. You can also just play with the MQTT.js code we used via your browser console and looking at the documentation on the website

If you want to take a look at a free instance of an MQTT Broker that you can play around with take a look at <a href="https://www.cloudmqtt.com/" target="_blank">Cloud MQTT</a> , and if you want to look at how you can take the concepts covered here today and play around with some IoT concepts as well then it may also be worth taking a look at this <a href="https://developer.ibm.com/tutorials/iot-mobile-phone-iot-device-bluemix-apps-trs/" target="_blank">IBM Developer Tutorial</a> using IBM Cloud and Watson IoT

If you're interested in spinning up your own MQTT broker with custom functionality you can take a look at <a href="http://www.mosca.io/">Mosca</a> or get an even deeper understanding of the MQTT Protocol <a href="https://developer.ibm.com/articles/iot-mqtt-why-good-for-iot/" target="_blank">on IBM Developer</a>

Conclusion

We've used a couple of different technologies and depending on your background you may not be familiar with everything here. The best thing I'd say to get to understand what we've covered would be to play around with the code as well as look at some of the other services and use-cases for message brokers and get a feel for what's out there

Good luck

Nabeel Valley

Intro to F# Web APIs

Wed, 30 Oct 2019 00:00:00 GMT

So we're going to be taking a bit of a look on how you can go about building your first F# Web API using .NET Core. I'm going to cover a lot of the basics, a lot of which should be familiar to anyone who has worked with .NET Web Applications and F# in general.

Along the way I'm also going to go through some important concepts that I feel are maybe not that clear from a documentation perspective that are actually super relevant to using this F# in a real-life context

If you're totally new to F# though you may want to take a look at <a href="https://fsharpforfunandprofit.com/" target="_blank">F# for Fun and Profit</a> or my personal quick reference documentation over here

Getting Started

Assuming you've got the .NET Core SDK with F# installed, you can simply create a new project with the following:

dotnet new webapi --language F# --name FSharpWebApi

code .\FSharpWebApi

Alternatively, if you're feeling a little unexperimental you can use the Visual Studio project creation wizard, psshhtt

Once you have the project open you can run the following command to launch the application:

dotnet run

Which should start the application on https://localhost:5001 and http://localhost:5000, you can see the current existing endpoint at /weatherforecast, this is handled by the Controllers/WeatherForecastController.fs file

Looking Around

Looking at the structure of the project files you should see the following:

FSharpWebApi
│   appsettings.Development.json
│   appsettings.json
│   FSharpWebApi.fsproj
│   Program.fs
│   Startup.fs
│   WeatherForecast.fs
│
├───Controllers
│       WeatherForecastController.fs
│
└───Properties
        launchSettings.json

So, mostly we see the typical Web API stuff that we'd expect for a C# project such as the startup and program files. In F# they serve pretty much the same purpose.

Looking at the Program.fs file we can see that it contains the main function and configures the Web Host, next we can see that the Startup.fs file contains the usual configuration methods. We should note that the method calls within these functions are piped to an ignore so the the functions to not return their respective Builders as this will break the application

The Program.fs and Startup.fs files can be seen below

Program.fs

namespace FSharpWebApi

module Program =
    let exitCode = 0

    let CreateHostBuilder args =
        Host.CreateDefaultBuilder(args)
            .ConfigureWebHostDefaults(fun webBuilder ->
                webBuilder.UseStartup<Startup>() |> ignore
            )

    [<EntryPoint>]
    let main args =
        CreateHostBuilder(args).Build().Run()

        exitCode

Startup.fs

namespace FSharpWebApi

type Startup private () =
    new (configuration: IConfiguration) as this =
        Startup() then
        this.Configuration <- configuration

    // This method gets called by the runtime. Use this method to add services to the container.
    member this.ConfigureServices(services: IServiceCollection) =
        // Add framework services.
        services.AddControllers() |> ignore

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    member this.Configure(app: IApplicationBuilder, env: IWebHostEnvironment) =
        if (env.IsDevelopment()) then
            app.UseDeveloperExceptionPage() |> ignore

        app.UseHttpsRedirection() |> ignore
        app.UseRouting() |> ignore

        app.UseAuthorization() |> ignore

        app.UseEndpoints(fun endpoints ->
            endpoints.MapControllers() |> ignore
            ) |> ignore

    member val Configuration : IConfiguration = null with get, set

Next we have the FSharpWebApi.fsproj file which contains references to the relevant code files. It's important to note that the order of the files in the ItemGroup specifies the order that files depend on each other. Lower files depend on files higher up

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="WeatherForecast.fs" />
    <Compile Include="Controllers/WeatherForecastController.fs" />
    <Compile Include="Startup.fs" />
    <Compile Include="Program.fs" />
  </ItemGroup>

</Project>

Lastly, we have a controller that resides in the Controllers/WeatherForecastController.fs with its types defined in the WeatherForecast.fs file. Looking at the WeatherForecast.fs file we can see that the type has a few simple properties and one function

WeatherForecast.fs

namespace FSharpWebApi

open System

type WeatherForecast =
    { Date: DateTime
      TemperatureC: int
      Summary: string }

    member this.TemperatureF =
        32 + (int (float this.TemperatureC / 0.5556))

Next up, we can see the controller which contains a single GET endpoint which delivers a random array of weather forecasts. Here we can see a few different things. First, the namespace is FSharpWebApi.Controllers, this pretty much follows the .NET standard of the Namespace being related to the Folder name, we can also see the ApiController attribute that adds <a href="https://docs.microsoft.com/en-us/aspnet/core/web-api/?view=aspnetcore-3.0#apicontroller-attribute" target="_blank">some useful functionality</a> for basic API handling and the Route attribute that states the controller route

The WeatherForecastController type defines the controller and that it inherits from ControllerBase, additionally the constructor requires the ILogger service which will be provided by DependencyInjection

Lastly, looking at the __Get method we can see the HttpGet attribute that specifies that this is a Get Method, and the __ shows that we don't care about references to the function's this, and the return type for the function is an array of WeatherForecast

WeatherForecastController.fs

namespace FSharpWebApi.Controllers

open System
open Microsoft.AspNetCore.Mvc
open Microsoft.Extensions.Logging
open FSharpWebApi

[<ApiController>]
[<Route("[controller]")>]
type WeatherForecastController (logger : ILogger<WeatherForecastController>) =
    inherit ControllerBase()

    let summaries = [| "Freezing"; "Bracing"; "Chilly"; "Cool"; "Mild"; "Warm"; "Balmy"; "Hot"; "Sweltering"; "Scorching" |]

    [<HttpGet>]
    member __.Get() : WeatherForecast[] =
        let rng = System.Random()
        [|
            for index in 0..4 ->
                { Date = DateTime.Now.AddDays(float index)
                  TemperatureC = rng.Next(-20,55)
                  Summary = summaries.[rng.Next(summaries.Length)] }
        |]

Creating a Controller

Creating a new controller is not particularly complex given that we have the above as a starting point.

Get Handler

We're going to create a handler that is able to return a simple message for an even param, and a 404 for a odd param in order to look at how we can return actual response codes in cases where we aren't always able to return something of a constant type

First, we can create a Controllers/MessageController.fs file with just some basic scaffolding to start with. We'll define a Get controller that just returns the id it receives as a route param multiplied by two if the the result shouldDouble param is set to true. Additionally we can see the sprint function used to format the output as a string

Before we can add the data to the actual controller we need to add the <Compile Include="Controllers/MessageController.fs" /> to the ItemGroup in the FSharpWebApi.fsproj file, :

FSharpWebApi.fsproj

  <ItemGroup>
    <Compile Include="WeatherForecast.fs" />
    <Compile Include="Controllers/WeatherForecastController.fs" />
    <Compile Include="Controllers/MessageController.fs" />
    <Compile Include="Startup.fs" />
    <Compile Include="Program.fs" />
  </ItemGroup>

And then we can put together the controller in the MessageController.fs file:

MessageController.fs

namespace FSharpWebApi.Controllers

open Microsoft.AspNetCore.Mvc
open Microsoft.Extensions.Logging

[<ApiController>]
[<Route("[controller]")>]
type MessageController (logger : ILogger<MessageController>) =
    inherit ControllerBase()

    [<HttpGet("{id}")>]
    member __.Get (id : int, shouldDouble : bool) : string=
        logger.LogInformation("I am a controller")

        let result =
            match shouldDouble with
            | true -> id * 2
            | false -> id

        sprintf "Hello %d" result

From the function's signature we can see that it has an id and shouldDouble values as inputs and that it returns a string. We have made these explicit however if we were to leave them out it would be fine too as F# would be able to infer the types, see that below:

We can open the following URLs in our browser and should be able to open the /message/3 and /message/3?shouldDouble=true routes and see hello 3 and hello 6 respectively

Note that if not specified the handler inputs will try to be parsed from the body

Now, if we would want to update this to return some sort of general HTTP Response Code when a user sends some kind of input, for example if the result is 4, we will need to modify the function such that we are able to reference the this and the return type of the function is now an IActionResult

[<HttpGet("{id}")>]
member this.Get (id : int, shouldDouble : bool) : IActionResult =
    logger.LogInformation("I am a controller")

    let result =
        match shouldDouble with
        | true -> id * 2
        | false -> id

    match result with
    | 4 -> this.NoContent() :> IActionResult
    | _ ->
        sprintf "Hello %d" result
        |> this.Ok
        :> IActionResult

From this we can see that we are using an additional match to either return this.NoContent() as an IActionResult or this.Ok with the piped message as an IActionResult. Just note that the following matches are equivalent:



// call the `this.Ok` function with
match result with
| 4 -> this.NoContent() :> IActionResult
| _ ->
    this.Ok(sprintf "Hello %d" result) :> IActionResult

// pipe the result of the format through
match result with
| 4 -> this.NoContent() :> IActionResult
| _ ->
    sprintf "Hello %d" result
    |> this.Ok
    :> IActionResult

// pipe the result of the format through on a single line
match result with
| 4 -> this.NoContent() :> IActionResult
| _ -> sprintf "Hello %d" result |> this.Ok :> IActionResult

Post Handler

We can also create a POST handler that will pretty much do the same as the above handler, we can pretty much just take the values from the function body and pass it to the previous handler we put together

Before we can create the handler, we need to create a type called PostData that can be used by the method to receive data, we can define this towards the top of the file, above the type definition for our MessageController. The type also needs to have the CLIMutable attribute so that the JSON deserializer can parse the data from the post body into it correctly

[<CLIMutable>]
type PostData =
    { id : int
      shouldDouble : bool }

Next we simply need to define the Post method with an HttpPost attribute which will just call the this.Get using the input params. this can be done pretty simply too

[<HttpPost>]
member this.Post(data : PostData) : IActionResult =
    this.Get(data.id, data.shouldDouble)

And that's really all that's needed

Conclusion

So yeah, that's pretty much it - Not that bad right? I feel like there are a couple of things that feel a little bit weird because of the pieces of OOP running around from C# that add a bit more overhead than I'd like, but it's .NET, that's inevitable

Still a few more to posts on F# to come, so stay in tuned

Nabeel Valley

Looky, a wild HTML!

Tue, 01 Oct 2019 00:00:00 GMT

Here at last

So, after far too long, we are here. A mostly-functional website, that I have not tested on a single iOS device which may prove to be problematic

Note to self, test on an iPhone or something

For the past six months or so I've looked at myself with a full stack of irony knowing that I, a web developer, do not have my own website. Or well, not one I'd actually want anyone to see (*cough cough* I'm talking to you, Weebly)

The Design Process

Well, the design process, especially when you're trying to satisfy your unattainable standard of perfection, can be difficult. I actually ran through a shocking number of iterations and end-to-end redesigns but I just could not seem to find something that satisfied me. Until one day, it all just sort of hit me at once

None of this nonsensical "cards and shadows" drama, no random floopy animations, none of that, just a blank slate, with some text and imagery.

I think I have a fair stock of those

Design v1

The first iteration that I have saved is this one, I had big dreams, there were going to be things moving all over the place, I was going to break the norm with horizontal scrolling, and just basically turn the site into a slide-show. But I didn't like the lack of fluidity with the overall design. Regardless, here are the first few screens:

During this initial phase I had sort figured out what my key goals were:

Unique experience on every device.
Unobtrusive elements
Clean lines
I needed to take emphasis away from the design, and onto the content

Basically, I wanted something basic

By now I had decided on the three most important things, #E44D90, Montserrat, and Nova Mono, but there's still a lot I was unsure of

Design v2

This version mostly morphed out of the first one, I was trying to address two things primarily

How can I not confuse people
Actually yeah, just the first thing

Personally, I hate this design. But It was definitely valuable in telling me what I didn't want this site to be

I've always disliked cards, and abusing it the way I was in that further deepened my hate. I felt kind of like "Well if everyone else does it, maybe I should give it a shot?", to be fair it's probably not the cards, it's me

Design v3

This is what you're seeing now (Assuming you're reading this in October 2019) I love how this design places a strong emphasis on the imagery and text, there isn't really much else in the way of it aside from occasional underline. Simplicity

I addressed my remaining goals by sticking to a basic site layout with a fairly aggressive grid to deliver interesting layouts for different devices

That being said, and my heart being content. It was time to code

The "Bashing your head against a wall" Process

I used React - Why? Because I used React. Sure, I could have went the Server-Side route and made sure I was sending normal HTML to people, but I wanted a sustainable component library that I could develop as well as super-quick responsiveness on the client, and from experience, I knew I could knock this out fastest with React. That being said, I kind of wish I had looked at setting up pre-rendering before I started, because I feel like it's going to be admin to add after (will basically need to refactor out all the <Suspense/>) tags

Now, as any front-end developer should know in 2019, you gotta start with npx create-react-app my-app and after about three years of node_modules you're ready to go

Content Management System?

I'm not going to go into the details of the development because there isn't really anything too fancy going on here - the site is mostly static anyway. I will however just talk about the Content Management System

Well ..., I think it depends on your view of a CMS. I really didn't want to go overkill on this so I'm making use of a fairly simple system. Each post consists of a post.md and a post.json file

The .md file is the Markdown file, it's the file that I write in as I write the post, and potentially every post in future

The .json file is also very simple at the moment and contains some metadata for the post such as the title and the date. I made this separation so that I would not have to get funky with my file processing, and I'm fairly happy with it

To process and render the Markdown I'm using a library that I fell in love with just over a year ago called Showdown.js, and to add syntax highlighting I'm using some flavour of Highlight.js

I've then simply rendered the content by fetching it from the Folder-cms and passing it through a Markdown Component

import React from 'react'
import { Converter } from 'showdown'
import * as showdownHighlighter from 'showdown-highlight'

import './Markdown.css'

const convertMarkdownToHtml = (text) => {
  const converter = new Converter({
    headerLevelStart: 2,
    parseImgDimensions: true,
    extensions: [showdownHighlighter],
  })
  const html = converter.makeHtml(text)
  return html
}
const Markdown = ({ text }) => (
  <div
    className="Markdown"
    dangerouslySetInnerHTML={{ __html: convertMarkdownToHtml(text) }}
  ></div>
)

export default Markdown

And that's pretty much it. It gets rendered into the page you're looking at now and that's all it takes really. It's a bit of a shame that my syntax highlighting CSS theme doesn't really do justice on JS, another TODO I suppose

Conclusion

There were a couple of new things I learnt during this entire process, simple little things really like how to change the colour of the scrolly thing on your browser, or how to put together a mildly-reasonable file system to store the posts, but more than anything I enjoyed being able to just take something from pure concept to a final build that lines up pretty well with my initial vision

That's all for now, stay tuned, lots more to come. But in the mean time take a look around, let me know what bugs you find (as I said, very untested)

Nabeel Valley

Nabeel Valley

HTML in WebGPU Shaders in Canvas

How Does It Work?

Usability Issues

Technical Challenges

Bad Bots

Symbols on the Ground

Photos from the Nederlands Fotomuseum

Quick and Dirty Object Access in Go

Async TUIs using Bubble Tea

Elm Architecture

A Sad Implementation

A Happy Implementation

Summary

Murder Mystery in Git

Git Tricks with Tri and Difft

Mildly interesting photos from Eid

Join the Webring!

Would You like to Join?

Web Component for Making Patterns

CSS Anchor Positioning

Example

CSS Identifiers

Reading Attributes in CSS

Anchor Positioning

Putting It Together

The Big Example

Accessibility Notes

Further Reading

Generator Generation

Promises

Creating Promises from Callbacks

Async/Await

Promises Vs Async

Promise.withResolvers

Iterators and Generators

Iterator

Generators

Async Generators

Unwrapping the Generator

Inside a Sync Generator

Inside an Async Generator

Creating Generators from Callback Functions

References and Further Reading

Parsing Helix logs in Nushell

Coat Rack

Shader Web Component

Snippet: Rachets, not Levers - Chris Krycho

Week 37, Server Management

Off topic

What I Found

Proxies

Application Management

Côte d'Azur

Week 36, Creativity

What I Found

Strudel

TouchDesigner

Use a password manager, they said

Week 35, Documentation matters

What I'm working on

What I found

Mask

Gauge

Pages CMS

Problems are better left solved

Scripting Text Manipulation

1. Literally just implementing MultiBuffers

2. A language for text manipulation

Bringing it together

Why?

but but but AI

Update 14 April 2026

Update 15 April 2026

Web Authentication API

Example

Code

Exploring the CSS Paint API

The CSS paint function

The Methods Available

The CSS `paint` function

Uncompiled Worker Using `ts-node`