Jacob Bednarz

Validating your OpenAPI specification with reality

2026-01-28T03:35:00+00:00

After spending a lot of time with OpenAPI schemas and trying to answer the question of “are my HTTP requests and responses actually conforming to what my users are sending and receiving?”, I set out to finally build the solution. Like all projects, it started out small, found some pitfalls, and eventually landed on a pretty solid approach that scales to any spec size.

My initial approach was to piggyback on existing log data. The idea was simple enough – parse the access logs, extract the HTTP information, then validate those against the OpenAPI spec. No code changes required, just point it at your logs and let it rip. It didn’t matter the size of the logs or the spec, it would just run in the background and churn through everything as it came in and notify you of any mismatches. The goal here was to provide a service that someone could, even temporarily, send their logs to get a sense of how well their traffic aligned with their OpenAPI documentation.

Note: I’m using Ruby here because it’s nicer to read for this kind of thing, but the concepts apply to any language.

# .. snipped for brevity
log_entries.each do |entry|
  validator.validate_endpoint(
    method: entry[:method],
    path: entry[:path],
    status: entry[:status]
  )
end

This worked…sort of. I could validate that endpoints existed in the spec and that response codes aligned with what was documented. But there were some pretty glaring issues:

This only worked for HTTP requests without bodies (like GET, HEAD) and didn’t allow inspecting the payloads
No way to validate request or response bodies against the schema
No header validation beyond what made it into the access logs
Timing was all over the place depending on log rotation, processing and size of the spec

While sounding great on the surface, this approach quickly fell apart when trying to validate anything beyond the most basic of requests. It was a good proof of concept, but ultimately not viable for real-world usage.

After digging into the problem a little more, I came up with the next iteration: inline middleware that captures the full HTTP context and ships it off to a background job for validation.

The middleware is dead simple - capture everything, enqueue it, and then validate asynchronously.

class OpenAPIValidationMiddleware
  def initialize(app, spec_path)
    @app = app
    @spec_path = spec_path
  end

  def call(env)
    request = Rack::Request.new(env)

    request_context = {
      method: request.request_method,
      path: request.path,
      query_params: request.query_string,
      headers: extract_headers(request.env),
      body: request.body.read.tap { request.body.rewind }
    }

    status, headers, response = @app.call(env)

    response_body = []
    response.each { |chunk| response_body << chunk }

    response_context = {
      status: status,
      headers: headers,
      body: response_body.join
    }

    OpenAPIValidationJob.enqueue(
      spec_path: @spec_path,
      request: request_context,
      response: response_context,
      timestamp: Time.now.utc
    )

    [status, headers, response_body]
  end

  private

  def extract_headers(env)
    env.select { |k, v| k.start_with?('HTTP_') }
       .transform_keys { |k| k.sub(/^HTTP_/, '').split('_').map(&:capitalize).join('-') }
  end
end

The background job then does the heavy lifting with the help of an existing OpenAPI validation library:

class OpenAPIValidationJob
  def perform(spec_path:, request:, response:, timestamp:)
    spec = OpenAPIParser.parse(spec_path)

    request_errors = spec.validate_request(
      method: request[:method],
      path: request[:path],
      query_params: request[:query_params],
      headers: request[:headers],
      body: request[:body]
    )

    response_errors = spec.validate_response(
      method: request[:method],
      path: request[:path],
      status: response[:status],
      headers: response[:headers],
      body: response[:body]
    )

    if request_errors.any? || response_errors.any?
      ValidationLogger.log(
        timestamp: timestamp,
        endpoint: "#{request[:method]} #{request[:path]}",
        request_errors: request_errors,
        response_errors: response_errors
      )
    end
  end
end

The background job approach gives us the best of both worlds:

Zero impact on request latency: The middleware overhead is minimal, just copying data and enqueueing a job
Full validation coverage: We have access to the complete request and response, including payloads or anything the client/server sends
Scalability: Background job workers can scale independently from web workers
Failure isolation: If validation crashes or takes ages, it doesn’t affect the user’s request
Flexible processing: We can batch validations, add retries, or even validate against multiple spec versions for non-production and production versions

The trade-off is that validation isn’t synchronous anymore. If you’re hoping to block invalid requests before they hit your application, this won’t help. But that’s not what this is for. This is about continuous validation of your production traffic against your OpenAPI spec to catch drift between what’s documented and what’s actually happening.

What I’m using this for

The main use case has been catching spec drift. You know, when someone updates an endpoint but forgets to update the OpenAPI spec. Or when a client is sending unexpected data that somehow makes it through. Or when error responses don’t match what’s documented.

You can then collect the validation failures in a metrics system and alert when patterns arise. For example:

If 10% of requests to an endpoint fail validation, something’s probably wrong
If a specific client consistently sends invalid requests, they might need help
If error responses don’t match the spec, the documentation is incorrect and needs attention

It’s also proven useful for finding endpoints that aren’t in the spec at all. Legacy endpoints that nobody remembered to document. Third-party integrations that were “temporary” five years ago.

The real win here isn’t just technical validation - it’s closing the loop between what you document and what actually happens in production. Requests flow through without overhead, specs stay accurate, and you catch drift before it becomes a problem for users. If you’re maintaining an OpenAPI spec, treating it as a living contract rather than stale documentation makes all the difference.

Restricted stock units: the hidden trap

2025-11-25T02:54:00+00:00

In today’s tech-driven job market, Restricted Stock Units (RSUs) are marketed like pots of gold that companies are willing to hand over to the right candidates. Companies love offering them because they can feel generous without requiring immediate cash outlay and it entices employees to have longer tenures and get their cut. Employees often accept them because there is the potential for a big payday if the company does well.

But here’s a hidden truth: RSUs only work in your favour if you treat them like a deliberate investment strategy — not a bonus. My accountant put it best: “If the company gave you an extra 10k in cash, unless your immediate plan was to buy company stock, you want to take the cash. Every time.”

If you don’t understand how they’re taxed, when they vest, how volatility affects you, or how they inflate your taxable income, RSUs can quietly cost you more than they help.

A higher base salary often works out better (especially after taxes)

Many people assume that “$X in RSUs + lower salary” is equal to “higher salary + no RSUs.” It’s almost never that simple.

Why a higher base salary can be better

Salary is predictable. You know what you’re getting and when.
Superannuation contributions (in Australia) are based on salary, not RSU value — so higher salary boosts your retirement savings.
You pay tax as you go, rather than being hit with a lump-sum taxable event on vesting day.
Cash flow is immediate, not tied to a volatile stock.

RSUs, on the other hand, introduce risks:

You’re taxed on the market value at vest whether you sell or not.
If the company stock drops after vesting, you’ve already paid tax on a higher amount than you actually get when you sell¹.
RSUs don’t help with borrowing capacity like salary does (banks discount them heavily or ignore them outright).

When you model the scenarios, a surprising number of people discover that a higher base salary creates more net income, more super, and less uncertainty, especially once taxes are factored in.

Tax hits at vest time (not when you sell) and capital gains apply later

This is the part that blindsides most people.

When your RSUs vest you’re taxed as if you received a cash bonus equal to the value of the shares on vesting day.

Example:

200 RSUs vest
Share price at vest = $50
You’re taxed on $10,000 of income, immediately.

It doesn’t matter whether you keep the shares, the price drops tomorrow or you actually wanted the stock. You owe the tax man now².

Then capital gains tax applies when you sell. After vesting, your cost base is the market value at vest.

If the stock goes up, you pay CGT on the gain.
If the stock goes down, you’ve paid income tax on “phantom value” you never really benefited from.

This double-tax structure (income tax now, capital gains tax later) means you need to manage RSUs proactively, not passively.

RSUs can cause hidden headaches with Centrelink and other income based benefits

Because RSUs are taxed as income at the moment they vest, they can unexpectedly inflate your taxable income for the year. This has real consequences beyond your immediate financial picture.

Services such as:

Family Tax Benefit
Child Care Subsidy
Parenting Payment

…all rely on your adjusted taxable income (ATI).

A large RSU vest can push you into repayment territory or out of eligibility altogether potentially causing an unexpected reconciliation debts months later.

And the worst part? Centrelink doesn’t care that you didn’t receive cash — you’re assessed on the taxable income, not liquidity. RSUs are counted for all these calculations, even if you never sold a single share.

So when do RSUs make sense?

Now, it’s not all doom and gloom. RSUs can be a valuable part of your compensation but you need to be intentional about how you handle them.

They make sense when

You have the cash flow to cover the tax at vest.
You plan your sales to manage CGT efficiently.
You treat RSUs as part of your investment strategy — not a surprise bonus.
Your employer’s stock is stable, or you diversify immediately.
You understand and model how RSUs affect subsidies, benefits, HECS, and overall tax position.

They don’t make sense when

You assume “free money” without planning for tax.
You’re depending on steady income and low volatility.
You’re eligible for income-tested benefits and want to avoid large Centrelink debts.
You treat RSUs as part of your salary rather than as a risky investment.

While RSUs are over hyped with plenty of hidden nuance, they are powerful wealth-building tools only if you understand how they work. For many people, especially those who prefer predictable earnings, a higher base salary is far safer, easier to manage, and often more profitable.

Yes, you do get a capital loss if the stock drops below the vest price, but that only offsets capital gains, not the income tax you already paid. So if you had no other capital gains that year, you’re out of pocket on the income tax side. ↩
Or there abouts to the current financial year. ↩

Skip the hype, sell the shovel

2025-10-16T03:18:00+00:00

In every generation, a new gold rush emerges.

It might be cryptocurrency. NFTs. A new social media platform. Or whatever hot AI trend is flooding your feed today. In each of these hype cycles, there’s a crowd rushing in to stake their claim. Most are driven by dreams of fast success, only to discover that hype burns bright — but often burns out even faster. But here’s the truth: the real winners are rarely the ones chasing the gold.

Let’s rewind to the 1800s.

In 1848, gold was discovered at Sutter’s Mill in California. Within months, hundreds of thousands of people from around the world poured into the region, desperate to dig their way to riches. This massive migration is what we now call the California Gold Rush.

Some did strike gold. Most did not. In fact, the vast majority of prospectors ended up broke, exhausted, or died from sickness or starvation.

There’s one person who became California’s first documented millionaire. But here’s the twist: he never mined for gold. His name was Samuel Brannan and he got rich selling picks, shovels, and supplies to the incoming wave of miners. He understood something most didn’t. It’s not always about digging for gold. Sometimes, the smarter move is selling the shovel.

Whenever there’s a wave of hype, the instinct is to dive in headfirst. Everyone’s talking about it, everyone’s investing, everyone’s launching something. And while some people do get lucky, the odds are slim, and the competition is fierce. Instead of diving in, observe the direction of the wave and build something that supports it. You don’t need to be at the center of the storm to profit from it. In fact, being at the edges often gives you more stability and less risk.

Creator economy? Instead of becoming an influencer, some built software that powers content creation, monetization, or audience engagement.
Cryptocurrency boom? While many speculated on coins, others built platforms, wallets, analytics tools, and payment infrastructure that could be used by anyone.
AI explosion? For every AI product, there’s a need for compute power, compliance tools, and API management.

Throughout history, the pattern repeats itself. The people who thrive long-term often build what the dreamers need to chase their ambitious goals¹.

There’s power in being adjacent to the action².

Scalable demand: As more people enter the hype, they need tools, services, or infrastructure to participate which is exactly what you’re offering.
Strategic leverage: You’re not competing with the hype, you’re enabling it. That means that when the hype succeeds, so do you. If it doesn’t, you can pivot to the next wave with your established customer base and expertise.

Just like Samuel Brannan did with picks and shovels, your path to success might not lie in chasing the gold. It might lie in helping others chase it.

So next time you see a gold rush forming, ask yourself: “What are they going to need and how can I be the one to provide it?”

Exceptions to this rule exist, of course. However, those exceptions are usually executed by people or companies who have deep expertise in a field or unique fundamentals that give them a significant edge over the competition. ↩
The idea of being “adjacent to the action” is inspired by the book The Adjacent Possible by Steven Johnson, which explores how innovation often happens at the edges of existing systems or using a combination of existing components rather than at their core. ↩

What Emperor Penguins taught me about building high performing teams

2025-09-17T01:05:00+00:00

In the frozen wilderness of Antarctica, emperor penguins do something extraordinary. Faced with wind chills that would kill most animals in minutes, they ensure the continuation of their species by forming tightly coordinated, groups that move as one.

When building high performing teams that need to operate in mission-critical environments, there’s a lot we can learn from these remarkable birds.

Trust Is the foundation, not a bonus

When emperor penguins form a huddle, each bird relies on the group for survival. There’s no micromanagement, no hierarchy. Just innate trust that each member will take a turn at the edge and that no one will be left behind.

In high performing teams, trust isn’t earned over time. It’s given early and strengthened through action. Team members:

Assume competence in each one other.
Share information openly.
Rely on one another in high-stakes situations without second-guessing motives or skills.

If your team doesn’t feel safe enough to rotate to the cold edge of the storm, you don’t have a high performing team — you have survivors. And survivors burn out.

The huddle: a living system of support

Penguin huddles aren’t static; they move in subtle, constant rotations. Each bird gradually shifting positions so that everyone gets equal exposure to the cold and warmth. Without the support of the huddle, they wouldn’t be able to survive due to not having enough in their fat stores to survive the storms alone.

This is exactly what high performing teams do:

They rotate responsibilities, spreading load fairly.
They support each other actively, stepping in before burnout happens.
They create fluid leadership dynamics, where roles shift based on context, not hierarchy or title.

You don’t need rigid structures when the team knows how to move together. You need awareness, empathy, and shared rhythm.

Shared purpose is the anchor

In the centre of a penguin huddle is something fragile and precious: the egg. It’s the reason the huddle exists. Every movement, every rotation, every act of endurance is done in service of protecting that purpose. The fathers must ensure the egg stays warm and off the ground or else lose their only offspring for that season.

High-performing teams are anchored the same way. They aren’t high performing because of perks, personalities, or process. They’re high performing because they know exactly what they’re protecting:

A mission-critical system.
A customer experience.
A promise to the public.

That clarity allows for autonomy. Teams don’t need to be told what to do. They intuit what to do, because they know what matters most. Unclear priorities or ever shifting focus is a deterrant to high performing teams.

Psychological safety (in the storm)

In Antarctic storms, visibility drops, winds surge, and the cold penetrates everything. Emperor penguins survive because they never break the huddle. They can’t afford to question if someone will do their part; they just do it, and trust others will, too.

In teams, especially under pressure, psychological safety is the equivalent. It means:

People speak up when they see a problem.
Mistakes are discussed, not punished.
Feedback flows without fear.

Without that, the team scatters under pressure. With it, they stay tight and move through the storm together.

Building the huddle takes intentional work

Penguins don’t randomly survive. Their huddle behaviour is evolutionary, practiced, and refined.

Similarly, high performing teams don’t just form—they’re built:

Through clear norms (how we work, talk, and decide).
Through shared experiences, especially hard ones.
Through rituals that reinforce connection: standups, retros, 1:1s, and even moments of celebration.

You can’t brute-force your way to trust. You build it by showing up consistently for each other, for the mission, and for the long winter ahead.

Survive alone, thrive together

The emperor penguin doesn’t survive alone. It survives because it belongs to a system where trust, shared purpose, and coordinated action are non-negotiable.

If you want to build a team that can withstand high stakes, tight deadlines, and constant pressure — you need more than skill. You need trust like the penguins have: deep, mutual, and weather-tested.

Because in the end, high performance isn’t about speed, or even individual brilliance.

It’s about knowing that no matter how hard the storm hits, the team will hold.

Automatically generating Cloudflare’s Terraform provider

2024-09-24T00:00:00+00:00

Lessons from building an automated SDK pipeline

2024-04-23T00:00:00+00:00

Gradual deployments, Rate Limiting, and new SDKs

2024-04-04T05:40:00+00:00

Lucky blueberries

2023-07-02T23:07:00+00:00

At NASA’s Jet Propulsion Laboratory (JPL), there is a tradition for “lucky” peanuts to be present at every launch. The tradition dates back to the early Ranger launches where a series of missions were either delayed or unsuccessful.

Ranger 1 (Launched Aug. 23, 1961) Launch vehicle failure; spacecraft could not leave Earth parking orbit
Ranger 2 (Launched Nov. 18, 1961) Launch vehicle failure; spacecraft could not leave Earth parking orbit
Ranger 3 (Launched Jan. 26, 1962) Radio contact lost; missed the Moon by about 22,900 miles (about 36,800 km)
Ranger 4 (Launched April 23, 1962) Onboard computer failed; impacted the Moon on April 26, 1962
Ranger 5 (Launched Oct. 18, 1962) Radio contact lost; missed the Moon by 450 miles (725 km)
Ranger 6 (Launched Jan. 30, 1964) Reached the Moon but cameras failed; impacted the Moon Feb. 2, 1964

On the morning of the launch for the Ranger 7 mission, a trajectory engineer, Dick Wallace, passed them peanuts in hopes of calming people’s nerves. When this Ranger performed flawlessly, a new JPL tradition took hold, and now peanuts are passed around the laboratory’s mission control centre during critical mission events.

So, what has this got to with lucky peanuts?

When we were preparing for the birth of our first child, we were told early on that first time mothers generally labour longer and that it is critical to stay hydrated and if possible, have small amounts to eat to have enough energy to keep going. In the weeks leading up to the due date, we made sure we had plenty of food for my wife and myself in the event we needed to grab something and go to the hospital during the night.

The big day finally arrived and as planned, I kept asking my wife if she would like to eat anything but each time, she immediately declined saying she didn’t feel like it. I was persistent so about an hour into labour, I offered her some blueberries and instead of declining she accepted.

Not even 6 hours after labour starting, we had our son (at home, totally unplanned). The midwife and attending paramedics were surprised at how calm the environment was and how smooth the birth went given the constraints. It was a pretty amazing time to say the least.

Much like the teams at JPL during critical missions with their lucky peanuts, I know the blueberries didn’t directly impact the outcome given the amount of work and preparation that goes into these type of events but, instead a new tradition to be honoured and carried forth.

So with our daughter due in just over a month, I’ve added lucky blueberries to our shopping list and even if my wife doesn’t accept them to eat this time round, they will be present at the labour. Just like the lucky peanuts.

Stop trying to convince people

2023-03-01T06:33:00+00:00

For much of my career, I’ve been in positions where I was sought out to provide guidance for larger or more complex tasks. That could be a functional spec for overhauling an existing system, a new vendor assessment or even how to approach a difficult or hard to trace bugs. This was all something I enjoyed as it gave me variety and breadth into the types of work I was doing day-to-day. Along the way, I built up common yet reusable processes for each of these tasks that allowed me to have a consistent and reproducible approaches to solving each of these issues.

Later on, I started offering my services to other companies who didn’t need someone like me full time but still needed someone in an advisory role on occasion. For the most part, this was great. I’d get insight into the problem, we’d chat about what options they would like to pursue and then I’d help them get there. However, there were a few times a company would get me in, talk about a problem, which options they would like and instead of executing the plan, nothing happened. I would follow up with questions about whether they were stuck, or we didn’t cover a particular portion, with the intention of making my value justified and not just another waste of money. In all bar one case, the responses I got were all along the lines of “Thanks for your help, we’re going to put this on the back burner for now but use this in the future”. Sure, it was a small portion that got to this point but this always frustrated me because I felt like the client had wasted their time, which was part of the reason I was there.

One day, while helping a friend out who was head of the technical direction at a startup, I got one of these responses from an engineer under him. Having the good rapport we did, I asked him “You’ve just spent all this money on getting me in, and now you’re not doing X. What can I do to help get this over the line?”. We went back and forth having a good converstion and towards the end he said “Well maybe, it was your job to plant the seed but not watch it grow”. Despite being a little philosophical, it gave me reprieve that regardless of the outcome of the engagement sometimes it just won’t get to execution – and that is totally fine for me and for the company.

I didn’t think too much more about that exchange until I was reading William Ury’s “Getting Past No” which covers negotiating with others that may be initially more on the adversary side. I won’t ruin the book for you however, one of the chapters focuses on the psychological ramifications of what happens when you use power to tell someone want you want. In a nutshell, because it is an external factor to their own decision making process, they are more adverse to it. The solution? Plant a seed for them to get where you want them to without directly telling them. For me, this was a lightbulb moment on a couple of levels. The first was with negotiation but the second, and more important to this story, is that my advising role was never about getting the outcome. It was about planting the seed so that in time they would come around to the decision on their own and want to run with the tasks. Further still, planting enough seeds so long after the engagement was finished, the company would still be benefiting because someone else would be pushing for the outcome.

While the book mentions it as a negotiation tactic, it is is something that I now (try) to apply more broadly. It also lends itself nicely with situations where the outcome isn’t immediately achievable due to other competition - priorities, budgets, etc. If you put effort into planting the seed, when there is an oppurtunity for it, you have the best chance of success to be picked up.

So, I ask you. Stop trying to convince people, instead plant the seed.

Terraform Providers and the Terraform Plugin Framework with Jacob Bednarz

2023-02-11T00:00:00+00:00

Piggybacking on container networking for debugging

2022-11-24T02:05:00+00:00

I regularly need a way of running a container (mitmproxy or tcpdump) against another running container.

Sure, you could stop the container, update the Dockerfile and build in your dependencies but this approach is very cumbersome and slow on most projects. Instead, how about just running a new container and attaching it to the existing Docker network?

To get started, build the container you want to run. Here is a way to build a tcpdump container.

$ docker build -t tcpdump - <<EOF
FROM ubuntu
RUN apt-get update && apt-get install -y tcpdump
CMD tcpdump -i eth0
EOF

Easy right? We have an image we can now use in Docker.

$ docker images --format "table {{.ID}}\t{{.Repository}}" | grep tcpdump
579f886818cc   tcpdump

For examples sake, let’s say we want to inspect some traffic on my Resque container. The container name is resque and we can piggyback onto it using the --net=container:<name> flag on docker run ... <image> command.

$ docker run --tty --net=container:resque tcpdump

And boom, we’re using the image’s default entry point of -i eth0. How easy was that?!?!

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
03:15:28.835766 IP 130330b90814.44222 > redis.dev-stack.6379: Flags [P.], seq 1560375946:1560375984, ack 3154187234, win 502, options [nop,nop,TS val 1153233662 ecr 4097934334], length 38: RESP "SMEMBERS" "resque:queues"
03:15:28.837174 IP redis.dev-stack.6379 > 130330b90814.44222: Flags [P.], seq 1:18, ack 38, win 509, options [nop,nop,TS val 4097939341 ecr 1153233662], length 17: RESP "default"

What about customising it, so we’re only seeing port 6379 traffic with ASCII output?

$ docker run --tty --net=container:resque tcpdump tcpdump -A 'port 6379'
 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
03:18:14.597643 IP 130330b90814.44222 > redis.dev-stack.6379: Flags [P.], seq 1560378553:1560378591, ack 3154187960, win 502, options [nop,nop,TS val 1153399433 ecr 4098100104], length 38: RESP "SMEMBERS" "resque:queues"
E..Z..@.@...... ........].|.........X}.....
D.z..D..*2
$8
SMEMBERS
$13
resque:queues

Astute readers will see the duplication tcpdump in there but it is intentional. The first is the image to use, and the second is the entry point override, here -A (for ASCII output) and then the filter of port 6379.

No redis-cli? No problem

2022-11-21T03:43:00+00:00

Today while performing some basic networking connectivity tests between Kubernetes namespaces, I found myself without redis-cli which is what I’d usually reach for to test connectivity to a Redis instance.

The container also didn’t have nc which left me in a bit of a bind as to what to do. I could have just installed the packages and shipped a new image but instead, I opted to use what I had: bash and exec.

I ended up with the following incantation which relies on abusing using bash’s inbuilt /dev/tcp device file for opening a socket to the hostname and sending the request.

exec 3<>/dev/tcp/<hostname>/<port> && echo -e "PING\r\n" >&3 && cat <&3

Command break down

`exec 3<>/dev/tcp/<hostname>/<port>`	This causes file descriptor 3 to be opened for reading and writing on the specified TCP/IP socket. This is a special form of the exec statement.
`echo ... >&3`	Build the request you intend to send to the remote. In my case, PING followed by a carriage return and newline.
`cat <&3`	Reads the response back to write out.

While used for Redis here, this approach can also be applied to UDP (swap to /dev/udp ) and HTTP requests (build a HTTP compliant request)!

Less exotic alternatives

# nc
(printf "PING\r\n"; sleep 1) | nc <hostname> <port>

# telnet
telnet <hostname> <port>

Migrating to auto-generated Terraform Registry documentation

2022-06-07T04:27:00+00:00

For the longest time, Terraform Registry documentation has been manually updated as the resource is created or modified. This has always been painful to remember and keep in sync given the number of sources of documentation already out there for the Cloudflare provider. However, that is all changing with terraform-plugin-docs.

terraform-plugin-docs uses the provider schema to generate consistent and automatic documentation. As of v0.9.0, the tool is mature and configurable enough for most providers. So, how do we migrate to it?

Repository changes

At a high level, terraform-plugin-docs has some inbuilt concepts for the directory structure.

docs/{resources,data-sources,guides}: This is where the generated output ends up.
examples/{resources,data-sources,guides}: Sub directories of examples. Each directory contains the Terraform resource examples that are used for the “example usage” sections of the documentation.
templates/{resources,data-sources,guides}: Templates are the files used as a blueprint to generate the outputs. In this directory, files with the extension ".md" (markdown) are copied over verbatim whereas files with the extension "md.tmpl" (templated markdown) are parsed through Go’s text/template package before rendering.

The Terraform documentation touches on some of these steps, however, leaves alot to be desired when it comes to migrating from an existing setup. Here are the steps I took:

Move website/docs to a top level templates directory. This will allow us to copy over the existing documentation verbatim.
Expand inner subdirectories to be their full representations.
- docs/d => docs/data-sources
- docs/r => docs/resources
Change file suffixes from .html.markdown or .html.md to .md.
Rename all files to match the resource/datasource names. This is important as otherwise, it will try to generate them all and you’ll end up with duplicate documentation.
Moved contributing docs to a top level directory.
Add make targets for generating documentation.

For a Pull Request demonstrating the change, check out cloudflare/terraform-provider-cloudflare#1680.

Using this approach allows us to keep our existing documentation until we explicitly cut over the resources to automatic generation.

Resource/datasource/guide changes

As for individual resources, there are only two (well, technically 3) steps:

Update the schema fields to be decorated with Description attributes.
Delete the old documentation from the respective templates directory.
Run make docs and commit the changes.

Again, for a Pull Request demonstrating the change, have a look at migrating cloudflare_rulesets via cloudflare/terraform-provider-cloudflare#1677.

Home network

2021-12-15T06:02:00+00:00

Since we purchased our new house in 2015, I’ve slowly been building up the home network and the components in it. Until this earlier month, majority of my network was run by Google Wifi and for the most part, it worked great. Unfortunately, between Google making some product changes which I didn’t really agree with and the more complex configuration, I outgrew it. Google Wifi is still my recommendation to family and friends who want a simple yet pretty powerful mesh and router combo.

Now, we’re a part of the Ubiquiti ~~cult~~ family.

Initially I wasn’t going to go all in on their gear due to the extra cost, however, after discovering how well it all integrated together, I was sold. I’m a sucker for when things Just Work™. Along side this, the other selling point was how well the UI/console makes some very complex topics straight forward to understand and use. As a bonus, you can SSH into the Ubiquiti hardware if you need to pull logs or do something complex the UI doesn’t expose.

House is wired with cat6 cable

The sweatiest portion of the upgrade; getting into my roof and running cat6 cable everywhere. Unless you have a need (like I did) I wouldn’t be too phased if you already have wired cat5e. It will work almost as well for most connections.

Wireless Access points

I opted for a mix of Ubiquiti in wall and in wall HDs depending on the need for wired connections in most rooms. For instance, my lounge room doesn’t have any wired connections so a regular is fine there however my office has a HD to connect up to 4 other wired connections if needed.

18U rack enclosure on casters

I purchased this quite early on and was glad I opted for the slightly larger unit given some of my tech wasn’t initially rack mounted and took up a bit more room. Nothing too fancy here but it keeps all my hardware less dusty and cooler than sitting on top of one another in a cupboard somewhere.

Keystone patch panel

Truly a MVP for the revamp. While looking at patch panels, I stumbled upon a video that showed a modular (or keystone) patch panel and I was instantly hooked. Unlike other types, this doesn’t require punching down the cables and instead, uses existing keystone connections to mount into the patch panel. This also means if I ever want to put a RJ11, HDMI or any other keystone connection into the unit, I’m not going to have to install a different blanking or vent plate to support it.

Ubiquiti Dream Machine Pro

Probably the most expensive part of the build and definitely the most exciting. I had been eyeing off the regular UDM for the better portion of 6 months after my Google Wifi configuration started to have issues with IPv6 and I wanted to replace it. Lots of people I spoke to loved their UDM however a couple mentioned they had random hardware issues that required a replacement unit. I nearly pulled the trigger on getting on until Ubiquiti announced a V2 model would be replacing it in the near-ish future. I decided to hold off until that was released but in the meantime, I managed to find a UDM pro locally for a bargain and couldn’t resist. Despite the UDM pro SE being available, the differences are negliable for my intended use so I didn’t bother with the “upgraded” version.

My configuration is primarily managed using Terraform and the Unifi Terraform Provider but in a nutshell, it looks like this.

Multiple LANs for devices based on their usage and what their intended network profile is. I.e. NAS dedicated LAN and can only connect to backup targets, IoT LAN for things like Roomba which cannot connect to the internet without manually adding a rule for it, Guest LAN to keep people off the internal devices like my NAS or computers.
VPN configuration on a single LAN for easy swapping or testing. This is still in the early days of use but it allows me to quickly jump onto a Wireguard VPN connection without any additional setup on the device.
Backup 4G LTE internet. Automatically fails over to connection when my primary internet connection fails (more on this below).

USW 24 port PoE switch

To support the other Ubiquiti hardware like access points and cameras, I added a PoE enabled switch. By default, the UDM pro doesn’t have PoE ports (SE does) so unless I wanted to add power inline to these devices somewhere, I needed a switch.

Netgear LB2120

While I have access to the NBN, I have always had a backup connection. This means that if the NBN or my ISP specifically goes down, I still have a connection. Up until now, the workflow was a little something like this:

Notice apps/webpages aren’t loading
Open Google Wifi, check mesh connectivity
Open modem, check internet connectivity
See internet failure, swap 4G/5G modem out for the modem and wait for the IP to be assigned

Not horrible, but definitely could use less human intervention.

With the UDM pro, port 10 is a SFP socket that, with the right adapter, (Ubiquiti RJ45 – SFP Transceiver Module, SFP to RJ45 1G) becomes a backup WAN port. Once the adapter is inserted the SFP socket turns into a standard RJ45 Gig-E port which a 4G/5G modem can stay plugged into. After some searching, the best bang for buck 4G modem I found was the LB2120 from Netgear. It doesn’t do a lot (which is perfect) but it does work in bridge mode which allows the UDM to do it’s thing in the event of a failed NBN connection.

Now I don’t even know when my primary connection falls over as the 4G automatically kicks in without my intervention.

Synology 420+ NAS

I’ve previously blogged in detail about how I use my NAS for my home setup. Check it out if you’d like full details.

APC 2U UPS

My need for a UPS came a few years ago when we started having grid upgrades in my area. What felt like every couple of weeks, we’d either have brown outs or total power outages resulting in things like my NAS abruptly shutting down. While the NAS survived a couple, I didn’t want to tempt fate more than necessary so I forked out for a 2200VA UPS. This doesn’t power much but it does have enough to hold power while my NAS shuts down gracefully and the network runs for about 3 hours. After that, it’s time to bust out the petrol generator (which luckily I’ve only had to do once in three years during storms).

I first purchased a non-rack mounted one but ended up swapping it to a 2U UPS in order to save a bit on the space side.

i7 16GB RAM NUC

I picked up this NUC off Facebook Marketplace for a bargain as it was thought to be completely busted. I replaced the RAM and came back to life so for once, a win on Marketplace. It currently runs nixOS (I’m wanting it full time eventually!). Previously, it was dual booting Debian and Windows as a test machine for my personal projects and cross compiled tools. On the device itself, I mainly just have a bunch of Docker containers running for things within my network. Grafana, Smokeping, etc.

Mac mini

Purely an Apple cache server and a playground for early Apple releases. Used to use it more when I was maintaining Boxen and needed to constantly wipe the OS to a clean slate but now is largely under used. The biggest advantage is the devices wanting to update are already on my network so the upgrades are faster and generally behind the scenes without anyone even noticing.

Raspberry Pi 4

Relatively new entrant running as a PiHole providing DoH for the network. (Scott Helme has a great article if you’d like to do this too)

Misc

Lastly, I’m working on a custom 3d printed 2U rack plate for the Mac mini, NUC and the Raspberry Pi to store them more efficiently than on their current shelf. By the end, I’m hoping to have it look like a HDD bay instead of a collection of different devices.

Terraform provider development overrides

2021-04-11T23:12:00+00:00

If you’re working on a Terraform provider, chances are at some point you want to run in against your configuration before pushing the changes publicly. Until recently, this involved sketchy and error prones solutions such as (ab)using the filesystem mirror directives or modifying your $PATH. These approaches were tedious and in some cases resulted in more debugging that what the code change involved.

Good news is that Terraform 0.14 has addressed this using development overrides for provider developers. The design feature is designed in such a way that you don’t need to fiddle with the filesystem and instead you toggle it using an environment variable.

Setting it up

Inside of our configuration file, we use HCL to define our provider override. Below you can see I’m overriding cloudflare/cloudflare which is the provider I’m working on. The provider name is keyed to the file path where the compiled binary is outputted; in my case, inside of my $GOPATH.

provider_installation {
  dev_overrides {
    "cloudflare/cloudflare" = "/Users/jacob/go/src/github.com/cloudflare/terraform-provider-cloudflare"
  }

  direct {}
}

Don’t forget to include direct {} at the bottom to allow other providers to be pulled in as normal. Without this, you will only be able to use those providers defined in the dev_overrides block.

I put this file at ~/.config/terraform/devrc for reference.

Now that we have that file in place, all we need to do is invoke our Terraform operations prefixed with TF_CLI_CONFIG_FILE=~/.config/terraform/devrc and boom! Terraform is using your development provider instead of the released version.

Personal data backup plan

2020-12-10T21:00:00+00:00

Like most of you reading this, I have important personal data scattered around on various devices (external hard drives, iOS devices, The Cloud™) and until recently, this had not bothered me too much. I take good care of my physical hardware and the cloud services I use have redundancy and durability built in. That was, until I got a text message from a friend who runs his own business earlier this week that yet another one of his server drives had failed and he was needing a hand to replace them. I’m not going to go into details on why these keep failing but I will say that this is the 5th drive in 3 months that has experienced a failure. While I can’t really explain why, this triggered me to start thinking about my own personal data backup plans.

The data I’m talking about here is stuff like family and vacation photos, contracts and legal documents. While the last two are also physically available elsewhere, I didn’t have any redundancies in place for my photos. I got to thinking about this and after a short while realised I needed to do something as I would be gutted if I was to lose those since there is no guarantee I would get them back if the devices were to fail.

So down the rabbit hole I went. As is pretty standard practice for important data, I chose to implement a 3-2-1 backup strategy. From the linked article, the strategy is summarised as

Three total copies of your data, two of which are local but on different mediums (read: devices), and at least one copy off-site.

For me, I decided I would opt for:

a local Network Attached Storage (NAS) device
a cloud provider
a physical offsite replication

This is in addition to the data on the initial devices themselves.

Network Attached Storage (NAS)

A NAS can be simplified to a collection of hard drives that are configured to either split or mirror data between them however present it in a unified fashion through a file system GUI. The intention is that you build the hard drive configuration in whatever way that suits your data and data redundancy needs.

Despite working in tech, I’m not a big “home tech” sort of bloke. This picture sums up my position pretty nicely.

As this would be the first NAS I’ve owned from new, I decided to scope out the market for my budget and needs and ended up settling on the Synology DS420+. There are plenty of great NAS brands out there however Synology provided some features that I couldn’t find in others and have a good reputation to back it all up so it was a win-win for me. To work out how many drives I needed, I had to first work out what configuration I would be storing the data in. After checking out the Synology RAID calculator (awesome resource even if you don’t use a Synology device!), I landed on RAID6 which would allow meet my durability and availability goals with enough redundancy built in that multiple drives could fail and I would still be able to rebuild the array without needing data recovery.

For the drive sizes, I collected all the important data I intended on storing in there and multiplied it by 5-6 times to factor in growth over the next 5 or so years and then purchased drives that made up that space. It’s important to decide on your RAID configuration before doing this as that determines how much usable space you will have available to you. If you decide to take this route yourself, be sure to get NAS specific drives as they are optimised for that particular workload and you’ll get better performance any longevity out of them.

Once the NAS and drives arrived, I configured it and started copying in the data feeling instantly better I had at least another copy.

Cloud provider

While I don’t have a “favourite” cloud provider (read: all of them have pros and cons with great uses for each) I tend to sway towards Amazon Web Services (AWS) as I got on board when Google Cloud Provider (GCP) didn’t have a huge range of features and I was already using it pretty heavily in my day-to-day work. GCP also doesn’t have anything that would cause me to up and jump ship at the moment. However, I didn’t chose either of these options for my cloud backup. Why? The balance of cost and data retrievability. In both AWS and GCP, you can store data cheaply and with excellent redundancy out of the box however they charge a fee for retrieving “cold storage” items which can quickly add up and have a slow turn around time on them.

Looking around the web, I stumbled on Backblaze. They have a few products but the one I ended up using was B2 Cloud Storage as they were the same price as the big cloud providers but didn’t charge any extra for quick recovery. This was a win in my books as it also allows me to validate the backup is working at any point in time without a delay.

With the Backblaze account created, I hooked up my NAS to automatically sync to Backblaze using Synology Cloud Sync. This is an automated process that watches certain directories for changes and then syncs them accordingly based on rules you create. I specifically made it a one-way sync so that even if I accidentally delete something on my NAS, it won’t be replicated to the Backblaze account.

During my evaluation phase, I also found that Backblaze offers shipping you the data on a USB device if you ever need the data back and it’s too large or slow to download. The great thing is, providing you return the device within 30 days, they refund the cost!

Physical offsite replication

This portion of the plan was something I ended up changing later in the process. Initially I was just going to buy another drive and have it constantly sync however a good friend of mine ended up also purchasing a Synology NAS around the same time which opened up the possible routes I had here. We ended up configuring Hyper Backup which allows syncing of one NAS to a multitude of other services (including another NAS). This meant I’d have another physical copy of the data within a short driving distance and it only cost me another drive. A bonus here is that hyper drive allows encryption of the synced folder which means that while he physically has access to the data, he (or anyone else for that matter) is unable to read the data I’m backing up there.

Other bits

Since I spent the time and money on a NAS, I wanted to ~~abuse~~ take full advantage of the capability. In addition to storing my important data, I also set up:

Backups for all the laptops and iOS devices in the house. Everyday at 1am, all of our devices will sync to the NAS making a restore reasonably quick when they are needed. It also means we don’t have to think about doing backups as they are automatically handled on a schedule.
I’ve added a list of “known backup devices” which is a collection of other storage mediums we have which when plugged into the NAS, automatically sync. Similar to the laptops and iOS devices, this means that backups automatically happen out of band without someone needing to explicitly do it.

In an upcoming article, I’ll go over the improvements to my home lab redundancy which the NAS helped shape and improve since it was a key piece of the infrastructure.

Surviving on-call

2020-05-12T21:23:00+00:00

For the majority of my engineering career, I’ve been on-call in some capacity. Whether it is looking after systems serving 50k rpm and customer facing, an internal service only handling internal traffic or serving as a subject matter expert for a technology in a bunch of systems I’ve rarely touched.

Much of the experience was trial by fire (my first couple of months at Envato included one of the longest sustained DDoS attacks we’ve seen), despite this I’ve managed to come out of it with lessons and tips I’d love to share.

Look after yourself

It should go without saying but here we are :slightly_smiling_face: Get enough sleep, eat properly and make sure you’re getting enough exercise. While it’s important to maintain this in your normal routine, it becomes increasingly important when you’re under a bit more stress than usual. You don’t have to be a power-lifting chef but you should be avoiding the usual unhealthy food suspects and getting at least some exercise to disconnect.

Don’t wait for alerts

My number one tip. Don’t sit around waiting to be paged. Being on call doesn’t mean you’re under house arrest, or even that you can’t do other activities. The premise of being on-call is that someone is available to take on emerging issues when they happen. Don’t sit on Slack (or whatever other team messenger tool you use). Don’t sit on your email. I’ve taken alerts on the side of the road, while herding cattle and while riding dirt bike trails. None of these scenarios changed the outcome of the event and if anything, helped formed the mindset that on-call isn’t an impediment on my day to day life but just another part of it.

Be prepared

Over the course of my on-call career I’ve come up with a few tools that have made my on-call experience more enjoyable and less nerve racking. Now, I want to preface this by saying that the tools don’t make the craftsman. You can do on-call without ANYTHING in this list but, if you can afford it and want to make you’re on-call experience that little bit better, this is a great start.

An on-call bag. Yeah, not the most fashionable thing but if you have a single place where your on-call stuff all lives, you can grab and go as needed. The only thing that I need to add to my on-call bag is my laptop which allows me to quickly grab it, put it in my bag and then take my bag knowing I have everything to be prepared for a call out. For a long time, I had the Cocoon Slim backpack which also had a nice Grid-It securing system in the front which worked a treat. I’ve since moved onto a Dakine backpack which I also use for air travel carry on and snowboarding. Any backpack will do though.
An internet connection. Pretty self explanatory. I use my iPhone via tethering but others have used dedicated hotspots (like MiFi).
Corded headphones. I’m a big fan of my Bose QC35s for day to day use but when I’m on call, I need something that doesn’t rely on batteries and can be used in my iPhone and my laptop. For this reason, I use the previous generation Apple earphones with the 3.5mm jack. Then, when I need to use it on my iPhone, I connect the 3.5mm jack to lightning adapter.
Battery pack. For whatever reason, the universe always decides the day you’re going to get a longer call out is also the day your devices aren’t going to be fully charged. I added a battery pack large enough that would also be able to charge my 13” macbook pro. This way, if I’m away from an outlet I can pop it onto the battery pack and get another couple of work done.
Charger for laptop. Complementary to the previous point. You’ll often be within reach of an outlet; take a charger and plug in where you can.
A mouse. I’m not a track pad fan, so any call out over a few minutes can be infuriating to be stuck on the laptop track pad while trying to fix issues.

The biggest improvement from this list was putting everything in a bag as I was able to sit that near the door and when I left the house, grab it and go without needing to think too much about it.

Use your secondary

Every on-call should have a primary and a secondary. This ensures that if the primary misses an alert, the alert is automatically escalated to another person to look at. However, secondaries shouldn’t only be used in those scenarios where the primary misses the alert. Sometimes life happens. Sometimes you can’t plan ahead. And sometimes, you just need a break (especially on those hectic shifts). Use these times to poke your secondary and ask if they can take the override and share the load. I’ve seen far too many people not utilise a secondary and end up in a situation where they weren’t fully present and end up making mistakes that draw out incidents because they didn’t think they needed to schedule an override. This is give and take so if you’re constantly offloading to a secondary, you need to pay it forward.

Have a preflight checklist

Do you know why first responders, pilots and astronauts all use checklists? It’s because they proven time and time again to be effective at mitigating human error and limiting on the fly decision making. Using a checklist ensures that when someone is going on call you can confirm the required access is assigned, silly or commonly overlooked things are covered explicitly and important details such as escalation or communication processes are made clear to the incoming on-caller. Regardless of how often you’re on-call, you should be going through the checklist every time. This ensures people don’t get complacent, miss critical updates and keeps the checklist self-updating.

Have you got something that you can’t do your on-call shifts without? Got a ProTip™ I missed? Let me know! I’d :heart: to hear them.

Organisation of information for my workflow

2019-12-23T19:47:00+00:00

In 2014, I was being bewildered and overwhelmed with the amount of options for a simple to-do list style application. My needs were pretty simple, I needed something with categorisation (work/personal) and a way to add items; that was it. Unfortunately, every system and application I looked at had those features plus more. This would have been fine however the features were all intertwined and hung off each other which made using a subset in isolation impossible. For example, you couldn’t have an item without a date or category assigned otherwise it wouldn’t save. All of these systems were too much for what I was looking for and in most cases, made the information super hard to digest. That was, until I stumbled across the Bullet Journal system.

The premise was straight forward. You would “rapid log” your daily tasks (using a legend for the various task types) and then “migrate” them to other components if you didn’t get them completed. Best of all, it was analog which meant I wasn’t locked into a single app with the potential of breaking changes for my workflow or needing battery power when I wanted to use it. Initially, I started off with the most basic setup possible. A notebook with daily to-do items and a monthly log. This worked great and helped me not only get my daily items in order but also what was worth holding onto in the list as I had to manually bring over any tasks that still needed to be completed. I would eventually expand upon this and use it for all sorts of collections which made the workflow concepts inadvertently sneak into other aspects of my planning process.

Bullet Journaling worked exceptionally well for me for the better part of 5 years and along the way, I’ve convinced quite a few others to give it a chance to sooth their current overwhelm due to the amount of to-dos they have.

I recently noticed that I was spending alot of time taking digital notes or events and transferring them into my Bullet Journal. This never used to be an issue due to the workload that I had however I was starting to feel the pinch of an analog system. A couple of the stand out issues I had:

Timed events were always really hard because there wasn’t a way the piece of paper could remind you when it was due.
Majority of my day is spent either on a laptop or nearby a device. Digital information shouldn’t need to be replicated to analog only to have it come back to digital form when I need it action it.

At this point, I knew the Bullet Journal wasn’t going to be able to solve all the problems but, the concepts might. I made a list (of course in the Bullet Journal) of the types of information I receive and what was important for each category type. I won’t bore you with the “what was important part” but my categories did boil down to:

Calendar: Work events, birthdays, holidays, car service reminders.
Email: I get far too many emails.
Notes: I take a bunch of random notes for topics that I find interesting or books I’m reading.
Tasks: Things I need to do or action.

Looking at the list, I can see why the Bullet Journal worked so well (with the obvious exception of email). The flexibility and customisation options ensured that you could mould whatever you needed to work with the system.

I started looking around and found that putting all of these into a single app or system wouldn’t work in the digital space without also creating some new headaches. Instead, I opted to split the information into single purpose applications. This would ensure that I’m not shoehorning a particular category into a system that wasn’t designed for it or that wasn’t the primary focus. It would also allow me flexibility to swap out an application or system that wasn’t working for a single category without needing to migrate everything with it.

So, which applications or systems did I end up on?

Calendar: For the most part, this was an easy decision. I have been using Google Calendar since I’ve had a Gmail account and my employer also uses Google Calendar heavily for meetings. The UI isn’t the best so I did opt to purchase Fantasical 3 or 4 years ago to improve that experience and have a desktop application. This allows things like reminders to pop up even if you don’t have the web page open.
Email: No application change here, I’ve used Postbox ever since Sparrow was shut down by Google. The difference was what I did with emails once they came in. Instead of sitting there waiting to be actioned, they end up routed into their correct organisational category.
Notes: This was one of the hardest categories to differentiate products as essentially anything can take notes. I ended up deciding on Evernote as using it felt more natural and I didn’t have to pay per feature or addon I may need. The pricing is straight forward and I won’t be restricted by “per component” or “per note” pricing like some of it’s competitors.
Tasks: By far the hardest (and most expensive) decision I had to make. I tried in the order of 10-15 task applications and majority of them were overzealous with features that I wouldn’t need or functionality that got in my way for a simplistic approach. The bang for buck also varies greatly as some of the more expensive applications in this category were actually the worse ones for my workflow. In the end, Things prevailed. Things came out on top primarily because of it’s simple UI and limited options of viewing the data. Yes, that’s right - I paid to have less options and as a byproduct, a more straight forward and easy to use application.

And now, the workflow.

When I was Bullet Journaling, that notebook came everywhere with me. There was definitely more than a couple of times that I busted it out while in a hardware store looking to pick up supplies for a weekend project. This was a bit of an inconvenience but I was pretty chuffed with how much I was able to use it for without even needing to modify or change the way I stored the data so I persisted with it.

Fast forward to recent times where all the systems I use also have an iOS application. This makes it far easier (and convenient!) as there is a good chance that I would already be carrying my phone with me when I need to add or check a task.

The calendar and note taking aspects don’t require much explanation as they are the destinations for the information but the email and tasks systems are worth explaining.

My email client is now essentially a reference point and tasks are created from it. Here is an example where I’ve got a new feature request for a project I maintain and want to create a task to get it done.

After installing the Things Helper, I can hit cmd + shift + n (in any application) and the task item is pre-filled and I can save it.

This also works if I need to save a URL to a task for later reference (I use Pocket for articles or links I intend to read later).

This leads us to the tasks (Things) which is where the bulk of my workflow adaption has taken place. When I have a new task that isn’t immediately important, I create it and don’t categorise it. Things automatically puts this into the Inbox. At the start of my day (or week depending on how busy I am), I comb through the Inbox sorting and making sure tasks have been prioritised and categorised (think G.T.D. method). Once the Inbox is clear, I move onto the Today list and perform any priorisation moves there before grabbing the first task and getting started on it. One of the perks of Things is that once the due date has arrived, the task automatically appears in your Today which saves you needing to sift through a calendar or another list to find them.

I do use the projects feature within Things to better organise some of the tasks as I find it’s handy to have a filtered list of tasks for some aspects of life (work vs personal vs recreational).

A good side effect of this is that you also have a type of “heads up display” when you select the area, here Personal, that outlines how many to dos in each project as well as any that haven’t been sorted.

Overall, I think the Bullet Journal is an excellent system and has shaped my workflow and thinking for the better however my line of work lends itself better to a more digital approach with GTD and Bullet Journaling sprinkled over it.

Taming travel

2019-09-03T01:40:00+00:00

Being a combination of remote worker and having a partner who loves travel has forced me to continuously evaluate my travel gear and setup to get the most out of the trip. This is a collection of tools and tips I’ve tested, updated, discarded, updated and then tested again throughout travelling both domestically and globally for personal vacations and work trips. The goal has been simple: do travel with as little as possible while still maintaining a stress-free and comfortable experience.

Note: None of the links I’ve used below is for affiliates or referrals or anything like that, just good ol’ fashion links to the thing you care about.

Gear list

At heart, I do favour using the right tool for the job and sometimes paying a bit extra for the better quality that will last you a long while over many cheap replacements. I’ve been caught out by buying cheap in the past and can assure you, when it comes to travel you can often be looking at more than the cost of a cheap replacement when they break down on you.

Hardshell Samsonite Roller bags: My wife purchased a set of 3 of these a couple of years ago and they have served us well. Built extremely light, they are also very tough so that your belongings can’t be damaged if the baggage handlers are having a rough day. We take these on our trips where we both travel or we’re going international.

Incase Travel Duffle: This is my go-to for trips under a week and all international flights as carry on.

Peak Design Tech Pouch: A relatively new addition to the tool belt however it’s already proven how efficient it is for storing the cables, adapters, external hard drives and power banks while still not being too bulky.

Fisher space pen: Carrying a pen is a must for travelling internationally. I purchased two Fisher Space pens a couple of years ago and they have never let me down. The AG7 is my daily driver and I use the 400B when travelling.

Bellroy Travel Wallet: This is the only part of my international kit that I’m still unsure of. I usually carry a very small wallet (the equivalent of The Low from Bellroy) so this wallet feels a touch too big to be carrying it everywhere but it’s great otherwise. The extra-deep note compartment for bigger notes, SIM card holder and ejector tool space have all been very handy. The pen is very convenient for filling out immigration forms. Before this wallet, I was using a Muji Passport holder which worked well but wasn’t something I took out of the hotel once I arrived.

Good quality sunglasses: I’ve worn a pair of polarised Ray Ban Justin’s for about 3 years now and take them everywhere with me. Whether it’s an early morning flight or walking about the city, these are a godsend to save your eyes.

Privacy and security

I’m what you’d call “privacy and security conscious”. As such, I have a pretty high bar for what I deem secure or where my privacy is concerned.

For those looking to increase their privacy and security stance, here are a few things I do when travelling.

Use a dedicated travel device. I started writing about this process but Filippo Valsorda has already beat me to it with a great in-depth article which I highly recommend checking out.

VPNs, 100% of the time.** I have both a commercial VPN provider as well as a handful of cloud instances that are running Wireguard, OpenVPN using stunnel and dsvpn. I keep the commercial one handy for quickly testing in multiple regions where I don’t have the infrastructure and something for my wife to easily use on her devices.

1Password Travel mode: For credentials and sensitive data storage, I use 1Password. This is great for crossing through airports or being in an environment where you don’t know who might end up with access to the device or data.

As a rule of thumb, I don’t take anything of a sensitive nature with me when travelling. This includes things like laptops which if they do come with me for work, end up pretty much wiped and set up upon arrival to ensure if a device copy is made, nothing important is taken.

Applications I use

iOS

Flighty: This application was only released this year however it has been a game-changer when it comes to notifications, updates and flight information. I rarely bother to check airline or airport websites now for things like delays or gate changes as this app beats them every time. The UI is also amazingly well designed and Just Looks Really Good when you’re using it.

TripIt: Trying to keep track of flight itineraries, taxis, hotel accommodation and any events information used to be a pain as I would keep them all in my inbox (or separate applications) and then just rely on my memory or notes to keep track of what time everything would be needed. This didn’t scale well and made things more stressful than what they needed to be. Now I send all of the information regardless of activity type to TripIt and it plans it all out for me. Whether it’s a 3-day stint to HQ or a 2-week multi-leg flight and accommodation with private transfers for vacation, it’s all organised and never forgotten.

TimeShifter: Australia is pretty amazing for the most part; you’ll struggle to find a better option for a country to live in. That is, as long as you never deal with timezones which is pretty hard because almost everywhere you travel outside the country is often a timezone shift by at least a few hours. Jet lag is never fun and I’ve had my fair share of tired sightseeing trips to know it’s a real thing and can get in the way of enjoying parts of your trip. While on holiday Thailand a few years ago, I was looking into the usual “how to beat jet lag” google links and stumbled on an approach that instead of relying on hope and supplements, used science-backed techniques to adjust your circadian rhythm. This has been great on my overseas trips since and beats waking up at 2 am and not getting back to sleep or needing to go to bed at 5 pm because of the timezone differences. The application makes this process simple and prompts you as to what actions you should be taking in the lead-up and arrival to your destination to get the most out of both timezones.

Airline entertainment apps: Most airlines have an application you need to download to access their in-flight entertainment. I only have a couple of these as we stick to the same airlines when travelling. Best to download these at the airport if you haven’t already got it as by the time they take off, it’s too late.

Foursquare: It’s always tough to find restaurants and activities when you’re not a local and don’t have a guide. I’ve been using Foursquare on all my trips and have found it to have the most accurate and honest reviews. I’ve used other apps in the past that were either quite biased or didn’t have the same level of information available about locations or activities. A bonus here is that things like price, WiFi speed and transport options to get to the destination are commonly included.

Apple Wallet: I download all my boarding passes to Apple Wallet to ensure I can’t lose them. This also means one less piece of paper to carry around. If you don’t have iOS, you can also take a screenshot of your boarding pass to present and this would achieve the same thing.

Mac

TripMode: Bandwidth is expensive when roaming. Don’t let your apps run up a bill on you. This application operates as a gatekeeper for your other applications and puts you in control of managing who can download while you’re out.

LittleSnitch: Little Snitch is a great tool that I use every day (regardless of travel) to know what process on my machine is connecting to the outside world. When I travel, I up the mode to be super locked down. This prevents my data from going out on things like hotel wifi but also saving bandwidth. This is similar to TripMode, just more in depth with finer grained controls.

Packing

Carry on

iPad mini: Before any travel (be it 2 hours or 12 hours) I load my iPad up with things to do on the flight. I do the majority of my reading on my iPad these days (aka an expensive Kindle) but I also download TV shows/movies in case the flight doesn’t have inbuilt entertainment.

Headphones: I pack two pairs of headphones. My Bose QC35s and the Apple wired earbuds.

All the adapters/chargers: I carry all the adapters/chargers/cords for my devices in my Peak Design Tech Pouch. The only specific one worth calling out here is having a shorter cable on the flight for your phone/tablet to avoid having an extra cable dangling and potentially tripping over.

Moisturiser: Your skin ends up dehydrated during and after flights. Help it out with some moisturiser.

Empty water bottle: Same as above, but for your insides. I take my Yeti and fill it up in the airport after getting through security. My Yeti also doubles as a muscle roller in my hotel room if my back ends up sore or tight. There are some whizz-bang alternative water bottles that collapse too if you’re looking for a space-savvy option.

Sanitiser wipes: While planes are visibly clean, they aren’t sanitised between every flight. If the person who sat in your seat on the incoming flight was sick, you could be one touch away from getting sick. Wiping down the area where you’re sitting avoids this and catches any of those sticky food spots before you find them mid-flight.

Wet Ones: Flights can be hot and stuffy. Do everyone a favour and freshen yourself up before landing or just after landing in the airport bathrooms.

Tissues: You can never find one when you need it.

Toothbrush and toothpaste or mouthwash: If you manage to sleep on that overnight flight, you’re going to wake up with goat breath. Don’t put flight attendants or the good people at immigration through needing to smell your breath.

Change of clothes: If you’re flying domestically, this could be all your clothing but it’s also handy to pack a change of clothes for international travel (especially if you’re wearing your comfy clothes on the flight or manage to spill something on yourself). At a minimum, fresh shirt, pants and underwear.

Power bank: I bought a power bank for my working from cafè days to charge my 13” MacBook Pro which used to come everywhere however I found it was a bit overkill for my domestic needs where I may only need to charge my iPad or iPhone (who needs to fully charge their iPad 5 times on a 3 hour flight?!?!?). I ended up buying a smaller one as well and swap them according to the type of travel I do.

Pain killers: I pack a handful of Panadol to account for getting headaches or dealing with backaches and pains.

Powerpoint adapters: I tried a universal all-in-one style of adapter only to find that some power points didn’t account for these very well and the configuration knobs would block the second outlet. I now opt for a dedicated one based on the destination but still on the lookout for a single solution here.

Domestic

For domestic flights where I’m travelling alone, I only do carry on. I’ve only had two occasions where my checked baggage was “lost” (in both cases, less than 8 hours) however it taught me a very valuable lesson that you can’t lose a bag that doesn’t leave your side. A side benefit of this is that I’m less likely to overpack because I’ll be restricted on the size and weight.

If the trip is longer than 4 or 5 days, I don’t pack for the full trip. Instead, I opt to use the hotels’ laundry service. This is a great, cheap alternative to overpacking. You get the laundry bag out of your room, put the dirty clothes in it and then let the front desk know you have something to go. My last bill for 3 days worth of clothing and a jacket was under $5.

When I’m flying with my wife, we do a setup close to my international approach as we can quite often share checked-in luggage and the carry on is just a nice to have.

International

Before travelling internationally, I go on a bit of fact-finding mission first. This ensures that I know what sort of hurdles (if any!) I’m going to run into while abroad. This preparation includes things like:

Being aware of any restrictions that are present in the country. Internet censorship, religious or cultural differences, etc.
Whether I need to brush up on my language skills. Many countries respond better to you when you use a basic language vocabulary in their native tongue. I don’t learn all the language, just enough to get by without butchering their language.
Expected prices and costs. This cost of living calculator from numbeo.com is great for knowing how much you should be paying and when to know if you’re getting ripped off.
What payment type the country prefers. I always take cash and a credit card however I do prefer to know which form of currency will be the most widely used so that I can load up more on that instead of needing to top it up while I am there.

My international packing is quite similar to my domestic except I also check in a bag for my clothing. My carry on does become a little more extensive though as it also needs to account for if my checked baggage ends up lost and I only have those things available to me.

I pack miniatures of my personal hygiene products in my carry on and full-sized items in my checked bag. For example, a 50g deodorant goes in my carry on and my usual 180g goes in my checked bag. This approach allows me to stay refreshed on long haul flights and not feel like I’ve rolled out of a gutter after a weekend bender.

Lounge access

Up until recently, lounge access wasn’t something I was too fussed with. That was until my wife’s employer paid for a business class flight and access came with the ticket. Boy oh boy, does it make a difference. Nicer chairs, easy access powerpoints and of course, a pretty stocked bar and food buffet. Lounge access is one of those things that make your trips a little less stressful as they often come with a dedicated screening process and direct access to the terminal where your flight is.

It is a luxury though at $65 for a one-off visit or $330 to join + $420 per year. This is something that I have been tossing up with for a year or so now as I could almost justify it with the amount of travel we do. Instead of paying the fees though, I’m giving an alternative a go by using my usual flights and normal expenditure to accrue enough status credits to get access. I’m not going to get into details about status credits here but if you’re interested in how it works, check out Velocity’s Status Credit overview (FWIW, we generally only fly Virgin Australia or partner airlines which makes this possible).

If you fly for work and can convince your boss this is something that will benefit you, I’d highly recommend it.

Interview with Jacob

2018-11-05T00:00:00+00:00

Migrating edge network providers

2018-05-06T00:00:00+00:00

Remedying the API gateway

2017-05-22T00:00:00+00:00

Getting Envato Market HTTPS everywhere

2016-08-05T00:00:00+00:00

Tracking down Ruby heap corruption in amongst 35 million daily requests

2016-05-19T00:00:00+00:00

Remote transition

2016-03-09T00:00:00+00:00

Just on 10 months ago I joined Envato as a Site Reliability Engineer and took the plunge into full time remote work. While I’m far from having it all worked out, I’ve managed to stumble across some useful information along the way that I think is worth sharing.

What I love

Spending time with my wife and family

Prior to working remotely my commute was on a train for three hours a day. Yes, 3 hours a day. Crazy right? During this time I used to work on open source projects or tidy up loose ends for the day but it was no substitute for the time I wasn’t spending with my wife and family. Simple things like going out for dinner was a pain because even if I managed to leave the office on time at 5pm, run to the train station and get the 5:10 train, I still wouldn’t be at the dinner location before 7pm. These days, I’m able to knock off work around 5 and still leave plenty of time to do what I need to and make it to dinner reservations.

Freedom of working where you feel like

I hate office cubicles. I especially hate office cubicles that have the horrid grey coloured material on them. I’m not sure where or when I developed these feelings but working in a cubicle has always made me feel restricted and enclosed. Having the freedom to work from a space of my choice has allowed me to adjust my environment based on the type of work I’m doing. Got some heavy duty thinking stuff to do? Alright, let’s head to the office and shut the door. Oh, now you want to do something creative? Let’s take a step outside onto the patio or take a skate to the nearest coffee shop for a few hours.

Taking care of errands

In November last year, I got married and during the planning stage there were multiple times I had to run out to sign paperwork or collect an item on very short notice which just wouldn’t have been possible if I was still commuting everyday. These days, it’s handy to be able to do things like pick up a parcel from the post office without needing to wait until the weekend.

Space to think about a problem

Working in Site Reliability you have alot of “WTF?” moments so it helps being able to take a step back from the problem and look it from a different angle. In an office space this isn’t always possible and I’ve found huge benefits in just being able to step out for a few moments just to clear you mind before heading back to tackle the problem.

Not so good aspects

You miss out on (some) company gatherings.

Envato are really good when it comes to organising get togethers and they don’t care where you are based - if you want to go, they’ll make sure there is a spot reserved for you. Unfortunately for me it’s not always possible to jump on a plane for a few hours to enjoy a night at the Aquarium with my work colleagues.

“Switching off” is hard.

I realllllllyy enjoy what I do. I know it sounds stupidly cliche but I love the work I do around performance, security and reliability at Envato. I enjoy being able to work a problem where the end goal is that someone can do their job better because of changes or decisions you make. The downside to this is that I’ve sometimes found it hard to stop working at the end of the day and go do other things. Lucky enough though, I have a wife who knows what time I usually finish and she makes sure we go and do something so that I’m forced to disconnect for a while. :)

Tips for those looking at going full time remote

If you’re going to take away anything from this, take these points.

Find yourself an ‘office’ space. The kitchen table doesn’t always work.
Ensure that you have a way to mark the end of your work day. Try exercising, taking your dog for a walk or skating to the local park. Switching off allows your mind to recharge at the end of your busy day instead of continue spinning into your time.
Buy good gear. Just like a mechanic will buy good set of spanners, you too need to buy good gear to get your job done. Sometimes this comes in the form of a $1000 chair, other times it can be a new router. If you’re serious about remote work you need to invest in the right tools. I’ve seen alot of people start cheap and never update and it comes back to bite them in the long run.

If you’ve got some remote work tips, hit me up @jacobbednarz. I’d love to hear what has worked and hasn’t worked for you!

Tracking SQL queries

2015-05-15T00:00:00+00:00

When you run a large application (or many large applications), it is important that you can effectively track down and investigate slow SQL queries when they arise. MySQL doesn’t receive context of where the query is coming from so in your logs you end up with something that will look like this:

# Time: 150517  6:26:05
# User@Host: mysql_user[mysql_user] @ db1.example.com [10.0.0.60]
# Thread_id: 4038366  Schema: example_table  Last_errno: 0  Killed: 0
# Query_time: 2.388451  Lock_time: 0.000041  Rows_sent: 0  Rows_examined: 1302  Rows_affected: 1302  Rows_read: 1302
# Bytes_sent: 13  Tmp_tables: 0  Tmp_disk_tables: 0  Tmp_table_sizes: 0
# InnoDB_trx_id: 2D57FA24A
use example_table;
SELECT *
FROM users

Sure, if this is a specific query you might know which URL or method is sending it, but what if someone not so familiar with the application needed to investigate it - would they be able to find it?

One option is to add a comment to your query that contains some useful information such as class name, method name or even a request ID. Consider the following query:

<?php

db_select('users', 'u')
  ->fields('u', array('mail', 'status'))
  ->execute();

# => SELECT u.mail AS mail, u.status AS status FROM users u

To add a comment to a query, you can use Drupal’s Query::comment method which will prepend your comment before sending the query. Below we add a comment that contains the module and function names of the caller.

<?php

db_select('users', 'u')
  ->comment('module:my_custom_module,function:some_slow_function')
  ->fields('u', array('mail', 'status'))
  ->execute()

# => /* module:my_custom_module,function:some_slow_function */ SELECT u.mail AS mail, u.status AS status FROM users u

Now should the query pop up in your slow query logs, it will be annotated with some helpful debugging information on it’s caller.

# Time: 150517  6:26:05
# User@Host: mysql_user[mysql_user] @ db1.example.com [10.0.0.60]
# Thread_id: 4038366  Schema: example_table  Last_errno: 0  Killed: 0
# Query_time: 2.388451  Lock_time: 0.000041  Rows_sent: 0  Rows_examined: 1302  Rows_affected: 1302  Rows_read: 1302
# Bytes_sent: 13  Tmp_tables: 0  Tmp_disk_tables: 0  Tmp_table_sizes: 0
# InnoDB_trx_id: 2D57FA24A
use example_table;
/* module:my_custom_module,function:some_slow_function */
SELECT *
FROM users

If Rails is your framework of choice, the marginalia gem adds this feature to ActiveRecord and can be configured to capture any data you wish.

Why post mortems matter

2015-03-12T09:00:00+00:00

Whether you run a service or provide a product to customers, there will be times when your service (or part of it) is unexpectantly unavailable. These things happen however providing a good post mortem can demonstrate transparency and build trust with your customers if executed correctly.

Leave blame at the door

If you don’t already use blameless post mortems, I suggest you go check out some articles from Etsy and Mathias Meyer as blameless post mortems will change the way you and your team view incidents. Instead of blaming individual people or tools, you will look to build an environment where people are safe to fail and people won’t be petrified of change. This type of environment will encourage people to discuss the problems and the best approach to solving them instead of throwing each other under the bus to avoid getting fired.

Apologise

Giving your customers a sincere apology is one of the most important things you can do for your post mortem. It shows you are human and it demonstrates you understand the impact your incident had on them and that you want to make it right.

Your apology should feature within the first paragraph (or there abouts) of the post mortem - giving an apology within the last sentence of your report isn’t good enough.

Provide a summary and technical account of the events

Despite what industry you are in there will always be a varied skill level of people reading your post mortem - which is why it’s important to include a summary of the events (usually in layman terms) and a nice technical dive into the incident itself. You should cover what components were affected as well as the steps taken during your diagnosis and investigation.

Another good use of this section is to provide a high level insight into how various parts of your service or products interact together. People love this type of insight and it offers the opportunity to turn this negative experience into a positive learning experience for the reader.

Remediation plans

For many people, this will be the most important section of your post mortem as they want to see what steps you are taking to prevent this incident from happening again. If you are writing a post mortem and don’t provide action items to remedy the situation, you are showing you are not committed to mitigating the problem from reoccurring and your customers are likely to take their business elsewhere.

Flip - Bringing feature toggling to Drupal

2015-03-06T06:35:00+00:00

When you are working on a large project sometimes you need the ability to gradually roll out a new feature to public facing users. Out of the box, Drupal doesn’t provide too much to assist you with this however one option is to riddle your code with ‘if-something-then-do-something’ checks but what happens if you need to update that condition? You’ve then got to go back through and redo all the conditionals and what happens if you miss one and suddenly your checks no longer match up?

To combat this, many other frameworks and applications implement feature toggles which allow you to implement checks around your code and manage a single source for the conditional. By using feature toggling, you still need to implement your if conditionals, however the checks are all maintained in a single place so this no chance of it getting out of sync with each other.

Now that you know what feature toggling is and how it’s going to benefit your site, you’ll need to go and install flip.

Putting it into action

Once the module is installed, you’ll need to register your toggle with the flip module - you can do this by implementing hook_flip_info within the module which the toggle will be used in.

<?php

function MYMODULE_flip_info() {
  return array(
    'example_toggle' => array(
      'name' => 'Example toggle',
      'description' => 'This is my example description for a toggle.',
      'condition' => 'my_condition_check',
    ),
  );
}

Be sure to check the docs on what each key does!

The main bit here is the condition key. This is the callback you wish to toggle based on - some examples used within these callbacks are: particular user roles, percentage of requests or time of the day. For simplicity sake, the conditional I will be using is that the username is “Jacob”.

<?php

function my_condition_check() {
  return ($user->name == "Jacob") ? TRUE : FALSE;
}

Once you have defined the hook_flip_info and flushed caches, all the implementing toggles should now be visible at admin/config/development/flip

Note: For a toggle to work, it must be marked as enabled in admin/config/development/flip AND the condition callback must return TRUE. This is quiet powerful as it allows you to disable a toggle via the adminsitration UI in the event of something going wrong and you not being able to make a code push to change it.

Now it’s time to test the toggle!

<?php

// Account for both outcomes. Handy for backwards compatibility.
if (flip_enabled('example_toggle')) {
  do_awesome_new_thing();
}
else {
  use_old_functionality();
}

// Just do something new.
if (flip_enabled('example_toggle')) {
  do_awesome_new_thing();
}

To explore this more, be sure to check out the following articles which demonstrate how powerful feature toggling can be.

Lightweight RESTful API's in Drupal

2014-10-22T00:35:00+00:00

Building RESTful services within Drupal isn’t a new concept and there are already a lot of existing projects out there that will allow you to build a simple REST service. However, one of the big problems with this approach is that because they are based on Drupal modules they require a full bootstrap process and in many cases, causes the application to pull in functionality that will never be used.

After digging through articles online and finding no decent options I set out in search of a solution with the following requirements:

Must be fast. The services on these endpoints will need to respond to thousands of requests per minute without skipping a beat.
Be somewhere within Drupal. Drupal is great at powering our existing sites and we don’t want to remove the opportunity to integrate with it later down the track.
Needs to be flexible and highly extendable. Like ctools but with less of the loading and registry problems we’ve experienced in the past.

The decision

A solid week of research and tinkering later, I had come to the conclusion that a custom solution with it’s own Drupal bootstrap process would be the best fit for our requirements. While this did take away some of the niceties of working with Drupal, it would give us the power and flexibility we needed to scale these services.

Benchmarks

During the decision making process there was a strong push to ensure this solution (whatever it ended up being) would be the fastest option available - so naturally, I benchmarked the shit out of it.

All traffic was sent as an anonymous user and profiled using wbench.

Basic Drupal

Fresh installation with no additional modules enabled.
No caching or aggregation enabled.

$ wbench http://localdev.drupal

Testing http://localdev.drupal
At Tue Oct 21 09:52:18 2014
10 loops
                                   Fastest   Median    Slowest   Std Dev
---------------------------------------------------------------------------

Host latency:
localdev.drupal:80                 0ms       0ms       0ms       0ms

Browser performance:
Navigation Start:                  0ms       0ms       0ms       0ms
Fetch Start:                       280ms     309ms     334ms     13ms
Domain Lookup Start:               291ms     320ms     344ms     13ms
Connect Start:                     293ms     321ms     346ms     13ms
Domain Lookup End:                 293ms     321ms     346ms     13ms
Connect End:                       293ms     321ms     346ms     13ms
Request Start:                     293ms     321ms     346ms     13ms
Response Start:                    666ms     710ms     771ms     34ms
Response End:                      667ms     711ms     773ms     34ms
DOM Loading:                       667ms     712ms     773ms     34ms
DOM Interactive:                   718ms     761ms     3360ms    780ms
DOM Content Loaded Event Start:    718ms     761ms     3360ms    780ms
DOM Content Loaded Event End:      774ms     819ms     3411ms    777ms
Load Event End:                    791ms     837ms     3428ms    777ms
DOM Complete:                      791ms     837ms     3428ms    777ms
Load Event Start:                  791ms     837ms     3428ms    777ms

Custom bootstrapped application just rendering a 200 status

No caching enabled.

$ wbench http://localdev.drupal/api/example

Testing http://localdev.drupal/api/example
At Tue Oct 21 09:50:20 2014
10 loops
                                   Fastest   Median    Slowest   Std Dev
---------------------------------------------------------------------------

Host latency:
localdev.drupal:80                 0ms       0ms       0ms       0ms

Browser performance:
Navigation Start:                  0ms       0ms       0ms       0ms
Fetch Start:                       278ms     297ms     341ms     16ms
Domain Lookup Start:               290ms     309ms     352ms     16ms
Connect Start:                     295ms     310ms     353ms     16ms
Domain Lookup End:                 295ms     310ms     353ms     16ms
Connect End:                       296ms     310ms     353ms     15ms
Request Start:                     296ms     310ms     353ms     15ms
Response Start:                    304ms     317ms     362ms     16ms
Response End:                      533ms     559ms     570ms     10ms
DOM Loading:                       544ms     562ms     579ms     10ms
DOM Interactive:                   545ms     572ms     584ms     11ms
DOM Content Loaded Event End:      545ms     572ms     584ms     11ms
DOM Complete:                      546ms     572ms     584ms     10ms
DOM Content Loaded Event Start:    545ms     572ms     584ms     11ms
Load Event End:                    546ms     572ms     584ms     10ms
Load Event Start:                  546ms     572ms     584ms     10ms

Views datasource returning 10 items from a database

Pulled 10 items with 2 string fields.

$ wbench http://localdev.drupal/views-json/10-items

Testing http://localdev.drupal/views-json/10-items
At Tue Oct 21 10:13:12 2014
10 loops
                                   Fastest   Median    Slowest   Std Dev
---------------------------------------------------------------------------

Host latency:
localdev.drupal:80                 0ms       0ms       0ms       0ms

Browser performance:
Navigation Start:                  0ms       0ms       0ms       0ms
Fetch Start:                       281ms     296ms     335ms     17ms
Domain Lookup Start:               292ms     310ms     346ms     16ms
Connect Start:                     293ms     312ms     348ms     16ms
Domain Lookup End:                 293ms     312ms     348ms     16ms
Connect End:                       293ms     312ms     348ms     16ms
Request Start:                     293ms     312ms     348ms     16ms
Response Start:                    856ms     922ms     1002ms    51ms
Response End:                      867ms     923ms     1004ms    50ms
DOM Loading:                       868ms     924ms     1005ms    50ms
Load Event End:                    870ms     926ms     1007ms    50ms
DOM Content Loaded Event End:      870ms     926ms     1007ms    50ms
DOM Complete:                      870ms     926ms     1007ms    50ms
DOM Interactive:                   870ms     926ms     1007ms    50ms
DOM Content Loaded Event Start:    870ms     926ms     1007ms    50ms
Load Event Start:                  870ms     926ms     1007ms    50ms

Custom bootstrapped application returning 10 items from a database

Pulled 10 items with 2 string fields.

$ wbench http://localdev.drupal/api/10-items

Testing http://localdev.drupal/api/10-items
At Tue Oct 21 10:18:15 2014
10 loops
                                   Fastest   Median    Slowest   Std Dev
---------------------------------------------------------------------------

Host latency:
localdev.drupal:80                 0ms       0ms       0ms       0ms

Browser performance:
Navigation Start:                  0ms       0ms       0ms       0ms
Fetch Start:                       270ms     287ms     311ms     13ms
Domain Lookup Start:               282ms     298ms     327ms     14ms
Connect End:                       284ms     300ms     329ms     14ms
Connect Start:                     284ms     299ms     329ms     13ms
Request Start:                     284ms     300ms     329ms     14ms
Domain Lookup End:                 284ms     299ms     329ms     13ms
Response Start:                    298ms     310ms     343ms     14ms
Response End:                      542ms     568ms     632ms     27ms
DOM Loading:                       543ms     573ms     633ms     25ms
Load Event End:                    555ms     583ms     645ms     25ms
Load Event Start:                  555ms     583ms     645ms     25ms
DOM Complete:                      555ms     583ms     645ms     25ms
DOM Content Loaded Event End:      555ms     583ms     644ms     25ms
DOM Interactive:                   555ms     583ms     644ms     25ms
DOM Content Loaded Event Start:    555ms     583ms     644ms     25ms

From the results above, you can see that the custom solution has come out with response times almost half that of the alternatives.

The code

Now that you can see it’s fast, let’s dive into what the code behind this looks like and how it all fits together.

Directory structure

To ensure this was (somewhat) isolated from the rest of the Drupal sites, I have put all the files required inside of a sub-directory called api. This does two things:

Create a namespace within the URI that only these endpoints will use. I.e. http://example.com/api/<service_name>. This helps with cache negotiation and preventing Drupal from attempting to route everything as it normally would.
Allow me to pick and choose what the service endpoints have file access to.

api
├── .htaccess
├── README.md
├── includes
|   ├── core.inc
│   └── route.inc
├── index.php
└── services
    └── example_service.inc

File overview

.htaccess: Handles pretty URL’s so you don’t end up with that ugly index.php in all of your requests.
includes/core.inc: Contains the functionality that all services have access to. Includes methods like outputting JSON and XML responses in a standardised manner.
includes/route.inc: Is for creating routes to the service endpoints with additional parameters like callbacks and arguments to pass through.
index.php: This is where the leg work to bring it all together and bootstrap Drupal to a low level is done.
services/example_service.inc: A demonstration on how to setup the service and what can do.

I’m not going to break this down line by line but the full code is available online for your viewing pleasure at jacobbednarz/lightweight-drupal-rest-services.

Caution!

This approach does have some caveats and before anyone embarks down this route I would recommend you at least be aware of them as your mileage may vary.

Firstly, and probably most importantly, this is outside the realm of just Drupal so nothing is given for free.
You will need to build up your own security around Apache and your custom .htaccess file.

Using the 'Accept' header for HTTP API versioning

2014-07-14T00:00:00+00:00

Something that all API’s need to consider (no matter how simple they are!) is how they will handle adding and upgrading functionality once someone gets a hold of it.

Sure, you could just wing it and rely on your test suite to give you some confidence in just rolling it out…but what about those weird and wonderful ways others may be leveraging your API? How will they cope with your update?

One approach that is often taken is to include the version number within your endpoint routes - an example would be https://api.example.com/v1/.... While this definitely works, it does create some issues with cache negotiation and clients holding onto outdated links.

Cache negotiation pitfalls

A large portion of caching identifiers are based on URI unless configured otherwise which means that https://api.example.com/v1/users will have a different cache key to https://api.example.com/v2/users. While this is desired behaviour you are potentially paying for the same cache data to be served even though it could be identical and when you pay according to the traffic offset by this service, it can quickly grow out of control. Should you be lucky enough to have a self hosted service you have the additional overhead of the same cached data being stored X times where X is the number of API iterations are accessible.

Outdated links

People bookmarking and holding onto outdated links is a problem that is as old as the internet itself and it probably won’t be going away any time soon. So help yourself out by not creating more issues when https://api.example.com/v1/... no longer works because version 1 of your API has been deprecated and the newly released version is now located at https://api.example.com/v2/... instead.

Now that you know the issues, what is the solution? You guessed it, the HTTP ‘Accept’ header.

The Breakdown™

Using the HTTP Accept header is relatively straight forward and looks a little something like this:

Accept: application/vnd.{app_name}.{version}+{response_type}

vnd (as per RFC 4288) is the method of registering as a “vendor” and that you plan to interchange using this context.
app_name is a field that identifies your application or product.
version will contain the version number you wish to use to distinguish between iterations or releases. Common values are ‘v1’, ‘verison2’ or even ‘beta’.
response_type covers what structure you plan to send the response as. E.g. ‘json’, ‘xml’, etc.

Using the HTTP Accept header

Now that you have a fair idea on why it benefits you to use the HTTP Accept header and what it looks like in the wild, it’s time to write some code to use it.

In this example an endpoint to return a user’s record has been setup. Unknown to the general public, a new field has been added to the API however while we are still within the testing phase we only want users who are a part of the beta program to see it.

get '/user/:id' do
  # .. Some magic to get the user based on ID. I will just assume this is done
  # and you have a User object under a local variable called 'user'.

  response_data = {
    name: user.name,
    email: user.email
  }

  if request.headers["Accept"].include? "beta"
    response_data[:plan] = user.subscription_type
  end

  response_data.to_json
end

To see the results all we need to do now is send a request to the endpoint specifying the correct HTTP ‘Accept’ header.

$ curl https://api.example.com/user/1

{
  "name":"Jacob",
  "email":"me@example.com"
}

Now, let’s specify the ‘beta’ HTTP Accept header so that we can take advantage of that new field.

$ curl -H "Accept: application/vnd.example_app.beta+json" \
  https://api.example.com/user/1

{
  "name":"Jacob",
  "email":"me@example.com",
  "plan":"medium"
}

Using monit for server monitoring

2014-03-19T06:35:00+00:00

Monit (as the name suggests) is a tool designed for monitoring systems, processes and filesystems. There are others out there which go far and above what Monit is capable of but with those more extravagant tools also comes more of a learning curve. Monit does an excellent job in being easy to setup and packs 95% of the requirements most developers will ever need to implement without a need to get too involved.

Installation

Monit is available within most major Linux flavours via the package manager so getting it installed can be done in a single line.

sudo apt-get install monit

or if you’re using yum

sudo yum install monit

Configuring monitoring

Depending on your Linux distribution, the configuration file will either live at ` /etc/monit/monitrc on Debian based systems or /etc/monit.conf` on RHEL/CentOS. One handy thing to note is that Monit allows this file to reside in a few different locations on the file system in case one of them are not available to you.

Monit ships with it’s own domain specific language that doesn’t require any programming experience however math operator knowledge will be beneficial should you try to create more in depth checks.

Examples of monitoring

The example below demonstrates how to implement some basic checks like checking for a process, ensuring system resources are not overrun and a simple log rotate.

# Ensuring a process (like Apache) is running
check process apache with pidfile /var/run/apache2.pid
  start program = "/etc/init.d/apache2 start" with timeout 30 seconds
  stop program  = "/etc/init.d/apache2 stop"

# Don't let CPU use over 50%
if cpu is greater than 50% for 5 cycles then restart

# Ensure the file system doesn't get over 80% full
check filesystem rootfs with path /dev/hda1
  if space usage > 80% for 5 times within 15 cycles
     then alert else if succeeded for 10 cycles then alert
  if space usage > 90% for 5 cycles then
     exec '/super/cleanup/script'

# Quick and easy log rotation
check file example.log with path /var/log/example.log
  if size > 50 MB then exec '/usr/local/bin/rotate /var/log/example.log myapp'

The full list of checks and available methods can be found at the monit documentation page.

Handy hits

Before reloading the configuration, be sure to run monit -t to check for any syntax/compilation errors. Reloading the configuration without checking this first main result in false positives.
Monit has a web UI! For those GUI lovers, Monit ships with a web UI and can be configured to be exposed on a particular port of your choosing.
The state of the machine can be quickly determined using monit summary.

Bootstrapping your projects

2014-01-10T03:00:00+00:00

Bootstrapping a project is a concept that I first seen derived from Ruby on Rails script/server and I was instantly hooked. Fortunately, this concept was beginning to catch on and many open source projects were getting on board with it.

The process of having a single point of contact for everything I needed to do in a project (outside of the actual development of course) was amazeballs. No longer would I need to worry about which environment dependencies I had to use to include for a project or whether I needed have another service running to make it function, it was already done by the maintainers and would only require me to run a single script regardless of the task at hand. This concept was applied to a range of functionality and helped new comers to the project get up and contributing much faster.

Here is an example of a typical script directory looks like:

script
├── bootstrap
├── console
├── cibuild
├── release
├── server
└── test

`bootstrap`

A series of tasks or dependency management steps that will run to enable a developer to start working on the project. This includes all required gems, environment variables and local services setup.

`console`

A script to load the current application into a console session - often use for debugging without a browser.

`cibuild`

The file that will setup the prerequisites for the CI server. Has very similar tasks to the script/bootstrap script often using mock or test information instead.

`release`

Tasks that determine how you will build and create a new release of the project. This isn’t used for individual commits for a feature branch, mostly for the project maintainers when a new version of the application is ready.

`server`

If your application is web based, you will need a something to serve it and this task would start it. Saves the developer fiddling with their own version of a web server.

`test`

A wrapping file for your test suite. Depending on your test suite, there are a number of ways to invoke it and if developer miss a flag or a certain syntax that your suite requires they may not be able to successfully run the test suite and not submit that code patch - leaving a sad developer.

Even though the example above is based on a Ruby workflow, it could just as easily be implemented for any other language. The required files and tasks may be slightly different however the underlying concept will remain the same and hopefully ease that learning curve of a new project.

Switching to chruby

2013-11-20T06:35:00+00:00

As all Ruby developers will know, dependency management and isolation in projects is paramount. Example, project ‘A’ is based on a Ruby 1.9.2 and has a version of a gem that only works with 1.9.2 whereas project ‘B’ is a bleeding edge Ruby 2.0.1 application that only works with a few custom gems that you have created.

Until recently I was using rbenv which worked great however it was incredibly slow at times - especially after I had accumulated 12 or so different versions that I used at one point or another. After drilling into the rbenv source code, it became obvious that the shim inside of my .zshrc file that rbenv required (eval "$(rbenv init -)") was actually doing a rehash of all the Rubies everytime a new terminal instance was started which would explain the ~1 sec additional load time when starting up. It can be avoided by passing the --no-rehash parameter however I also didn’t want to start remembering to rehash everytime I made an update.a

After this digging I decided it was time to re-evaluate rbenv and check to see if there were any better solutions that had popped up which I had missed. Not long into the search I stumbled upon a little project called chruby.

chruby has a lot of cool features and all in ~90 lines of code!

Updates $PATH.
Correctly sets $GEM_HOME and $GEM_PATH.
Users: gems are installed into ~/.gem/$ruby/$version.
Root: gems are installed directly into /path/to/$ruby/$gemdir.
Additionally sets $RUBY_ROOT, $RUBY_ENGINE, $RUBY_VERSION and $GEM_ROOT.
Calls hash -r to clear the command-lookup hash-table.
Fuzzy matching of Rubies by name.
Defaults to the system Ruby.
Optionally supports auto-switching and the .ruby-version file.
Supports bash and zsh.
Does not hook cd.
Does not install executable shims.
Does not require Rubies be installed into your home directory.
Does not automatically switch Rubies by default.
Does not require write-access to the Ruby directory in order to install gems.

Ok, cool. So now what? I started by removing all traces of rbenv (including the Ruby binaries as I wanted to start completely fresh). To install chruby, I used homebrew so it was as easy as brew install chruby however it is available from the source as well as a setup script if you are looking to get up and running quickly.

After that completes, you then need a way of installing additional Rubies. My pick was to use ruby-install due to it being simple and the easiest to get going with - there are other options such as ruby-build that will also work. Now that you have chruby installed and at least one of the many Rubies available, it is time to tell your shell that it needs to start using it. Add the following line to your .bashrc or .zshrc.

source /usr/local/share/chruby/chruby.sh

Additionally, chruby also supports auto switching based on a .ruby-version file that is located in the current or parent directory. Even though it doesn’t ship with it out of the box, it is also easy to enable. Just add the following line to your .bashrc or .zshrc below the previous declared chruby line.

source /usr/local/share/chruby/auto.sh

Excellent! Now to set a system wide default so we can get off 1.8.7. There are a couple of ways mentioned in the documentation on how to achieve this but I opted to use the .ruby-version file in my home directory as it was simple and I can add it to my dotfiles under version control ;) .

echo "ruby-1.9" > ~/.ruby-version

Restart your shell and bam, chruby. You can see what Rubies are available by running chruby.

$ chruby
  ruby-1.9.3-p392
  jruby-1.7.0
  rubinius-2.0.0-rc1

And manually switching can be done using chruby <version>.

$ chruby
  ruby-1.9.3-p392
  jruby-1.7.0
  rubinius-2.0.0-rc1

$ chruby ruby-1.9

$ chruby
  * ruby-1.9.3-p392
  jruby-1.7.0
  rubinius-2.0.0-rc1

If you are migrating from another tool such as rbenv and don’t want to remove all your existing Rubies, you can just point $RUBIES to wherever you had them previously installed by defining it in your .bashrc or .zshrc. E.g.

RUBIES=(
  ~/.rbenv/versions/*
  /opt/jruby-1.7.0
  $HOME/src/rubinius
)

Sinatra inside of Rails

2013-07-28T06:35:00+00:00

I recently started building a new service in Ruby on Rails and had the need to build an API that users can interact with for their own applications or tools. Unfortunately, Ruby on Rails is a little bit heavy for this so I needed to find something that not only played nicely with my current setup but could also leverage the funcitonality I have already built.

I ended up settling on Sinatra. Why?

I would mainly be returning JSON, not full blown views.
Needed a clean and easy nested routing system - Rails can get very condeluded if not done correctly.
Routes are based around the HTTP verbs (GET, POST, PATCH, DELETE, etc) so it is intuitive to write routes and conditions.
You can use conditions on routes so the same endpoint can return different pieces of data based on user agent, host name, etc.

So I’ve decided on using Sinatra for my API, now, how to get it into Rails without blowing away my existing work? Thankfully, a few different repositories on GitHub had already done this but with pretty minimal documentation so here is the more documented version.

Add Sinatra to your Gemfile

To get Sinatra into your application you will need to add the following line to your Gemfile.

gem 'sinatra'

Don’t forget to run bundle install after updating to get the new gems installed.

Adding in the Sinatra endpoint

Now that Sinatra is available, let’s add the Sinatra file to our Rails lib directory inside of it’s own api directory so that we can keep all of the API files together. Inside of the api directory create a file called core.rb. This will be the base of our API.

$ mkdir lib/api
$ touch lib/api/core.rb

The directory structure should now look a little like this:

app
└── lib
    ├── api
    └── core.rb

# .. snip

Once you have created the core.rb file it is time to write the code that will enable your first response from the Sinatra application. Below is my endpoint that will render at http://localhost:3000/api.

module Api
  class Core < Sinatra::Base
   get '/api' do
     "Howdy, from your Sinatra API."
   end
  end
end

Autoloading for the lib directory

To ensure your Sinatra application is loaded on boot, you will need to add the following line to your application.rb.

config.autoload_paths += %W(#{config.root}/lib)

Routing to your application

Now that the lib directory is being autoloaded, you need to update routes.rb to tell Rails that the /api endpoint is going to your Sinatra applicaton.

match 'api' => Api::Core

And that’s it! You can now open your browser to http://localhost:3000/api and it should render your API response.

Why you should be virtualising your development environments

2013-07-04T06:35:00+00:00

Within my daily workflow I can work on 5-6 projects over the course of the week and the likelihood of anyone remembering the exact dependencies and requirements for each environment is close to impossible. In addition to this, I need a development environment that will replicate the production environment so I spend less time debugging inconsistencies and more time shipping code. The solution? Virtualisation.

There are multiple methods of virtualising environments however I am going to cover using vagrant as it is simple with nil to very minimal performance overhead.

Prerequisites

Vagrant is a CLI tool for interfacing with VirtualBox so be sure to download it prior to trying to use vagrant. I am going to be using my mac however the instructions should be pretty well cross platform.

Installing vagrant

Vagrant is available for download however it is easiest to install via RubyGems.

$ gem install vagrant

Adding a base box

Once that has been installed successfully, you will need to add your box which will house your configuration and OS for the virtual machine. Luckily, there is vagrantbox.es which allows you to get started with a pre-configured base box with an OS already on it. If you are feeling adventerous or need to make your own, there is documentation for creating a custom one! I will be using a default Ubuntu Lenny 32bit base box within the guide so remember to swap it out for your actual box path and name.

To make the box available on your system, you need to add it to vagrant. Easy enough:

$ vagrant box add lucid32 http://files.vagrantup.com/lucid32.box

You should see some output similar to the following. This is Ruby grabbing the box on your behalf so it can be used within your project. Don’t worry where you do this, it will make it available globally.

[vagrant] Downloading with Vagrant::Downloaders::HTTP...
[vagrant] Downloading box: http://files.vagrantup.com/lucid32.box
[vagrant] Extracting box...
[vagrant] Verifying box...
[vagrant] Cleaning up downloaded box...

Creating the Vagrantfile

The Vagrantfile is the source of configuration for the vagrant instance. You can configure anything to do with the box via this single file. Everything from forwarding ports to sharing directories can be achieved and is all backed by solid documentation. To create a Vagrantfile within your project using your newly added box, simply change to the desired directory and run:

$ vagrant init lucid32

Here is an example of a Rails application with just port 3000 forwarding onto the virtual machine.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant::Config.run do |config|
  # All Vagrant configuration is done here. The most common configuration
  # options are documented and commented below. For a complete reference,
  # please see the online documentation at vagrantup.com.

  # Every Vagrant virtual environment requires a box to build off of.
  config.vm.box = "lucid32"

  # The url from where the 'config.vm.box' box will be fetched if it
  # doesn't already exist on the user's system.
  # config.vm.box_url = "http://domain.com/path/to/above.box"

  # Boot with a GUI so you can see the screen. (Default is headless)
  # config.vm.boot_mode = :gui

  # Assign this VM to a host-only network IP, allowing you to access it
  # via the IP. Host-only networks can talk to the host machine as well as
  # any other machines on the same network, but cannot be accessed (through this
  # network interface) by any external networks.
  # config.vm.network :hostonly, "192.168.33.10"

  # Assign this VM to a bridged network, allowing you to connect directly to a
  # network using the host's network device. This makes the VM appear as another
  # physical device on your network.
  # config.vm.network :bridged

  # Forward a port from the guest to the host, which allows for outside
  # computers to access the VM, whereas host only networking does not.
  config.vm.forward_port 3000, 3000

  # Share an additional folder to the guest VM. The first argument is
  # an identifier, the second is the path on the guest to mount the
  # folder, and the third is the path on the host to the actual folder.
  # config.vm.share_folder "v-data", "/vagrant_data", "../data"

  # Enable provisioning with Puppet stand alone.  Puppet manifests
  # are contained in a directory path relative to this Vagrantfile.
  # You will need to create the manifests directory and a manifest in
  # the file base.pp in the manifests_path directory.
  #
  # An example Puppet manifest to provision the message of the day:
  #
  # # group { "puppet":
  # #   ensure => "present",
  # # }
  # #
  # # File { owner => 0, group => 0, mode => 0644 }
  # #
  # # file { '/etc/motd':
  # #   content => "Welcome to your Vagrant-built virtual machine!
  # #               Managed by Puppet.\n"
  # # }
  #
  # config.vm.provision :puppet do |puppet|
  #   puppet.manifests_path = "manifests"
  #   puppet.manifest_file  = "base.pp"
  # end

  # Enable provisioning with chef solo, specifying a cookbooks path, roles
  # path, and data_bags path (all relative to this Vagrantfile), and adding
  # some recipes and/or roles.
  #
  # config.vm.provision :chef_solo do |chef|
  #   chef.cookbooks_path = "../my-recipes/cookbooks"
  #   chef.roles_path = "../my-recipes/roles"
  #   chef.data_bags_path = "../my-recipes/data_bags"
  #   chef.add_recipe "mysql"
  #   chef.add_role "web"
  #
  #   # You may also specify custom JSON attributes:
  #   chef.json = { :mysql_password => "foo" }
  # end

  # Enable provisioning with chef server, specifying the chef server URL,
  # and the path to the validation key (relative to this Vagrantfile).
  #
  # The Opscode Platform uses HTTPS. Substitute your organization for
  # ORGNAME in the URL and validation key.
  #
  # If you have your own Chef Server, use the appropriate URL, which may be
  # HTTP instead of HTTPS depending on your configuration. Also change the
  # validation key to validation.pem.
  #
  # config.vm.provision :chef_client do |chef|
  #   chef.chef_server_url = "https://api.opscode.com/organizations/ORGNAME"
  #   chef.validation_key_path = "ORGNAME-validator.pem"
  # end
  #
  # If you're using the Opscode platform, your validator client is
  # ORGNAME-validator, replacing ORGNAME with your organization name.
  #
  # IF you have your own Chef Server, the default validation client name is
  # chef-validator, unless you changed the configuration.
  #
  #   chef.validation_client_name = "ORGNAME-validator"
end

Getting onto your new machine

Once the Vagrantfile has been generated and you are happy with it, all you need to do is SSH into the machine using vagrant ssh (be sure you’re in the correct directory with your Vagrantfile when trying this).

$ vagrant ssh

Once inside, you can start your services, get new projects rolling, etc. You can also visit your site by visiting http://localhost:3000 (providing the application is running of course :p ). Most people will be fine leaving it at this stage but if you need a solution for multiple developers or other services, you might need to look at integrating puppet or chef into your workflow for provisioning tools.

Writing good commit messages

2013-02-04T06:35:00+00:00

One thing that is overlooked and under rated in team projects and especially open source collaboration is a good commit message and the benefits that come from that good commit message. Writing a good commit messages will be the difference between understanding the changes and needing to see the diff to understand the changes. Consider the following commit message:

[project/master_branch] updated database configuration parameters

Ok, so the database configuration parameters were updated, simple? Not exactly.

From this commit message we cannot understand why the commit was made or the scope of the issue that it resolved. Sure, it could just be a single one-off database parameter change or it could be another issue in another part of your project.

Want the solution? Write a better commit message. Here is an example of a good commit message and I will explain why in just a moment.

Updated database configuration parameters for new user

Changed the user credentials for database connection since 'user_123' is no
longer an available SQL user account.

Now everyone that reads this commit will know why the commit was made as well as now understanding the scope of the issue that was resolved and how this issue could effect other parts of your project.

Tips for writing good commit messages

The commit message

Use the first line to sum up the change
Leave an empty line (this is a formatting nicety for many clients)
Explain the commit in detail (what was involved in the commit, what the new feature does, etc)
Reference tickets, support lines if they were involved.

General

Format the commit message to be no more then 80 characters long. This will help readability and not require horizontal scrolling in many clients.
Don’t use “misc changes” or something similar as a commit message. It defies every rule previously mentioned.
If you cannot look back 5 years from now and understand why the commit was made, you have not wrote a good commit message.