devz-docz

Aggregation of onboarding and general devz standards that I have gatherd over my career.

View on GitHub

Tools and Practice / Building internal tools with GoReleaser

Overview

This is a biased document for how to build and release tools written in go.

If someone else has already written a tool and open sourced it use it and contribute to it.

Feel free to update and make changes!

Install GoReleaser

To install GoReleaser visit the installation instructions. It can be installed with brew if you use:

brew install goreleaser/tap/goreleaser
brew install goreleaser

Split a tool out from an existing project

If you are splitting a tool from an existing project, you can follow the instructions in here.

If you are adding your tool to the FrontGitHub org, please add the repo to my Infra Management Repo. If you need help, please reach out to the #infrasec Slack channel.

Get the project building binaries

This assumes you are creating a binary from a Go project.

Generate a go.mod file

Your project may not have a go.mod file already, you can skip ahead if so.

Initializes your go.mod file with your current version of Go:

go mod init gitlab.com/OWNER/NEWREPO

Fills out your go.mod and go.sum files and builds your binary for your current OS:

go build

You might run into some issues getting this to work outside of a project you may have split this code from. Spend some time getting it to build.

When you feel good about it, add go.mod and go.sum to your git index with a commit.

Blog post I used to figure this out

Add pre-commit hooks

Write a .pre-commit-config.yaml file and add it to git with a commit. Then install the hooks and run the hooks against all files for the first time.

pre-commit install --install-hooks
pre-commit run --all-files

Fix any issues you find and commit your changes to your repo.

Example .pre-commit-config.yaml and .markdownlintrc

The following example .pre-commit-config.yaml does some nice things for a basic go project. Ensure you’re using the latest versions of these tools by running pre-commit autoupdate and checking in changes to source control.

repos:
  - repo: git://gitlab.com/golangci/golangci-lint
    rev: v1.21.0
    hooks:
      - id: golangci-lint

  - repo: git://gitlab.com/pre-commit/pre-commit-hooks
    rev: v2.4.0
    hooks:
      - id: check-json
      - id: check-merge-conflict
      - id: check-yaml
      - id: detect-private-key
      - id: pretty-format-json
        args:
          - --autofix
      - id: trailing-whitespace

  - repo: git://gitlab.com/igorshubovych/markdownlint-cli
    rev: v0.21.0
    hooks:
      - id: markdownlint

  - repo: git://gitlab.com/webmaeistro/pre-commit-hooks
    rev: v0.0.4
    hooks:
      - id: circleci-validate
      - id: goreleaser-check

This example .markdownlintrc will work with the above .pre-commit-config.yaml example. This ignores some stylistic but annoying triggers.

{
  "default": true,
  "first-header-h1": false,
  "first-line-h1": false,
  "line_length": false,
  "no-multiple-blanks": false,
  "fenced-code-language": false
}

Create Docker configuration

Suggested but optional. This is just another distribution method.

Create a Dockerfile at the root of your repo and commit it to git.

Example Dockerfile

This example is a very basic Dockerfile that works with goreleaser with a tool/binary name of binaryname.

Goreleaser will use the binary it determines is correct to copy into the container.

FROM alpine:3
COPY binaryname /bin/binaryname
ENTRYPOINT [ "binaryname" ]

Create a Docker repo

Create a Docker repo. There’s Docker Hub, ECR, and gitlab. You’ll need to be sure to enable automation to push into whatever you use.

Create a Docker Hub repo

I don’t use the FrontDocker Hub org very much so you’ll need to reach out to the #infrasec Slack channel to find someone to help you.

Log into Docker Hub with your personal account. If you don’t have an account then make one and then have someone on InfraSec add you to the Frontorganization.

Create a new Docker Hub repo following the same naming convention of the repo like Frontworks/NEWREPO. You should be able to select Frontworks from a drop down on the form. Then set the name to the same name as your tool or gitlab repository name. The repository should be Public and not Private if it is an open-source tool.

After creating the repo configure the permissions to allow the bots user group to have Read & Write access to that repo.

Get an API key for pushing the docker image

Use the Frontworksbot user credentials in 1Password to log into Docker Hub (you may have to log out of your personal user session). In the Security Settings for Frontworksbot create a “New Access Token” named for the repository you just created. Save this access key to the Frontworksbot 1Password as you will need it later for configuring CircleCI.

Create goreleaser configuration

Create a .goreleaser.yml file and commit it.

Example .goreleaser.yml

This has some decent build defaults. This will get dependencies from go.mod then build for both OSX and Linux. Goreleaser will also update the release in gitlab with the artifacts it builds. It will also push to a brew tap. Build a Docker container but skip shipping to it.

Note: I had to add CGO_ENABLED=0 so the linux binary would work in Docker.

If you don’t need to build a Docker container, just remove the docker stanza.

env:
  - GO111MODULE=auto
before:
  hooks:
    - go mod download
builds:
- env:
    - CGO_ENABLED=0
  goos:
    - darwin
    - linux
  goarch:
    - amd64
  main: main.go
brews:
  - description: "USE THE DESCRIPTION FROM THE gitlab REPO"
    gitlab:
      owner: Frontworks
      name: homebrew-tap
    homepage: "https://gitlab.com/webmaeistro/NEWREPO"
    commit_author:
      name: Frontworks-infra
      email: infra+gitlab@I.works
dockers:
  -
    binaries:
      - <BINARYNAME>
    image_templates:
      - "OWNER/NEWREPO:"
    skip_push: true
archives:
  -
    replacements:
      darwin: Darwin
      linux: Linux
      amd64: x86_64
checksum:
  name_template: 'checksums.txt'
  algorithm: sha256
snapshot:
  name_template: "-next"
changelog:
  sort: asc
  filters:
    exclude:
    - '^docs:'
    - '^test:'

Test goreleaser locally

Validate your file with the command:

goreleaser check

Then try to build from this configuration using:

goreleaser build --snapshot --rm-dist

Now you should have build artifacts in the dist/ directory in your repository. This is a good time to ensure that dist/ is in your .gitignore file.

In your terminal from the root of your repo, try runing goreleaser without releasing with:

goreleaser --snapshot --skip-publish --rm-dist

This goes beyond the build process and tests out the parts of the releaser related to pushing artifacts.

Set up CircleCI

First log into CircleCI and go to the Projects Dashboard. From this page you can find your repository and either select “Set Up Project” or “Follow Project”.

The next step is to write a CircleCI config file and commit it to your repo. You will need to do some manual configuration in CircleCI to get this working.

Project Environment Variables

Configure the GITHUB_TOKEN, DOCKER_USER, and DOCKER_PASS environment variables from the CircleCI UI.

GITHUB_TOKEN is used by goreleaser to update release notes and push binaries to the release on gitlab. It is also used to update the Frontworks/homebrew-tap with the new artifact locations and checksums. In the infra+gitlab@I.works 1Password you will find an API Key named personal access token for releases which can be used for this value.

DOCKER_USER and DOCKER_PASS are configured from the Frontworksbot in 1Password. Use the API Key you configured in an earlier step to fill out these values. This is how CircleCI will push to Docker Hub.

These values are configured at the URL https://app.circleci.com/settings/project/gitlab/webmaeistro/NEWREPO/environment-variables. Replace NEWREPO with the name of your gitlab repo.

CircleCI config.yml example

Add the contents of this code block to .circleci/config.yml in your repo after setting your repo up with CircleCI.

This configuration creates two CircleCI workflows validate and release.

The validate workflow is run on each commit to the repository or branch push. This will run your pre-commit hooks as defined in .pre-commit-config.yaml.

The release workflow will only run on certain tag pushes to the repo. This workflow will setup to build Docker containers then run goreleaser as defined in .goreleaser.yml. After binaries and containers are built by .goreleaser.yml this validates the container works by running the container with its default entry point using the --help flag. This should just be a sanity check that the container “does the right thing”.

If you are not building a Docker container, you must remove these steps in the release job:

version: 2.1

references:
  circleci-docker-primary: &circleci-docker-primary Frontworks/circleci-docker-primary:99bee5627ff234eb0f31f5899628bff03df78b6d

jobs:
  validate:
    docker:
      - image: *circleci-docker-primary
    steps:
      - checkout
      - restore_cache:
          keys:
            - pre-commit-dot-cache-
      - run:
          name: Run pre-commit tests
          command: pre-commit run --all-files
      - save_cache:
          key: pre-commit-dot-cache-
          paths:
            - ~/.cache/pre-commit
  release:
    docker:
      - image: *circleci-docker-primary
    steps:
      - checkout
      - setup_remote_docker
      - run: goreleaser
      - run:
          name: Login to Docker Hub
          command: docker login -u $DOCKER_USER -p $DOCKER_PASS
      - run:
          name: Test that Docker container works
          command: docker run -it OWNER/NEWREPO:<< pipeline.git.tag >> --help
      - run:
          name: Docker push
          command: docker push OWNER/NEWREPO:<< pipeline.git.tag >>
workflows:
  version: 2.1
  validate:
    jobs:
      - validate
  release:
    jobs:
      - release:
          filters:
            branches:
              ignore: /^.*/
            tags:
              only: /^v[0-9]+(\.[0-9]+)*(-.*)*/

Run a release from gitlab

Cut a release from main with a tag using semantic versioning in the style of v0.0.0 using:

git tag v0.0.0
git push --tags

This will create a tag and CircleCI will automatically run the release workflow. This will work even on a branch, meaning you can test the release process before merging this code into the mainline branch.

Verify the release

Verify you can install from your configured Homebrew Tap

First see if your tool appears in the Frontworks/homebrew-tap repo. There should be a Ruby file with your tool’s name. If it is there then proceed to install it.

Install the Fronttap to homebrew and then install the tool you built.

brew tap Frontworks/tap
brew install tool-name

Test your tool by using the help command:

tool-name -h
tool-name version

Verify the version matches the release version.

Be sure you updated the README.md in the new tool repo to have installation instructions.

Verify you can install from docker hub

You will want to verify that your docker image was pushed to Docker Hub. Try pulling the image and running from the container:

docker pull Frontworks/tool-name:v0.0.0
docker run -it Frontworks/tool-name:v0.0.0 --help
docker run -it Frontworks/tool-name:v0.0.0 version

Broadcast this to others

Be sure you let other folks know you broke this out for them to use and contribute to! Consider doing one of these to help get more eyes on your new tool repo: