chore: automate checking that the quickstart works as expected

[leggetter]

TODO

• Move to acceptance-tests/docs/quickstart.sh
• Remove clean.sh and clean up in quickstart.sh
• Add docker-compose ... down
• Create .env in appropriate place in the examples directory and use that instead of the top-level example
• Update to use PostgreSQL b41be77
• Add GitHub action to run on release. If action fails, then release should fail.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
outpost-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 28, 2025 9:33am
outpost-website	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 28, 2025 9:33am

[alexluong]

Hey @leggetter, a few notes after I went over this quickstart test here.

First I just want to catch up on the .env.example conversation. I recalled you bringing that up a few weeks ago about the Docker example .env.example vs the root .env.example. It seems you removed that in favor of the top-level one a while ago (commit). I just want to revisit this decision a bit.

My POV is that the top-level .env.example is for Outpost contributor who needs to be very familiar with Outpost available configuration and needs the .env to tweak it as they work on the service. There seems to be a lot going on in this example and therefore may not be the best thing to introduce to a user who just wants to get things going.

That’s just my 2c here. Ultimately, it’s up to you whether you think the top-level .env.example is suitable for the quickstart. Regardless, I’d love to know your thoughts so that I can adjust how we evolve that example. As of right now, when I make changes to .env.example, I don’t think about a potential user use case, so it would be helpful to know so that I can account for that audience as well.

---

Back to the automation changes, the biggest thing I want to highlight is the main objective of this script. I understand we want an automated way to test if the quickstart is working correctly.

When do you want it to run? (manually, GitHub action, etc.)

The thing is the quickstart is set up to use a specific release, so I just want to make sure we’re on the same page that it won’t catch changes between commits if a new release is not provisioned.

The git clone line sort of led me to assume that you want to test every commit change, in which case this won’t solve that. We may need to rethink how we want to go about that, so would be great to discuss a bit more high-level objectives if possible too.

---

A few smaller feedback

1: There are 2 scripts, quickstart.sh and clean.sh, and it feels like they’re supposed to be run a bit differently.

For example, I have to run from the root dir $ ./test/quickstart.sh for it to work. However, with clean I have to cd into the test dir first from the root: $ cd test && ./clean.sh.

2: We should consider docker-compose ... down to clean up the Docker containers usage as well?

3: I wonder if test is a good directory name for this. I think it’s probably okay. It just feels like a pretty generic name and I would expect a bigger test suite of some sort for a test dir.

[leggetter]

@alexluong

Back to the automation changes, the biggest thing I want to highlight is the main objective of this script. I understand we want an automated way to test if the quickstart is working correctly.

When do you want it to run? (manually, GitHub action, etc.)

The ability to run manually to check things are working as outlined in the docs and upon release/merge into main. Not on every commit.

My assumption was that merging to main was the point at which we run a release. So, there's probably a discussion we should have about that.

1: There are 2 scripts, quickstart.sh and clean.sh, and it feels like they’re supposed to be run a bit differently.

I added clean.sh as a utility because I was creating a lot of files during dev. However, I think just clearing up after a successful run is fine and leaving the files there if a failure occurs is a better approach. So, I'll delete the clean script.

2: We should consider docker-compose ... down to clean up the Docker containers usage as well?

Good idea.

3: I wonder if test is a good directory name for this. I think it’s probably okay. It just feels like a pretty generic name and I would expect a bigger test suite of some sort for a test dir.

The goal is to ensure the release is inline with the quickstart. So, it is kind of top-level. How about acceptance-tests/docs/quickstart.sh?

[alexluong]

My assumption was that merging to main was the point at which we run a release. So, there's probably a discussion we should have about that.

Yes, but in order for the script to work, it must:

1: wait AFTER the release is live on Docker
2: somehow updating the version to the latest

Currently, we specify the version in the quickstart example.

---

Can you clarify again why we need to git clone the entire project before each run?

---

The goal is to ensure the release is inline with the quickstart. So, it is kind of top-level. How about acceptance-tests/docs/quickstart.sh?

Ultimately I don't want to nit too much on this, so whichever works for you! We can always change later if necessary. Just a suggestion, maybe something with quickstart as part of the dirname to specify? Or maybe scripts/quickstart-test or something along those lines, within a generic scripts dir?

[leggetter]

Yes, but in order for the script to work, it must:

1: wait AFTER the release is live on Docker
2: somehow updating the version to the latest

I believe we should treat docs as part of the product, so should not allow a "full" release to go out if the docs are inaccurate.

I see we release on a tag being applied. Could we also create a new docker image when a merge to main takes place?

We should also ensure that the YAML is up to date. But there's definitely a chicken-and-egg problem here 🤔 Is there a @latest option with docker compose?

[alexluong]

Could we also create a new docker image when a merge to main takes place?

We can but would that "clutter" the Docker page? Maybe that doesn't matter.

Is there a @latest option with docker compose?

Yes, there is an option. We pinned a specific version because I wanted to make sure the quickstart would always work and was concerned "latest" won't work if documentation is outdated. I do agree with your sentiment that it shouldn't happen, so we can adjust there.

One minor note is that latest can be cached so if there's a new version and you have already pulled the image before, you would have to manually pull again.

---

Let's circle back to the workflow here. My understanding is you want to run this on every PR before (or after) a commit is merged to main. I don't think pushing to Docker every time makes sense here, especially if we don't consider it a release.

Or maybe we should run this before making a new release?

Regardless, I think one solution is to build the Docker image in CI and run the quickstart off of that image.

Let me know if you'd like my help setting something like this, or a workflow that works for you. Happy to dig in, just need to make sure we're on the same page first.