Developer Operations

[arneschreuder]

This is the discussion board for and with Developer Operations.

[arneschreuder]

Please follow: https://nx.dev/ci/monorepo-ci-github-actions

[arulagrawal]

Hi, we have some questions related to general git workflow.

My current understanding is this:

• Everyone in the team forks COS-301/graduates
• Everyone makes the branch feature/team/feature_name
• Each person commits their work to their own fork on the branch feature/team/feature_name.
• DevOps merges everyone's feature branch with upstream feature branch (via pull requests)
• Either DevOps or Project Owners merge from feature branch to develop.

Is that the general idea?

[arneschreuder]

Hi There. That sounds almost right. Let me correct where wrong:

• Everyone, like literally everyone forks the repo. No work will be done on upstream unless through PR.
• Everyone, like literally everyone that works on anything related to code, makes feature branches using the convention feature/{whatever}, no team, nothing like that in the name.
• Correct
• DevOps is not responsible for this, this is the PMs job, but you can certainly assist. However, you need to keep an eye on ALL PRs, since some PRs might break the pipeline or conventions are not correct etc, then you need to step in. If a PR breaks the pipeline, this team, and the DevOps from that team must figure it out. To be specific PRs are made from arneschreuder:feature/whatever to upstream:develop.
• This is the PMs and the person who made the PR's responsibility, DevOps just participates if there are maybe things wrong, missing etc.

I will show more examples of this on Friday.

[arneschreuder]

The key cultural thing to get right here is that:

• PRs should be MANY and should be SMALL. Not few and large. So collaboration happens through getting stuff in develop, not through working on feature branches together. If anybody is working on a branch together, that means they are missing the point and you can help mitigate this.
• In the beginning, develop might break a lot, since there is a lot of dependencies between teams etc. But those should be catered for using mocks. If I am waiting for the auth team, for some service that they will have, they should've already provided some mock service that I can at least temporarily use. So allow for mistakes in the beginning ok, people will fall into the motion of things.

[arulagrawal]

Thanks, that clarifies quite a lot! Since things like tests aren't setup yet (and therefore no automated testing via CI) what are some of the things DevOps can do to provide value to the team in the beginning?

I remember you talking about gitmoji, would for example a check to ensure commit messages follow gitmoji be something doable? Or is that more just on the side?

[arulagrawal]

@arneschreuder

[arneschreuder]

Sorry, I see I never replied here @arulagrawal . Apologies. DevOps needs to work with CI/CD team to set up the automated testing ASAP. That should be one of the first commits. Even if there are no tests (which there should be some tests already afaik). The pipeline is agnostic whether the implementation is there of not. It just provides the plumbing. So I would say that is an urgent todo. Those repo chores that you mention are also important, but those only come after CI/CD has implemented their part. The DevOps person needs to take this pipeline to the team, and make sure the team is prepping for it: by having tests, linted, etc. Look at like Husky, to do precommit hooks.

[DYL-WF]

@Alexander-Muendesi, @COS-301/go from what I have gathered your team is responsible for CI/CD. Please let me know if you need anything from my end as a devOP.

I eagerly await any updates from your end. At the moment, your wiki is empty.

[Alexander-Muendesi]

Hey there.

We are still in the phase of "System Requirements Definition" and hope to be done early next week. Once that is done the wiki will be updated accordingly and we will coordinate with DevOps members.

In the mean time did you perhaps visit the conversation I had with Arne about the whole CI/CD and devOps thing in the Project Manager discussion board? That could clear things up a bit if anything is foggy

Link: #3 (comment)

[DYL-WF]

@Alexander-Muendesi is there a WhatsApp group? if so can I please get a link to join it? I'm sorry, I missed the first meeting.

[Alexander-Muendesi]

There is no group that I am aware of. Maybe ask the other DevOps people. I dont know why there is a need for one though? My team will be adding some scripts most probably by the end of Sunday to the pipeline.

Also keep in mind I am a PM for CI/CD, not DevOps. Maybe if you guys have a lead devOps rather message that individual

[10416260]

@sinothandomabhena Please take a look at this thread

[Alexander-Muendesi]

@sinothandomabhena if you have some specific questions just drop them here and I will answer them best I can. If you require a discord voice chat I feel like you will gain more clarity talking to my DevOps member. He is more in tune with things

[10416260]

@Alexander-Muendesi Please help by getting in touch with @sinothandomabhena , he would like to have more discussion when it comes to the DevOps role, he's actually a UI engineer but his role involves working closely with the DevOps team in terms of deployment

[arulagrawal]

@arneschreuder

Should a DevOps member be providing team wide github actions? Or should github actions be left to the CI/CD team?

If I should provide some team wide automations then what is the flow to do so? 2 ideas were brought up in the DevOps meeting

1. I make the actions on my fork, and everyone forks my fork to have access to my actions.
2. I make the actions on my fork, and everyone PRs to my fork to get the actions to run. When the actions pass, we make PR to upstream.

Which of the 2 is preferable? Should I even worry about github actions as someone who is not on the CI/CD or DevOps team?

[arneschreuder]

No need for team-based actions. The global CI/CD actions is all we need. Your role is then to sure that you help get your team's code into develop. If your team's PRs break the development branch, I am going to come ask you. So you have to ensure that the team confirms to global CI/CD pipeline and developer processes.

[arulagrawal]

Okay so just to be sure, I would have like 0 commits since I am not providing code nor actions?
Also, I thought about writing a contribution guide in the project wiki. Something that overviews what a developer (or anyone contributing) should do to get their changes into upstream. Essentially an overview of the develop process. Is this a good idea? @arneschreuder

[arneschreuder]

Absolutely.

Also, it doesn't necessarily mean you will have 0 commits. You might help you team to get their PRs in upstream, and that might require you jumping in to help.

[arulagrawal]

Perfect thanks

[MichaelViljoen]

@arneschreuder
We understand that we need two docker files, one for the client and one for the API, is it appropriate for us to store them in the root, and do they need a specific name.

[arneschreuder]

They need to be in apps/api/ and apps/client/

[nimGaluu]

@arneschreuder,

Concern about friday's Demonstrations - in short:
Since our major task for each team has already been allocated as a feature, CI/CD, and not all teams have had a lot of git movements that we might need to monitor or 'supervise'. According to the playbook, Implementations start in sprint 2 and I believe then any DevOps member would have things to do since that will be the time PRs would be sent in and a lot of pipeline maintenance would be needed.
WIth that said, I am afraid not much is there to present since we have minimum to none contribution as a Role. Those who have double roles would have advantage over those who have chosen this role only. Is that fair?

Please advice

KInd Regards,
_en

[arneschreuder]

Hi There. Thanks for bringing this up. I suspect we might need to broadcast some clarity as to what is expected for Friday. But basically the project managers need to come show up what their team did in sprint 1. These could be meetings, discussions, contributions to the wiki, code, code reviews, etc. We will probably on our side pull the repo log to just verify. Nonetheless, you should think of sprint 1 evaluation as a small progress update. We will give a small mark (like 3 of the 10) for spring 1 evaluation and the only requirement is that you have deliberately worked towards your deliverable.

So in the case of DevOps. You should have had meetings now, talked and discussed how CI/CD is going to be working with the CI/CD feature team. Ensure that everybody is coming right with the repo. If there are PRs, assistance here helps. Most importantly, if you have tasks allocated to you, you should be fine.

[DYL-WF]

So my API has just done somework on his end and has now made a pull request. Can someone please confirm with me that our process was correct.

-> Updated Fork
-> Pull from fork

->created new branch(shorts-api) off develop -> added API files
-> commited
-> pushed to origin(Fork)

-> Made PR to upstream from FORK/feature-branch

[arneschreuder]

Sounds good

[arneschreuder]

Hi @COS-301/developer-operations you have your first challenge. So the CICD team added the basics of the GitHub actions last night. Well done to them. So the pipeline is currently broken. Now this is where as Developer Operations come in, we jump in, investigate the issue and help fix it. You will see it is mostly linting issues.

[RobynHancock]

@arulagrawal Will you be compiling the code of conduct?

[arulagrawal]

I can

[RobynHancock]

@arulagrawal Okay. I can also help, if you want it

[arulagrawal]

@RobynHancock not gonna lie, I was gonna use one of the two templates offered by GitHub.

[arneschreuder]

@arulagrawal you can use mine sure. Yeah, this PR #284 fixed the pipeline. So now CI/CDs stuff is in and the rest of the PRs will be strict. People can not merge in if stuff fails.

[priyo5]

Hey guys

Please take note that my team and I have added new tests to the repo

Main purpose is to check meta tags of a PR

If one of these tests fail all you have to do is add the related tag it fails on

[arulagrawal]

Possible breakage of pipeline:

I noticed PRs #498 and #500 failing with strange errors on the lint test.

I cloned upstream develop and ran the test locally:
git clone https://github.com/COS-301/graduates
yarn install --frozen-lockfile
yarn nx run-many --target=lint --all

and the result of this:

$ /tmp/main/graduates/node_modules/.bin/nx run-many --target=lint --all

Unexpected error:
TypeError: Cannot read properties of undefined (reading 'hashCommand')
    at /tmp/main/graduates/node_modules/@nrwl/linter/src/executors/eslint/hasher.js:11:32
    at Generator.next (<anonymous>)
    at /tmp/main/graduates/node_modules/tslib/tslib.js:117:75
    at new Promise (<anonymous>)
    at __awaiter (/tmp/main/graduates/node_modules/tslib/tslib.js:113:16)
    at run (/tmp/main/graduates/node_modules/@nrwl/linter/src/executors/eslint/hasher.js:7:34)
    at TasksSchedule.<anonymous> (/tmp/main/graduates/node_modules/nx/src/tasks-runner/tasks-schedule.js:117:19)
    at Generator.next (<anonymous>)
    at /tmp/main/graduates/node_modules/tslib/tslib.js:117:75
    at new Promise (<anonymous>)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.```

[DYL-WF]

can someone please confirm whether the pipeline is broken or not please? cause i was planning on completely reseting my commits and starting from scratch. Id hate to do that if the problem wasnt on my end this whole time. @arneschreuder @COS-301/developer-operations

[arulagrawal]

@DYL-WF, have a look at #502 and #504, Arne says he will look tonight

[DYL-WF]

@arulagrawal should i hold of until tomorrow then for my pull request?

[DYL-WF]

but thanks. i was stressing that i broke something

[arulagrawal]

@arulagrawal should i hold of until tomorrow then for my pull request?

holding off is probably best, any new PRs at this time would fail anyway

[Daniel-J-Burgess]

Hi everyone
I just want to confirm do we still need to link an issue to each PR?
It appears almost no one is doing this.

[Alexander-Muendesi]

you definitely should be doing that

[arneschreuder]

Hi @COS-301/developer-operations please fix for us. The post PR CI tests are failing, see error in actions:

[Alexander-Muendesi]

Hi @arneschreuder. With regards to that error, its is a very weird one. The first time we tried implementing actions to run on merge, on our forks we never got such an error then when we moved it to repo it started giving that error and hence we(CI/CD) removed it. Then yesterday we did some changes to see what was causing it and made a PR to bring it back to the pipeline. It was working fine afterwards and all of a sudden from PR692 it starts complaining again. Sometimes it wants to work sometimes it does not want to work.

My team is struggling to see why its doing that. Any feedback would be appreciated

[arneschreuder]

Maybe it is the cache thing? Not sure. Maybe try removing the cache to see if that fixes it. That would be dumb, because then all the builds are going to take super long.

[DYL-WF]

should I hold all incoming PRs from my team until this issue is resolved @Alexander-Muendesi ?

[Alexander-Muendesi]

You can continue with them. We want to see with fix added if it will break again. The fix was just added now

[tshegomotlatle]

Could the team responsible for the student-explore page shed some light on this error that is currently experienced when the Full CI tests run

[tshegomotlatle]

@COS-301/java

[Daniel-J-Burgess]

Appears to be caused by the recent database changes, we will update it.

[Daniel-J-Burgess]

@COS-301/sas the company profile API will need to be updated as well.

[TimovdMerwe-18]

Yes, the database has changed. I'll fix it asap

[MichaelViljoen]

@arneschreuder what is the turnaround time for reviewing pull requests?

[arneschreuder]

You should ask your own team… your own team should review and approve the PR.

[MichaelViljoen]

Sorry, will sort it out in my side.

[lindzik]

@arneschreuder for PRs that are now being made into release we need to have 2 approving reviews. Are you the second reviewer or can a team member review it for us?

[arneschreuder]

For now I will review it, but if it becomes much I will set it to one reviewer

[lindzik]

I got this error for the affected e2e tests for my team's PR. client shorts team do you have a fix for it.