SLC-1 dev

[g105b]

Biff's going to be supplying the wireframes for the accepted aggregation functionality, and while he's doodling those I've been working on two things:

1. Trackshift's tidy - now we have a clear path of where the development is going, there were a few things that needed sorting on the project before we progressed. Notably, I've got the test bed passing again (so we can be fully TDD), and introduced two new quality tools (Code Sniffer and Mess Detector) that will highlight any mistakes and areas of complex code, before they make their way into the codebase.

2. Spotify API - I started building a type-safe API for Spotify, so it documents itself rather than have to be back and forth in documentation pages. I'm only building the functionality out for what we're using in Trackshift, as there's a lot more of the API that we won't use (audiobooks, playlists, etc.). One day I'll code the rest of the functionality, but it's not needed yet.

[g105b]

I spent yesterday working on the SLC-1 issues and a few hours today. The functionality is complete, but there are a few improvements that need to be added to make this release ready.

The first thing that I tackled was getting the data ready for the new simple way we're working on it. A description of the layout of our Entities is in issue #82.

---

After our Entities were all in place in the system, I could work on getting the aggregated usages output in the correct tables again. This is done similarly to the first prototype version, but now it's backed by a database so we have proper integrity and a full audit trail back to where the calculations are made.

Not much to shout about for this issue - details in #84.

---

The next issue that was tackled was to handle Cargo files. The new Entities made this a breeze to implement - take a look at the CargoUpload class - I'm sure these Upload classes will become more complex over time, but for now we can make a pretty safe assumption that the uploaded file is a Cargo file if it matches the few known column headers.

Something I want to work in to a future release is the ability to upload a zip file containing the Cargo data (I'm told this is how the files are received by labels). For now, it requires the royalty_extended.csv to be used.

---

Now we're working towards getting this product in front of people, this tabbed interface is introduced:

It works in a similar way to before, but we're going to have the rows clickable to open the Product view, and the Balance/Output/Profit will be editable in the next release.

---

I'm leaving the fun task until last. An implementation of Biff's fun animation idea:

[g105b]

What's next?

There are three things I need to work on before it's release-ready:

1. Album artwork proved to be too slow using the standard approach. My plan is to load the top 5 earning products' artwork at import time, then whenever the browser sees a music symbol without a matching artwork, it will request the artwork to be cached in chunks of 5 or 10 products.
2. Speed of uploading. At the moment, I've just done all the processing on the same thread, during the upload phase, and I said I'll improve it if/when desired. Well, after testing with files that are a few megabytes in size, I think it's time to shift the processing onto a background thread, so we can get to the product table as soon as possible and see it updating in front of our eyes.
3. Deployments. There's a few too many moving parts with the server and deployments, so it isn't as automated as it needs to be. I really want to see the public release done automatically through Github, so I still need to do some work on this part of the system.

[g105b]

The first thing I've started on today is Cargo imports. Currently, the files import and are handled by the importer, but the products are not assigned correctly. I'm building up some test data to include with the Behat tests.

While I'm in the tests, I need to finish off getting them bulletproof for Github Actions. There are a few flaky tests in there (Code Sniffer and Mess Detector) that need buckling up so Github Actions is happy running them, so we get nice automated deployments.

Once all of this is completed, I'll get on to Spotify and getting background work running so any heavy lifting can be shifted away from the foreground process.

[g105b]

Github's happy with the changes I've made to Actions, and all the test scripts are happily running. Now on to the deployment, so when I update the Spotify API stuff we'll be able to see it deploy onto the slc1 branch for testing.

[g105b]

STRESS TEST! I managed to break the Cargo importer by adding way too many lines, and it's clear where improvements are to be made. I'm not going to worry about handling this much data, as there's no point in optimising until optimisation is necessary, but I do need to prevent it crashing. Once this is done, I'll make a deployment to the SLC1 branch, then get the Spotify artwork loading in the background.

[richardbirkin]

how many lines was it?

[g105b]

About 500,000

[richardbirkin]

ah, not a problem yet

[g105b]

For SLC1 to be release-ready, it needs to be fast and accurate. Those two points are what I'm working on now, to get the codebase bulletproof and ready for release.

I would like to create some test files with lots of fake, generated data, so they can be used in integration tests, as it's a bit too easy to test files with < 10 lines. Maybe we can come up with something together @richardbirkin ? I'm talking about a real-looking file with a million rows, so we can default to testing with these huge files, building in headroom for when we get a user dropping something of that size.

One feature that was added last week was the Spotify API integration, so we can start looking up metadata such as the album artwork. I actually found a weird bug in Spotify's API, which was meaning every so often we'd get seemingly random artwork back, but luckily we're building our own library for Spotify API usage, so we can fix it in the library rather than have a hack in Trackshift, or have to wait for a fix from Spotify.

The new Spotify code is integrated in our library, and this is being used directly by Trackshift. Now when I drag in a big statement, all tracks that have matching artwork are correctly matched, and all of this is done through the lazy loading script - to the user, there isn't really any noticeable delay in waiting for the artwork now.

On to the primary focus of today - the processing speed. I've built the current iteration with the least amount of code possible. I think that's a great style of development, as it lets you expand and optimise when necessary. It's necessary now because of the introduction of the database.

Each time an operation is performed on a database, it's done in a transaction. It takes a fraction fo a second per transaction to work with data in a database, and currently each record is being inserted individually, which adds up to a lot of fractions of a second. I'm working on a mechanism to perform all this in a single transaction, so we're only going to be limited to the upload bottleneck.

[richardbirkin]

Steps for this week to get SLC1 wrapped up:

• Processor speed optimisation
• Loading screen user journey
• Spotify artwork matching for complilations and reported mismatches (easy wins only)
• Oustanding bugs

Processor speed / Loading

The simplest solution:

There may be a third step where IF TIME = > 60s then we'll email you when it's done. But for now the above will suffice for SLC1. SLC2 can have the next iteration if we see processing times of longer than 30s on the current server.

We should also discuss the cost of processing on a super server separate to the app screens.

[richardbirkin]

There's also another couple of outstanding points about file uploads:

• check that Cargo Digital -royalty-extended.csv is using column K Reported Royalty for earnings.
• add Cargo Physical 123456_YYYY_Q1.xlsx to importer to allow users to add physical sales to TS

This last point will be a good segue into SLC2 as physical distributor accounts often contain manufacturing costs (if that has been done by the ditributor on behalf of the label) and therefore can be added to ProductCosts at import.

[richardbirkin]

Cargo Physical relegated to SLC2 as it is an .xlsx file that needs to be converted to CSV by the importer before being processed

[richardbirkin]

Also:

• Merge biff-style with slc1 and then master
• fix deployment so that tests pass and trackshift.app is live.

[richardbirkin]

• Unhappy paths (micro-UI to explain things going wrong)

[richardbirkin]

• Add Tunecore importer

[g105b]

I've been setting up the production server. We're all ready to go now, so I'll grab something to eat and DEPLOY!

[g105b]

Need to merge everything into master first, including new tests, and ensure the tests are all up to date with the terminology changes we've made over the last few sessions.

[richardbirkin]

I've tested the delete-me functionality.

Deleting all files = DONE

Extending period by 3 weeks = not working.

[richardbirkin]

To do:

• add container to delete-me
• style delete-me buttons same as the rest of the site
• extend by 3 weeks test

[richardbirkin]

Since all we're doing in SLC1 is dragging and dropping files, I don't think we need to make a big deal of the 3 week thing. So I've just removed it from the page entirely.

SLC2 involves some data entry from the user, but even then I don't think we need the expiry button. Instead we turn off that functionality.

What do you think @g105b ?

[richardbirkin]

I've commited and pushed all my changes and am happy for this to go live @g105b

[g105b]

Amazing! I'll take one last look and then I'll press the button 😨

[g105b]

Wrong emoji but it looks funny so I'll leave it.

[richardbirkin]

haha. one day we need an actual physical button under a locked glass bowl.

[g105b]

Two little tweaks made: both chrome-specific. 1) background position of body was clipping on the right, 2) mask-image needs a -webkit prefix like it's 2012.