Blog

Entries tagged "Programming"

I put off writing music by writing...a shopping cart

Posted on 2023-01-23 10:09:00 by Kyle and tagged with Programming

One of the things I really enjoy about being a self-publisher composer is that there’s a variety of tasks to work on (read: get distracted by) when I’m not composing. This is a story about one of those tasks.

You may know that I’m addition to my work as a composition instructor at NDSU and my work as a composer, one of my side gigs is web development—I write software (Liszt) for running schools of music. So of course, one of my composing tasks is maintaining a website where people can buy my music. And to make that happen, you kind of need some sort of shopping cart.

For the past several years, I’ve used a great shopping cart platform called Snipcart to handle the shopping part of my websites, both at KyleVanderburg.com and at NoteForge. It’s kind of a drop-in solution: include some code, and as long as your “add to cart” buttons are coded right, they take care of everything. It’s $10 a month, and they package up all the purchasing information and send it off to Stripe, my payment processor.

Recently though, Snipcart has been unable to charge my card for the monthly subscription. When I’ve pressed for diagnostic information, they’ve pointed me to Stripe, and I figure if I’m going to have to deal with Stripe anyway, why not save $10 a month and build my own shopping cart?

The past few days have involved just that. Here’s how it has worked.

First off, I started off with some things that helped out already:

My sites are built on Liszt, which means I have a lot of control over the databases and how the site can communicate with the database.
I already have a database with info about prices, shipping weight, product images, etc. I’m not tracking stocking info because most of my work is print-on-demand at this point.
I have a pre-existing relationship/development account with Stripe due to building a payment gateway for invoicing several years ago.
I don’t want to use the pre-existing payment gateway I've built because…we’ll, I just don’t. It would make things complicated (collecting name and address information for example). We'll just let Stripe handle it.
My websites already have a built-in off-canvas “drawer” component that I can use (on the NoteForge site, it comes up when you click the perusal score button).
I want to take advantage of Stripe’s checkout feature, so the only thing I need to worry about is the cart functionality.

So with those constraints, the first thing I need are a couple of databases, one for the “cart” and one for “cart items.” Cart doesn’t need much more than some sort of identifier, while cart items need fields for cart id, item, and quantity. (It occurs to me on this write-up that there might be a way to build this without a Cart database, but too late now).

Next up, we need a cart page on the website to handle all the cart functions. I decided to program it in PHP because I’m faster at that than writing it with JavaScript. The first time the cart page (let’s call it cart.php) is loaded, it creates a Cart database record, gets a Globally Unique Identifier (GUID) and writes that GUID as a cookie to the user’s browser. (I considered using HTML local storage or PHP sessions, and cookies seemed like the easiest.) Liszt just generated GUIDs for every table row anyway so that’s easy.

In pseudocode: (Note: none of this is actually production code, but it's close enough to explain it)

build(); //This will create the record and the GUID.
	setcookie("NoteForgeCart",$cart->guid,time()+60*60*24*30,"/"); /* expires in 30 days */
	$cartid=$cart->guid;
}else{$cartid = $_COOKIE['NoteForgeCart'];}
?>

I opted to use url constructions like cart.php?addItem=myAwesomeScore to manipulate the cart. This requires that every change to the cart cart involves a page reload, but the code is lightweight enough to where I’m not worried about performance. I could have written this in JavaScript and done some sort of Ajax call but…this was faster. Oh and of course instead of myAwesomeScore, I’m using the GUID of the product we’re adding.

getByGUID($_GET['addNewItem']);
	if(!empty($row)){
		//Add to Cart
		$cart_items->build();
		$update['cart']=$cartid;
		$update['item']=$_GET['addNewItem'];
		$update['qty']="1";
		$cart_items->update($update);
	}else{
		//Invalid Product ID
		echo "Invalid Product";
	}
}
?>

From a user interface level, this means that product links can just be to cart.php?addNewItem=guid. Some css styling to load that page in an iFrame in the drawer I mentioned, and it’s an easy implementation.

Retrieving cart contents is easy since we can just do a database query for all the rows in the Cart Items database with a certain cart I’d. That code goes on Cart.php last.

getByCart($cartid) as $cartitem){
	$item=$product->getByGUID($cartitem['item']);
	/* display cart items here, prettily */
}
?>

Deleting a cart item is easy to work out: a button which loads cart.php?removeItem=item reloads the cart and removes that row from the database.

getByGUID($_GET['removeItem']);
	$cart_items->delete();
}
?>

Adding an item to the cart that’s already in the cart proves a challenge. When an item is added to the cart, the cart contents needs to be loaded to see if that item is already on the cart, and if so, to increase the quantity by one. This requires some additions to the addNewItem method.

getCartContents($cartid);
if(in_array($_GET['addNewItem'],$contents)){
	//Item In Cart, Update Quantity
	$cart_items->getSingleItem($cartid,$_GET['addNewItem']);
	$update['qty']=$cart_items->row['qty']+1;
	$cart_items->update($update);
}
?>

Changing quantities poses the next problem. A simple way would be to include a text field for quantity, and then add a handler for when it changes, to make a database update. That was a little more complicated than I wanted it to be. I considered + and - buttons, but if the page reloaded every time, it would be obnoxious for large quantities. I considered +1, +10, and +100 buttons, but that seemed similarly awkward. I opted for + and - buttons that ask the user how much to add or remove from the cart.

The next challenge is the actual checkout process. We need an intermediate page between cart.php and Stripe to format the data—something like cart-process.php. This will package the cart in a format that Stripe understands and pass it off to Stripe. Since the cart ID is just in a cookie, we can use that. This takes a bit of time to figure out the nested arrays, but the Stripe documentation (and the Stripe errors in the Apache logs) are well-written.

Once you can get Stripe to catch the data, you’re home free.

There’s a lot of things I haven’t sorted out in this quick and dirty process: shipping prices, whether products need to be shipped, ~~taxes (though I think Stripe is doing that for me)~~(I figured that out since this write-up), digital assets, and so on. Snipcart used to automatically send out download links for digital goods, and I think I’ll just have to not have that for a bit.

I've been using a GitHub project to track everything, here's what that looks like:

There’s some room for improvement, but it’s not bad for several hours of worth over the weekend to save $120 a year by writing 200 lines of code.

This code will (hopefully) go live later this week.

New Login system for Liszt

Posted on 2021-01-26 16:38:00 by Kyle and tagged with Programming

When I first built Liszt, it required (logically) that users have a username and password. Well, more correctly, it required that I have a username and password, since I was the only one using it. As it grew and I got better at programming, I kept slightly upgrading the credential process and permissions system--the last iteration used a salted password based on a proprietary hashing algorithm.

When storing usernames and passwords, the worst way to go about it is to store passwords in plain text. If your password is "password", then you just put "password" in the database. Hashed passwords are a little better. It involves taking a one-way Hash, which is a way to take a variable length string like "password" or "pwd" and maps it to data of a fixed size, like 36 characters. The problem here is that if multiple people have the same password, if a bad actor were to steal the User database in Liszt, they could cross reference hashed passwords with password hints.

Liszt never had password hints though, so…but whatever.

Encrypted passwords are alright, but since encryption has a sibling named decryption, it's safe only as long as the encryption key is safe.

The safest way to store passwords oneself is to combine the user's password with what's called a salt. In the case of Liszt, I took a hashed combination of the users ID number, combined with a hashed combination of the user's password, hashed the result for good measure, and saved that in the database. That way, even if everyone had the same password, it wouldn't show up in the database as the same password. Everyone gets their own gibberish.

Much of my philosophy on how to properly authenticate users comes from YouTuber Tom Scott--most specifically, his "How Not to Store Passwords (https://www.youtube.com/watch?v=8ZtInClXe1Q, which I've just explained in brief) and "The Fictional Day Google Forgot to Check Passwords" (https://www.youtube.com/watch?v=y4GB_NDU43Q). And as I onboarded more and more students, it made more sense to start using alternative providers--Since most clients used Microsoft Office 365, having Microsoft handle the authentication made the most sense. That way, I never have to deal with the passwords.

I wrote much of the code to handle the OAUTH login myself, based on snippets I found online and bolting them to Liszt's Single Sign On system. Liszt doesn't use external frameworks for the simple reasons that 1) They either didn't exist and I wasn't smart enough to use them when I get started, and 2) It would be a massive undertaking now, and Liszt is kind of its own platform. Because of this, I didn't use a lot of third-party libraries.

In late 2020, I finally integrated the PHP package manager Composer because I needed it for some storage work I was doing (maybe a blog post in there too), which opened up the possibility of using some existing packages and modules I previously either didn't use, or integrated manually. One of these things is the League of Extraordinary Packages' OAUTH Client.

At the beginning of 2021, Liszt contained four login systems. The first was the original Liszt username/password combo, the second was Microsoft 365 for registered users, the third was Microsoft 365 for students and clients, and the last was

A hacked-together beta version of the Google OAUTH code, which worked fine on the Google end, but I never really properly wrote the Liszt end. My work in January has been to reduce Liszt to one login system that can handle multiple providers. Which is…really what I should have done at the beginning.

This sounds reasonably simple, but here are some of the challenges:

The Liszt Login code is one of those things that I play with every couple of years, and as long as it works, I don't question it. So the code--which does a lot of things including allowing for cross-domain logins (logging into Liszt automatically logs you into Liszt Swipe, Liszt Studio (defunct), Liszt Account Management, ScoreShare, AudioAtlas, and anything else that uses the Liszt login system). This is…pretty hairy.
Liszt accounts and Liszt student/client accounts are totally separate entities, and it's possible to have access to both. And what's more, it's possible to have access to multiple client accounts. Undoubtedly what would happen is that a student would try to log into Liszt instead of the Liszt Portal, and then that ends up being an email.
There's no onboarding process for Liszt, and only a rudimentary one for Liszt Portal.
There's also no account sign-up.

Also, I wanted the following:

I wanted the ability to add service providers (Google, Apple, etc.) later.
I wanted the ability to have a third class of authentication--There are registered users, there are registered clients, but I also wanted a basic identity provider--unregistered visitors. People who might need to be authenticated in some way, but I don't need a full account for. I call this "fingerprinting."

So there's just a ton of access control flow going on. Here's what that looks like now.

For starters, when I coded the original Liszt Office 365 login setup, I used my personal email account. Since we're upgrading, I moved it to NoteForge. Sound easy? Yes. Is easy? Microsoft now requires that commercial or business accounts be verified, which is a reasonably straightforward, though undocumented process. What that ultimately means is that instead of this:

Liszt now asks for permissions like this:

Even though it's the same app, in the view of Microsoft it's different, so all users will have to re-authorize Liszt. Only this time, it comes with a nice blue checkmark.

So Microsoft (or Google or whoever I end up using as providers) verifies the login--Liszt never sees the password or login information, it just gets Profile information, who you are, what your email address is, etc. Then Liszt has to figure out what to do with you. And that flow is, simply, complicated.

The most straightforward cases are either registered users or identified clients/students logging in--It just passes the info on to the appropriate part of Liszt. Next are users or clients who haven't logged in before and need their Microsoft username applied to their Liszt profile. Reasonably simple.

What about Clients who are trying to log in as Users? Or vice versa? Now Liszt tries to nudge you in the right direction:

ACCOUNT ERROR
Liszt couldn't find a client account with your credentials. but did find a Liszt user account with your information. Are you perhaps
looking to log into Liszt?
Liszt

Alright, we have those. Next up are people who are clients in multiple Liszt sites--which are arguably few, but better to take care of this now while I know what the code does. If you're logging in at the wrong place, Liszt will now try to nudge you towards the right client site.

At NDSU, we tried allowing students to self-enroll in 2019, and that worked really well. But, it was a completely manual solution. As in, I wrote some code that said "enroll at this website, get attached to the Marching Band." With this system, I've built in hooks to allow students to self-enroll by inputting a code, which will grant them the appropriate access and assign them to the appropriate groups. That's not hooked up yet, but it's ready to be written.

People who are logging in who just need to be fingerprinted? That's easy--since we're not checking them against a database, we just package their data and send it to the appropriate Liszt-built system.

Finally, there's the user onboarding flow, which is designed for creating new registered user accounts. That group will either want to get access to an existing site, or to create a whole new site. That last group, I'm not ready to automate yet, so Liszt collects their information and creates a workorder. Those new users to an existing site, however, now get the option to type in a secret code (provided by the administrator) which creates a permissionless user for that site. Admins can go in and assign the required permissions afterward.

Complicated? Sure. Better in the long term? Way better. In fact, I coded the Google login code for the system in about ten minutes this morning.

Progress? Very yes.

The new version of the Liszt login system goes live later this week.

Liszt upgrade to 2.4.7

Posted on 2014-10-26 22:10:00 by Kyle and tagged with Programming

FINALLY, after ten full months, Liszt has received a major upgrade which improves the interface and calendar components, re-introduces the Liszt Rehearsal Scheduler, and allows for better single-sign-on for Liszt/AudioAtlas/ScoreShare apps.

Single-Sign-On integration for Liszt

Posted on 2014-06-03 16:06:00 by Kyle and tagged with Programming

For those of you that are keeping track and using Liszt/ScoreShare/AudioAtlas, moving between applications is now easier. Once you're logged into one app, your login transfers to other Liszt apps automatically!

Map Integration and Updates

Posted on 2014-05-30 16:05:00 by Kyle and tagged with Programming

As part of the AudioAtlas Project, I've put together a map of my past performances over on the map page. I'll be updating this with future events as I get word of them. There's also some sample syllabi over on the teaching page, as part of a project to design an electroacoustic music curriculum.

Yet Another Website Redesign

Posted on 2014-05-04 18:05:00 by Kyle and tagged with Programming

As part of my "putting off studying for my dissertation defense" I decided to redesign KyleVanderburg.com again. I've been updating all of the NoteForge (and NoteForge Apps) sites to Twitter Bootstrap because it works well and is pretty easy to use, but I wasn't terribly happy when I moved the .com over to a new design. I ended up losing the old note cluster, and my name wasn't as large as I'd like it (something something composer ego), but all of that is updated with the recent redesign. There's a new footer to go along with the new header, some javascript controlling the menus over on the compositions page, a new footer, and the entire thing works pretty well on mobile too. It's also fun to compare this version to Version 14 and Version 13.

Introducing NoteForge Hammer 2.4

Posted on 2012-11-13 04:55:00 by Kyle and tagged with Programming

In 2007, I started putting my marginally-useful programming skills to use building a simple Content Management System (CMS) to manage my website, similar to Drury’s CMS (named, originally, DCMS). As my programming skills improved, the CMS grew, adding modules for composition management, event notification, invoicing, and something called “Opportunity Organization” (which is directly responsible for my increased performances in 2012) among others. The original simple CMS has evolved into Hammer, which is both a Composer Management System and a platform for building similar applications such as Keys (for private studio management), DAVID (Document Management), and a few more starting in 2013. Today, that system, just shy of its 5th birthday, was upgraded to version 2.4 in what is the largest cosmetic and backend upgrade of the system to date. Here’s a brief look at some of the new features.

Improved Navigation

One of the complaints about Hammer 2.3 was the static left-side navigation. On modules such as opportunity organization, the user was forced to scroll up to access the main menu. This has been completely rewritten in Hammer 2.4 with the change to the Twitter Bootstrap framework with CSS3 Microsoft Modern Buttons styling. The navigation stays at the top of the page, and divides modules by category.

Smart Buttons

Another complaint of 2.3 and previous versions was unstreamlined functions. After creating a new item, the user would have to scroll up to the menu, click the correct module, and then click a link to create a new item. While not annoying for a few items, this becomes tedious when inputting a month’s worth of invoices. This has been fixed with the addition of New and View All buttons when accessing a module that allows new items.

User Menu

Building different applications on the same platform (or at least, this same platform) means that all applications share a codebase and a user database, allowing one Hammer account to access all of the applications. The new user menu allows for fast switching between Hammer applications.

Account Management

The addition of multiple applications to Hammer necessitated a rewrite of the user and configuration modules, and the creation of a permissions database. These are all controlled via the new Account application in Hammer.

Improved Search

I finally rewrote all the code to make searching possible (and useful)

UI Improvements

Like these buttons:

And these forms:

Hammer now has a consistent look across all applications, and the CSS probably sucks less.

Completely Rewritten Code

Every module in Hammer has been recoded. All of them. Every single line. While developing DAVID (Digital Access to Vanderburg Internal Documents. Yeah, I know. It’s a fantastic name) I discovered a better way of coding the modules that use 25% of the code I was using before. As in, a module with 800 lines of code can be done in just 200 lines. It’s far more streamlined, and easier to add fields.

Proprietary Hashing Algorithms

Passwords and other hashed information (such as invoice access numbers) are calculated via a new proprietary hashing algorithm.

New Lemon Scent

This might be false. So yeah. That’s what I’ve been doing with all my time when I’m not writing music. And now, off to build more things.

Website Redesign

Posted on 2012-06-23 13:04:00 by Kyle and tagged with Programming

Soooooo I'm still in Memphis for the Belvedere Chamber Music Festival (which will get a full write-up once I'm back in Norman, needless to say IT'S AWESOME), but I've had the spare time to put the finishing touches on a new design for my website at http://kylevanderburg.com. The old version of the site (Version 12) was put together sometime shortly after I moved to Norman and I've been slowly upgrading it, but I think this new version is pretty cool, and it works similarly to http://kylevanderburg.net (my newly-revised personal site).

Composition Submission Tracking

Posted on 2012-05-30 10:49:00 by Kyle and tagged with Programming

As it turns out, there are quite a few composition contests, calls for scores, conferences, and other opportunities available to composers each year. And, following the advice of Eric Whitacre, I try to submit things as often as I can. But I kept falling into this pattern:

Find a contest.
Prepare everything for the contest.
Let the paperwork sit on my desk until deadline has passed.
Repeat.

This obviously didn’t work that well for me. So at the beginning of this academic year, I decided to try something new. I thought, if I can somehow automate (or nearly automate) the contest submission process, then I’ll be spending less time sending off compositions (or rather, less time not sending off compositions) and more time composing. Or watching police procedurals. Or both! So I built HOE.

The Hammer Opportunity Engine or HOE is essentially a composer’s to-do list on steroids. It tracks composition opportunities, which pieces to send, whether the deadlines are postmark or receipt, and it prints the mailing and return address labels, all coded with QR codes. It plugs into Hammer’s composition database (which drives kylevanderburg.com) so new compositions are added automatically. And it basically reminds me incessantly when submissions are coming due.

While I’m sure I could go on about how HOE works, it’s probably more interesting to talk about what I’ve learned by using it. Here’s a list of things in no particular order:

The system actually works. Since using HOE, I’ve won the Belvedere Chamber Music Festival Student Composers Competition (BCMFSCC?) and was selected to have Creatures from the Black Bassoon performed at ICMC2012. (which means travel to Memphis and Slovenia, both of which are famous for their barbecue. Or I might have made part of that up.)
Anything looks more legit with a well-placed QR code and a logo.
I’ve sent out ~40 submissions this year, rather than the 2-4 I usually send.
A good chunk of contests don’t respond when they’ve picked winners, which means they sit in a “submitted” state until you search for them.
Contest fees range from the free to the astronomical. The reasonable amount I pay is somewhere between the two.
EasyChair is kind of a ridiculous service, though it seems to work.

So there it is. My programming geekiness meets my composerlyness.

Screwing up is the real mother of invention...

Posted on 2010-06-06 10:26:00 by Kyle and tagged with Programming

As chronicled in my last post, in the massive amount of shuffling around that has happened to KyleVanderburg.Net lately (and vandermusik.com and kyledavey.com and and and...) I managed to drop the wrong database, losing all of the posts from Poor Little Misled Notes. Did I have a backup? Of course not. The entire KyleVanderburg.Net server structure is a wonderfully convoluted system that I sometimes copy to an external drive if it's a full moon and the planets are aligned. And backing up things like databases (which run the ~8 WordPress blogs, applications like my library and photo album, the Vandermusik billing system, and of course the KV.Net core)...who backs up databases? I mean, who would be so stupid as to accidentally delete... Oh. So in a blazing moment of clarity (and by that, I mean an evening of binge-coding) I wrote a file-copying-database-exporting backup script. It's ugly. It's weird. Any real programmer would have an aneurysm. But it works. I was going to post news about the ongoing piano sonata, but SVU is on and I want a sandwich.