Organizing Many Projects

I do independent development for both work and pleasure across several languages and frameworks.

I usually break my work into several broad categories of project type: product, library, learning, experimental, and templates. In addition I sometimes fork other OSS projects for examination or minor bug fixes.

I also work across a few machines to suit different work conditions (local dev laptop, local dev on desktop and remote workstation).

This results in a fairly large number of distinct code projects to manage in at several locations.

I get easily distracted and lose focus, so I am particular in defining and using patterns to help organize and reduce the cognitive load and diffusion navigating across these dimensions can cause. Consistent hierarchical categorization and placement, naming conventions, etc.

This results in a sort of ‘meta-project’ organization that I prefer, which also tends to evolve over time as I decide on re-categorization or further refinements in support tooling.

Options

There are of course an infinite different number of ways to organize, and I’ve tried many.

Manual Management

This was the very thing that I found burdensome and distracting. I tend to slowly evolve my organization over time. That with the fact that I bounce between machines so much, that things would desynchronize enough that it would disrupt my flow.

Management Scripts

Nothing about my structures are complicated beyond some fairly simple scripts. But I do tend to evolve things at a steady pace so it did mean some constant updates and tweaks. Working across machines, you then have to keep the management scripts in sync as well. Like many things, LLM assisted coding does make this much easier so it’s not a huge burden but I wanted to do something with less overhead.

Monorepo

I went to a monorepo for a brief period but this ultimately proved too heavy weight. To support the various types of projects I ended up wanting or having to build more complicated tooling then I wanted to. While this has gotten vastly easier with LLM assisted coding, I found the tooling overhead when combined with the ‘smearing’ of commit history across all projects less than idea.

I DID very much enjoy the moon tool and will continue to use it for appropriate use cases. I think I may apply it across my submodule solution as well for certain cross-cutting ‘meta’ tasks.

Singular Development Server

I would consider moving but I am working on some desktop applications that target multiple platforms which makes this a no-go as a total singular solution.

‘Meta-repo’ and Submodules

Blending the approach of entirely isolated git repositories and a monolithic ‘monorepo’, I have settled on a single ‘dev’ repository structure for meta-organization and embedding my other projects with git submodules.

It looks something like this:

dev (its own repository)
  companyA
    customer1project
      docs
        plans
        todo
      repo (submodule)
    customer2project
      docs
      repo (submodule)
  companyB
    productA
      planningDocs
      repo (submodule)
  oss
    projectA
      repo (submodule)
    projectB
      repo-public (submodule)
      repo-private (submodule with in progress work)
  templates
    template-tauri-project
      repo (submodule)
    template-astro-project
      repo (submodule)

It’s not perfect. There is still the likelihood of drift between workspaces, and git submodules can be a bit unwieldy if you make a mistake.

However the single canonical organizational structure with isolated git work history is proving the best approach i have found so far.

Advantages

Canonical organizational structure

Existing Tooling

No scripts or additional configuration files to write or manage. It is simply just folder structure and git.

Conceptual Isolation

Projects retain their own independent work history. It doesn’t get smeared or interleaved across a single monorepo. This wasn’t hard to deal with using various git log filter tools, but it is one less thing to have to deal with.

Cross Cutting Tooling

I like to try to maintain project similarity with regards to tasks, build actions, ci, etc.

Once I download the parent repository, and then properly initialize it I can perform cross cutting updates easily enough.

NOTE: You will have to perform multiple commits and pushes however. See [[#Unweildy|below$a.])

• Still allows cross cutting tooling

Cons

Error Prone (at first)

It required learning and understanding how git implements submodules as I made some mistakes and ended up in some inconsistent error states in my repository early on. (More coming in another post) It is well explained in the herehowever.

Unwieldy

You need to ‘double-commit’. If you make changes in a submodule you have to commit and push those changes, and ALSO be sure to commit and push changes in the parent module (as it retains a pointer to the particular submodule hash)