Tommy McMichen

Automatic Data Enumeration for Fast Collections

Thu, 29 Jan 2026 00:00:00 GMT

This post is based on a paper of the same name, published at CGO 2026.

Data collections are everywhere! They provide a powerful abstraction to organize data, simplifying software development and maintenance. However, choosing how we implement each collection in our program is critical for both performance and memory usage.

While general-purpose data collections (the ones in the C++ STL, Abseil, etc.) provide drop-in replacements for quick performance tuning, the real value comes from specialized implementations. Specialized implementations (bitsets, prefix trees, bloom filters) provide superior performance and memory usage compared to their general-purpose counterparts. However, they require additional constraints on the kind of data stored, and the methods used to access them.

For example, let's look at bitsets. In this domain, other implementations pale in comparison, as shown in the speedup table below. However, a bitset requires dense (or better yet, contiguous), integer keys.

Implementation	Insert	Remove	Union
`std::unordered_set`	1.00	1.00	1.00
`absl::flat_hash_set`	1.61	0.40	1.71
`boost::dynamic_bitset`	9.08	1.24	5817.38

Those are some pretty nice performance boosts, and if my set is dense enough we'll get good memory savings too. But, what if I'm in a domain that doesn't have the necessary guarantees to use a bitset? Is there any hope for the folks processing non-integer data types, or sparse domains?

The good news: There is.

The bad news: It's not free.

Data Enumeration

If we don't have inherent data properties guaranteed by our domain to rely on, we can instead engineer the desired property.

Take the following code, which iterates over a list of strings (items) and prints unique items by tracking duplicates in a set (seen).

items = ["foo", "bar", "foo"]
seen = set()
for item in items:
    if not seen.has(item):
        seen.insert(item)
        print(item)

If we want to use a bitset instead of a set, we can do a bit of transformation (shown below). First, we'll assign unique identifiers to each item and maintain a bidirectional mapping (enc and dec) to convert between the ID and the actual data. Then, we convert the data stored in our input array to use IDs (items). Now that we are iterating over IDs (which we've assigned to be contiguous), we can convert seen into a bitset. Finally, we need to patch up any use that requires the original (the print statement) by decoding the identifier. Et voilà!

enc = {"foo": 0, "bar": 1}
dec = ["foo",    "bar"]
items = [0, 1, 0]
seen = bitset()
for i in items:
    if not seen.has(i):
        seen.insert(i)
        item = dec[i]
        print(item)

We've successfully converted our program to use a bitset, instead of a set. Now, we can reap the performance and memory benefits! Additionally, we can perform comparisons between our items much faster by using their ID instead of the string.

However, we've gotten ourselves into a bit of a pickle now by applying this manually. I have to juggle my IDs and values everywhere in the program. Additionally, I need to make enc and dec available anywhere that I need to translate between IDs and values. Furthermore, I'm left with a bit of a performance headache figuring out if I should use IDs or values in different parts of my program. Yeesh, this sounds like the job for a compiler!

Automatic Data Enumeration

Instead of performing data enumeration manually, I propose that we do it automatically. Looking at the example above, the transformation itself is rather mechanical. We just need the proper substrate to implement it. LLVM is a bit too low-level, and there's no existing MLIR dialect for this stuff, but I've heard of a cool new compiler IR on the block: MEMOIR!

MEMOIR is nice here because it preserves high-level semantics about data collections, which we'll need to transform how we store our data. Additionally, its SSA form makes the analysis we need very easy. Sounds like the perfect substrate to implement Automatic Data Enumeration (ADE). I won't get into all the nitty gritty about how we mechanize the data enumeration transformation here, but I will give the high-level overview.

Fundamentally, ADE transforms the program to build an on-the-fly enumeration (the bidirectional mapping from IDs to values). This means that whenever you encode a value for the first time, it will be assigned a unique identifier. ADE is responsible for determining where we should use IDs instead of values, and then inserting translations using the enumeration. As you would expect, this introduces a lot of redundant translations, which can be eliminated:

enum_A = ...
seen = bitset()
for item in items:
    id_1 = enc(enum_A, item)
    if not seen.has(id_1):
        id_2 = enc(enum_A, item) # redundant, replace id_2 with id_1
        seen.insert(id_2)

Eliminating redundant translations is the main performance driver of ADE, so we introduce a few transformations that introduces more redundancy. One of these is sharing (shown below), where we have two sets share the same enumeration. Sharing has the added benefit of amortizing the cost of constructing the enumeration.

enum_A = ...
seen = bitset()
seen_twice = bitset()
for item in items:
    id_1 = enc(enum_A, item)
    id_2 = enc(enum_A, item) # redundant, replace id_2 with id_1
    if not seen.has(id_1):
        seen.insert(id_1)
    elif not seen_twice.has(id_2):
        id_3 = enc(enum_A, item)
        seen_twice.insert(id_3) # redundant, replace id_3 with id_1

There are more optimization, with full details available in Section 3 of the paper.

Evaluation

So that's all great, but how about the performance? In short, ADE provides significant performance improvements in the majority of our benchmarks. While ADE incurs an increase in memory usage for a few cases, we find that the majority of benchmarks experience no memory explosion, with some seeing significant memory savings! For a more detailed evaluation on both Intel and ARM machines, along with comparisons to Abseil swiss tables, see Section 5 of the paper.

Whole-program speedup of ADE over std::unordered_set.

Maximum resident set size of ADE over std::unordered_set.

Tuning

In addition to fully automatic ADE, we provide ways for users to guide the optimization. This lets you enable/disable enumeration for a collection, and prevent/force sharing between collections. By overriding the static cost model, users can fine tune data enumeration for their specific application.

We did a brief case study for LLVM Andersen-style points-to analysis (PTA). In short, we identified a case where sharing with a specific collection was detrimental to performance. The full details and exploration are in Section 5, RQ4 of the paper.

By injecting a single noshare directive, ADE achieved a speedup of 78x (13.7x over fully automatic ADE). In addition to the performance gains, memory usage (max resident set size) was reduced by 71% (compared to 49% reduction with fully automatic ADE).

Conclusion

Automatic Data Enumeration provides a fully automatic, transparent solution to one class of memory optimization in the compiler. With ADE we are able to achieve significant performance improvements without the need for developer input, though tuning can unlock further savings. Thanks to MEMOIR, the implementation is straightforward (and fun!). I believe that data enumeration is just one of many memory optimizations that can be automated by the compiler, and I hope for a future where developers can trust the compiler to handle most of them.

Running Out-of-Tree LLVM Passes at LTO

Wed, 14 Jan 2026 00:00:00 GMT

I had a long time trying to find the proper way to run LLVM pass plugins at LTO. There were a lot of times where I found a solution and it turned out to be flimsy and break under some circumstances, or not fully support all the features I needed. This post gives your my solution and concludes with some thoughts about why this solution works. If anyone knows that I am correct/wrong please confirm/correct me. Otherwise, take the explanation with a grain of salt, I have not extensively tested it.

My Solution

In the examples, I will use a hypothetical LLVM pass plugin where:

MyPlugin.so is the shared object containing the LLVM pass plugin.
MyPass is the name of a pass in the plugin (registered with the new pass manager).
--my-flag is a compiler flag that is registered in the plugin.

A typical invocation of this pass using opt would look like:

opt -load-pass-plugin=MyPlugin.so \
    -passes='MyPass' \
    --my-flag \
    ...

So, if you want to run this pass under the linker, it would look a little different:

ld.lld --load-pass-plugin=MyPlugin.so \
       --lto-newpm-passes='MyPass' \
       -mllvm=-load=MyPlugin.so \
       -mllvm=--my-flag \
       ...

The above solution comes from an lld test: https://github.com/llvm/llvm-project/blob/main/lld/test/ELF/lto/ltopasses-extension.ll

So now, let's do the same thing for clang with LTO enabled:

clang -flto=full -fuse-ld=lld \
      -Wl,--load-pass-plugin=MyPlugin.so \
      -Wl,--lto-newpm-passes='MyPass' \
      -Wl,-mllvm=-load=MyPlugin.so \
      -Wl,-mllvm=--my-flag \
      ...

My Understanding

The first -load-pass-plugin=MyPlugin.so loads your pass plugin with lld, which runs pipeline parsing on --lto-newpm-passes='MyPass' to construct the llvm::PassBuilder. So, if you did not load your plugin, this would result in an error during parsing.

The second -mllvm=-load=MyPlugin.so is passed to the actual invocation of the LLVM optimizer, making your pass plugin available internally. From my debugging, you only need this -load to populate your flags (e.g., --my-flag), the first -load-pass-plugin is sufficient to run your pass without flags.

I ran a couple of tests to back up these claims:

Omitting the first -load-pass-plugin=MyPlugin.so:

LLVM ERROR: unable to parse pass pipeline description 'MyPass': unknown pass name 'MyPass'

Omitting the second -mllvm=-load=MyPlugin.so:

ld.lld: error: --mllvm: ld.lld: Unknown command line argument '--my-flag'.

Concluding Remarks

LLVM's new pass manager is very {un,under}documented, I will try to share my own experiences with it here when I learn something new. If you similarly get something to work, it would be great if you could share it with the community! I have found a surprising lack of community notes (blog posts, mailing lists, etc.) about the new pass manager, which is a little concerning. There are a lot of "weird" LLVM use cases and a critical mass of documentation is one way we can get around individually debugging the same mechanism ad infinitum.

Private Fork of Git Repo

Wed, 14 Jan 2026 00:00:00 GMT

I find myself creating a private fork of public repositories rather frequently. There are a variety of benefits to this over just create a public fork and working there, but that's not what this post is about. I have found two solutions to this problem, which I will go over in this post.

`git-private-fork`

The first solution works when making a private fork of any git repository:

# Clone the version control data of the public repository.
git clone --bare ${PUBLIC_REPO} ${PUBLIC_REPO_DIR}

# Enter the cloned directory
cd ${PUBLIC_REPO_DIR}

# Push an exact copy of the git repository.
git push --mirror ${PRIVATE_REPO}

# Leave the cloned directory
popd

# Clean up.
rm -rf ${PUBLIC_REPO_DIR}

git clone ${PRIVATE_REPO} ${PRIVATE_REPO_DIR}

if [ ! -z ${SET_UPSTREAM+x} ] ; then
    pushd ${PRIVATE_REPO_DIR}
    git remote add upstream ${PUBLIC_REPO}
    git remote set-url --push upstream DISABLE
    popd
fi

I will usually go ahead and set the public repo as my upstream too:

git remote add upstream ${PUBLIC_REPO}
git remote set-url --push upstream DISABLE

You can find this solution with some other little improvements at github.com/tommymcm/git-tools/. Feel free to open any issues/PRs there.

GitHub Import

There is one known issue though, which comes up when forking rather large repositories (like llvm/llvm-project). For large repositories, you may hit single-push limits (2GiB on GitHub). I have toyed around with making a general-purpose solution in git-private-fork, but I couldn't ever get one that handled all the edge cases (rate-limits, size-limits, granularity).

Instead of fixing that, I typically just fallback to importing the repository on GitHub. Disclaimer, that I don't know of a solution for other hosts like gitlab or codeberg because I don't use them. I'd assume they have a similar feature though.

To create your private fork by importing, go to github.com/new/import. It's self explanatory.

Disclaimer, this is a pretty slow process for large repositories. However, it's the perfect thing to run while you go grab lunch, or write a blog post.

Acknowledgments

I'd like to thank 0xjac and ConfoundingVariables on GitHub for providing the original solutions in a Gist and the comments.

Deploying Eleventy site with GitHub Actions

Wed, 17 Dec 2025 00:00:00 GMT

I recently moved my website from hand-written HTML over to Eleventy. It's been really nice to work with, the transition process is smooth, and their template system is intuitive. I'm now able to generate all of my web content from a single source of JSON files! ¹¹ You can find the source code here

After finishing up my port, I was interested in automate the process of publishing changes, which is what this post is about. I originally did this work with two repos because I wanted to have the materials (unpublished papers, course materials, etc.) not be public. Since then, I moved over to a public repo to avoid hitting any GitHub Action limits²² I am a penny pincher. and use a private fork to hold uinpublished items.

Here's the step-by-step process that I took:

1. Create a Personal Access Token (PAT)

First, you need to create a token that allows the private repo to push to your GitHub Pages repo:

Go to GitHub Settings → Developer settings → Personal access tokens → Tokens (classic)
Click "Generate new token (classic)"
Give it a name like "Eleventy Deploy"
Select the repo scope (this gives full control of private repositories)
Generate and copy the token (you won't see it again!)

2. Add the Token to Your Private Repo

Go to your private repo's Settings → Secrets and variables → Actions
Click "New repository secret"
Name it PAGES_DEPLOY_TOKEN
Paste your personal access token
Save it

3. Create the GitHub Action Workflow

In your private repo, create .github/workflows/deploy.yml:

name: Build and Deploy to GitHub Pages

on:
  push:
    branches:
      - main  # or whatever your default branch is

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout private repo
        uses: actions/checkout@v4
      
      - name: Setup bun
        uses: oven-sh/setup-bun@v2

      # bun is fun! if you disagree, you can use node:
      # - name: Setup Node.js  
      #   uses: actions/setup-node@v4
      #   with:
      #     node-version: '20'  # or your preferred version
      #     cache: 'npm'
      
      - name: Install dependencies
        run: bun install
      
      - name: Build Eleventy site
        run: bun run eleventy
      
      - name: Deploy to GitHub Pages repo
        uses: peaceiris/actions-gh-pages@v3
        with:
          personal_token: $
          external_repository: you/you.github.io  # TODO: your pages repo
          publish_branch: main  # or gh-pages, depending on your setup
          publish_dir: ./_site  # Eleventy's default output directory
          cname: yourdomain.com  # TODO: custom domain (optional)

4. Configure Your GitHub Pages Repo

Make sure your GitHub Pages repository is set to publish from the branch you specified (usually main or gh-pages) in the repository settings.

That's it! Now when you push to your repo, it will automatically build your Eleventy site and deploy it to your GitHub Pages repo.