Running a Lightweight Research Paper Club

We’ve been running a weekly paper club at Promptless for the last 4 months. Every Friday, we pick a research paper and read it together in-person. It’s been surprisingly useful, so we wanted to share how we set it up in case it’s helpful to anyone else.

We’re a small team (two technical founders and two engineers), and even though we’re not doing research ourselves, we’re trying to extract as much intelligence and capabilities from LLMs as we build AI agents. While reading research papers isn’t in the critical path for our product development, following along with research is important to us.

How We Run It

Operationally, this kind of paper club is super easy to run, but a few details made a ton of difference:

• Print the papers ahead of time. Get a $100 laser printer if you don’t have one in the office. Print one-sided, so that people can always read adjacent pages side-by-side.
• Bring pens for people to mark up and take notes.
• Leave the office, or at least leave the desks in the office. Two reasons: (1) makes it harder for people to get distracted with computers, and (2) changing locations can help facilitate better brainstorming. Try to pick somewhere indoors (wind + paper = bad), not too loud, and well lit.
• Just have one person responsible for picking a paper. At first we did round robin paper selection, but not everyone is usually equally plugged into research. Of course, anyone can suggest or nominate a paper if they want.
• Spend the first ~45 mins in silent reading, before diving into open-discussion. There was no expectation for anyone to have read the paper ahead of time.

Why This Has Been Helpful

Here are a few tangible improvements to our agent that came directly or indirectly from paper club discussions. Interestingly, many good ideas came up during discussion even when they weren’t directly related to the paper we were reading:

• Fine-tuned visual grounding models can be great at guessing pixels and bounding boxes for elements in product screenshots (very relevant for how Promptless automatically updates screenshots in product documentation)
• Reflecting on past experiences and extracting learnings from the user’s feedback helps with verbal reinforcement learning. We let Promptless maintain its own learnings as it gathers feedback from users and evolves its own knowledge about the customer’s workflows and preferences.
• The original LoRA paper inspired us to fine tune customer-specific models to rewrite Promptless documentation suggestions in each customer’s unique voice and style. This ended up removing AI-slop completely, making Promptless’s writing much more concise and human-sounding.

Papers We’ve Read

We’ve covered everything from foundational ML papers to very specific papers related to our domain:

• LoRA
• Reflexion
• Visual grounding
• Agent early experience
• WritingBench
• On-policy distillation

It might seem daunting to pick out a new paper each week, but we wouldn’t over-index too heavily on the specific paper you choose. One of the biggest upsides of our weekly readings has simply been having a dedicated forum to talk about and debate approaches to agent design.