It started with a mundane problem: I wanted a better way to filter stocks. Not trading signals, not price alerts — just a structured monthly process to surface equities worth looking at based on fundamentals and momentum.

So I built a screener. It pulled financial data, normalized it across four signals, and ranked tickers by a composite score. Simple enough.

But then I got curious about something more interesting than the stocks themselves.

What if instead of me reasoning through the output, I let a model do it — and watched how it thinks? What would happen if I gave an LLM full context about a company and asked it to make a decision? More importantly: what would happen if I tracked whether those decisions were right, and fed that information back?

That question turned a weekend project into a multi-month rabbit hole about LLM reasoning, decision-making under uncertainty, and what it actually takes to build a system that gets better over time — not by swapping in a more powerful model, but by designing better scaffolding around it.

The result is an open-source multi-agent stock screener. But the stocks are almost beside the point. This is really a project about how to make AI systems learn from experience.

Technical Decisions and Tradeoffs

The Debate Architecture

The core insight early on was that asking a single model to reason through a hard, ambiguous problem is a bad idea — not because the model is incapable, but because it has no structural reason to surface its own uncertainty.

A single LLM asked "should I buy this stock?" will give you an answer. It will sound confident. It will cite relevant data. And you will have no idea how strong the opposing case is.

So I split the reasoning across three agents: Bull, Bear, and Judge.

Bull and Bear run in parallel. They have the same data but opposing mandates — Bull must make the strongest possible case for the stock; Bear must find every reason it's a bad idea. Judge reads both arguments and adjudicates: BUY, SELL, or HOLD, with a stated margin of conviction and a decisive factor.

The tradeoff here is cost and latency. Three LLM calls per ticker instead of one. At S&P 500 scale with 10 top picks, that's 30 calls per run — manageable, but it means you need to be thoughtful about which model you use for each role. I ended up running Bull and Bear on a cheaper, faster model (Claude Haiku) and reserving a stronger model for Judge when needed.

The more important tradeoff: this architecture makes disagreement visible. When Bull scores 72 and Bear scores 68, you know the pick is contested. That's information a single model call buries.

Grounding with SEC Filings

One of my early frustrations was that LLM debate arguments were fluent but unmoored. Bull would make a compelling case about a company's competitive moat. Bear would counter with macro risk. Neither would cite anything specific.

I added a RAG layer that indexes 10-K and 10-Q filings from EDGAR monthly. Each filing gets chunked (512 tokens, 10% overlap), embedded, and stored in a vector database. Before the debate starts, relevant chunks are retrieved and injected into Bull and Bear's context. Both agents are required to cite which sources they used.

The tradeoff here was complexity vs. grounded-ness. The EDGAR pipeline is the most operationally fragile part of the system — SEC filing formats are inconsistent, embedding costs add up, and retrieval quality varies. But it was worth it. Arguments grounded in actual filings are categorically more useful than arguments generated from parametric knowledge alone.

One decision I'm still not sure about: I used a single embedder (OpenAI's text-embedding-3-large, 3072 dimensions) rather than making it swappable like the LLM. Embeddings need to be consistent across index and query time, which makes the backend stickier than I'd like.

Episodic Memory

This is the part I find most interesting to think about.

Every verdict the system produces — BUY, SELL, or HOLD — gets stored against the ticker and the month. A separate eval job runs monthly, scores prior picks on a 0–100 rubric (accuracy, confidence calibration, directional bias), and writes those scores back to the same ticker's memory.

After four scored months, the debate adapts. If Bull has been more accurate for a given ticker, Bull's arguments get weighted more heavily with the Judge. The weights shift based on evidence, not intuition.

This is episodic memory as a behavioral input, not just a log. The system doesn't just remember what it said — it remembers how well that turned out, and uses that to change what it does next.

The tradeoff: this mechanism only activates after four months of history. For the first few runs, the system has no prior track record and behaves like a stateless agent. You're investing in future improvements that aren't visible immediately.

The Eval Feedback Loop

The eval job is what turns this from a one-shot system into something that can improve over time.

Each month, it scores the prior month's closed picks. Not just accuracy — it tracks directional bias (is the Judge systematically bullish?), confidence calibration (do high-confidence picks actually outperform low-confidence ones?), and citation quality.

All of that gets injected back into the Judge prompt the following month as eval_context. The Judge reads its own historical track record before making the next decision.

The analogy I keep coming back to: this is closer to how a good analyst actually thinks than any one-shot prompting approach. A good analyst doesn't just know the current data — they know their own blind spots.

Infrastructure

The pipeline runs monthly on Cloud Workflows (GCP), with three Cloud Run Jobs (scoring, EDGAR indexing, debate) and a Cloud Function for eval. Cloud Scheduler triggers the whole thing on the first Friday of each month.

The decision to use Cloud Workflows over something like Airflow or Prefect was pragmatic: the dependency graph is simple (four sequential steps), and Workflows gives you retry logic and state management for free without running a persistent orchestration server.

Storage is abstracted behind a DAO interface — Firestore by default, but swappable to S3 or OpenSearch. This was more work upfront but the right call. I've already had to migrate storage once during development.

What I Learned

Open-loop AI systems are the norm, and that's a problem. Most production AI applications I've encountered generate outputs but have no mechanism to evaluate whether those outputs were good. The eval loop is the thing I'd advocate for most strongly if I could give one piece of advice about building AI systems: close the loop. Track outcomes. Feed them back.

Adversarial structure is underused. The Bull/Bear architecture isn't novel — red-teaming and debate patterns appear in the research literature. But I rarely see it implemented in production applications. It's more expensive. It adds latency. But it makes the model's reasoning legible in a way that single-agent systems simply can't.

Structured outputs change everything. Using Pydantic models with .with_structured_output() — so every agent returns a typed, validated object — eliminated an entire class of parsing bugs and made the data model explicit. I should have done this from day one instead of parsing freeform text.

Prompt engineering hits a ceiling. Early on I kept tweaking Bull and Bear's prompts trying to get better arguments. Eventually I accepted that prompt quality matters less than the feedback mechanisms around the prompt. The system got meaningfully better when I added eval feedback than it did from any prompt change.

What I Would Do Differently

Start with the eval pipeline. I built it last. It should have been first — or at least designed upfront. The data schema I chose for picks and performance had to be partially refactored once I understood what the eval job needed to measure.

Make the embedder swappable from day one. I hardcoded OpenAI's embedding model early because it was easiest. Swapping it later would require re-indexing all EDGAR chunks, which is expensive. The LLM is swappable via config; the embedder should be too.

Run the debate on fewer tickers, better. I default to top 10 picks. In practice, the debate quality on picks 7–10 is noticeably lower than on picks 1–3. I'd rather run a high-quality debate on 5 tickers than a lower-quality one on 10. Quality over coverage.

Add a human review step earlier in development. I spent months evaluating the system programmatically before I sat down and actually read the Bull and Bear arguments for a few tickers. When I did, I caught systematic issues immediately — Bear was systematically hedging by citing macro risk regardless of the ticker. That took one read to spot and five minutes to fix.

The project is open source. If you're building in the agent memory or eval loop space, I'd love to compare notes.

https://github.com/thisisnish/multi_agent_stock_screener_opensource