Gen AI Intensive Course Capstone 2025Q1 – Kaggle x Google
This project describes the concept of Dictionary Serialization and introduces the PyPi package jsoncons to accelerate interoperable design when handling JSON-related data structures such as COBOL. The project was extended with an advanced implementation that is now the Eventflow Forecasting Package.
The Kaggle Project Notebook related to this post is here:
https://www.kaggle.com/code/dandersonmsf/dictionary-serialization-in-python-and-jsonco
I created a PyPi Project named jsoncons and extended the data structure concept to a MVP for a serverless trading application.
The following post was generated with the help of Google AI Studio as part of the Google x Kaggle Generative AI Capstone Project 2025Q1. This is a concept demonstration for potential used by licensed operators or DCMs to make markets on a wide variety of event-based contracts.
See a video of the extended Eventflow API in use with the Streamlit app available at:
https://eventflow.streamlit.app
From Theory to Distributed Trading Floor: Python Dictionary Serialization Powers the Eventflow API
We often learn fundamental programming concepts in isolation – mastering loops, understanding classes, or, as we’ve explored recently, grasping the essentials of dictionary serialization in Python. Tools like our own jsoncons utility help solidify how to convert Python dictionaries to formats like JSON, Pickle, or YAML, handle data types, and manage persistence. But the real magic happens when these fundamental skills become the building blocks for complex, real-world applications.
Today we’ll look at how the core principles of dictionary serialization aren’t just academic exercises – they are the critical engine driving Eventflow API (eventflowapi.com), a serverless template designed for building high-performance, peer-to-peer event markets.
The Challenge: Orchestrating Real-Time Event Markets
Imagine building a platform where users can trade contracts based on real-world events – elections, sports outcomes, financial milestones. The heart of such a system is often an order book: a constantly updating list of buy and sell orders. This data needs to be:
- Structured: Orders have specific attributes (price, quantity, timestamp, user ID, direction). Dictionaries are a natural fit in Python.
- Persisted: The state of the order book must survive beyond a single function execution or server instance.
- Exchanged: Data needs to flow between users, the API, and backend processing logic.
- Fast: In trading, microseconds matter. Data handling must be incredibly efficient.
This is where effective serialization becomes non-negotiable. How do we represent that order book dictionary? How do we save its state between serverless function invocations? How do we send updates to clients?
Eventflow API: Serverless Trading Orchestration
Eventflow tackles this challenge head-on, providing a lean, serverless backend (< 1000 lines of Python!) designed for orchestrating these event markets. It leverages a modern stack to achieve high performance and scalability:
- Python: The core logic language.
- Pydantic: For defining clear data structures (our “dictionaries” with rules) and handling validation and serialization/deserialization automatically.
- Upstash (Serverless Redis): For ultra-fast, persistent storage of the order book state and other critical data.
- Vercel: For deploying the serverless API functions globally.
- GitHub: For source control and CI/CD integration.
- AWS Route 53: For DNS management.
Dictionary Serialization in Action within Eventflow
The concepts we practiced with jsoncons are amplified in Eventflow:
- Defining the Structure (Pydantic): Instead of raw dictionaries, Eventflow likely uses Pydantic models. Think of these as supercharged dictionaries. They define the expected fields (like price: float, quantity: int, order_id: str), their data types, and validation rules. This directly addresses the “input validation” best practice we discussed. Pydantic models have built-in .dict() and .json() methods, providing reliable serialization out-of-the-box.
- API Data Exchange (JSON): When users submit orders or query market data via the API endpoints deployed on Vercel, the data is almost certainly exchanged as JSON. Pydantic excels at converting incoming JSON into validated Python objects (deserialization) and converting outgoing Python objects back into JSON strings (serialization) for the API response. This aligns with JSON being the web standard.
- State Persistence (Redis & Serialization): How does a serverless function “remember” the order book? It needs to persist it externally. Upstash provides serverless Redis. Eventflow likely serializes the current state of the order book (perhaps represented as a complex dictionary or a Pydantic model instance) into a format suitable for storing in Redis – often this is still JSON (as a string) due to its flexibility, though for extreme performance, formats like MessagePack could be considered, or even custom binary formats managed directly by Redis. The choice depends on the trade-off between human readability, interoperability, and raw speed/storage size – exactly the factors we considered when comparing formats. When a new request comes in, the state is deserialized from Redis, updated, and serialized back.
- Performance Considerations: Eventflow’s choice of Redis highlights the importance of serialization speed. While JSON is readable, serializing/deserializing large, complex order books frequently requires optimization. This might involve careful data structure design, potentially using more performant JSON libraries (like orjson), or choosing different formats for internal state vs. external APIs. Eventflow’s focus on performance, despite a known issue mentioned in its documentation, underscores that real-world systems constantly balance theoretical best practices with practical implementation constraints.
From Learning Tool to Production Principles
Mastering dictionary serialization, understanding format trade-offs (JSON vs. Pickle vs. MessagePack), knowing how to handle custom types, and prioritizing security through validation aren’t just steps in a tutorial. They are the essential skills that enable the construction of robust, performant systems like Eventflow.
The jsoncons tool, including its COBOL-to-JSON capabilities, serves as a practical sandbox for these ideas. Eventflow demonstrates their application at scale in a demanding, serverless environment.
Ready to see more?
- Explore the live API documentation and examples: https://eventflowapi.com
- Look out for the Eventflow codebase release, planned for the near future, to dive deeper into the implementation.
Understanding how data is structured, validated, and moved using serialization is key to unlocking powerful applications in Python, from simple scripts to sophisticated trading platforms.
