General Data Flow for Futures Exchanges
By this point in the series you should be quite familiar with this diagram of the general data supply chain for futures and options trading:
Recap: A trader submits buy/sell orders into the exchange. These orders match with opposing orders to create a trade. Trade data is then sent to the clearinghouse and FCMs before being made available via an API.
You can find more details in the top section of this post.
Whereas this series has focused on the institutional segment of the data supply chain (Exchanges, Clearinghouses, FCMs), here we are going to dive into the last mile. This is the piece that every company builds differently and where things can really go awry.
Avoid Tripping on the Last Mile
Should you use your trading sysem’s exchange connectivity? The answer, of course, is that it depends.
- Do you need these trades in other places? Risk, quants, margin, position limits, and compliance all need raw trade data.
- How will those systems be fed data? Are they fed off of the back of the CTRM system? Or, can your CTRM vendor’s feed fork the raw data in their feed?
And here is the whopper: Do you trust your CTRM team? Maybe it’s an internal team. Maybe it’s an external vendor. But the trade feed is the life blood of the entire trading organization. So, we will ask again: Do you trust your TRM team?
Experienced executives consistently guide their firms to use a large TRM at the core, but then surround it with a set of best-of-breed tools.
There are some obvious efficiencies from an enterprise system that can scale, but containing its scope means it won’t be a bottleneck.
One of the most obvious steps to scope containment is using generic trade feeds. Trade feeds that can only plug into your TRM are a door to the walled garden.
The Walled Garden
However, without folks on the team who have been burned before, companies continue to plug trades directly into a large risk system. It usually stems from a fragmented supply chain.
- Executives are trying to move the organization towards a goal – “We need a TRM system to raise more capital, scale, and stay compliant.”
- Operations groups are swamped – “The TRM system is going to do everything for us! Pump it full of data and let’s get going!”
- IT would probably want to plan something out but… “Execs and Ops need the TRM pumped with data now. They are going to use it for everything! No time to plan for the future. Go go go!”
- Traders – “WHERE ARE MY TRADES?”
What the rush fails to contemplate is that there will always be other consumers of trade data. It is the lifeblood of a trading company. Taking data from the TRM to satisfy other data needs is like trying to figure out how many red peppers you used by looking at the final stir-fry. TRM systems digest and rip apart trade data for their own purposes and aren’t a good place to get the raw data.
What happens as the company grows and…
- New quantitative analysts want to analyze past performance or build machine learning models
- Compliance officers need raw trade data to respond to regulatory inquiries or for recordkeeping requirements
- Operations teams need to work through margin calculations
- IT needs to debug issues in the TRM, but the firm doesn’t have a test environment with production data
- The firm wants to test a new TRM system
If the data is all locked in a TRM, you can forget about moving the enterprise forward with data hungry initiatives.
The real secret in the TRM game is that after a company goes through the lift to implement a large, costly, complex system, there is little appetite to revisit that effort. So if a company is disappointed with their TRM system, that disappointment will last. There is too much effort required to swap it out. Welcome to the walled-garden.
So how do we avoid getting stuck?
Tear Down the Wall
Data data data. It’s all about the zeroes and ones moving. The easier you can access raw data, the faster you can make analytics dance.
First save all the raw trade data into a trade datamart/datalake/datastore and then feed the TRM from there. With the right software, a TRM and a datastore can even be fed in parallel and save the extra hop.
Putting the raw trade data into a normalized data store means you can now feed the new TRM in parallel to your current system. Test them side by side to your heart’s content.
Any other consumers of this data will be able to access it as needed without disrupting the critical flows.
Cloud data stores like Snowflake and BigQuery provide analytical capabilities for tomorrow in what feels like a traditional relational database. Kafka provides some similar capabilities but operates like more of a messaging queue. Given the low cost and scalability, these options are a no-brainer.
Here to Help
Want to keep the learning going or get answers in real-time? We’re here for you. Take advantage of our complimentary office hours—yes, really—or subscribe to our blog below and receive all upcoming posts.
Be sure to catch up with the rest of the posts in our series, including Where Are My ICE Trades?, ICE Trade Capture Unpacked, Real-Time Trade Data from the CME, and Real-Time Trade Data from the Nodal Exchange.
Office Hours with Vivek
Office Hours with Vivek
Sign up now for a complimentary meeting with our ICE trade expert for real-time TRM help.
Trade Data Supply Chain Demystified: The Latest Updates
Understanding The Data Supply Chain, From Start to Finish.
Trade Data Supply Chain Demystified Part 1: Where Are My ICE Trades?
New Definitive Guide & Checklist for ICE Trades Part One of Five This is the first in a series of five blogs examining the futures
Trade Data Supply Chain Demystified Part 2: ICE Trade Capture Unpacked
Tracking Trade Data from an Order to Instant Risk/PnL Part Two of Five In this second post in our series of blogs discussing the futures
Trade Data Supply Chain Demystified Part 3: Real-Time Trade Data from the CME
General Data Flow for Futures Exchanges This is Part Three of our five-part series on demystifying the trade data supply chain. Be sure to read
Trade Data Supply Chain Demystified Part 4: Real-Time Trade Data from the Nodal Exchange
General Data Flow for Futures Exchanges If you aren’t already familiar with the general plumbing for exchange futures trading, be sure to read the top
