Every trade surveillance program starts at step one: Where’s the data?

You can come up with all the reports, alerts, and visualizations the heart desires, but absent the right data it’s just a list of empty requests.

Data is the lifeblood of a good surveillance program. While much of the focus for surveillance is around the logic for finding patterns of activity, here we’ll focus on step one, the data integration challenges.

Let’s start with data for listed products, such as futures and options.

Go to the source. Get your firm’s unaltered data, direct from the horse’s mouth, the exchange.

To protect your firm, you need the same data the enforcement groups and regulators have at their disposal. You don’t need all the data. Google has that covered. Think fit for purpose.

You could get derailed by well-meaning teammates who tell you all the trades are in this risk system, or quant database, and already being used for X, Y, and Z. It’s appealing. It seems like less work to do. However, risk and P&L systems almost always save trade data with a different structure than an exchange sends them. Ripping apart some trade structures and aggregating others does of course help the math go faster. Trying to stitch it all back together to figure out the raw trade data is three left turns instead of a right one.

Too many compliance projects run sideways for months and even years because management decides obtaining trade and orders data from the exchanges is too expensive or complex.

Major exchanges provide both a feed for executed transactions as well as one for electronic orders that gave rise to these transactions. If an exchange calls you with questions, that’s the data they are looking at. You want to be looking at the same data. You also want to check the box on any record-keeping obligations.

Okay, you have all the listed transactions sourced. What’s next?

Now, let’s tackle bilateral trades and physical transactions.

You’ll need to leverage that internal system, since it’s the only place to find this data.

In this world, there are no market enforcement teams with which to align. In the case of privately negotiated trade terms, the focus is less about individual transactions, and more about overall positions and their relationship to your listed trades activity.

You want to ensure that listed activities are well correlated to the physical activity, and that there aren’t events happening on the exchange that may have an unfair influence on your bilateral positions. For example, driving up the price of a listed contract as it settles for the month could have sizable benefits on bilateral swaps, which settle using the final futures settle price.

By this point, you have both listed and bilateral data sourced.

Integration: Bring it all together.

Now comes the real fun of data integration.

Each source looks completely different. At times, new pieces of data and trade types will come across that connection. So, you’ll have to build some technical connections to keep flowing this data from the various sources. This needs to be real-time, streaming ETL (Extract Transform Load).

The trading world moves fast. The technical infrastructure you put into place to feed your surveillance system can’t just sit there for 50 years like a city bridge across the river.

What kind of integration tools do you have at your disposal to transform and integrate all the data? And what happens when there are technical changes to these data sources?

If you solve all the above but can’t handle the inevitable changes, you’re crippling your compliance program. While you may be able to punt on some changes and hack your way past others, the resulting ROI will be zero.

Here is where you must have some cross-functional support. Typically, someone technical has to be aligned with compliance for the long haul to handle ongoing changes. Building is just the first step for this house. It will need maintenance.

So far, we’ve been focused on the challenges. But if you’ve made it this far, you’re in good shape to take advantage of the opportunities.

Cost savings for your surveillance spend.

Enable compliance teams to self-serve using low-code ETL tools. Why low-code? It empowers business users and places less burden on technical teams who are generally stretched thin.

Make the raw data available to trading as well as risk. This allows your business to take all the spend from above and turn it into a mirror to deliver value. Giving traders insights into how their order patterns result in certain activities is a excellent use of this integrated data set.

Visualization: The fruits of your labor.

All your data is organized, but you must integrate it directly into a reporting layer for compliance personnel to inspect. Every question leads to data inspection which raises new questions leading to more analysis. It can be simple with small static data sets. Think of a big excel sheet for example.

Streaming data from electronic markets and integrating it with slower moving bilateral data, requires the right tools to be in place so you realize value from all your efforts. Again, the last thing you want to do is put in all the work to integrate data only to get stuck at this final step of gaining insights.

Just like the data integration piece is amenable to low-code solutions, visualization tools are no different. If anything, the options here are simpler and further ahead.

We hope this leaves you feeling optimistic about solving your data integration challenges and delivering a successful surveillance program.