Since the online revolution, we have collectively flooded the world with unprecedented amounts of digital data. Several years ago, it was estimated that by 2020, the amount of information in the digital domain would reach 44 zettabytes. By the way, that’s 1,000,000,000,000,000,000,000 bytes. A truly astronomical figure, since it is also forty times more than the number of stars in the observable universe.
Another sobering statistic is that every day we produce an incredible 2.5 quintillion bytes of data. This means that 90% of all global data has emerged in the past two years alone.
Data in the e-commerce industry
With trading now being an almost exclusively online activity, you would expect the same general trend to be present in this industry. However, online commerce actually sits at the crossroads of a number of related trends that amplify the effects of this data deluge.
This industry is also benefiting from trends in mobile telephony, market access and widespread adoption which have significant impacts on the cost of sourcing, managing and storing data.
One of the significant changes is the adoption of cryptocurrencies. New asset classes mean that thousands of new symbols must be integrated, transmitted and stored. If you’ve been around long enough, you’ll remember that the cost of market data used to be so negligible that it barely factored into the business logic of most outfits. Nowadays, an entire industry has sprung up to help market players organize this most crucial area of their operations.
For example, Devexperts‘ Market data subsidiary, dxFeed, currently stores petabytes of historical market data and this will only grow as markets continue to evolve. To that end, I’d like to give some ideas to companies that are just starting to think about how to optimize their use of market data.
Regularly audit your use of market data
This should involve conducting regular audits to identify which supplier flows are critical to the business, their performance, and which have increased their prices. Also, do your vendors overlap in what they offer? Are you paying for plans that you are not currently using and do not intend to use in the immediate future? Finally, do long-term contracts really save you money? How flexible are they? Do you need to meet the needs of a changing market by sourcing different additions to your available assets elsewhere?
Only this kind of regular auditing process will give you the information you need to identify inefficiencies and then you can fix them by negotiating with existing suppliers or getting a better deal for your business elsewhere.
Get the most out of what you pay
As your market data usage is regularly audited, you should be able to make cuts to account for unnecessary redundancies and price feeds that are not currently in use. After that, the next step is to make the most of the market data you want to keep.
If you currently subscribe to feeds consisting of unrefined market data, you can do much worse than use compression algorithms to smooth out price spikes that are unique to the feed being used and therefore not representative of the market anymore. large. This is done by comparing different streams and filtering out peaks that occur within a given percentage.
This has the effect of adding value to the data you are already paying for, which will be greatly appreciated by traders whose positions are affected by said spikes.
Manage your subscriptions
While the added value is significant, the biggest savings for your business are likely to come from paying more attention to your subscription costs. In our experience, this is where third parties (including dxFeed!) have been extremely disruptive and can often offer much more cost-effective solutions than you might currently think.
This is because third parties are equipped to provide data that approximates the original real-time exchange streams. This has the effect of massively reducing trading costs as the derived data used cannot be used to reconstruct the original tick data.
Third parties also allow for greater personalization of the data you subscribe to. The big players in providing market data can be inflexible, forcing you to subscribe to certain default symbol collections, many of which may not be of interest to your traders. In contrast, third-party market data providers are more likely to allow their customers to select only the specific feeds the business needs.
Consider storage costs
With so much emphasis on accurate, down-to-the-second price data, the importance of detailed histories can easily be overlooked.
How high should brokers go? At what resolution? What exactly should you keep? This can sometimes come down to IT budget constraints, whereas a solid third-party provider can offer options that take the guesswork out and can be tailored to their merchants’ needs. These include on-demand streaming of historical price data, as well as detailed market replay data that is crucial for advanced clients who want to backtest to optimize their trading strategies.
Reduce transmission costs
Retail brokers are sometimes limited by their platform providers in how they distribute data, but there are a number of hacks that can be used to optimize transmission costs.
On the one hand, quote merging and removal algorithms can be used to sift through tick data and pass only non-duplicate ticks, as well as remove quotes that are no longer relevant. The result is a feed that looks identical to the underlying market data, but with costly duplicates and redundant data filtered out. This frees up bandwidth and reduces latency, especially during peak traffic times.
Delta encoding is another way to reduce bandwidth and storage costs by transmitting only the differences between sequential values instead of transmitting the entire stream, including redundancies. So instead of a stream of price ticks that appear as: “411, 414, 416, 413, 415”, after delta encoding they would look like this: 1, 3, 2, -3, 2 By only sending or storing deltas, this method reduces the number of bits needed to store and transmit historical market data.
The above are just a few introductory ideas and optimizations that we, Devexperts and dxFeed, have put into practice over the years while working closely with brokers to revamp their market data infrastructures.
Every brokerage business is unique, with different inefficiencies that need to be identified and addressed. But if there’s one thing we’ve found almost universally true, it’s that all market data frameworks could use at least some optimization. In our experience, there is always money to be saved and cost savings to be had. After all, we’re in the early innings of the exponential data trend we started this article with, and you might be surprised how much optimization there is already to be done under all this data.