All investment strategies generally follow a similar lifecycle. Strategies are thought up (generation), they are tested against historical data (validation), if testing proves well they are put into practice (implementation), and once implemented they are subject to continual monitoring and analysis (confirmation). Last, the cycle repeats itself as strategy improvements follow the generation, validation, implementation, and confirmation loop.
In 2015 we were contacted by a client that had already made a full trip around the strategy lifecycle with an investment process that was originally conceived of in the early 1980s. At a time when quantitative and computer-driven strategies were exceedingly rare, this firm leaped ahead of their competition by implementing a strategy in the Fortran programming language that scanned thousands of individual data points to generate lists of portfolio candidates. In the more than three decades since then, the strategy had proven itself a winner and the firm continued to utilize the program’s outputs as a part of their larger investment process.
With a winning strategy and a nearly unprecedented amount of historical performance in hand, this firm had little to complain about, except for the fact that the program was still written in Fortran.
The program was written well and had operated nearly flawlessly since it was created more than thirty years ago, but it was becoming increasingly difficult to adapt to newer operating systems, required a significant amount of training to operate, and could only ingest data that was hand-input. The system also represented a single point of failure since its database was hosted on whatever machine it ran on. Attempts to clone the database and share it amongst multiple instances of the program had failed, and even if they had worked would represent only a temporary fix to the larger problem.
It was clear that the program needed to be brought into the 21st century. The new system needed to recreate the original program’s analysis with a new user interface in a cloud-based environment that supported multiple users, automatically ingested market data, and that was redundant in case of system failure or corruption. Additionally, the client expressed an interest in adding backtesting capabilities that would allow flexible testing of a variety of the existing strategy’s criteria.
To meet this challenge we broke our process down into three phases:
- Reverse Engineering & Replication: Documenting & testing the old FORTRAN code, creating comparisons in a new language, and measuring outputs against a wide variety of inputs for accuracy.
- Re-creation: Using our reverse-engineered core logic, re-creating the system’s functionality in a new language in a cloud environment, with new means of interaction more in line with the present state of web application technology.
- Performance & Feature Improvement: Improving system performance for wide-area data collection and analysis, adding features to improve strategy performance and visibility.
Using the platform’s existing data, the 600+ page dot-matrix codebase we’d been handed, and an understanding of the math from one of the strategy’s founders, we began to create a loose map of what exactly was going on under the hood. Given some doubts we had about the closeness of the codebase to the actual functionality (the book appeared to be from an even older version of the code), we had to navigate by feel.
Eventually, through many rounds of prototypes, tests, and comparisons, we were able to get the our proof-of-concept strategy’s mathematical component to match that of the FORTRAN code 1:1 under every circumstance we tried, which led us to our next phase.
For remaking the platform anew, we chose to use a combination of the Laravel PHP framework and Python, using MySQL as our database and Redis as a cache. Laravel enabled us to efficiently create the user-facing portion of the web application as well as handle server-side tasks such as scheduling ETL processes, managing worker pools, and efficient object relation. For some portions of the backtesting engine, we employed Python as it allowed us to perform some calculations and data handling processes much faster than PHP was able to. Specifically, matrix operations in PHP can be a pain.
Performance & Feature Improvement
After nailing the strategy’s mathematical component, and replicating the base level functionality in a web application hosted on Amazon Web Services (AWS), we were finally into the really fun part. This is the phase where we get to push every system component to its limits.
We started by separating the system’s core logic away from the frontend. Command and control would happen from a small server hosting the web interface, and the heavy lifting would be done separately. In doing so, we were able to utilize an AWS service called Elastic Container Service to scale and shrink our lifting capability based on demand. If there isn’t much to do, very few resources are used – the system just checks in occasionally to see if there’s anything to do. Once a big job hits the tape, the system sees it and begins to step on the gas. Depending on job size, thousands and thousands of container instances (small computers) are spun up all at once while the command and control server begins to delegate work and collect results.
As a result of this, we’re able to achieve supercomputer-level insights about the strategy’s performance in a variety of circumstances that otherwise would be impossible, and can do so quite affordably. Because of this transformation, our client was able to go from an overworked laptop running code from the 1980s to advanced statistical analysis and machine learning generated insights in an impressive amount of time. They have been able to leverage these new insights and technologies to re-orient a sizeable portion of their business to focus on this strategies and its several variants.
Check out the dashboard we built for monitoring mass backtesting below. Thanks for reading, and as always please get in touch if you’d like to learn more!
On @AWS (ECS + Aurora + Elasticache), we’re able to use 30TB+ of RAM and 5,000+ vCPUs to perform thousands of years of simulated trading for one of our clients. Check out this time-lapse of our @datadoghq dashboard that recorded over three hours of analysis. pic.twitter.com/qn53cO9o3l— Oscillas Technologies (@OscillasTech) January 16, 2020