Rising Fintech Credit Lending Firm Hires FP Complete during the Pandemic to create a seamless transition from legacy code to modernized cloud architecture
FP Complete takes a credit lending technology firm from a physical server environment to a modernized cloud architecture platform with zero downtime.
FP Complete jumped in during this firm’s restructuring to oversee engineering leadership, project management, design, and implementation and assist in the hiring and training of new software engineers.
The customer had several urgent requirements for the project:
- The new solution needed to be proven to produce the same exact results as the legacy code, given the same inputs, but also integrate modern DevOps technology such as CI/CD.
- The project had to be completed with no production downtime to avoid financial processing errors that would trigger costly credit expert reviews.
- Achieve the above during a period of high employee turnover. Departing employees included the CEO, the CTO, and the developer of the original legacy code.
Corporation Type
Credit Lending Technology
Industry
FinTech
Project Type
Credit Policy Software
Business Issues
- Client needed the solution to integrate modern DevOps technology such as CI/CD while using the same inputs to perform the same core business operations.
- Client needed the project completed with no production downtime to avoid potential financial processing errors that would trigger costly credit expert reviews.
- Client needed a smooth transition during a period of high employee turnover. Departing employees included the CEO, the CTO, and the developer of the original legacy code.
Project Outcome
Arranged a Knowledge Transfer
Arranged a knowledge transfer with outgoing employees to avoid losing their institutional knowledge. Ensured the current code and legacy workflow were well-documented and thoroughly understood before implementing any changes.
Project Outcome
Successful Generative Testing when Monitoring Output
Rewrote the legacy code incrementally and carefully over multiple months, using generative testing to monitor output samples to avoid accidental logic changes. Completed the new code without causing a single unexpected bug.
Project Outcome
Facilitated Candidate Interviews and New Employee Onboarding
Assisted client by conducting technical interviews with prospective DevOps candidates to fill open positions, and provided project-specific training to speed up the onboarding process upon hire, resulting in a competent and proficient team to run the new software.
TECHNOLOGY USED
AWS, EC2, Haskell, Ruby, Microservices, PostgreSQL, AWS Lambda
FP Complete’s Solution
To prevent the Client’s Credit Policy engine from drifting into chaos, FP Complete was asked to take over the engineering management, project management, and programming until the project stabilized.
- We arranged a knowledge transfer with the employees leaving the company, ensuring the current code and legacy workflow were well understood before making any changes. We did a thorough write-up of this knowledge to remove the existing single points of failure.
- We enacted a policy of “full output equivalence”, meaning that code refactorings to the project could not change computational results. To ensure this, we used generative testing to generate millions of input-output samples, covering well the probability distribution of possible loan applications, and built them into the Continuous Integration pipeline. This warned us of any accidental logic changes.
- With this safety harness in place, we rewrote the legacy code incrementally and carefully over the course of multiple months, without causing a single unexpected bug.
- We helped the company fill their open developer positions by conducting technical interviews for their candidates.
- We assisted the Client in finding a few highly skilled programmers and then performed project-specific training to speed up onboarding. This included sharing our years of experience with fully remote work, helping the company to stay organized during the Corona virus lockdowns.
- With the legacy problem solved by the clean rewrite elaborated above, the newly built team could perform the architecture switch from VMs to AWS Lambda. Because the cleaned code was written in a Functional Programming (FP) style that Lambda fundamentally embraces (benefitting from statelessness and referential transparency), this migration went through without surprises.
- We incorporated common DevOps techniques FP Complete employs routinely to reduce the risk of downtime. Git-push-to-auto-deploy as our Continuous Delivery framework, linear version history for regression bisection, structured logging, and property-based integration testing were especially useful on this project.
NEW CHALLENGES FOR FP COMPLETE
- The unexpected departure of company employees, including our daily project contacts.
- Taking on additional responsibilities beyond the initially planned code implementation by providing project management, product development, and engineering management.
- Navigating a wildly grown network of legacy databases and microservices that our project had to interface with.
- Protecting a large amount of Personally Identifiable Information (PII) during the advent of new EU privacy regulations (GDPR).
The Conclusion
Despite a rocky start and staffing changes, the clean code rewrite and cloud architecture switch was successfully executed in a year-long development process. The microservice had no unplanned downtime, and customers reported no outages or service issues. The customer received a revamped cloud platform with modern, reliable code and a solid new team to tackle future projects.