In the first part of the case study, we outlined the purpose of the performance testing for the Bulgarian DSK Bank, described the selected systems, then highlighted the dilemma of the appropriate sizing of the banking systems, and also presented the special difficulties encountered by DSK Bank. We now provide an overview of the actual performance measurement, then of the way of cooperation with internal experts, of communication with the management, and finally of the expected and actual results.
DSK Bank made an acquisition, not a completely new system. The expansion resulting from the acquisition of the other financial institution is a significantly easier task for the testers than the control of a newly created system, as it means a lower level of risk. In such cases, the task is essentially to look at the operation of a known system in an unknown load range.
In the case of a new system, completely new developments, new components and new functions are introduced, which mean a lot of uncertainty and risk. At DSK, we were able to start working with the increased data volume of an existing system, with existing hardware, software, and existing operators, relying on their experience. (The joint team of KPMG and ProofIT has already performed performance testing of new systems - most recently at MKB Bank.)
Performance tests and measurements can be of many types,
The questions to which we can get answers can be quite simple questions to be decided, for example: does the system comply with the pre-established performance indicators, but they can also be quite complex, for example: where are the bottlenecks and under what circumstances does the system run into them, what impact do they have loads of different services operating on the same system group?
For example, while searching for upper load limits, we load the system more than usual, and then see how the response times and transaction throughputs develop. If we detect an overload, we start adding extra resources - either hardware or software - to see how it affects maximum performance (the relationship here is usually not linear). A significant part of our measurements was also aimed at checking the results of modifications and adjustments when testing DSK Bank's system, and based on these, we made recommendations for further modifications.
Outsiders rightly ask the question, if it is possible to know in advance that the load will increase by 30-60 percent, then why can't the resources simply be increased by that amount? Designing such complex systems is not an exact science. There are aspects that can and should be taken into account during planning. However, in reality, it is only during implementation that it is possible to finalize what type of resource and how much extra capacity must be installed in which layer in order to realize the planned extra capacity.
Most banking processes are extremely complex. Dozens of systems are involved in a single sub-process, for example a mobile bank transfer. If there is a bottleneck somewhere in just one of these systems or hardware components, it affects the entire process. During testing, therefore, most measurements examine the behavior of the system, under extreme conditions, and after these modifications, we measure back whether they achieved the expected effect.
Testing a financial system - especially when there is so little time available, as in the case of the DSK project - is unthinkable without the intensive cooperation of internal specialists and experts. At the Bulgarian bank, in addition to department heads and group heads, we were also able to enjoy the support of the IT manager (also one of the bank's deputy CEOs). It is important that all management levels accept the goals of performance testing and then participate in resource allocation and task prioritization. (It is clear from all this that although the testing is performed by an external, independent company, it is not an external job that can be outsourced without the internal experts noticing it.
Although performance testing is only a small part of the entire migration project, the fact that the testers place heavy loads on the systems affects its entire operation, so there is no question of going unnoticed. Of course, the work at DSK Bank was not done on the live banking system, but on a test system, but we were not alone here either. Therefore, the performance tests had to be strongly coordinated with the other tests running on the system and with other current activities, so that the possible extreme loads of the performance measurements do not hinder the other tests, and the other tests do not affect the numerical results of the measurements.
A performance testing project requires very different competencies. These may change during the course of the project. The joint team of KPMG and Proofit worked with a total of 9 people on the DSK Bank performance test.
For the work, the bank provided the high-performance hardware infrastructure, with the help of which we were able to simulate simultaneous transactions from tens of thousands of customers and thousands of bank clerks.
Performance testing of a bank cannot be done on a small or large scale. This task cannot be reduced. This means that a lot of information has to be accessed and processed. These factors also basically determine the duration of the project. The team's previous, similar projects lasted approximately 1 year each. Roughly this amount of time is required to see into the operation of the bank, plan, develop, communicate, and perform dozens or even hundreds of checks.
Now, in order to be able to do all this at DSK Bank in a quarter of the time, i.e. 3 months, we had to make serious compromises - in such a way that they did not jeopardize the usability of the result. Compromises have already started with the identification of target areas and the definition of performance indicators - compared to the ideal methodology, their scope had to be narrowed. Due to the tight deadline, we had to demand continuous, extreme performance, extraordinary agility and availability from both ourselves and the Bulgarian contributors. Only with the high-level support and coordination of both organizations could the project be realized in such a short time.
On extreme days, this meant up to 6-8 daily meetings with middle managers and 1-2 with senior managers. Experts and operators had to be mobilized even at night, on weekends, or on holidays. Such intensive employment of external colleagues would not have been possible without adequate support from 2-3 management levels. By prioritizing tasks from the bank's side, preparing people on time, they seriously laid the foundation for our work as well.
DSK Bank's extended system started operating at the beginning of May 2020. That's when it became clear how the transition was successful. (Succeeded.) When measuring the results, in addition to response time, maximum load and utilization, many more indicators can be determined. The point is that they have a benchmark and that they can be verified.
While the business processes of banks are very similar, their implementation can be very different at the technical level. It follows from this that there are boxed performance measuring devices, but they only cover standard technologies. There is no such ready-made measuring device for measuring a specific system, manufacturer-specific protocol, or other special conditions.
Proofit has its own expandable platform for both performance testing and functional testing, based on these it creates an independent test lab environment customized for the given task for all its projects, including bank projects. The drivers, sensors and the evaluation graphs designed for the task are placed here, as well as the configurations defining the load patterns, message patterns, parameters, test data, as well as the raw and evaluated results.
Measuring a more complex industry or system often requires certain unique developments. In such cases, we also create components that are able to communicate with the given system, send messages in a correct and repeatable way, read and interpret the responses received to the messages, so that it is sufficient to decide whether the communication was commercially successful or not. then what type of error did it end with.
When testing a financial institution system, the measurement objectives can also be very layered. About twenty types of measurements were also made at DSK - each of them has a different need for evaluation.
In a measurement project, an important requirement for the customer is the provision of measurable systems. In a banking environment, there are usually 4-5 such copies of the main system reserved for different purposes, but there are usually no copies specifically dedicated to performance measurement. Therefore, the performance measurement usually has to share with other types of tests the environments in terms of space - time - configuration, it is usually a serious task to create an integrated environment in which there is also a copy of the main systems involved at the same time
The developed measurement systems determine which processes and sub-processes of which system, which message types are suitable to represent the real load. They are able to direct the load into the appropriate channels. They also determine where the responses and load results of the background systems can be read.
At a large commercial bank, we usually have to count with hundreds of business processes and the hundreds of message types required for their operation. For testing, these processes and the traveled paths must be reduced to a few dozen, while still covering the main paths. This narrowing must be carried out while taking into account industry and local conditions, priorities, frequencies and statistics.
In general, the used test lab should be able to produce as much load with 20-40 message types as all (200-400) transaction types would produce on the real system.
The interdependent measurement sessions are complex in themselves, each containing up to 20 sub-goals. The tests may include simple response time measurements, but they may also include endurance tests lasting several hours and covering 5 subsystems.
In such a project, it is not enough to submit a final report. In order to support developers and operators, it is important to ensure, in more frequent, extreme cases, results that are updated on a daily basis or several times a day, on which configuration and optimization decisions can be based.
Although we produce a lot of numbers and spectacular graphs based on the tests, we are aware that our client does not need this, but mainly their interpretation. Based on well-founded information, they can modify the affected systems, processes, and parameters in time.
Our team performed an enormous amount of elemental measurements. After the first, preparatory month, measurements are required on average every second day. In the second and third months of the project, we modeled more than 20 million transactions with a total of more than 700 measurement runs.
We have verifiably and repeatedly shown on the more important systems and process types that the system is not in danger at loads 2-3 times higher than the extraordinary loads expected during the start-up week. (The sharp start at the beginning of May proved this.)
