Published at September 1, 2020by Jörg Brandeis

This blog series is, for once, not about SQLScript, but primarily about ABAP. Transformation routines in SAP BW can be implemented in both programming languages. And there have always been, and still are, many customers who like to use ABAP for this. There are various reasons for this. One major strength of ABAP is dynamic usage, which I cannot achieve in this form with SQLScript.

In this series, I want to share my experiences from my recent BW projects. I have repeatedly been fortunate to work in greenfield projects at very large customers with high requirements for SAP BW and, in particular, for backend processing. As a consultant with a focus on software development and strong BW and HANA knowledge, this is of course ideal for me.

The Problems in BW Transformation Routines

In SAP BW systems that have grown over many years, sometimes more than 10, you often find a wide range of different BW transformation routines. Some of them are programmed very cleanly and efficiently. Others are barely maintainable anymore or have massive performance problems. I would like to present a few typical cases here that most people have probably seen before.

The Spaghetti LOOP

Unstructured or poorly structured code is referred to as spaghetti code¹. As the length of the code section increases, readability decreases considerably. In transformation routines, the problem is exacerbated by the fact that these normally consist of one large LOOP over the ResultPackage². I call this unfortunate combination the Spaghetti LOOP. A typical problem here is, for example, that values from a previous loop iteration are stored in variables and reused in later iterations. This means I always have to search the code before and after the current line to see whether a variable is being changed.

I have actually seen Spaghetti LOOPs of more than 2000 lines, and they have brought troubleshooting and the processing of new requirements almost to a standstill. Because every correction produced unpredictable effects on the other requirements.

50 shades of modularisation

Unfortunately, ABAP offers a great many different ways to prevent all the code from ending up entirely in the transformation routine. Unfortunately, all of these modularization techniques are also used:

• Instance methods
• Class methods
• Function modules
• Subroutines
• Macros
• Includes (!)

I have actually seen transformation routines that consisted only of an INCLUDE statement. The same include was then used in different routines. That was the ugliest application of the DRY³ principle I have ever seen.

Performance Problems

Generic Overengineering

Generic and dynamic programming is very much possible in ABAP. You can do very useful things with it, but you can also make simple requirements extremely complicated. The readability and maintainability of the code then suffer as a result. And often you see that the same generic function was reinvented multiple times by different developers. Or that generic functions are called exactly once.

TYPE STANDARD TABLE and ASSIGN

Moving code out of the transformation routines brings a typing problem with it. The type of the ResultPackage is generated in the routine class. That's why the signature of methods typically uses TYPE STANDARD TABLE. Access to the rows of the table is then done with LOOP ASSIGNING, and access to the fields of the rows is again done dynamically with ASSIGN. With the field symbols we can then write directly into the table in the current row/column. This is convenient, but it also allows changing fields that we originally only wanted to read. So there is a lack of clean documentation of what is read and what is changed. And the code for each field used is relatively long (and slow):

    FIELD-SYMBOL <lv_value> TYPE ..
    ASSIGN COMPONENT 'FELDNAME' 
        OF STRUCTURE <resultline>
        TO <lv_value>. 
    ASSERT <lv_value> IS ASSIGNED. 

This can be done more elegantly.

The Causes

There are several causes for these (and other) problems that can be observed repeatedly in projects.

Occasional Programmers

Almost always, the transformation routines are developed by BW consultants⁴ who have only limited programming experience. That is even sufficient to find an adequate solution for perhaps 70% of the requirements. But when things get more complex, solid ABAP and programming skills are required. You don't learn those on the side; instead, you need an experienced programmer.

Sometimes BW consultants also overestimate themselves and simply give it a try. And sometimes that even leads to a more or less functioning solution. But in terms of maintainability, things then look really bad.

ERP ABAP Developers

If the BW consultants fail, the next cause of problems often comes into play: ERP ABAP developers are not used to working with real mass data. That's why functioning, clean, maintainable, but slow concepts are implemented at row level.

Moving Targets

Often the requirements are not clear from the start. On the one hand, the requirements change, and on the other hand, new things keep getting added. These changes are always woven into the existing code. This happens even when the existing code is no longer 100% understood.

Too Many Cooks

Long software lifecycles and long projects give rise to the next problem. Since development in a BW is ongoing, many different people are involved over time. This means the wheel is reinvented again and again, and the number of variations increases. For the complex transformations from the early days, a real expert can rarely be found.

No Modern Concepts and Methods

In the SAP world, modern concepts and methods for developing software catch on more slowly than is the case in other areas. These include, among others:

• Test Driven Development (TDD)
• Clean Code

This applies to a particular degree to BW transformation development. On the one hand, because of the aspects mentioned above. On the other hand, also because the requirements are often underestimated, along the lines of: it's just not worth it for that small routine.

Applying TDD is also made more difficult when you have to program against the database state. Unfortunately, this is the case in BW.

The Transformation Framework

We have already brought the transformation routines into a clean structure in various projects. A framework that enforces a clean and uniform approach helps with this.

The most important point is that the complexity of the requirements is broken down into small, simple steps. And that each of these steps is implemented in a separate method. Some of these steps can be implemented generically and can be reused again and again in different data models. Other steps are specifically necessary for exactly one transformation only.

This decomposition allows us, on the one hand, to move the control of the steps and the parameterization out of the code. In other words, a separation between:

1. What should be done
   1. Which steps in which order
   2. Which fields are read in a step, which are changed
   3. (Which records should be changed in a step?)
2. How the step should be implemented.

Through the decomposition into small steps and the outsourcing of the control of these steps, many orthogonal requirements can be implemented. Among others:

• Runtime measurement per step
• Generation of documentation
• Targeted breakpoints before a step
• Test Driven Development
• Reusability of steps
• Generation of AMDP code

In the following blog posts:

• The decomposition into steps - At first this looks as if a lot of overhead is being added. But that's nonsense. Runtime and maintainability improve, even if there is a bit more source code.
• The control of the whole thing - A set of tables with which you describe the transformations and the important parameters of the steps.
• Error handling
• New fields are needed - so that the requirements can be broken down into small steps, we sometimes still have to build additional fields into our ResultPackage

Footnotes

1. https://de.wikipedia.org/wiki/Spaghetticode ↩

2. I write ResultPackage because it is the most common case. Depending on the routine type, the SourcePackage may of course also be meant. ↩

3. Don't Repeat Yourself ↩

4. The term occasional programmer sounds very negative, but it is not meant to disparage the work of my colleagues. Most BW consultants are very competent and experienced, and I have great respect for their work. But as a developer who deals with ABAP 8 hours a day, I have a different understanding of programming than a typical BW consultant. ↩