Step 1: Model search. Searching for models of trading systems using traditional methodologies is a tedious, trial-and-error-process. There are no recipes for success and every trader is alone in this highly demanding task. For example, some traders rely on visual inspection of charts and attempt to identify pattern formations that can be modeled as a set of simple rules. Others adopt the algorithmic route and attempt to forecast future price direction using indicators or other mathematical techniques. Thus, the complexity of models can vary widely. Many trading system developers use prepackaged systems and indicators available in software programs used for back testing. However, it is highly unlikely that a combination, or even a variation, of known formulas, indicators, or pattern formations can result in a trading system with profit-making potential. It is often the case that trading rules based on experience or on the visual inspection of price charts can offer more potential than indicators or other fancy mathematical forecasting models.
Step 2: Implementation. After a model of a trading system is available, the next step is to implement it and back test it. In the context of the methodology discussed, it is not enough merely to discover a
trading system model that one might be able to use in actual trading. It is also essential that the model logic is in a form suitable for coding in a programming language. High-level programming languages designed for this particular purpose, and offered with popular back-testing software packages, significantly reduce the time and effort required for developing and debugging programming code. Still, some trading system developers prefer to write their own code due to the limitations in the functionality of high-level-language programming.
Step 3: Back testing. Back testing involves driving the model of a trading system with historical data input and generating market entry and exit signals. The entry and exit signals in conjunction with the input are then used to calculate a set of performance parameters. Most software programs with built in back-testing functionality also offer advanced graphics capability for plotting entry and exit points on a chart and displaying the equity curve. The use of high-level programming languages provides virtually unlimited flexibility for coding and calculating custom performance parameters and running advanced analysis to determine the statistical significance of the results.All the computational power and implementation flexibility available nowadays is useless unless one can find a model of a trading system that will provide the much-needed competitive advantage. As mentioned previously, back testing is the process of determining and analyzing the hypothetical historical performance of a trading system it does not guarantee future performance.
Step 4: Validation. It is highly recommended to always validate that the results obtained from back testing correspond to the intended operation of a trading system. The validation step does not try to address
the limitations and pitfalls of back testing, but attempts to identify programming and implementation errors.Specifically, validation involves the manual inspection of the back-testing results in order to determine whether the hypothetical historical operation corresponds to the intended operation. Quite often, trading system developers skip the validation step because it involves some tedious manual checking of entry and exit points and a considerable amount of calculations.A random sample of entry and exit points followed by manual calculations based on the mathematical formulation of the model usually suffices to determine with high probability whether the implementation is correct. However, such a method of validation cannot assure that all entry and exit signals the model should have generated were actually generated. The typical reaction of back-testing software program developers is that missed signals do not matter as long as the trader follows the generated signals based on which the historical performance was calculated. Conditions may develop in the future that can trigger entry and exit signals, resulting in a degradation of performance.
Step 5: Analysis. The objective of this step is to determine the suitability of a system in actual trading based on hypothetical historical performance results. Analysis is more of an art than a science, and its effectiveness depends on many factors, including skill and experience. Moreover, the results of back testing can be analyzed only in the context of the model’s intended operation. Thus, the decision of whether to accept or reject a trading system depends on the results obtained but also on the trader’s objectives.For instance, if a trading system designed for use in short-term trading ends up being a longer-term trend-following system, it should be rejected even though the analysis of the results indicates acceptable performance. Many trading systems developers analyze results in absolute terms and not in relation to intended operation. This can result in a conflict between the objectives of the trader and the operation of the trading system. Therefore, the analysis of results should also include determining whether the actual operation of the system conforms to the intended operation in the trading time frame applicable. As an example, some trading system developers use ad-hoc methods to force a system to operate in the trading time frame of interest. This includes, among other things, placing a time limit on open positions. Such practice often results in gradual actual performance degradation due to random factors that can cause, among other things, future volatility to vary significantly from that of historical prices used in back testing.
In the case of back-testing results obtained for trend-following systems, the most important parameter to consider is the profit factor Pf.
A large profit factor indicates that the trend-following system achieves its objective by minimizing losses during sideways-moving markets and maximizing profits during trending markets. The profitability P is
of secondary importance in this case, high values may be needed to avoid degradation of actual performance due to future market conditions. On the other hand, in short-term and intraday trading systems, it is quite hard to obtain a large profit factor. In these trading time frames, the value of the profitability P in conjunction with the average win to average loss ratio RWL are the important parameters to consider, although a large profit factor is always welcome. Similar considerations apply to the analysis of drawdown levels. Trend-following systems must be able to sustain larger drawdowns, but that should not be the case with intraday and short-term trading systems, where increased values probably indicate an unacceptable streak of consecutive losers. Since the maximum number of consecutive losers that can occur in the future is a random variable, it is desirable that the number obtained from back testing is as low as possible.
Rejecting a trading system because of unacceptable back-testing results is much easier than rejecting a system that seems to be acceptable, but chances are it will fail in actual trading. As mentioned previously, analysis of back-testing results is a fairly complex process and its effectiveness ultimately depends on skill and experience. Eliminating subjectivity from the analysis process is as important as developing mechanical systems for the purpose of eliminating emotions from the trading process. If a trading system is accepted but the analysis was not performed properly, although emotions are eliminated from the trading operation, performance is already compromised and the benefits of systematic trading may not be realized. This is one reason it may be preferable to make the analysis of back-testing results automatic; this naturally leads to the concept of synthesis of trading systems.
Step 6: Modifications. After the results of a back test are properly analyzed, the trading system developer may attempt to improve performance by making appropriate changes to the model. For instance, different position exit schemes and money management methods can be implemented in an attempt to determine the best one for the particular market and intended operation. It is important to understand that the model modification step should not be confused with performance optimization. While the former is a useful practice, the latter is not. It is very hard to establish rules for determining when a specific change made to a model transforms it into a completely different model. The only way this can be deduced is by analyzing back-testing results. For example, changing the exit logic from a profit target based on a fixed percentage of the entry price to a trailing stop may result in an increase in the profit factor while at the same time turning a short-term trading system into a trend follower. Thus, all changes made must remain within the domain of intended operation. This is a rule not always followed by trading system developers, who often analyze the results in absolute terms and ignore whether any changes made to the model transform it into a different model not conforming to initial specifications, including trading time frame considerations, initial trading capital requirements, and risk/reward parameters. If one is not careful, such practices may cause trader–system incompatibility, which can be the source of significant losses. Optimization and implementation of ad-hoc methods to reduce losing trades must be avoided in this step. An important empirical rule is that any changes made to a model that reduce the number of losing trades without a proportional increase in the number of winning should be viewed with great suspicion. The reason for this lies in the possibility that such improvement may be just a filter of losing trades based on a limited set of conditions that just happened to be present in the historical data. When the system is employed in actual trading, new conditions may emerge not subject to the same filtering used and performance can degrade to the point of reversing from profitable to unprofitable. This can happen when the system was tested on market data spanning the period of a major prolonged trend. For example, if the developer changes the system logic to filter out short trades by eliminating signals that occur while the difference between a fast and a slower simple moving average is positive, the number of losing trades may be reduced significantly. However, in actual trading conditions, the trend may reverse to a prolonged downturn and the system may end up filtering out short trades occurring near the peak of short-term reversals (those offering the best potential of profit in a downtrend).
By the way, this appears to be a serious limitation of many systems designed for trading equities using data from the period of the longest stock market rally in recent history, from 1993 to early 2000. As a matter of fact, traders who used black-box trading systems based on such native design methodology after 2000 ended up losing money. The best way of avoiding indirect optimization due to special conditions reflected in the historical data is by selecting markets that exhibit several cycles of upturns followed by downturns, such as commodities and currencies. This is especially useful for short-term and longer-term system developing. In the case of intraday systems that do not use other time frames to filter out trades, the effectiveness of such methods to reduce the number of losers may be higher. Model modification is another step in the trading system development process that is plagued with subjectivity and errors and will also be eliminated by the synthesis method.