ASSESSMENT OF EVALUATION METHODS FOR SPACE TECHNOLOGY CONCEPTS more

IAC-11.D3.4.5 ASSESSMENT OF EVALUATION METHODS FOR SPACE TECHNOLOGY CONCEPTS Egbert Jan van der Veen DLR, Germany, Egbert.Veen@DLR.de This paper identifies, analyses and appraises a range of technology evaluation methods for their applicability in evaluating space technologies concepts. Evaluation methods are mostly used to support and enhance decisions regarding investments in technology development projects. Examples of evaluation methods include: Delphi Method, S-Curve Extrapolation, Decision Trees, Analogy- and Patent Analysis and many others. The paper identifies methods which are most suitable for evaluating space technologies in order to improve investment decisions in the space sector. Investments in space technologies can be categorized according to a mission-focused (“pull”) and transformational (“push”) influenced decisions. Especially the push technology investments require a great deal of strategic planning, as their time horizon is longer than that of pull investments. Because of the presence of breakthrough innovations (also named Game Changing Technologies or Disruptive Space Technologies), the need for the usage of the most accurate evaluation methods is apparent. In literature many evaluation methods exist, however these methods are just partially applicable to the space sector, because of its different market dynamics. This difference causes the factors influencing the diffusion of innovations (agents’ decision-making process, the product characteristics and the structure of interactions between agents) to be different from the non-space sector. For example, because of a relative low frequency of space technology usage, the method of extrapolating trends is deemed impractical for evaluating space technology concepts. This paper identifies which factors influence the effectiveness of evaluation methods within the space market structure and assesses which methods show the highest potential to be an accurate tool for space technology investment decisions. The application of the results of this paper will lead to more informed investment decisions and thus, improvement of the technological capabilities of the space sector as a whole. INTRODUCTION Developments of space technologies have enhanced our society in the form of understanding of our universe, communications, navigation, healthcare and many other areas. The space sector is working continuously on improving these space technologies so that they can benefit our society even further. Because of budget constraints, only a small portion of the technology ideas can be developed and tested. Even less continue to be actually developed into spaceflight proven technologies. Decision makers within the space sector face the problem of identifying the technology concepts which will maximally benefit the space sector and thus justify an investment. There are many tools available for evaluating or forecasting the development of technologies. These all differ in their approach, their time horizon and their results. This paper will document on an assessment of evaluation and forecast methods for their use on technology evaluation for the space sector. The diffusion of technologies depends on the dynamics of the market. Almost every market has different market dynamics, which is depending on the interactions between the agents’ decision-making process, the product charac- teristics and the structure of interactions between agents (Janssen & Jager, 2003). This paper starts with an analysis of the market dynamics of the space sector, along the interactions mentioned before. This will give insight in the processes that influence technology development. After this, different technology evaluation and forecasting methods will be assessed for their applicability on space technologies. These methods will be taken from the overview provided by the futures methodology framework of Vanston & Vanston (2004). The results are then summarized into a framework which can support decision makers in their choice of evaluation tools. MARKET DYNAMICS The space sector has a complex market structure which is highly influenced by governmental entities. Across the world, there are over 50 space agencies, more than 40 commercial operators and several institutional entities which are procuring launchers, spacecraft and ground related services. To clarify the market dynamics of the space sector, different factors described by Janssen & Jager (2003) are used: the agents’ decisionmaking process, the product characteristics and the structure of interactions between agents. 1 Agents’ Decision-making Process In this section, the agents’ decision-making process is described. The space sector is a very diverse sector, but can nonetheless be divided into two fields, military and civil. The civil field is divided again into a commercial and scientific field. The military and scientific fields receive the majority of their funding from governmental instantiations. The commercial field however, has citizens and companies as direct customers and receives income through them as well as from governmental instantiations. This also means that the market dynamics in the scientific and military field differs from the commercial field and is caused by a different customer seller relationship. Smyrlakis et al. (2011) and Szajnfarber & Wiegel (2007) have confirmed this by classifying the space sector as being highly influenced by governmental entities. They even state that, in the case of science missions, the space sector resembles a MonopsonisticOligopolistic market structure. This entails that a monopsonistic buyer, which in the space sector is a governmental institution, faces a small amount of possible sellers (Oligopolists), which in the space sector are prime contractors. Although it might be tempting to see a government as one single entity, this might be an oversimplification of the market. This is mainly because often multiple governmental entities exist, which are procuring and/or providing funding for space technology development. For example in Europe the space sector has the following governmental entities with their own unique strategies and visions for funding:  ESA with 18 participants, 1 associate member and 12 cooperative members  The European Commission  18 National agencies across Europe Although this might mean that the space sector is not a monopsony, it still is highly governmentally influenced. For the decision making process this means that all decisions are influenced by governmental policies, laws and regulations. This means that within evaluation methods, governmental influences have to be taken into account. Product Characteristics The main products of the space sector are payloads and launchers, with addition of their components and associated services (Pagkratis, 2011). Space is an environment which is difficult to reach and to operate in. This creates unique constraints in form of performance levels exceeding those required for terrestrial technologies. Performance requirements unique for space technologies include resistance to: o High energy radiation (both ionizing and electromagnetic) Extreme temperatures Large and frequent temperature variation Micro impacts Vacuum Shocks and high acceleration (during launch & reentry) Additionally, the high costs and risks involved with moving objects into space results into: o High quality and flight heritage (technology has proven itself as reliable for operating in space) requirements o High testing costs and long testing times o High costs and fewer applications These factors result in a vicious circle of higher costs and testing times within the space sector: o o o o o Figure 1: Vicious circle of space technology cost. This vicious circle has the following consequences for investments in space technologies:  The long development time of a space technology means that the risk and return on investment is equally high, this is a barrier for new startup companies, as this makes it difficult to find investors.  Space technologies often have a significant amount of equipment purchases, development costs, proprietary knowledge and human capital invested into them. These costs lead to a reluctance of incumbents to cannibalize existing technologies for new technology developments (Kamien and Schwartz, 1982). Structure of Interactions In the space sector, a difference can be made between basic technology development and mission specific technology development. When looking through the “technology push” and “demand pull” model, the basic technology research and research in early stage innovations can be identified as the technology push factor while technology developed for specific missions can be identified as the demand pull (Summerer, 2011). Especially push investments result in breakthrough technologies while the pull investments result in more incremental innovations (Nemet, 2009 and Carayannis & Roy, 2000). 2 Figure 2: Key features of evaluation methods EVALUATION METHODS Technology evaluation methods are used to justify decisions in technology investments. After evaluating technologies for their potential for future success, a forecast can be made. Because of this, forecasting methods are used as a synonym for technology evaluation methods. Forecasting methods use information from the past to forecast events in the future as depicted in Figure 2. This information can come in many forms like e.g. experience, performance data, intuition, trends, patterns. This process is based on the assumption that powerful feedback mechanisms in human society cause repetitive processes (i.e. future trends and events to occur in identifiable cycles and predictable patterns based on the past). Accurate evaluation methods can contribute to the selection of the most promising alternatives leading to the most beneficial possible future state of the system. Because of this clear benefit, an extensive amount of literature has been written on this subject. Within this vast amount of literature, different approaches to gaining insights on future developments can be identified. In general, these views can be categorized as being either quantitative, qualitative or a combination of both. Quantitative methods are based on a hard sys- tems approach to reality (Jackson, 2000). Supporters of hard system approaches believe reality is composed of ontological entities and that based on this view an accurate forecast of the future can be made. The qualitative methods are based on a soft systems approach to reality and believe accurate forecast depends on human non-quantitative inputs. In Figure 3, the axis of quantitative and qualitative methods is used to provide a categorization of methods. On the axis from quantitative to qualitative you have the Extrapolators, Pattern Analysts, Goal Analysts, Counter Punchers and Intuitors. These different categories contain several methods which will be described and assessed on their advantages, disadvantages and applicability as evaluation methods for space technologies. Extrapolators Extrapolating methods are based on the view that the future state will be a logical extension of the past. Complex forces will drive the future in a predictable manner which can be used in creating forecasts based on analyzing trends from the past. Due to this fact, a forecast can be made by extrapolating the past according to mathematical principles. The methods that follow this hard view on systems are highly quantitative in nature. Examples of extrapolators are Technology trend analysis, Fisher-Pry analysis, Gompertz analysis, Growth limit analysis and Learning curve. The Technology trend analysis is an analysis which maps the performance of a technology on a linear scale. It is based on several observations of technologies advancing according to an exponential improvement process. Famous examples of these are: Moore’s Law which predicts the amount of transistors in processors and Haitz’s Law which predicts the improvement of light-emitting diodes. Figure 3: Five Views of the Future strategic analysis framework adapted from Vanston & Vanston (2004) 3 The Fisher-Pry analysis, Gompertz analysis and Growth limit analysis are all based on the statement of Beer (1981): “Technological change can be categorized as a series of overlapping S-Shaped curves”. They are different formulae to predict the S-Curve development of technologies predicted by Beer (1981). Learning curves are cost modeling forecasting techniques which state that the more items of a certain type are produced, the lower production costs tend to be. The learning curves are often illustrated as a straight line on a log-log graph paper which makes prediction relatively simple. Advantages/Disadvantages + Forecasts based on extrapolation methods are backed up by historical data which is usually highly accurate and relatively easy to obtain. + Very accurate predictions can be made if it can be proved that technologies follow a certain rate of improvement over an extended amount of time. + Predictions are usually made relatively quickly, as they do not rely on expert inputs. + Using only quantitative data eliminates factors of bias. Prediction of performance gain only works for technologies focusing on one performance dimension. Methods do not take physical restrictions into account. Methods relies on a status quo of the market (i.e. major changes cannot be foreseen) Changes in driving forces might change the development of a technology (i.e. technologies can become strategically more important) A law like Moore’s law can become a selffulfilling prophecy, once it gains popularity (i.e. companies assume the rule to be true and adjust their funding level to the rule as not to fall behind) Applicability of extrapolators to Evaluate Space Technologies Extrapolator methods are hard to use for forecasting STs because STs often compete on several performance dimensions (i.e. two rocket engines cannot be compared merely by the amount of thrust or specific impulse they offer). This makes the extrapolation of a trend of the past in the performance of the primary performance dimension useless. Additionally for extrapolation a forecaster would need many accurate data points over a relatively long period in order to extrapolate the trend, which, for many technologies, is not available in the space sector. This is caused by the irregular use of space technologies which makes identifying trends very complicated and rarely accurate (usage of space technology is highly dependent on missions). Because of this, extrapolation methods inhibit the evaluation of most technologies as no past data can be extrapolated. On top of this, Sood & Tellis (2005) found out that technologies often do not follow distinct patterns and often jump in performance after reaching a plateau for some time. Because of these reasons, it is not recommend using extrapolator forecasting methods for the evaluation of Space Technologies. An exception would perhaps be frequently used technologies with relatively a highly dominant performance dimension (i.e photovoltaic cells). Pattern Analysts Pattern analysts believe that future technology adoption will reflect the process of past technology adoptions. This view of reality has led to a method of identifying and analyzing analogous situations of the subject technology and applying the found patterns to predict the future development of the technology. The adoption of color television, for example, closely followed the pattern of adoption of black-and-white television and which, in turn, followed the pattern of radio adoption. Thus, one might reasonably forecast the pattern for future adoption of high-definition television by examining the pattern of past adoption of color television. On the other hand, it is quite possible to choose an invalid analogy, and, in any case, future developments never exactly replicate past analogies. This field is equal to extrapolators in the way that it assumes that occurrences in the past will be an indicator for future development but differentiates itself, by being broader in its application area, i.e. focusing on more than a single performance dimension or technology replacement. This kind of analysis also takes market factors in account and can be highly accurate if the right analogy is available. Analogy Analysis is based on the view that patterns of diffusion of a technological innovation are often analogical to that of similar technologies in the past. In this analysis forecasters identify appropriate analogies and analyze similarities and differences. Generally, having more examples minimizes the probability of selecting false or inappropriate analogies. The result of the Analogy Analysis usually does not predict a certain event but rather future where a range of events is possible. As a result this method is semiquantitative in nature. Precursor Trend Analysis makes a forecast based on the fact of recurrence of a time lag between technolo- 4 gy developments in different fields. In other words, it projects the timeframe future developments in a new technology by correlating them with previous developments in a related leading technology. For example, the first application of technical advances in passenger cars (Anti-lock Brake System, Traction Control, Fourwheel Drive etc.) typically occurs approximately four years after their application in race cars. Similarly, the applications of new technologies in commercial products tend to follow laboratory demonstration by a relatively constant period. This had also been researched by Agarwal and Bayus (2002) who state that there is a general relation between invention, commercialization, firm takeoff and sales takeoff. Using this analysis the future of the lag technology can be determined by analysis of the precursor technology and the past similar technology replacements. Results of this technique are highly quantitative and rely on the selection of the right past similar technology replacements and precursors for its accuracy. Morphological Matrices provide a structured method to detect new product and process possibilities. The first step of the process is to identify the essential functions of a product or process. The second step is determining which of these functions can be realized by cross-checking them with each other in a morphological matrix. The third and final step utilizes this matrix to identify new, reasonable combinations of these means that could result in practical new products or processes. Results of the application of this technique are qualitative in nature. This technique is a helpful tool in assessing non-obvious new opportunities for a company. Feedback models are techniques which allow the modeling on interactions of technical, market, societal, and economic factors as the future unfolds. It utilizes a computer model which mathematically specifies relationships between each of the relevant factors. For example, advances in science may result higher performing technology that could result in increased sales and resulting more funds for science. This technique is highly quantitative, however it is often used to examine qualitative consequences of trends, events, or decisions. Advantages/Disadvantages + The usage of repetitive patterns of the past can give accurate forecasts of the future. + Because the forecast is based on past data it has a high validity. + When the correct analogy is picked, a relative high accurate forecast can be made. Picking the wrong past event for analogy will lead to a false forecast. - It is very difficult to find an analogy which completely fits to the present situation. The quantitative approach to qualitative problems might lead to one sided forecasts. It is complex to take the complexity of markets into account using this technique, which may lead to choosing the wrong analogy. Applicability of Pattern analysts to Evaluate Space Technologies Pattern analyses are reasonably applicable to evaluate Space Technologies. Especially Precursor Analysis especially might be applicable in determining the timeframe of a technology replacement. This trend is probably hard to determine for space technologies due to the relative low frequency in which space technologies are applied. For this same reason Analogy Analysis is not that applicable because of the inaccuracy of past data and the major differences between space technologies. Morphological Matrices might be a good tool for one company to assess new fields of technology development, but it is not applicable as an evaluation or forecasting method. Feedback models are unsuitable for an evaluation method for STs, due to lack of data concerning the relationships between factors. Goal Analysts Goal analysts believe that the future is caused by the beliefs and actions of various individuals, organizations, and institutions. The future is therefore not determined and is susceptible to alteration by one or several of these entities. Because of this, a forecast can be made using the stated and implied goals of the various decision makers and trendsetters. Examples in the space sector would be International Traffic in Arms Regulations (ITAR) and the European space policy. Impact analysis is a brainstorming technique used to assess the impact of an event (innovation) on its surroundings. It is a relatively simple formal method which takes the fact into account that, in a complex social system, trends, innovations, and decisions often have consequences that are neither intended nor foreseen. Because of this, brainstorm techniques are applied to identify the range of impacts an innovation might have. Brainstorm sessions force the participant to follow alternative thought processes and in this way allows for the detection of the unforeseen. Content analysis is a technique for weighting the relative importance of social, political, commercial and economical events through measuring the amount of media attention the event receives. If a trend can be found involving the media attention of column-inches 5 in newspapers, time allocated on television, and, more recently, number of items on the internet. Forecasters can extrapolate this trend and make a forecast of future events. In the area of technological innovations, this technique can be used to project advances in new technologies, through measuring the attention an innovation receives. Although this is a qualitative technique, the results are often displayed in quantitative format. Recently many of these content analysis techniques have begun surfacing. Most notably are Google Trends, Recorded Future, Shaping Tomorrow, Quid and the Web Bot project. Stakeholder analysis is a formal method which measures the influence that individuals and institutions have on future developments. This technique analyzes the importance that each individual or group assign to these issues and the relative influence that they might have on events. The resulting analysis is partially quantitative and is often used to test the validity of forecasts that might be impacted by unexpected opposition or support. Patent analysis can indicate the interest companies and researchers have of a certain technology. It is based on the assumption that an increasing number of filed patents are early indicators for the success of a technology. When analyzing these trends, maybe in combination with an analysis of analogous technologies, a forecast for the future can be made. The results of this analysis are presented in a quantified manner but their use in decision making is based on a qualitative evaluation. Advantages/Disadvantages + The success of an innovation is determined by the market adoption, therefore an extensive evaluation of the market (through stakeholders, content and patent analysis) can be an accurate forecasting method. + Content analysis will potentially prove to be a very accurate technique in the future as this provides a quick real time indication of interest in a certain technology. + Stakeholders analysis goes to the core of decision makers concerning technologies. + Patent analysis is a clear indication of the academic interest in a certain technology. An indication of interest by the development parties alone has no direct influence of the success of a technology. Merely an increase in content concerning the technology is only an indirect indicator for the success of a technology, as it might only be hyped. - A count of the amount of patents might not be indicative for the interest in the technology as some patents are more extensive than others. Applicability of Goal Analyst to Evaluate Space Technologies In general Goal analysts are very applicable to use for forecasting Space Technologies, because they analyze markets instead of focusing only technology characteristics. Impact analysis however, has little predictive value as it merely describes the impact of an innovation if it would be successful. Because of this, it is not usable for predicting Space Technologies. Content analysis might be very helpful in the future as these techniques develop, but are not useable now because of the lack of detailed analysis on the relevant date they give. Stakeholder analysis is an applicable tool, as the stakeholders for technology development in the European space sector, especially governmental entities exert a lot of power on the technology development. The complexity is that it is hard to model entire entities as one consistent stakeholder. Patent analysis is also applicable for predicting the success of innovations as it determines the interest of academia and the amount of research being done on it. As space technologies are mostly developed by academia, this is a very useful tool for evaluating STs. Counter Punchers Counter Punchers believe that forces shaping the future are highly complex and therefore future events are essentially unpredictable. They propose that the best way of handling the future is by identifying a wide range of possible trends and events by monitoring changes in technical and market environments. The way to cope with changes from an unpredictable future is by maintaining a high degree of flexibility in the technology planning process. Scanning, Monitoring, and Tracking techniques are based on the principle that for most new technologies a considerable amount of time is required from invention to innovation. When considering this, an alert organization, through the techniques discussed before, can take advantages of this lag-time. While all techniques involve the scanning of the environment, they do differ in purpose, methodology, and degree of focus. For example the scanning technique involves a broad scan of the environment in order to detect trends. Monitoring follows different trends and is therefore more focused than scanning. Finally, Tracking involves the continuous observation of developments in a specified area (specific technologies, market developments etc.). Results of these techniques 6 can be highly quantitative to basically qualitative, depending on the technique used. The Alternate Scenarios technique is a structured method in which a number of individual forecasts are combined into a series of comprehensive, feasible narratives about how the future might develop. Monte Carlo models are computer models which rely on the view that all future states are fundamentally, probabilistic in nature. In this technique, all of the steps involved in the development of a new technology are identified, and their inter-relationships specified in a mathematical model. Quantitative values are assigned to the probability of occurrence for an event occurring in various different ways and to the length of time it will take each event to occur. The model runs multiple iterations to determine the probability of various results. The results of these models are highly quantitative and are highly applicable used to project technology development times and patterns, to allocate resources and track development of new technologies. Decision analysis techniques involve the detailed analysis of a decision using multiple criteria and weights originating from the operations research field. These methods can use a combination of quantitative and qualitative inputs to deliver a quantitative result. These methods can be used for a variety of decisions but for the field of innovation and technology development they are mostly used as investment decision methods. Examples of the methods include Analytic Hierarchy Process (AHP) and Concept scoring (Urich and Eppinger, 2003) Advantages/Disadvantages + Applicable for long term forecasts. + Scanning, Monitoring, and Tracking allows for the continuously tracking for trends decreasing response time to innovations. + Allows a range of possible scenarios instead of one fixed scenario, which teaches decision makers to deal with the uncertainty. + Decision analysis allows for an evaluation on several criteria and can therefore encompass a range of other methods and measure them together. Mostly applicable to long term forecasts A wide range of trends or alternative scenarios are not applicable to specific cases. Often there is not sufficient data available for Monte Carlo modeling. Applicability of Counter Punchers to Evaluate Space Technologies Out of the counter punchers some techniques are more applicable to forecasting Space Technologies than others. Scanning, Monitoring and Tracking techniques are very applicable indeed to organizations wishing to follow particular technology developments over time. Although this technique might lead to relevant results, the amount of effort it takes to continually track the development of all technologies within the space sector might make it impractical. Therefore an evaluation at a single point in time might be more efficient. The alternative scenarios technique is not applicable to forecasting Space Technologies as an investment decision maker has no use of different scenarios in the case of a technology disruption. More applicable is a single scenario technique, however this has the disadvantage of providing a definitive view on the future and thus closing an investment decision maker’s mind to alternative scenarios. Monte Carlo models are not applicable to forecasting Space Technologies as their data input is impossible to determine. This input data would be data on the long term future of the space sector which is impossible to accurately calculate. Decision analysis techniques are very applicable to evaluation of Space Technologies because their aim can be seen as evaluation for investment decisions. The AHP for example can be used to rate technology concepts on different criteria and weigh them according to expert opinions. The concept scoring method in turn can be used to provide a scoring matrix for ratings on different criteria and compensate them with weights. Intuitors Intuitors are a group of futurists that believe that the future will be realized through a complex mixture of trend, random occurrences and the actions of individuals and institutions. Because of this complexity they believe that no technique can provide an accurate forecast of the future. Therefore they usually rely on the subconscious information processing capability of the human brain and use this to provide useful insights about the future. They do this by feeding the brain with information and allow intuition and experience (tacit knowledge) of experts to make judgments on the likelihood of a future. The Delphi method is a process of gaining consensus from a group of 15-20 experts while maintaining their anonymity to decrease social desirability bias (Hsu & Sandfort, 2007). Social desirability bias means that opinions of individual participants are influenced by group dynamics. It was developed during the 1950s by the RAND Corporation while involved on a U.S. Air Force sponsored project (Ayres, 1969). Wechsler (1978) defines the Delphi method as (translated from 7 the original text in German): “a survey which is steered by a monitor group, comprises several rounds of a group of experts, who are anonymous among each other and for whose subjective-intuitive prognoses a consensus is aimed at. After each survey round, a standard feedback about the statistical group judgment calculated from median and quartiles of single prognoses is given and if possible, the arguments and counterarguments of the extreme answers are fed back...” As this quote explains the Delphi method involves a coordinator which sends the relevant data and criteria to the experts, their comments are shared among the group, discussed, and, hopefully after several rounds, a consensus is reached. Nominal Group Conferencing is a formal technique used to structure expert opinions. The technique resembles mechanisms used in Brainstorming techniques, but requires the active participation of all the participants. Just like in Brainstorming, it focuses participants to use their brains along different paths than usual. Examples of this include: to generate new ideas, to assess the ideas of others, to jointly examine the implications of new ideas, and to formally evaluate a series of options. The technique is often used to project future developments and the results are usually semi-quantitative. Structured and Unstructured Interviews are well known methods of gathering information from experts concerning their thoughts and opinions on how the future will unfold. The structured interviews are methods such as surveys and opinion polls. It entails that the interviewers know beforehand what they would like to know from the interviewees. In contrast, the unstructured interviews the subject is broadly set but the details are less defined. The results of the interview are determined by the interviewee’s answers to each question. Advantages/Disadvantages + Forecasts are made on data from reliable experienced resources (experts). + Forecast results have a high validity + Using multiple rounds (Delphi method) may reduce bias factors. + Uncertainty of the future and multiobjectivity can be incorporated Often the view of experts is one sided which may lead to blind sightedness to unexpected events. Bias factors are large and difficult to mitigate for intuitive forecasts. Requires extensive data gathering Applicability of Intuitors to Evaluate Space Technologies In general, Intuitors are very applicable in forecasting Space Technologies. Especially the Delphi technique is applicable since it involves a group of experts which have to reach a consensus on a question. It depends however on the questions asked if the Delphi method could be useful since Space Technologies are sometimes unexpected and therefore experts have to be questioned according to disruptive indicators. Additionally the Delphi method benefits from anonymity which decreases bias and allows for long distance communication. Especially the latter makes the Delphi technique very applicable for the space sector as experts are often located all over the world. This also makes the nominal group conferencing and interview techniques less applicable as traveling for interviews might be highly time consuming. This problem can be solved through the usage of teleconference techniques although some people argue that this decreases the effectiveness of communication. CONCLUSION This paper analyzed the market dynamics of the space sector and performed an assessment of evaluation methods applicability to evaluating space technologies. Overall qualitative methods seem to be the most applicable for evaluating space technologies. Especially the Delphi method and decision analysis methods seem to be the most applicable. In table 1, the assessment of each group of evaluation methods are summarized in table form. This table contains all discussed methods along with their applicability in forecasting STs, the required information input, the type of forecast and in what way a method is quantitative of qualitative. 8 Table 1: Summary of applicability of space technology evaluation methods Method Applicability Information input Extrapolators Technology Trend Analysis 2 Performance data Fisher-Pry Analysis 2 Adoption data Gompertz Analysis 2 Adoption data Growth Limit Analysis 2 Performance data Learning Curve 1 Cost data/performance data Pattern Analysts Analogy Analysis 2 Technology analogy data Precursor Trend Analysis 2 Adoption times Morphological Matrices 1 Technology functions Feedback Models 1 Environment factors Goal Analysts Impact Analysis 2 Brainstorm session Content Analysis 3 Trends in media attention Stakeholder Analysis 4 Stakeholder information Patent Analysis 4 Patent trends Counter Punchers Scanning, Monitoring, and Tracking techniques 3 Past and real-time technology information Alternate Scenarios 3 Various sources Monte Carlo Models 1 Computer models Decision analysis: Analytic Hierarchy Process 5 Expert opinions Concept scoring 5 Expert opinions Intuitors The Delphi Survey technique 5 Expert opinions Nominal Group Conferencing 3 Expert opinions in brainstorming form Structured and Unstructured Interviews 4 Interviews 1 Not applicable, 2 Slightly applicable, 3 Reasonably applicable, 4 Applicable, 5 Very applicable 9 Type of forecast S-Curve graph S-Curve graph S-Curve graph S-Curve graph Cost graph Adoption pattern Adoption time Morphological matrix Relationship factors Unforeseen events Future interest Influence by stakeholders Future scientific interest Quan/Qual Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Quantitative Qualitative Quantitative Partially quantitative Quantitative Continuous forecast Scenarios Forecast Relationship model AHP model Potential rating Expert ratings Expert ratings Expert ratings Mostly quantitative Partially qualitative Quantitative Mostly qualitative Mostly qualitative Qualitative Qualitative Qualitative REFERENCES Ayres, R.U., (1969) Technological forecasting and long range planning. McGraw-Hill, New York Beer, S. (1981). Brain of the Firm, John Wiley & Sons Carayannis, E.G. and Roy, R.I.S., (2000) Davids vs Goliaths in the small satellite industry: the role of technological innovation dynamics in firm competitiveness. Technovation, vol 20, pp. 287–297 Hsu, C. & Sandfort, B.A. (2007) The Delphi Technique: Making Sense Of Consensus. Practical assessment, research & evaluation, vol 12, no. 10, 1-8 Jackson, M.C. (2000) Systems approaches to management, Kluwer Academic/Plenum Publishers, New York Janssen, M.A. & Jager, W. (2003) Simulating Market Dynamics: Interactions between Consumer Psychology and Social Networks. Artificial Life, Vol. 9, No. 4, 343-356 Kamien, M.I., Schwartz, N.L., (1982) Market structure and innovation, Cambridge university press: New York Nemet, G.F. (2009) Demand-pull, technology-push, and government-led incentives for non-incremental technical change. Research policy 38, 700-709 Pagkratis, S. (2011) Space Policies, Issues and Trends in 2010/2011. Vienna, Austria: ESPI. Smyrlakis, N., Summerer, L., Latronice, L., (2011) Innovation Dynamics in a Monopsony Structure – Insights based on a simplified model of the European space sector. IJSTMI, vol 1, 24-43 Sood, A. & Tellis, G.J. (2005), Technological Evolution and Radical Innovation, Journal of Marketing, Vol. 69, 152–168 Ulrich, K.T. & Eppinger, S.D. (2003) Product design and development. McGraw-Hill, New York. Vanston, J.H., Vanston, L.K., (2004). TESTING THE TEA LEAVES: EVALUATING THE VALIDITY OF FORECASTS, Research Technology Management, September—October, 33-39 Wechsler, W., (1978): Delphi-Methode, Gestaltung und Potential für betriebliche Prognoseprozesse, Schriftenreihe Wirtschaftswissenschaftliche Forschung und Entwicklung, München. 10