Criteria for Selecting a Challenge Exemplar

Broad Coverage.  Despite the flurry of research activities and advances, most agent-oriented methodologies are still in their early stages of development. There is considerable diversity in concepts (e.g., no consensus on the notion of agent), approaches (formal, informal, hybrids), origins (extensions to object-orientation, knowledge acquisition methodologies, etc.), lifecycle coverage (requirements to implementation), or in even overall objectives.  While a standardized exemplar can be a stimulus for movement towards convergence, the exemplar itself should not lead the convergence in any particular direction.  As much as possible, it should allow any approach to be presented and explored to the fullest extent. An exemplar that is problem-focused would avoid any bias.  The problem should be described in terms of the application domain as much as possible.  Where references to technology systems are made, they should be construed as suggestive rather than definitive.  Mention of methodological concepts and steps should be avoided as much as possible.

Showcasing Agent Orientation.  The exemplar should allow the distinctiveness of agent-oriented approaches to shine through.  Agent-oriented systems can offer greater flexibility, robustness, adaptiveness, and many new functional capabilities. Agent-oriented concepts during the development process can offer social organizational views and analyses of relationships among software agents as well as among human agents. Agent abstractions allow higher level analysis and encapsulation of issues and greater independence from implementational issues.

Overcoming legacy methodology concepts. Conventional development methodologies are typically organized into stages such as requirements, design, implementation, testing, and so on. While the broad categories may still apply, their meanings and specifics will likely undergo substantial revisions under agent orientation.  What is a requirements specification if agents have autonomy?  What is the proper role of architectural design if a multi-agent system is dynamically configured and embedded in equally dynamic human social organizations?  Where is the boundary between requirements and design if agent behaviours are expressed in terms of goals and beliefs at both levels?  What decisions are made at run-time by intelligent agents as opposed to at design-time by human designers?  The exemplar should provide opportunities to explore these questions without being confined by traditional methodological concepts and terminology.

Real-world Richness, Extensible, Subset-able.  The exemplar should reflect real-world concerns.  For pedagogical purposes, examples are often simplified and stripped-down versions of real-world settings.  As agent-oriented approaches could depart considerably from traditional approaches, the use of simplified examples could potentially introduce assumptions and biases specific to some particular agent-oriented approach (or class of approaches).  Since agent-oriented approaches are supposed to be able to handle richer and more demanding types of environments, the exemplar should offer sufficient richness for these capabilities to be demonstrated.  For this reason, the exemplar also needs to be open-ended to allow further richness to be added. An open-ended real-world example can stimulate and inspire additional capabilities for emerging methodologies. On the other hand, an open-ended exemplar can grow unnecessarily and become unwieldy.  Efforts will be needed by the community to refrain from adding extensions whose methodological issues are already well covered.  For some methodological issues or activities, the exemplar should allow subsets to be defined for more focused examination and discussion.

A Representative Application Domain. To be concrete, the exemplar will necessarily be grounded in some application domain.  It will not be easy to find a domain and application setting that can encompass all the relevant issues in a comprehensive way.  Further, it should be easily understandable to non-domain-specialists. The general reader should be able to relate to the domain issues and ideally be able to imagine oneself as a stakeholder in the domain. Selecting a generally accessible domain could also leave room for research teams to do further elicitation with actual subjects in the domain. While grounded in a particular domain, the issues and scenarios should be easily imagined in other domains by analogy or extrapolation. 

Allowing comparison with Non Agent-oriented approaches.  For agent orientation to be widely adopted, its strengths in relation to more conventional approaches must be made easy to comprehend.  The exemplar can play an important part here by facilitating and encouraging comparisons with non agent-oriented approaches. If the exemplar is presented only in terms of application domain issues, it would be open to any methodological approach.  If one can work out the exemplar using different approaches side-by-side, one will be able to appreciate exactly where the benefits arise, what issues are addressable, supported by techniques and tools, etc. The comparison will therefore be at a fine-grained level. The decision to use software agent technologies, for example, could come about as a result of applying a methodology, rather than as an a priori decision before a project starts.  A methodology could also lead to the choice of a mix of agent and non-agent software technologies, or a staged transition from one set of technologies to another.  Even among agent-based technologies, there should be guidance from the methodologies on the choices and mixes of technology types, representing different kinds and degrees of mobility, intelligence, autonomy, coordination and communication capabilities, etc. 

Neutrality With Respect To Agent Orientation. To succeed in the above objective, it should be clear that the exemplar is not biased in favor of or against any approach. This can be achieved by staying close to the application problem domain. While the exemplar should provide plenty of opportunities to showcase the benefits of agent orientation, those benefits (and limitations) should arise from the application of a methodology, and not inherent in the exemplar itself.

Prominence of Generic Software Engineering Issues and Criteria. While highlighting the variations among agent-oriented approaches, and their contrasts with non-agent-oriented approaches, one must keep in mind that all of these approaches must face the same generic issues that are faced by information systems development and software engineering in general.  These should be prominent in the exemplar. For example, performance, costs, scalability, reliability, time-to-market, manageability of development project and team (including management of schedules, risks and priorities, manpower, knowledge and skills, training, learning curves, etc.), quality, maintainability, traceability, evolvability, and reusability of code and other project artefacts – all of these will continue to be central issues. Methodologies – agent-oriented or otherwise – can be differentiated on how (and how well) they address these issues.

Including Non-Technical Issues. Many of the issues crucial to the success of a system or project are non-technical, and they are intertwined with technical issues.  The exemplar should provide sufficient complexity for these to be identified and explored.  Real-life systems typically have many stakeholders with conflicting interests.  There are political, economic, socio-cultural, legal, and regulatory issues.  Commercial systems have competitors, as well as other players upstream and downstream along value chains. They operate under different business models, and go through occasional (and increasingly frequent) changes and upheavals. Consortiums, coalitions, and associations (within, among, or across industries, professions, consumer groups, government agencies, and international bodies etc.) might establish guidelines, set standards, and promote or resist change. Dominant players could set de facto standards. Once dominant players can go out of business,  start-ups can become dominant players.  Non-technical issues are also inherent within project organizations – differences and changes in organizational forms and styles, individual capabilities and limitations, local versus distributed teams, budget and schedule changes, team turnover and reorganizations, mergers and acquisitions, etc. – these are part and parcel of any real project.  An exemplar can help illuminate how well the different methodologies perform under these complex, but real-life conditions.