Performance Evaluation in the Brazilian Public Sector

The Brazilian Public Sector is being pressured by society to provide more and better services to citizens. Thus, this research is motivated by the need to provide management tools to improve the performance of public administration for better use of public resources. The research explores a constructivist methodology of performance evaluation as a tool for decision aiding in a Brazilian public organization. It highlights propositions identified in qualified literature to justify the use of constructivist approach in public management, as the need manager actively participate in the model of construction in order to expand his/her knowledge about the context and the need to recognize the uniqueness of the resources and moment instead of seeking generic models of evaluation. The development of the model itself, provided a detailed overview of the aspects understood as needed and sufficient by the decision-maker. It was able to disclosure the uniqueness of the context, the objectives of the sector and the construction of indicators for the performance evaluation of aspects understood as important to the public manager. It was observed the theoretical contributions to the area of public management knowledge, especially the key role of public manager to build evaluation models, recognition of limited rationality in decision making and uniqueness as a major element in the decision within the public administration.


Introduction
Especially in the last two decades, the Brazilian public administration is being pressured by society to provide immediate solutions to pressing problems, and to provide more and better services to Brazilian citizens (Motta 2013).Thus, it has been expected of Public Administration a forward-looking role for the citizen as a customer and focus on public management by results (Poister and Streib 1999).
For those reasons, the public sector is increasingly adopting a performance management system (Latham et al. 2008;Astrini 2014;Agostino and Arnaboldi 2015) and the evaluation of public management has been understood as a necessary tool for the success of administrative reform in the different spheres of government (Catelli and Santos 2004).It is noticed that the desired quality for the public service could be achieved through greater involvement of citizens with public institutions, as well as monitoring the performance of governments and public authorities (Moisés 2010).
Therefore, this research is motivated by the need to provide management tools to improve the performance of Public Administration to closer monitoring and better use of public resources.In this sense, researchers have noted the need to address the lack of knowledge of the manager about the context, the creation of innovative strategic actions and prioritize them according to what is most relevant to the decision maker for their management.(R. T. d.O. Lacerda et al. 2014a) For this propose, performance evaluation was adopted as a tool for decision aiding in public administration, where there is need to expand the understanding of the public manager about his decision context and the consequences of their decisions on their goals, contemplating generation of improvement actions in a continuous management process.(Fernandez and Rainey 2006;Janssen et al. 2004;Rainey et al. 1976).
The question of this research is "How to build a model to support management decisions in the public sector from a constructivist approach?".
With intent answer this question, this study is aimed to develop a performance evaluation model to aid management decisions of a Brazilian Court of Justice.Such evaluation model requires the use of cognitive methods to increase the understanding of the manager about your goals from their values and preferences.

Theoretical
This section talks about the performance evaluation in the public sector and presents some scientific challenges of this area of knowledge.

Performance Evaluation as a Decision Aiding Tool in Public Administration
The Performance Evaluation has been understood as a tool available to managers for the continuous improvement of organizations.It is believed that the current valuation of the instrument arises from a fundamental change: the transcendence of the costs of phase to values phase in organizations (Neely 1999).However, the corporate world uses methods to evaluate their organizations that are not appropriate for the public sector (Behn 2003;Fryer et al. 2009).But it can not rule out the use of performance evaluation within public organizations, especially when the evaluation criteria are related to the objectives of public officials and their decision-making contexts (Behn 2003).
The decision aiding is regarded as a science that seeks to develop a network of concepts and procedures that is able to structure a body of knowledge that can act as keys to guide decision making (Roy 1993).With respect to the results produced with the use of decision support methods, researchers can follow some approaches.In the normative approach, guided by rationalism, the researcher takes the existing literature on the subject and/or expert knowledge as the main sources to define the performance evaluation criteria.The descriptive approach also based in rationalism searches the criteria that influence the desired result by the decision-maker, in the previous experiences The above rationalistic approaches are associated with the decision-making process while the two approaches are presented herein are affiliated to the process of decision aiding.For prescriptive approach, the concerns underlying the proposed evaluation model emanating from the decision-maker over a learning process that focuses on the facilitator.For this approach the central source of data is the decision-maker, however the knowledge is built on the facilitator.Finally, there is the constructivist approach that understands that the constituent criteria of the evaluation model, its scales and the integration of these must emerge from the decision-maker's values and preferences (L.Ensslin et al. 2012).
Within a constructivist vision of the world, the concept of performance evaluation is based on building knowledge in decision-maker, regarding the specific context that proposes to evaluate.Within this perspective, two important emerging issues related to performance measurement in the public sector.
The first results from the role of public manager who is assigned to the function by his superiors and the latter wish that this organization is conducted from the perception of the reality of designated manager within the legal and ethical constraints that entails his/her function.The second question emerges from the uniqueness of the decision-making contexts in public organizations.Within these analysis lenses, it is the following subsections discuss about how the scientific literature is applying the performance evaluation as tool in the public administration.Broadbent and Laughlin (2009) based on conceptual models developed in the literature to suggest the advances "New Public Management," in United Kingdom.Yetano (2013) uses the scientific literature to propose an array aimed at institutionalizing of performance evaluation tool in organizations in local governments.The analysis of these models, it is concluded that there is little or no participation of the manager in the Performance evaluation process, as in the case of Gao (2009) which describes the performance evaluation process through analysis of official documents in China.It is clear from these studies that the role of public manager corresponds to observe the imposed national rules or following generic evaluation models, which may bring risks to operationalization of management by results in the public sector.Moura andSá andKanji (2003 and2007), mitigates this risk by proposing the application of a KBEMS assessment model within the Portuguese Municipal Governments and incorporating the perceptions of stakeholders, understood as the driving force improving the quality and organizational excellence.Van Aken et al (2005) even if external factors are used as evaluation criteria must have an involvement of managers so that the model becomes legitimate to use in a given context.

Role of the Public Manager in Performance Evaluation Process
Reinforcing the perception of managers, Moynihan (2006) estimates that the doctrine of "Managing for results" was partially adopted in the US, but has been neglected a portion of doctrine related to the increase of managerial authority.The author believes that changes concerning the definition of clear objectives and managers' accountability by the resources used and the results produced would be needed.
In this line of thought, Behn (2003) points out that what is measured by the organizations is executed, but usually what is measured is not what people want it to do.The author believes that public managers should define the goals that they desire.After this fact, public managers would be able to select criteria to assist them in plans proposition.
Studies of Linna et al (2010) have researched perceptions of the Finnish municipal authorities as well as literature related to the topic of measuring productivity in the public sector.The authors contemplated in their studies the interests of managers in identifying the criteria to be assessed.Each of the above research carries some working assumptions that influences the performance evaluation is applied and consequently as the public manager can make use of this knowledge area to assist in their administrative activities (Tsoukiàs 2008).
In a realistic approach performance evaluation, the public manager uses mathematical formulas, or economic statistics to find optimal solutions, and these rationally accepted as best or proven successful in past or successful experiences in different contexts from those that the manager is situated (Tsoukiàs 2008).This approach is recommended for decision-making contexts where there is variable to be optimized and there is unequivocal causal relationships between the conditions and the results.That is, the decision-maker has absolute knowledge of the situation and can infer the solution that stands out over the others in all aspects from the perspective of all stakeholders involved (Roy 1993).
Another way to understand the public manager in the use of performance evaluation in decision process is to resort to prescriptive approaches, where the decision-maker uses a facilitator to organize the objectives pursued by the public manager in a structure cognitively accepted by the decision-maker and the other actors of the situation.Thus, the decision-maker can verbalize his/her problem and each constituent part, but can not structure a model able to clearly identify the best alternative to a given situation, since no alternative surpasses all other alternatives in every respect.In prescriptive approach, the facilitator makes use of interviews and schemes for him/her to understand what the decision-maker want and proposes solutions based on this speech.The search is for consistency between discourse and decision (Tsoukiàs 2008).
In the constructivist approach to performance evaluation, there is a recognition that the decision-maker needs expand his/her knowledge to explicit their goals and better alternatives (R. T. d.O. Lacerda et al. 2014b).Thus, the decision-maker uses a facilitator to make use of cognitive methods to build an understanding in decision-making about their own context and, after generating this understanding, propose appropriate actions.From this perspective, inconsistencies between speech and actions of public managers are not seen as problems but opportunities for discussion with the other actors and generate understanding of what would be the best targets for a given situation (Keeney 2009).Thus, the decision-maker wants to understand the problem and actively participate in its structure and not just choose among the possible alternatives.
Proposition 1: The public manager must actively participate in the construction of performance evaluation model to expand their understanding of the consequences of their decisions and giving legitimacy to the reasons why he/she was designated.

Uniqueness of the Decision Context in Public Management
The study Broadbent and Laughlin (2009) is based on the scientific literature.Thus, it is assumed a generic model applicable in different contexts from a convergence of ideas prominent authors in the area.Another way to meet multiple points of view in performance evaluation is exemplified by by Moura andSá andKanji (2003 and2007), which proposes the establishment of a model from existing excellence models.
With another look, Moynihan (2006) argues that the performance evaluation criteria and targets set for public managers are assessed on a case to case than it is concluded that the research results in the singular analysis of the context for the development of a performance evaluation model.Behn (2003) states the need to be defined objectives to be achieved in organizations so that the criteria to be measured are defined later, in order to allow the use of information to support decision of managers.In this sense, the author recognizes the uniqueness of the decision-making process in the public sector.Linna et al (2010), proposes a form of action which shows the uniqueness of the context where the actors are identified and interviewed by the researchers.Yetano (2013) observed the need for some degree of uniqueness to involve senior managers and other employees to assist in evaluating the generic model.In the same vein, Van Aken et al (2005) propose a model made up of diverse experiences within the excellent performance of public organizations, but also through the customization of the scoring method and checking the validity of the methodology proposed by the stakeholders specific context.
In a descriptive way, Gao (2009) recognizes the uniqueness in making the valuation model from specific documents the context and states make use of such a model only for the context in which it was formulated.
Analyzing qualified literature, it can observe two important ways to understand organizations and how science helps to public management.In a preliminary way, the natural sciences dramatically influenced the behavior in public organizations using scientific methods that make use of heterogeneous units samples to result in general solutions, commonly accepted as appropriate for any similar context of the sample.Within this way of thinking, organizations are the same context and that there is a perfect resource mobility (Barney 2001;Teece 2007).
Within a constructivist view, generalizations of public contexts for use in specific situations should be used with caution and always with due regard to the specificities of each context, especially the objectives pursued at any given time, resources and awareness of unintended consequences in a certain action (R. T. O. Lacerda et al. 2011).
In short, what worked in one situation will not necessarily have benefits in other contexts (Eisenhardt and Martin 2000).It is sometimes even counterproductive playback behavior in the same organizations that similar, and this is the primary role of public manager who was assigned: to analyze what happens in the external and internal environment of his/her organization so that it can take decisions consistent with the reasons that the public manager was appointed.
Proposition 2: Performance evaluation model must emerge from the specific needs of the context to be evaluated.
Proposition 3: The performance evaluation process should provide a way to elicit the evaluation criteria from the uniqueness of each context.

Methodology
The presentation of the research methodology employed in this work will be developed into two sections, called 3.1 Methodological framework; and 3.2 Intervention instrument adopted.

Methodological Framework
As to the purpose of the research, this research is exploratory and descriptive, and the articles nature of the practical role.The research logic falls as inductive in the structuring of the evaluation model and deductive regarding the phase of the recommendations.
Data collection was conducted with both primary and secondary data and the problem of approach was carried out qualitative and quantitative manner, and the results can be characterized as an applied research.The used technical procedures are based on case study.

Intervention Instrument Adopted
The Multicriteria Decision Aiding Constructivist -MCDA-C was the intervention instrument chosen to develop the model, and even to carry out the interpretation of the data collected.
The choice for the use of MCDA-C in this context results from the realization that the social processes involving people, values and perceptions, that is, are considered complex situations by straddle multiple and conflicting criteria (Roy and Vanderpooten, 1996) (S.R. Ensslin et al. 2013).
The referring methodology assumes that the activity of decision process is developed in three phases: The first phase is for the understanding of the problem and the physical context researched, through the identification of the problem, the actors and the generation of knowledge in decision-making, represented by a hierarchical value structure (L.Ensslin et al. 2012).In the second phase, using cardinal scales, we seek to evaluate the current performance, according to the preference of the decision-maker between the impact levels (de Azevedo et al. 2012).Finally, in the third phase are proposed performance improvement actions in relation to current performance (Keeney 2009).

Research Results
In this section, we seek to present the results obtained by applying the MCDA-C to develop a performance evaluation model in a department of a Brazilian Court of Justice.
The Santa Catarina Court of Justice is the holder of agency jurisdiction in that State of Brazil and responsible for the infrastructure of other regions installed in its scope of action, contemplating the provision of physical infrastructure, qualified staff, equipment, furniture, software, services and hiring other mechanisms needed service to society.It is composed of judges and administrative units responsible for carrying out the services described above.The Judicial Academy, context chosen for the development of this research, is the sector responsible for training and development of knowledge, skills and attitudes of judges, civil servants and employees of the judiciary.

Structuring Phase
For the first phase, called Structuring Phase, meetings were held which were attended in order to generate knowledge in decision-maker by understanding the problem, represented by a hierarchical value structure.

Context
The contextualization step involved the identification of the actors, in preparing the presentation of the problem and justification of its relevance.
The identification of the actors for the researched context, the decision-maker was considered the Executive Secretary of the Judicial Academy; as actors, their advisors; the facilitators were the authors of this research; acted were considered, judges, servants and employees of organization.
Regarding the presentation of the problem and justification of the relevance of the study it was highlighted that the managers of the Judicial Academy verified the need for a tool for decision aiding in the creation and prioritization of actions that impact positively on the performance of the area and organization.The scarcity of resources and increasing social pressure to provide more and better services were reported with justification for the job.

Hierarchical Structure of Value
In this step, based on open interviews, the Primary Elements of Evaluation (PEE) were compiled, corresponding to concerns that the decision-maker sees in their routine activities.Sixty PEE have been identified.
After identifying the PEE, it has been proposed to expand the understanding of these elements by constructing concepts.The concepts establish the direction of preference of decision-maker and demonstrates the expected outcome as well as the consequence to minimize called psychological opposite pole (de Azevedo et al. 2012).Some examples of concepts can be cited:  "57 Disseminate knowledge with use of educational technologies rather than to Limit participation of employees"  "28 To develop the knowledge, skills and attitudes of judges and servers who enter the career rather than to Prejudice the adjudication and people management"  "53 Evolving work processes according to the current situation rather than to ignore the innovations and brought understandings with time " After this stage, the facilitators together with the decision-maker, identified strategic objectives of the sector in which the research was conducted, based on global institutional understanding built at the meetings and in the developed concepts.These objectives were organized into a hierarchical structure.
On this occasion, the goals were tested regarding the necessity and sufficiency of the developed concepts.With regard to necessity, the decision-maker related each concept to a goal (need).With respect to fill, no concept ran out related strategic objective.

Construction of the Performance Indicators
The development of cognitive maps for each strategic objectives enabled the identification of evaluation criteria related to the decision-maker's interests.Seeking to better elucidate this activity, shows the Figure 1, which shows the cognitive map of means-ends relations to the strategic objective "Organizational management" -Part 1.

Evalua
The evalua scales.

Anal
To perform attractiven the model mutually in

Cons
To perform constructio For this, i attractiven

Identification of Compensation Rates
The integration of performance evaluation model is made from the identification of the contribution of each criterion to the performance of superior point of view.This is performed by changing the lower reference level of normality to upper level of each criteria.The MCDA-C proposes that the assessment of the contribution of a performance indicator for the model is performed with the definition of compensation rates.
The process was conducted the same presented by R. T. d.O. Lacerda et al. (2014b), which consists of the comparison of the reference levels for criterion within each set of indicators of the model in the hierarchical structure.The facilitator presents a hypothetical situation where all the criteria are on the lower level of normality and calls for the decision-maker to choose which criterion considered more attractive to rise to the upper level of normality.The crucial difference of this method is the need to use the reference level for each criterion to establish the preferences.MACBETH method was used again to transform the ordering of preferences into cardinal scales, thus generating the model compensation rates.

Identification of the Impact Profile of Alternative
After the conversion of ordinal scales into cardinal ones and the definition of compensation rates, it has become feasible to assess the overall performance of the studied context.This evaluation was performed with the determination of the status quo by collecting information related to the measurement of the performance indicators, through which it sought the performance objectively achieved by the organization for each target criterion concern by the decision-maker.
Thus, the overall cardinal assessment of the current context was valued at 15 negative points, i.e. an overall assessment below normal.Two strategic objectives were within the normal range: Education with 5 points and Institutional Relations with 3 points rounded.The other objectives were to compromising performance, with the objective Internal Management evaluated to -19 points, objective Organizational Innovation with -1 point and objective Research with -2 points.
With this expansion of understanding provided for the construction of the compensation rates, it can sort criteria according to their overall contribution, using the following equation:

  
Where: Increase of performance in criterion i to pass the current level (SQ) to the upper level (Good) of normality.
Global measurement when the performance of criterion i is on the upper level of normality and the other criteria at the current level.
Global measurement when the performance criteria of all current model current level.
 i = 1, n Criteria of the model  n = Number of Criteria of the model Table 1 shows the five indicators with greater overall potential increase to improve performance for upper reference level, for what they were considered most priority indicators of context, and they summed provides an opportunity an improvement of 50 overall points rounded.

Recommendations Phase
For the MCDA-C methodology, Recommendations phase intends serve to support the decision-maker to propose actions and evaluate the consequences of implementing these actions in light of the strategic objectives (S.R. Ensslin et al. 2013).
Based on the criteria with greater global impact potential, as shown in the previous section, the manager can focus on improvement actions, using more effectively the public resources available.The priority actions are explained in Table 1.Facilitators stressed that the decision-maker other indicators could have improvement actions, but it is necessary to focus on projects that will be controlled more effectively and they must have a target performance.Education -E04 -% of the courses were evaluated by participants with higher index of 4 (on a scale 1-5), in all aspects, in the last 6 months.54 9.2 This indicator will change the mentality evaluation of area courses.Today, the average is applied and there is no a structured process of monitoring the improvements.Barriers to excellence must be mapped and meetings with the manager accompany each aspect that is below normal.

Analysis of Results and Theoretical Contribution to the Field
After describing the case study that presents a methodology of a performance evaluation in a public organization under the constructivist worldview, it can analyze the results in light of the above in the theoretical section.
Regarding Proposition 1, the MCDA-C methodology starts from of the recognition of limited rationality present in public managers when their decisions.Thus, the methodology presented requires the active participation of the public manager so that it can clearly display their perceptions regarding the objectives to be pursued by the organization and, in the process, be able to expand their understanding of the consequences of their actions these goals.In this line of thinking, the public manager legitimizes the model and especially its legitimate function for which it was designated.In the case study, these questions can be mainly observed in the elicitation of primary evaluation elements, building concepts and hierarchical value structure.
Regarding Proposition 2, the case study includes some elements that are peculiar to the assessed context and it becomes difficult reproduction in other contexts, even if similar.This is mainly due to path dependence concerning the historical and cultural factors associated with time and people during the performance evaluation.This can be exemplified by the G01 indicator -% of critical work products of unit has formally documented processes -in which there was a great reflection and discussion with the manager, collaborators and facilitators to take into account the current maturity of the organization with respect to process management.In the first instance, this criterion could have been assessed by the maturity process management, but the manager looked concerned that the organization has a strategic plan to put in place the management processes of corporate form and understood the department under his management would still need to evolve a lot in this matter.Thus, after many arguments exposed and discussed together with the cognitive map, the decision-maker understood that if there was an attention to at least document the critical processes, would be a major breakthrough that criterion.The maturity of the team at that time was a specific asset that context it was taken into account when building the model.
In reference to Proposition 3, the case study presents the activities conducted by the facilitators for the evaluation criteria emerge from the values and preferences of the manager in question, respecting the peculiarities of context, such as the wishes of senior management of corporation, budgetary constraints, culture and maturity of the team.The disclosure of this theoretical proposition can be observed in the elicitation especially when organizing the concepts of the construction of cognitive maps and construction of the measurement scales for evaluating a line of reasoning.

Concluding Remarks
This research presented a case study in a public sector organization, to demonstrate the usefulness of developing a performance evaluation model through a constructivist methodology.
This methodology enabled the development model to identify, measure and integrate the aspects considered relevant in the perception of decision-making, enabling a comprehensive assessment of the context analyzed.
The model of development itself, performed in a participatory manner allowed to build knowledge in the decision-maker, providing a detailed overview of the aspects understood as needed and sufficient by the decision-maker.It was able to disclosure the objectives of the sector and the construction of indicators for the performance evaluation of aspects understood as important to the public manager in question.
It was also observed the theoretical contributions to this area of public management knowledge, especially the key role of public manager to build evaluation models, recognition of limited rationality in decision making and uniqueness as a major element in the decision within the public administration.
For future research, showed up the opportunity to analyze the sector's performance in the face of the performance of other similar contexts, found in the Brazilian and world public to verify the application of the methodology for public contexts and demonstrate its viability as a public management tool.

Figur
Figur intensity of the attractiveness difference between the ordinal scale levels for all performance indicators, in the perception of the decision-maker.
course, students should be evaluated about the change of attitude in its sector, making this a factor of applicability of the knowledge.The courses with little application must be revised or no longer offered.Internal Management -G01 -% of critical work products of AJ has formally documented processes (process flow, an institutional program of business process management that are aligned with the strategic planning of the organization.The proposition is to set the area as an benchmarking of this program and increase the maturity of employees in process management theme.registered should be evaluated not only by meeting their research objectives, but also of its publications.Thus, an awareness of the importance of this indicator should be made and the obstacles to publication should be managed.system is being selected by the manager and the evaluation model constructed by this research will help him in this product selection.However, the manager has already ordered collaborative spreadsheet with the results of the indicators are published on the network to a spread of the expected results.

Table 1 .
Prioritized criteria to propose improvement actions