Previous Page

Next Page

Table of Contents

6. Findings by SIDPD Objectives -Round 1 Outcome Findings

The outcome evaluation examined three main areas: Relevance, Achievement of Results Against the Objectives and Effectiveness. Each of these areas, and their sub-sections, will be examined below.

6.1 Ongoing Relevance of SIDPD Objectives

Three main issues were examined under the subheading of the on-going relevance of SIDPD projects: (1) the extent to which departments and voluntary organizations articulated an ongoing need for projects-based programming to support the work of SIDPD; (2) evidence that SIDPD projects addressed departmental policy priorities; and (3) evidence that the projects addressed voluntary sector policy priorities.

Not surprisingly there was some variation in responses both between and among the federal government and the voluntary sector. There appeared to be unanimous support from within the voluntary sector concerning the continued need for such programming. Likewise, many departmental representatives indicated that the continued engagement of the voluntary sector would require an infusion of resources into their policy branches. Continued engagement with the voluntary sector in such an active fashion was reflected in only a very few departmental responses.

The majority of voluntary sector respondents also indicated that the projects had met both their own, and their federal department's priorities, with a few indicating that over the course of the project the policy area became a priority for either the department, the voluntary organization, or both. This issue of addressing policy priorities is somewhat difficult to determine with real accuracy given that for some projects, voluntary organization representatives described the project process as highly department-driven, particularly at the proposal-writing stage in Round 1, whereas in Round 2 it was left up to voluntary organizations to choose from identified departmental policy priorities and to shape their project proposals accordingly. In a small number of cases, the SIDPD projects provided an opportunity for voluntary sector organizations and federal departments to jointly identify some policy priorities. Overall, there was general consensus that the aims of the SIDPD continued to be relevant, with the understanding that there was still much work to be done in order to achieve a shared understanding of some of the objectives, and the associated responsibilities for each sector.

6.1.1 Emerging Policy Issues

The identification of emerging policy issues is seen as one potential outcome of greater voluntary sector input into the federal policy development process. The federal government and voluntary sectors execute their roles in different ways. It is often because of grass roots, front-line involvement with the daily challenges of Canadian citizens that voluntary organizations are able to articulate emerging policy issues. By contrast, the federal government does not generally have as direct a service-delivery role, and departmental policy agendas are often constrained by government mandates and priorities. In Round 1 the departments largely identified the policy issues to be addressed by the SIDPD projects and solicited involvement from voluntary organizations that were also identified by the departments.

Due to these and other factors, it is not surprising that Round 1 did not result in the identification of any emerging policy issues. However, several projects dealt with current policy issues from very new perspectives; prostitution and the drug trade and native battered women are two projects that took fresh approaches to the policy work.

6.2 Achievement of Results Against SIDPD Objectives

6.2.1 NGO Capacity Development

A key evaluation question concerns the extent to which voluntary organizations involved in the SIDPD projects increased their policy development capacity. This increased capacity could have taken many different forms including the ability to assess departmental policy priorities and to provide input into various stages of departmental policy development processes. According to the analysis of Round 1 project final reports, 62% of projects indicated that their projects had resulted in increased policy development capacity within their organization. Evidence from interviews with project proponents, shows that fully 100% reported that their SIDPD project has increased their organization's capacity to contribute to federal policy development. The following quotes are examples from these interviews:

One project in particular concerning immigration serves as an illustration of the positive impact of SIDPD. Prior to the SIDPD project, the immigrant settlement organizations were highly fractured and disparate in their capacities and their experience in working together, not to mention in terms of their ability to work with the federal government. "The SIDPD project has resulted in a much more focused and united sector that has increased our capacity for collective and collaborative action".

Another project working with street prostitutes was equally positive about what it had achieved:

The substantial increase in voluntary sector policy capacity was underscored at the December 2003 focus group; where there was considerable participant agreement concerning the amount of policy capacity development achieved through the SIDPD projects. Research capacity was the one area of policy development seen as requiring particular and ongoing attention. Because of the high level of expertise needed to undertake rigorous, valuable research, many voluntary organizations simply did not include this as one of their activities. Focus group participants agreed that nevertheless, the close link between good research and policy development requires that voluntary organizations serious about policy development also need to increase their research capacity.

Yet, some respondents did have reservations concerning their partner federal departments' willingness to use the increased capacity within the voluntary sector. For example, one respondent stated "To some extent, the project has assisted [our organization] in improving its effectiveness to have input into policy development. However, for the most part, our efforts to provide comment and expertise are ignored by the department". This tune was a minor theme that ran through approximately one third of the interviews. Despite positive responses to specific outcomes, a number of respondents had reservations concerning the sustainability of their newfound relationship with their federal partners, and questioned the political commitment to furthering the VSI ideals for true collaborative partnering which emerged from the SIDPD projects.

6.2.2 Government Capacity Development

The evaluation also examined the issue of whether the SIDPD projects contributed to the capacity of federal departments to engage voluntary organizations in policy development, and to then use that input in a meaningful way for departmental policy work. When the SIDPD Initiative was first conceived there was a great emphasis on the requirement to improve the policy development capacity of the voluntary sector. Yet, from the perspective of many voluntary sector organizations, there is considerable variation among federal departments in their ability and willingness to not only engage the voluntary sector effectively, but also to then use the information and knowledge gained from that engagement process. Indeed, from the perspective of many voluntary sector focus group and interview participants, there was an equivalent need for capacity development within the federal government, and there was a perceived requirement for SIDPD to address the need within both the government and voluntary sectors.

Based on interviews with voluntary and government sector representatives, there are mixed reports concerning whether the projects contributed to the federal government departments' capacity to engage knowledgeable voluntary organizations in policy development. On the one hand, some respondents from the voluntary sector indicated that yes their projects did achieve this goal, and that "they now have more respect for our opinions and provide us with opportunities to speak". On the other hand, others indicated that one partner department was already good at engaging the voluntary sector, while another partner department has been 'shown the way' through the SIDPD project and that it is now up to them to decide to walk the new path or not. In fact, there was considerable ambiguity in the responses to this question. Some organizational representatives suggested that federal capacity was enhanced, but that this "varies region to region and depends on how the department works". This difference highlights the fact that departmental culture has a large impact on how 'engaged' departments are willing to be with the voluntary sector. In a small number of cases, respondents were decidedly negative in their responses. They indicated that while the potential was there in the department to increase their capacity to engage with and use input from the voluntary sector, it "wasn't acted upon". Further, one voluntary organization representative stated that

6.2.3 Influence on Policy Development Process in the Federal Departments

Enhancement of policy capacity — a key SIDPD objective — was also measured in terms of the extent to which projects were able to input into and to thereby influence the policy development process with their partner federal departments, or in other jurisdictions. According to project final reports, six in ten projects (62%) indicated that they had input as a result of their SIDPD project, while close to one third (29%) reported that through this input they had influenced the policy process within their federal department. Most importantly, more than half indicated that it was too early to tell, which is not surprising given the short project duration. These somewhat different results are suggestive of the fine line between having an opportunity for input, and the extent to which that input is incorporated into the policy process. While the vast majority of voluntary sector respondents have rated their opportunities for input favorably, only time will tell whether this input has resulted in real voluntary sector influence on the federal policy process.

Of those projects that did influence policy development, this influence occurred at different levels of government. A small number of projects became highly engaged at the municipal level (such as one project focused on building the policy capacity of the youth environmental community). Many projects had an effect at the regional level, with about half affecting policy at the federal level. These results are seen as highly positive given the short project duration. Indeed, a couple of projects have far exceeded the expectations concerning policy development. One Round 2 project is poised to have an impact on legislation currently being developed in the area of non-profit corporations that was before Parliament at the time of interview. According to the federal government respondent, "this work was directly informed by the department's SIDPD projects and the involvement of its partners. This step was a major development and will have a huge impact on the sector (liability issues etc.)". Two other Round 1 projects — one project concerning Creating Options Aimed at Reducing Sexual Exploitation and another concerning Housing Issues for Veterans and Seniors — have each resulted in the development of highly successful programs that have been recommended as models for other regions.

Of note, a limited number of senior government and voluntary sector people said that they do see a trend toward a change in the way some government departments conduct the policy process. Interviewees indicated a new openness to dialogue and a willingness to work more collaboratively. They further indicated that they attributed that attitude to the work of the VSI in general and the roll out of the Accord and Codes, and not just to SIDPD projects.

6.3 Effectiveness

In addition to its two major objectives SIDPD had several another objectives that were used as indicators of how effective SIDPD was overall.

6.3.1 Innovation

The first point to be made with respect to innovation, in the context of SIDPD projects, is that innovation is frequently in the eye of the beholder. There were virtually no representatives from the voluntary sector who said in their interviews that their projects were not innovative. And indeed, many of them were innovative, particularly when one considers that, in some sectors; little effort had ever been made previously at developing policy capacity. For some sectors, merely bringing a group of concerned individuals together to discuss what needed to be done could be considered as innovative.

An analysis of the SIDPD database created for this evaluation indicates that almost half —43% - of projects were innovative. Innovation took many forms: for example, projects innovated in their approach to partnership development. In a few cases, partnership development was done at multiple governmental levels excluding the federal level, with real innovation occurring through the involvement of municipal or provincial governments. This finding was substantiated in a number of interviews and through the focus groups, where participants spoke extensively about the strength of their regional relationships, in some cases with little or no federal involvement. This lack of federal involvement²⁴ meant that projects were unable to progress significantly despite extensive buy-in and involvement at the regional level and from other partners.

Across all the various projects in SIDPD, "innovation" appears to come primarily in the form of:

• Bringing together groups who have never, or have rarely, worked together on projects (as many organizations and/or departments continue to operate as distinct "silos" with little opportunity to share knowledge and expertise with other organizations or sectors).
• Innovation in approach occurred within at least two projects engaging marginalized groups: prostitutes and battered native women; two groups who in the past would have seldom if ever been deliberately engaged in policy issues.
• The collection of data that had never been collected before, the sharing of existing data that had never been shared before between groups, or simply the packaging of data in ways that it could be used more readily and effectively by the voluntary sector.
• The use of new technologies for improved communication and cooperation between voluntary sector organizations and their partner departments - in one case to develop policy using the Internet.
• The development of a professional policy capacity at new and different levels within some voluntary organizations. As one voluntary organization contact said, this was the first time they offered "professional development in policy" outside of the organization's head office. "Rather than promote policy development at the national headquarters level, we chose to develop expertise across the organization."
• New focuses for policy within departments. For example, one project involving the Canada Mortgage and Housing Corporation (CMHC) had an impact in expanding the agency's focus from simply affordable housing to affordable home ownership as well.

6.3.2 Dissemination/Knowledge Transfer/Replicability

Dissemination, knowledge transfer and the replicability of project learning are considered important indicators of the achievement of SIDPD results. Measures of success include evidence of dissemination of project results and lessons learned interdepartmentally, inter-sectorally (between the government and voluntary sectors) and within the voluntary sector; evidence of a change in the level of understanding within voluntary organizations of the policy process and of government priorities; within government of policy priorities within the voluntary sector; and finally of the actual or potential replicability of project activities and outcomes.²⁵

At the time of the process evaluation, many departments indicated that they intended to disseminate the final reports once projects were completed. Now at the outcome stage, approximately one third (33%) of projects reported dissemination to the relevant parties, though fully 48% of final reports were ambiguous on this point and indicated a 'don't know' notation. As such, it is possible that greater dissemination of lessons learned and findings will take place. Dissemination of results, however, can mean more than simply the sharing of a final report. For example, at least one department included regular meetings and communication between the four to five organizations carrying out the projects as part of the management of its portfolio of SIDPD projects. Their goal was to ensure that all of the organizations were kept aware of what other projects were doing, what information they were using, and what kinds of results they had achieved or were expecting to achieve. All project representatives interviewed indicated that they placed a value on dissemination and knowledge generation and they intended to complete their planned activities in this regard.

Knowledge Transfer

In terms of knowledge transfer between the federal government and the voluntary sectors, information gathered from the interviews and the focus groups indicate that within each sector, there has been a large shift in the level of understanding of the other sector's policy priorities and processes. Many voluntary organization respondents indicated that they had learned a great deal throughout the life of their project, right from the submission of their funding application to the submission of their final report. This experience rang particularly true for those organizations partnering with the federal government for the first time, or engaging in a new partnership. A key success factor identified by the final focus group was the achievement of significant learning achieved by both sectors, especially where an iterative and ongoing dialogue opened up between the voluntary organization and the federal department. Despite such successes, other focus group and interview participants articulated a continuing need for the government to clarify the differences between advocacy and policy development. One organization in particular experienced a challenge to its charitable status as a result of its project work funded by the SIDPD that was perceived by one department as advocacy work rather than policy development. For some stakeholders, the legal limbo and definitional confusion raised questions concerning the appropriate role of voluntary organizations in policy development.

At the same time, there was also considerable consensus that there should be more opportunities and mechanisms to share lessons learned. Some respondents suggested that the departmental 'champions' should have facilitated the dialogue within and between projects, and should have been charged with facilitating de-briefing sessions whereby projects and departments could have learned from one-another.

Replicability

Finally, replicability is another dimension of knowledge transfer and specifically deals with whether project activities, other organizations, departments and jurisdictions can use outcomes and lessons learned. The evaluation examined both actual replicability of project activities and the potential for change. More than half of Round 1 projects (57%) indicated that the experiences and solutions gained from their projects could be replicated, while just one project out of 21 reported that replication had already taken place. This finding is not surprising given that many of these projects were only just starting to engage in serious policy development related activities by the end of their projects. They had spent much of the start-up time developing networks, partnerships and getting the projects up and running. One project dealing with Family and Children in Poverty was originally seen as being a networking model that would be set up in the regions across Canada. Though by the project's end this expectation became questionable due to the absence of sustainability funding.

6.3.3 Horizontality and Collaboration at the Departmental Level

Projects, which work across sectors and issues areas, across disciplines and departments as well as across jurisdictions, are by force of circumstance compelled to operate in a horizontal fashion. In an increasingly complex world, it is rare that an issue is not related to other issues. Horizontality requires the development of collaborative working relationships and without it there is only the appearance of horizontality.

For the purposes of the outcome evaluation, horizontality was measured in terms of the degree of new or strengthened evidence of collaboration in the area of project development or policy development, within or between departments and voluntary organizations. Horizontality as a way of working played out at various levels that were examined: within departments, between departments, within the voluntary sector, and between the voluntary sector and the departments.

As has been previously stated, there was no formal mechanism or process within SIDPD to bring federal departments together in order to facilitate either intra- or interdepartmental project development or monitoring, let alone policy development. Where successes occurred at the individual project level, they did so as the result of a champion within a department and/or within the voluntary organization partner that understood the requirement for broad, multi-sectoral involvement.

Regardless of the potential for horizontality through the SIDPD projects, there is mixed evidence of the departments working in a truly collaborative fashion. For example, there were no efforts to bring the departments together at the SIDPD Initiative level to share project progress reports until the evaluation brought them together. Nor were individual projects adequately resourced to ensure that opportunities for information sharing and exchange were embedded in their projects.

However, there were a significant number of projects where project contribution agreements indicated the existence of federal partners. At the level of project execution, a number of these 'paper partnerships' did not come to fruition due to a number of factors including the absence of formalized Steering Groups or lack of staff resources to nurture the work. This apparent contradiction in the findings suggests in part that there was not sufficient understanding or recognition of the time, resources and skills required for working collaboratively and managing horizontally. By the same token, other projects acquired some partnerships along the way, and though not indicated in their contribution agreements, such partnerships were real and played critical roles in the projects. Certainly, it clear that working collaboratively was a key project success factor, and that the idea of collaboration is closely tied to focus group participants' assertions of the need for equity within voluntary sector /federal partnerships.

6.3.4 Creation of Partnerships/Networks and Collaboration Among the VSOs

The range of partners engaged by the voluntary sector in projects reflected an extremely broad cross-section of institutions representing Canadian society. These groups often included other organizations such as umbrella organizations made up of smaller member organizations, informal (and formal) networks and partnerships between voluntary organizations, and federal government departments. In some cases, provincial governments were involved as partners, as were some municipal governments, Aboriginal governments, police departments, labour unions, chambers of commerce and other business groups, and universities, colleges and independent think-tanks and academics. A small number of projects also included private businesses as project partners.

Typically, partners in a project had worked together previously. In such cases, SIDPD offered an opportunity to solidify their existing relationships. But many projects involved partners who had never worked together previously. They often cut across sector lines by bringing together groups working on different issues with common interests in developing a policy capacity in cooperation with other groups. One project even planned to stretch across international boundaries, involving voluntary organizations from Latin America and Africa.

The participation of these groups occurred at all points in the life of the projects. Partner groups were involved in designing or conceptualizing projects; contributing a particular area of expertise; providing services to specific projects, such as training or facilitation services; or in providing secretarial services and meeting space. Many projects planned for the involvement of their key partners throughout the life of the project, such as in co-managing the project or participating as a member of the project's steering committee. Others brought stakeholders in for only specific parts of their work, such as participating in workshops, round tables or symposia, contributing data, or providing input into a particular piece of the project's work.

The vast majority of respondents from voluntary organizations said that working together with other groups was vital to the success of their projects. They also suggested that strengthening existing relationships with other voluntary organizations, or creating relationships with new partners contributed the most to developing a policy capacity within their organization(s). Contributing to a "horizontal" environment and collaborative working relationships among SIDPD participants, and among voluntary sector participants in particular, appears to be one of the most successful and positive outcomes of the SIDPD Initiative.

There was also consensus about the importance of equity within the partnerships that were developed, in particular where government was a key player at the table. The partnerships most open and engaged in continuous dialogue, the better the relationship between the voluntary organization and the government, the better the project. By contrast, inequitable relationships led to a perception that the voluntary organizations were coming hat-in-hand, rather than as equal partners and there was some palpable resentment from representatives of the voluntary organization in such instances.

Participants at the final evaluation focus group also raised the issue of the benefits of the voluntary sector and the federal government working together in terms of the knowledge that voluntary organizations bring to the relationship. Specifically, voluntary organizations were clearly able to make great contributions due to their presence within their communities, their abilities to engage with a range of other groups and partners, and their ability to pull together diverse communities and perspectives. For some projects, the voluntary sector partner was instrumental in forging coalitions with other organizations that benefited the projects through sharing of resources and expertise — critical elements of project success that a federal department alone would have been unable to tap, due to its distance from citizens and communities. For most SIDPD participants, SIDPD was about relationship building as a critical building block for the establishment of effective, equitable ways of working.

6.3.5 Sustainability

Sustainability of the SIDPD objectives was examined from two main perspectives for the outcome evaluation: within federal departments, evidence of senior departmental management actions to engage voluntary organizations in new and ongoing ways; and within the voluntary sector, evidence that input into departmental policy development had increased as a priority for the organizations and whether the work itself could be sustained.

The majority of respondents in Round 1 indicated that there had been limited evidence of real change in senior management actions concerning voluntary organization engagement, with many reasons provided. For example, one respondent indicated that "[w]e continue to deal with mid-level public servants; senior managers are not fully engaged with us; the voluntary sector is not their core business..." Another respondent stated that such leadership is "sporadic at best because there is not much reciprocity of resources, capacities and understanding...there is no champion, and therefore no consistency...." However, in a few of the projects, senior management has been very involved and has been actively engaged from the beginning. These few projects have met with considerable success. In addition, there is some indication that more engagement may have taken place in Round 2 projects. The nature of the change will be better understood once the projects are completed and their outcomes are evaluated.

At the level of the voluntary sector, there are mixed views as to whether the SIDPD projects made engagement with the federal government around policy development a greater priority. Several contacts reported that engagement was always a priority for their organization, and that the SIDPD project was merely a fulfillment of their mandate. For others, the project enhanced the organization's interest in a particular policy area, one that the organization saw as sustainable into the future. For a very few, it was too early to tell whether engagement with the federal government on policy development would become a higher priority for them — possibly because it is too early to tell whether the organization's efforts have yet been effective.

Other respondents suggested that sustainability also needs to be understood in terms of an ongoing voluntary sector involvement in policy development across the board. In order to continue to attract voluntary sector representatives, and in particular to engage organizations who may be able to "think outside the box", it was suggested that the government should consider a measure such intervener funding that would support the costs associated with engaging in federal policy development. Intervenor costs could include participation at meetings and researching and preparing policy papers. Further, one respondent suggested that "[p] perhaps ... a government-wide policy (is needed) which takes the position that input from the voluntary sector, the department(s) should be required to support (voluntary sector) participation. There needs to be some way to compensate them".

As a case in point, it appears that where multi-jurisdictional networks were formed as a result of the SIDPD, they will have difficulty sustaining their work over the long term. These projects include PolicyLink in New Brunswick, VOICE in Manitoba, and the Sports Collective at Heritage Canada. In all three examples, it has become clear that partnerships and networks, like collaborative working relationship, are not sustainable without additional resources or supportive mechanisms.

Another respondent stated that resources should be allocated not only "to seek policy input, but to also be able to communicate with the voluntary organizations concerning how their point of view was taken into consideration irrespective of final political decisions". Focus group and interview participants echoed this perspective on sustainability, and the discussion raised additional issues concerning the difference between democratic access to the policy development process, and "a perceived expectation that involvement in policy development automatically means having one's views or recommendations accepted". Many felt that this issue will require more discussion and a longer period of working together, but that the SIDPD projects had opened the channels of communication in a very positive way.

Previous Page

Next Page

Table of Contents