Dimensions of open research : critical reflections on openness in the ROER 4 D project

Open Research has the potential to advance the scientific process by improving the transparency, rigour, scope and reach of research, but choosing to experiment with Open Research carries with it a set of ideological, legal, technical and operational considerations. Researchers, especially those in resource-constrained situations, may not be aware of the complex interrelations between these different domains of open practice, the additional resources required, or how Open Research can support traditional research practices. Using the Research on Open Educational Resources for Development (ROER4D) project as an example, this paper attempts to demonstrate the interrelation between ideological, legal, technical and operational openness; the resources that conducting Open Research requires; and the benefits of an iterative, strategic approach to one’s own Open Research practice. In this paper we discuss the value of a critical approach towards Open Research to ensure better coherence between ‘open’ ideology (embodied in strategic intention) and ‘open’ practice (the everyday operationalisation of open principles).


Introduction
The Research on Open Educational Resources for Development (ROER4D) project was established in 2013 to contribute to a better understanding of the adoption and impact of Open Educational Resources (OER) in South America, Sub-Saharan Africa and South Asia by developing a body of empirical evidence on OER activity in the Global South.The project comprises 18 sub-projects with 86 participating researchers and research associates across multiple time zones.It is coordinated by a central Network Hub team based at the hosting universities: the University of Cape Town (UCT), South Africa, and Wawasan Open University (WOU), Malaysia.
Since its inception, the ROER4D Network Hub team, under the direction of the Principal Investigator, has aspired to adopt Open Research practices, recognising a natural affinity between OER and the other 'Opens'-Open Access, Open Data, and Open Research.Based on the belief that 'performing research in the open' would lead to greater transparency, accountability and rigour, this approach manifested in an intention to release interim outputs, process documents, and other research products throughout the research process.
In the course of project activity, certain initial assumptions about Open Research practice were revealed to be too optimistic, inappropriate, or difficult to implement.This engagement with Open Research practices led to specific choices in licensing, communication and dissemination that were more nuanced than initially envisaged.
In this paper we discuss the value of a critical approach towards Open Research to ensure better coherence between 'open ideology' (embodied in strategic intention) and 'open practice' (the everyday operationalisation of open principles).Building upon Hodgkinson-Williams and King's (2015) four domains of openness approach as a framework, we demonstrate how initial planning, The development of a shared, open ideology as a guiding principle for a particular research project carries with it a set of legal, technical and operational imperatives.

Ideological openness
The set of practices that characterise Open Research rest on a foundation of beliefs about the purpose and value of openness in research: an 'ideology of openness' (Gibbs, Rozaidi & Eisenberg, 2013).While the concept of 'ideological openness' is employed in the literature, it is not always clearly defined (Chandra & Patkar, 2013), or is sometimes used to describe broader 'ideological self-disclosure' (Klein 2011)-disclosing one's ideological commitments and judgments.Tapscott and Williams (2013) popularised the idea of 'radical openness': the belief that reducing insularity, bureaucracy and secrecy in government, industry and research leads to better, faster innovation and development.However, Resnik (2006) considers that while openness may be essential to the scientific endeavour, there are many reasons for maintaining secrecy and protections.
Indeed, the ideology of openness interacts with other ideologies of practice that exert in research, such as discourses around ethical practice, responsible research conduct, and what constitutes 'good' or 'quality' research.While openness is believed by many (Munthe & Welin, 1996;Poynder, 2015;Resnik, 2006)  Research may conflict with established doctrine in certain research cultures and with traditionally closed parts of the research practice, such as data production and early release of findings.
Our use of the term 'ideological openness' is centred on the belief that Open Research adds value to the research process, which is made more transparent, accountable and verifiable to a wide scholarly and/or public audience through persistent and barrier-free access to the research outputs.

Towards ideological openness in the ROER4D project
Developing and explicating a coherent ideology of openness is an important foundational step towards Open Research practice.We found that before open research practices could be operationalised, we needed to develop a shared understanding and vocabulary of what openness was in the context of our project.In multilingual and cross-regional projects, the development of a shared vocabulary or understanding can be difficult to negotiate.The dominance of English in scholarly and scientific communication may lead to erroneous assumptions that the terminology employed when discussing Open Research (such as Wiley's 5 Rs) is universally translatable and comprehensible.We found that language issues complicated attempts to build a shared research lexicon, due in part to the use of largely common-language terms (open, share, reuse, etc.) as key concepts within OER research; yet these English terms do not necessarily have equivalents in other languages.Furthermore, key concepts (such as "reuse" or "revise") are used inconsistently across languages.Despite considerable effort to standardise terms across languages by OER advocates such as David Wiley (2014), the aspiration for a comprehensive shared set of descriptions in multiple languages for key concepts has not been entirely successful.
Such 'on-the-ground' understandings of openness in the research process influenced how the project's ideological openness was conceptualised and how it changed over time.Reflecting on how ROER4D's ideological openness, expressed in various presentations and public interactions, such as OCWC 2014 (the annual conference of the OpenCourseWare Consortium) and the Open Education 2014 conference, indicates not only the intention to share early and often but also an evolving critical approach, in which the commitment to sharing openly by default is on condition that the sharing is valuable, legal and ethical.
This condition refined the initial implicit and simpler open ideology expressed in the project proposal and scoping documents in which the alignment between OER and Open Research was alluded to but not explicitly defined; nor was there a precise strategy explaining how we intended to enact Open Research principles.The refinement came about as a result of reflection on the interplay (and misaligments) between a desire to conduct Open Research, the limited time and resources of the Network Hub ('if it adds value'), and the need to protect researchers and research subjects ('if it is ethical' and 'if it is legal').Even though to 'make open by default' remained as the core principle, awareness of these misalignments began to emerge as a result of the self-evaluatory practices adopted by the Network Hub (Goodier, King & Hodgkinson-Williams, 2015).
Negotiating ideological openness has been an iterative process, and one in which understanding power relations between and the need for capacity development amongst research participants is particularly important.This is especially important in network-based or geographically dispersed projects where researchers pursue their own objectives, partly or largely in isolation from their project peers.While we argue that a strategic approach to openness is advisable, researchers will need to revise the overall strategy as their research progresses, particularly when engaging at-risk or vulnerable groups who may be uneasy about the release of interim research outputs or open data.
Within this framework of ongoing negotiation, ideological commitment is enacted through the three other domains: legal openness, technical openness and operational openness.

Legal openness
Open licensing-such as the use of Creative Commons licences-provides the legal framework for Open Access, OER and Open Research.The ROER4D Network Hub's commitment to sharing research outputs and data was enacted in the sub-grant agreements signed with sub-projects, which stipulated that (where possible) all outputs and findings would be made available under a Creative Commons licence to ensure the greatest possible development impact.
The initial licensing terms allowed for a fairly generous interpretation of legal sharing in order not to compel researchers to contribute their research outputs and data openly, but rather to encourage a spirit of openness and responsible research conduct.This consideration regarding the level of openness and the readiness of the associated research community in the contracting process means that legal openness needs to be considered not only by project researchers, but also by institutional lawyers and senior authorities who endorse these agreements.
A critical approach to legal openness allowed the project to determine which legal permissions would be feasible given the subject matter of the research, what was valuable for both the producers and users of the research, and what was practical given the levels of legal expertise and resources available internally or externally.The ROER4D Network Hub drew on prior documentation (Hodgkinson-Williams & Gray, 2009) and experience from other 'Open' projects at UCT-e.g.Opening Scholarship and OERUCT (Czerniewicz, Cox, Hodgkinson-Williams & Willmers, 2015)-to inform decisions.

Towards legal openness in project documentation, research data and project outputs
Publishing a selection of ROER4D research data in line with open principles proved to be more challenging than originally anticipated.From nascent initial plans to share research data in principle there emerged a nuanced and complex data publication strategy following the employment of a full-time Curation and Dissemination Manager to address the widely acknowledged challenges of Open Data (Floca, 2014;Pampel & Dallmeier-Tiessen, 2014).
The project recognised early on that openness as pertains to data sharing and legal permissions needs to be considered carefully in order to abide by ethical principles and protect research participants.A core ROER4D contractual provision stipulates that at no point will raw data (i.e.data still containing disclosive information that could be used to identify individuals) be shared, including exchanges between the sub-projects and the Network Hub.Subsequently, the project entered into a publishing agreement with DataFirst, an internationally recognised data service, whose expertise in data preparation and verification has supplemented the Network Hub's data-sharing efforts.
Open licensing of internal reporting documents has also presented a challenge, especially in the licensing of technical reports (i.e.those reports describing details of project implementation) by the Network Hub for the project funder.Although our contractual commitment expressed the desire to license outputs as openly as possible, in the case of technical documents of this nature we applied a Creative Commons Attribution No-Derivatives (CC BY-ND) 4.0 International licence as we felt the information needed to be stand as a complete record and that adaptation or derivation would compromise the intergrity of the information.
While a critical and flexible approach to licensing of research outputs was possible in the project, research projects in the future may be constrained by funder mandates regarding licensing which are more prescriptive.This may limit options for flexibility.Therefore, early awareness of this at the contracting is beneficial as it has ramifications for implementing a contextually appropriate approach to legal openness.Dimensions of open research: critical reflections on openness in the ROER4D project Open Praxis, vol.8 issue 2, April-June 2016, pp.81-91

Technical openness
Technical openness refers to the use of open file formats and open software development standards to ensure equitable access and discoverability of research.Depending on their technical format and/or mode of publication, outputs can manifest varying degrees of openness not necessarily aligned with their licensing provisions or the ideological commitments of their creators.The PDF format, for example, while ubiquitous, does not allow for easy revision or remixing unless the user possesses the necessary proprietary software, while outputs placed in obscure or inaccessible locations have minimal chance of being used, reused or revised regardless of their creators' intentions.

Towards technical openess for collaboration, availability, revisability and verification
ROER4D's approach to technical openness revolves around four elements that facilitate Open Research practice: 1. Collaboration: ensuring that, where appropriate, project documents can be written, edited and commented upon collaboratively within and beyond the ROER4D network.2. Availability: ensuring that outputs are hosted on stable, secure platforms that facilitate open licensing and provide adequate metadata according to recognised international standards, thereby maximising their discoverability and no-cost accessibility.3. Revisability/remixability: utilising open file formats (supported by open licensing) to facilitate access in a non-proprietary software context, allowing users to make changes, extract text or images, or otherwise alter the content.4. Verification: the tools and instruments that support the analysis are freely available and facilitate interrogation of the research results.
At the start of the project, we explicitly intended to address two key elements-availability and verification-while issues of collaboration and revisability emerged as a result of internal critical examination of our research processes.Our growing understanding of the importance of technical openness and a need to adhere to all four elements prompted the elevation of curation and dissemination from an ambition to a core project objective in order to address these issues systematically.

Affordances of technical openness for collaboration
One of the principal intentions of Open Research is to enable and support collaboration (Maurer, Rai & Salie, 2004;Woelfle, Olliaro & Todd, 2011).To this end, the ROER4D Network Hub has used the cloud-based authoring tool Google Drive as its main research collaboration platform.This enables ROER4D researchers and mentors to create, review, edit and comment on shared project documents asynchronously across 16 time zones.As the Network Hub develops project documents, we open these up via Google Drive for input.While the use of Google Drive has been valuable where researchers are comfortable with the technology and have good Internet connectivity, researchers with limited connectivity or insufficient expertise have not always been able to use Google Drive effectively.The primary users of the document authoring features have been the Network Hub team, but Google Drive remains useful as an open storage space accessible by our researchers to monitor and comment on documents in progress.We maintain an agile approach to technical openness for collaboration and have on occasion used less technically open tools (such as Microsoft Word) if it aids collaboration with a particular researcher.

Affordances of technical openness for discoverability
Utilisation of stable curatorial platforms for sharing project outputs is vital in order to ensure that research outputs remain accessible and discoverable after the project ends.There are several online, publically accessible, open repository platforms (such as FigShare, Zenodo and Slideshare) which support different output types and disciplines, and can be adapted for a range of curatorial and publishing activities.We evaluated various repositories to determine the best fit for the project according to the following criteria: 1. Affordances for supporting open licensing.2. Ability to accommodate multiple content types and genres.3. Assurance of long-term stability.4. Zero cost associated with deposit or access.5. Use of an international metadata standards.
While institutional platforms such as UCT's open institutional repository (OpenUCT) were investigated, the ROER4D Network Hub has chosen to use Zenodo as the public curatorial space for its outputs, due to its stability, comprehensive licensing and metadata features, and capacity to accommodate a wide range of outputs from a cross-institutional group of researchers (institutional repositories such as OpenUCT only accept outputs from UCT-affiliated authors).As the ROER4D project is a grant-funded (and therefore time-bound) initiative, we needed to pursue options that enable longterm, free access to the materials under open licensing provisions without interrupted access.

Affordances of technical openness for remixing
File formats exist on a spectrum of technical openness, which impacts upon their accessibility, revisability and remixability.While certain, more 'closed' formats (e.g.PDF, EPUB) can usually be viewed with free software, it can be difficult to extract components of documents in these formats for revising and remixing without proprietary software.Open formats, such as ODT, ODS, HTML, XML and SVG, usually allow for access and remixing of constituent elements using open source software; while Microsoft Word is a popular choice for reuse and revision due to its ubiquitous use and familiarity, but requires proprietory software.
Initially, the choice of ROER4D output formats was opportunistic, using what we had at hand and what we were familiar with.We become aware of the tension between our choice of document formats and the kinds of reuse we wanted to encourage, with a particular tension between PDF format and the ability to remix outputs.Due to lack of familiarity and the technical skills required to utilise open formats in the broader research community, we have decided not to default to the use of open formats while producing project documents, although we have committed to releasing final outputs under a range of formats to maximise revisability and remixability.Our approach to technical openness is informed by the ambition to provide end-users with affordances for revision and remixability of the outputs.

Affordances of technical openness for verification
Open research facilitates a value-added component to the research process: the verification of research through interrogation of open data.This means that access to the data that underpins the analysis and conclusions of the research process will also be shared openly where possible, enabling third-party analysis of the results and facilitates longitudinal and latitudinal studies without needing to contact the researcher to gain access to the data.Dimensions of open research: critical reflections on openness in the ROER4D project Open Praxis, vol.8 issue 2, April-June 2016, pp.81-91 We faced numerous challenging decisions around what data to make available (both quantitative and qualitative), how to best de-identify the data, which platforms to use, and which metadata to provide in order to optimise visibility and reusability.To guide decision-making a set of Data Publication Guidelines was devised (Willmers, 2015) as part of the ROER4D Open Data Initiative.
Sharing data openly opens the research to in-depth scrutiny, and requires that data not only be comprehensive and accurate, but also that ethical procedures be conducted rigorously.As this area of activity is novel and potentially intimidating for researchers, the ROER4D project has not mandated that all sub-projects release their data openly, but supports those who do wish to do so.

Operational openness
For the ROER4D Network Hub, operational openness entails the enactment of the ideological, legal and technical principles in the course of conducting research.This can take many forms, including: early and frequent communication about the the project; sharing bibliographies, literature reviews, conceptual frameworks, and interim and preliminary findings; and actively developing networks of interested readers, colleagues and potential collaborators prior to the final publication of research outputs.Operational openness therefore refers to the openness practice that emerges as a result of critical reflection on when to be more or less open as the specific context dictates.To this end, the phrase 'if it adds value' has been a touchstone as we grappled with how and when to enact our commitment to ideological openness and where a more critical and nuanced approach towards openness was required.

Towards operational openness in project activities and sharing outputs
Within the Network Hub, we have adopted an agile approach to operational openness in order to support the research management process.An early activity was to make our weekly project administration meeting minutes available as a Google Document to which both partnering universities and the project funders had access.This small research management activity set the tone for our open practice and provided a platform to explore the experience of writing about research activities while others read and comment upon them dynamically.This process increased the rigour of our discussions as well as the accountability of our decisions, and highlighted the many sensitive decisions required when enacting ideological openness in research.The ROER4D Network Hub openly shares a range of outputs, including project proposal documents and technical reports (with summaries of individual sub-project reports) or extracts from technical reports that provide insights into the project without disclosing financial or contract details.
In line with the guiding principle of 'adding value' while not unwittingly exposing researchers, we tried to model our operational openness through the activities of the ROER4D Network Hub, but did not require the same level of operational openness from others in the network For example, we created and maintained a publically shared ROER4D Bibliography 1 , in which the references used in the project were categorised, but did not require that sub-project researchers share their own bibliographies.An unexpected benefit of this activity came from outside the project in the form of an offer from John Hilton III, an OER researcher in the United States, to incorporate his 1 000-item OER bibliography with our then 450-itemed list.In this case, our operational openness resonated beyond the project before it resonated within the ROER4D community.Subsequently, one of the sub-projects has released their project's annotated bibliography as a public document.
Early and sustained communications about ROER4D was an attempt to operationalise open communications as part of Open Research practice.The project appointed a Communications Consultant to develop a communication strategy and engage with audiences through strategic use Open Praxis, vol.8 issue 2, April-June 2016, pp.81-91 of social media, the ROER4D website, newsletters, weekly emails and SlideShare.Outward-facing stakeholder engagement and internal networking was further facilitated by the ROER4D Network Hub team's attendance at conferences.This process of engagement was initiated well before interim outputs were scheduled for release in order to develop links with stakeholders and build a network of Global South OER scholars.The ROER4D communications strategy was therefore designed to be 'dialogic' rather than 'transmissive', in that the purpose of the communication is to develop an interested and interactive community of stakeholders whose commentary feeds back into the research process, rather than a one-way process of information delivery.
Whether engaging in open research practices internally within the network or externally with potential research recipients, operational openness requires flexibility to account for possible changes in the methodology and refinements to the research process.Strategic decisions need to be made with regards to which open practices will be practical for the project given fluctuations in research formulation and progress.
We found that a significant complicating factor in adopting operational openness is about timing and the difficulties associated with sharing interim data and analysis,.Releasing initial analyses may be misrepresentative of later analyses due to their partial nature, meaning that decision-making around the optimum time for data sharing is not straightforward.Moreover, some of our researchers have expressed the desire to 'mine' their own data before releasing it publically.The ROER4D Network Hub does, however, endeavour to release data and accompanying research instruments as soon as possible.An example of a ROER4D sub-project dataset (incorporating micro-data, instruments and extensive metadata) published via DataFirst is shown in Figure 1.

Discussion
Criticality in research can provide a self-reflexive lens for researchers and managers of research projects to examine how their practice aligns with their goals, surfacing areas in which there are tensions or contradictions in their Open Research practice and providing indicators for how to advance their openness in a way that is appropriate to their project context.Initially, aspects of the ROER4D Network Hub's practice did not fully align with our ideology, a problem which only became visible after attempting to put our commitment to openness into practice.While this prompted changes in our practice, it also required a re-examination and subsequent refinement of our ideological position to one that adequately reconciled our belief in the value of openness with our commitment to quality, ethical research.In this sense, the interrelationship between the four domains was (and continues to be) iterative and coherent to various degrees at different stages of the project.
Coherence is a worthy aspiration, but it may be elusive.As a new research orientation, Open Research ambitions are tempered by other priorities such ethical protection and quality assurance, and against the reality of limited resources.Individual researchers' awareness and acceptance of Open Research also differ.Attempting to align all 86 researchers from different regions and across contexts to a single vision of Open Research seemed impractical and a distraction from the primary project deliverables.In ROER4D, the Network Hub acknowledged the difficulty of building consensus on appropriate Open Research practice amongst the broader researcher community, and so aimed to demonstrate our vision of Open Research practice and inspire the sub-projects to enact those aspects appropriate to their contexts and capabilities.
We acknowledge that our context-operating under the auspices of a well-resourced institutional host, with human resources specifically allocated for expansive Open Research activity, and a focus on coordinating and supporting research-has allowed us to interrogate and develop our own practices in a way that less well-resourced projects or individuals may struggle to do.

Conclusion
ROER4D's critical approach to openness is informed by the principle that research is only valuable if it is used.While some aspects of Open Research may seem like a departure from traditional methods, much of it speaks to the foundational aspiration to increase rigour and better communicate findings to maximise uptake and use-increasingly reflected in the growth of funder mandates that tend towards (mainly legal) openness.The ROER4D Network Hub's experience suggests that adopting an Open Research strategy may be a way to improve the transparency and reach of research while simultaneously increasing rigour and building research capacity.
Although a comprehensive Open Research plan is valuable, research contexts frequently change.An agile, iterative and strategic approach to openness is likely to better serve researchers than a rigid strategy, allowing researchers the freedom to adjust their ideological, legal, technical and operational approaches to improve their congruency.We term this approach 'critical openness' as a thinking tool to enable iterative strategic planning.
Finally, we would argue against a single correct way of conducting Open Research.Contexts vary too widely and other pressures in the research process exert too strongly to support a prescriptive approach.Rather than a radical open approach, we argue for the importance of striving for congruency between the different domains of openness that researchers identify for themselves as valuable in their context.We encourage researchers to engage in Open Research not for the sake of openness, but as a tool for enhancing transparency and rigour and expanding the impact of their work.

Figure 1 :
Figure 1: Example of a ROER4D dataset 2 published on the DataFirst Data Portal to serve the mandate of producing quality research, a commitment to Open Dimensions of open research: critical reflections on openness in the ROER4D project Open Praxis, vol.8 issue 2, April-June 2016, pp.81-91