Here, we detail the functioning of our data monitoring and evaluation program.

Mission

The mission of the Nyaya Health data program is to:  

• provide online, public access, up-to-date data from our programs

• innovate in the realms outcomes monitoring, data transparency, and epidemiological collaboration. 

• use data to drive improvements in our evidence-based clinical services

• provide community members with data that can empower them to take action to lobby the government, private, and non-profit sectors for improved and expanded services

• provide donors and supporters with an accurate and timely assessment of their social return on investment

• collect data that sufficiently protects patient privacy

We believe that a rigorous data system is fundamental to our aims of providing effective evidence-based clinical and public health services locally and of driving innovation in global health practice. 

Strategy

Our Approach:  Telemedicine Collaboration

The Sanfe team is responsible for the collection and electronic transmission of data to the US data team.  The US team is responsible for analysis and reporting, including public access wiki posts and government reports.  The reason we have structured this as such is that it is not sustainable nor possible to have a full-time statistician or epidemiologist on site for extended lengths of time.  Meaningfully engaging professionals who want to contribute but cannot be based in Achham is also part of our building a broad-based social movement.  The core outputs-- government reports, internal organizational review, and public access reporting-- all are integrated to maximize efficiency and minimize errors.  The key to this functioning is reliable and consistent communication.  We follow a these core principles:   

• Public access to the data is key to efficiency, transparency, and collaboration.  Providing our data in an open-access format allows for rapid communication of our results to team members and collaborators without having to manage passwords and access controls.  Further efficiency is gained by having one set of data outputs for government, public, internal organizational, and donor reports.  It also maximizes our potential impact through potential dissemination to other collaborators and supporters.  
• The half-life of unprocessed data is very short.  As such, data should be evaluated on a monthly basis.  The inputs are reviewed by our data team and the outputs are reviewed by our generalist clinical team.
• Extraneous data produces noise in the system that taints the entire data program.  The minimum amount of data should be collected to get at the core objectives.  Each data point should be rigorously evaluated to decide whether it is worth collecting or not.  
• We must design data collection instruments that are appropriate to our setting.  Many of our staff have minimal formal education and almost none had computer training prior to their work with Nyaya Health.  We must take care to ensure that our data strategy utilizes our staff's strengths and skill levels.
• Effective, rapid, and open communication is essential to continually refine our data program.  Data programs are dynamic.  No survey instrument, database, or analytic strategy will work as desired without continued and sustained attention.  Any break in the data chain can ruin the entire program.  Maintaining an active and open dialogue between the US and Achham teams is therefore essential. 
• Being responsive to evolving community needs requires an efficient and up-to-date data system.  A key reason for why we require monthly analysis and review of data is that our clinical teams need to use these data to adjust their practices.  Our patients cannot afford for us to wait six months to make essential changes to our clinical operations.
• Community members themselves must have access to up-to-date and understandable data.  These data can help better inform citizens as consumers of healthcare and can better position them to be advocates for their communities. 

Social Return on Investment

Quantifying the social return on investment requires accurate assessment of both expenditures and outcomes.  For the most part, the desired outcomes for SROI are those at a public health level, for example, assessing the effect of a certain interventions on long-term trajectories in infant or maternal mortality.  As such, donors concerned about SROI (which is really all donors at some level, although most don't articulate/quantify their objectives) have a time horizon on the order of years.   At the same time, there are intermediate SROI process indicators and "surrogate markers" that help the investor decide in the short-run whether the investment is likely worth the risks involved.  So, even as we make available all our financial and clinical data, we also aim to create metrics combining the two-- that is, assessing the costs and risks of achieving potential social outcomes.

Community-based Data Programming

Dissemination of accurate, up-to-date, and accessible data at a community level can lead to a more informed and empowered citizenry who can then advocate for better services.  One randomized controlled trial in Uganda, for example, demonstrated a 33 percent reduction in child mortality after one year following the start of a community-based monitoring project in which citizens were provided with aggregate service delivery and health outcomes indicators.  The idea is that the availability of data at a community level can increase utilization of healthcare and can empower citizens to push for better services, from the government and from Nyaya Health.  Achieving this in practice is predicated upon an up-to-date, easily aggregated database.  Once this is established, we then have to put a considerable amount of time and effort in identifying appropriate data points that the community desires; forums for their dissemination; further education on the meaning of these data; and strategies that can translate the data-knowledge into action.  If done in a meaningful way, this can be a critical component of ensuring that both Nyaya Health and the government are accountable to the citizens covered by our health services.

Data Domains

Current data domains include the following.    

Financial

Financial data catalogues the key independent variable in the social return on investment function.  Clinical and public health outcomes are the dependent variables.  Understanding in what ways and how much we are spending money is essential to effective and transparent operations.

Input: Accounting data from quickbooks

Output:

-wiki: Budget, updated monthly

-nepal taxes, processed annually in July

-US taxes, processed annually in February

-US auditor, processed annually in February

Pharmacy

Pharmaceuticals represent a sizable chunk of our operating expenses, and they are the main interventions that we provide at the clinic level.  Monitoring their use helps to drive our procurement and prescribing patterns.

Input:  mSupply database (currently); transitioning to MS Access database

Output:  

-wiki pages: Pharmacy Data

Patient-level Data

These are by far the largest aspect of our program.  

Input:  Comprehensive patient database in MS Access

Output:

-wiki pages: Patient Flow Data, Geospatial data, Maternal and Child Health, Mortality, Laboratory, Tuberculosis, 

Malnutrition, Community Health Worker Program, Inpatient Services (all updated monthly)

-Nepal government Health Management Information System (HMIS)

-outcomes data for use in health services projections, grants, and donor reports

Device Utilization

We monitor certain critical capital machines for their use.  The main outcomes here include throughput, which is useful in procurement planning, and quality assurance, which are important generating useful clinical information for our clinical team.  

Input:  Access device utilization databases

Output:

-wiki pages: Ultrasound, QBC (awaiting achham data), i-Stat (on hold owing to device issue)

-reports to donors of the devices.  

Data Measures

There are four primary categories of data measures. 

Clinical process measures

These inform how we are delivering care. They are descriptive in that they tell us, for example, how many tests we ran, how many trainings we conducted, knowledge of how many diagnoses we made, which medicines were prescribed for which diseases, or how many patients we saw.  They help to provide insight into what our potential weaknesses may be.  For example, we may notice that, during the harvesting times of year, we have fewer pregnant women come into the clinic for antenatal check-ups because they have so much work in the fields.  This would indicate to us a need for greater outreach during those times.  At a clinical level, we may review our pharmaceutical data and notice that we are prescribing the wrong drugs for certain diagnoses. We can then rectify the situation through improved staff training.

Clinical outcomes measures

These inform the effectiveness with which our care changes outcomes at a patient level.  Several clinical outcomes measures for chronic conditions—such as hemoglobin a1c or blood pressure readings or follow-up rates among abscess patients—are surrogate outcomes that by monitoring and improving we hope to impact the ultimate clinical outcomes we are targeting—reduction in morbidity and mortality.   These are critical in assessing whether the individual treatments we are providing are having an impact on our patients.  Evaluating these are central to meeting the fundamental mission of serving and healing our patients. 

Public health process measures

These inform us as to the reach of our services at a demographic level, and, although we think they are important, they do not necessarily tell us about the broader public health impact.  Public health process measures include coverage of infant vaccination, antenatal care, malnutrition screening, tuberculosis detection, or HIV testing.  

Public health outcomes measures

These are ultimately what we are attempting to impact at a broader, demographic level: infant mortality, maternal mortality, life expectancy, fertility rates, the incidence of diseases such as HIV, TB, malnutrition.  It takes a longer time both to assess and impact these.  Only by regularly monitoring the other three types of measures, can we have an impact on the long-term public health outcomes.  

Achham Data Team:  Collection

Objectives

The Achham team is responsible for the collection and entry of high-quality data.  We are developing a solid culture of accurate and honest data recording on the part of Achham-based staff.  We are working with our clinical team to emphasize that if they don't record the patient's information, then effectively they did not see the patient.  The core functions of the Achham team include:

• designing data instruments and training staff on their use
• collecting the data  and identifying errors or inefficiencies in the data collection strategy 
• entering the data into on-site databases
• uploading the data to the US  team on at least a monthly basis

Leadership

On site, ultimately our Executive Director is responsible for the quality of the data coming out of Achham.  While the Executive Director herself typically will not enter nor collect the data, with her oversight, our staff perform the essential data functions locally.  She helps to set the culture and tone of the entire data program.  

US  Data Team:  Analysis

Objectives

The US team is responsible for the analysis and presentation of the data.  The US team also investigates new technologies and strategies to improve our ability to meet our broad data mission and objectives.   The core functions involve:

• critically reviewing the academic literature, industry and non-profit websites, and any other sources to identify new strategies to pursue in our data system
• review, upload, analyze, and present the data each month from Achham.  
• provide feedback to the Achham team on the quality of their data and areas for improvement.
• to provide sufficient oversight to ensure timely production of the data reports
• respond to Nyaya team's requests for improvements in the presentation of data

• to help the Nepal-based team with trouble shooting any database
• to provide statistics and projections for grant writing

Team Members

The US-based data team, consisting of non-clinical epidemiologists/statisticians, will work closely with the broader Nyaya clinical team to ensure that the data being presented are truly relevant and help to drive clinical care.  Through dialogue, the data team and the clinical team can work together to improve the relevance of the data to patient care. The Director of Operations is ultimately accountable for the quality of the data program, although he is not responsible for processing or analyzing the data himself.

Note on the need for monthly reviews

The half-life of unprocessed data is about one month (unpublished observation).  If data are not reviewed monthly for quality, outcomes, and bugs, it does not matter how thoughtfully designed the data instruments or databases are, the data output will seriously suffer.  This is magnified in a situation such as ours, where many of our field workers collecting the data have minimal education, where the medical culture is not one of rigorous documentation, and where our data enterers have often just recently become computer literate (through Nyaya Health training).   Furthermore, to make the data useful to our clinical teams and to rapidly respond to community needs, we must have access to up-to-date information.  

Note on patient confidentiality

A major challenge in providing relevant, micro-level data in a public access format is achieving sufficient patient confidentiality.

Protocols 

HMIS Protocol

Analysis spreadsheet: SharedNyaya/Clinic/Data/hmis_monthly.xls

Inputs: Access Database; mSupply (soon to be overhauled); Quickbooks

Overview

All of the calculated fields are summarized in "calcs"; all automatically generated fields on pages 1-7 refer to this sheet.  

Red values are input by the data manager to generate the result; note there are a few on the "calcs" page pertaining to mortality.  Mortality data should be reviewed every month and entered by hand.  Yellow values are automatically calculated output on the HMIS forms. Orange values on the calcs page are those that we collect for our own purposes and are not linked to the HMIS forms.   Role of Achham data entry team: enter the data accurately and send the databases to US team monthly; sign and deliver the HMIS form to DHO monthly.

Role of the US data analysis team: review the accuracy of the data and generate the final HMIS for submission by Achham team each month to the DHO.  

Steps 

1. Establish Dates for Analysis.

Fill in the dates in the red boxes on the calcs page. You can find the Nepali dates on the "Dates" tab. All calculations are made through the Gregorian dates.  The Nepali dates are there because the government reporting scheme is based on those.  After changing the dates, there may be some time to wait for computation time, depending upon speed of your computer.

2. ANC Input

Take output from "ANC HMIS Export" query and paste into "ANC_input".  

Notes

Page 3, 2.3. "Number delivered at a basic and comprehensive EOC institution" will be marked as "x" until we have C/S capacity since we are not a comprehensive EOC facilty yet. This query includes the entire ANC registry, since there is no easy way to exclude old records owing to the multiple dates fields. This is not a problem currently since the ANC data are relatively small, but it is sub-optimal.

To Do:

[Aditya]Columns BD and BE are calculated within excel.  If the sheet has greater than 2500 entries, you will need to drag the field down. This is not optimal; preferred that "Total Iron" (sum of all antenatal iron tablets dispensed at any of the six antenatal visits) and "Total PNC" (sum of whether a new mother had Post-natal care visits) are both calculate automatically with Access.

[Aditya]Need to get the midwives to record GTPAL.  This is helpful in estimating the fertility rate among women visiting our clinic. 

[Aditya]Need to change the HMIS paper form to include all the AMDD outcomes, as per "Labor and Delivery Protocol".

3. OPD data

Paste OPD HMIS Export query (filtered with the dates for that month with all ages) into OPD_input.

Could do: 

[Jennifer] Use of sumproduct function adds to computation time.  Could perform using pivot table instead.  

4. Peds input

Paste  OPD HMIS Export query (filtered with the dates for that month + ages 0 to 5) into Ped_Input sheet

To do:

[Aditya] Once new pharmacy data is rolling, can use to improve the calculation of ORT, zinc, IV fluids, vitamin A, anti-helminthics.

[Jennifer] Perform an automated check of weight/height/age data with the malnourished check box that staff use (they have paper growth chart). 

5. Lab data

Take output from "Lab HMIS Export Query" and paste into lab input.   

6. Pharmacy Data

This is from msupply; will be overhauled by mid-March with the new pharmacy system.

A.  Open the mSupply database.  

B.  Go to Reports.  Open SharedNyaya\Clinic\Data\msupply\msupply_allitems.

C.  Open the exported file and paste into "Stock_input" on the excel sheet.

D.  Go to Reports>>Stock by Date.  Select your date and select category "HMIS".  Paste output into "Stock_input".

E.  Go to Reports>>Custom Report.  Open SharedNyaya\Clinic\Data\msupply\msupply_allitems; create tab-delimited text file.  Paste the output into "Pharma_input".

7. Budget Data

The budget data are exported by Achham team as per "Financial Management" note.  Paste these data into "Budget_input" tab. 

To do: 

[Aditya] Eliminate msupply and incorporate all pharm data into the current Access Database. 

Budget:  Wiki

Input:  monthly gnucash database export

Processing Files: NyayaBanking/nyaya accounting DATE.xlsx

donors' worksheet:

http://spreadsheets.google.com/a/nyayahealth.org/ccc?key=pWMfEM1cjNv18GuSXCoSyNw&hl=en

budget worksheet:

http://spreadsheets.google.com/a/nyayahealth.org/ccc?key=p-TJjzE7A-O7vvlOQZMrgCw&hl=en

Output: Budget page

OPD Database:  Wiki

Input: Monthly access database export

Processing File: SharedNyaya\Clinic\Data\HMIS\opd_codes

The current wiki output, ClinicData, is descriptive and set for the entire life of the clinic.  

1. Copy and paste the new OPD values from OPD HMIS export into opd_codes "input" sheet.

2. Copy new opd hmis export values onto the spreadsheet:

http://spreadsheets.google.com/a/nyayahealth.org/ccc?key=p-TJjzE7A-O4wT-vh-N_0Rw&hl=en

Both "OPD codes" and "OPD tally" should be filled out.

3. Then paste OPD HMIS export into registry_data to fill out "OPD Demographics"

4.  Use pivot table on the dates column of the OPD export to obtain the number of patients per day; paste these into the "OPD tally" tab on the spreadsheet and make sure that http://wiki.nyayahealth.org/ClinicData#NumberofPatientsoverTime is updated properly.

To Do:

[Aditya] Incorporate automatic caste construction within Access. 

Inpatient, Mortality: Wiki

Both of these are simple copy-pastes for now.  Important to upload to wiki via 

http://spreadsheets.google.com/a/nyayahealth.org/ccc?key=p-TJjzE7A-O4wT-vh-N_0Rw&hl=en

so that we can review the quality of the data and make suggestions on management.

Pharmacy:  Wiki

Input: msupply (old); now transitioning to the Access database

Processing File: SharedNyaya\Clinic\Data\msupply\msupplyexport.xls

Output: Pharmacy wiki pages, uploaded via

http://spreadsheets.google.com/a/nyayahealth.org/ccc?key=0AtZQBHyI2oBYcC1USmp6RTdBLU80Xy15RVZrU0w0S1E&hl=en

Spatial Analysis: Wiki

See protocol on SpatialMapping

Device Utilization:  Wiki

Input:  Ultrasound database

Output:  Ultrasound data are sorted by date and then uploaded into their respective (non-OB and OB) tabs on   

http://spreadsheets.google.com/a/nyayahealth.org/ccc?key=p-TJjzE7A-O4mq2jHoKGEAw&hl=en#

These data are accessible via:

http://wiki.nyayahealth.org/UltrasoundData

Input:  QBC database

Awaiting response from Achham team  

Mortality Data: Wiki 

Mortality data consists of 1) data on the cause, timing, and demographics of deaths; and 2) mortality reviews.  These are outlined on MortalityData page.   An overview of the subject is posted on our blog: http://blog.nyayahealth.org/2009/10/29/mortalityreview/

The protocol is as follows.  For each death, a mortality leader on-site is identified to lead the discussion among staff and to document for those team members currently outside Achham.  An email thread is initiated over the team@ list. After 10 days of discussion, the results of the thread are compiled by a US-based volunteer, saved within the google docs folder (nyaya login required): 

http://docs.google.com/a/nyayahealth.org/Doc?docid=0AdZQBHyI2oBYZGY4cmdwbmJfMzJoa3EybjRkZw&hl=en

which is linked to the MortalityData page.  

Every three months, a mortality review blog post is posted by the US-based volunteer.  

Posting data on the wiki

The strategy we have taken to maintain open-access, rapidly updateable data streams is through google apps' docs, spreadsheets, and gadgets.  There are a few tricks that are important to making these work for us.    The first time you set up a google gadget presenting data, you will have to test it out a bit.  Thereafter, it should update fine as you update it with additional data. However, the title of the gadget is embedded in the source code, thus if changing the title of a gadget, you will need to update it in the code/republish the gadget in order for the update to appear on the wiki. For all data sources, you should click "publish as webpage".   To facilitate collaboration within Nyaya, for sharing>>"share with nyayahealth.org".  Allow people from within Nyaya to edit without signing in.  Do not, however, allow others from outside the nyayahealth.org domain edit without signing in (i.e., "share with the world")

To embed spreadsheets, click "publish as webpage">>"more publishing options">>select "HTML to embed">>copy the resulting code>>then in pbwiki, "Insert Plugin>>"HTML & Gadgets">>"HTML/Java script". 

For charts, just click on the chart and then "publish" and grab the "img" code.  Then the easiest way is to click on the "Source" button in pbwiki and paste that code directly into the page.   For gadgets, under more>>"Get query source URL".  

For time series charts, you often have to replace http://spreadsheets.google.com/a/nyayahealth.org/tq? with http://spreadsheets.google.com/pub?  (note removal of /a/nyayahealth.org) to allow for access to the public.  Otherwise people will see "access denied" in their browsers (even if in your browser, being signed in with access, you see the gadget properly).   Make sure to check the wiki in a browser in which you are not signed into your google apps @nyaya account.   Then, go to http://www.google.com/ig/directory and find the time series or whatever gadget you need, select webmaster>>embed, fill out the specs (typically use about 480px width by 320px) and copy the "script" code.  On the wiki, edit your page and click "insert plugin">>HTML & Gadgets">>google gadget.  Then paste in your code.  Always check your webpages in a separate browser in which you are not logged into your nyaya google apps account, to make sure that the gadget is viewable by everyone. 

The Current State of Nyaya's System

Overview

Achham team sends exported text, csv, or excel files.  This is to save bandwidth from transporting entire large Access or Quickbooks databases.  US team then uploads to google spreadsheets and links these to the wiki.

Strengths

-the Access database is feasible and appropriate to our setting

-the exporting of simple files is easy and efficient, particularly given our bandwidth limitations

-the wiki provides an excellent format for internal and public dissemination

Next Steps

-need to better incorporate key process and outcomes measures into the maternal database

-need to lock out all the VDC/ward information so that there are no data entry mistakes for our geospatial aspects

-better outcomes/performance data for laboratory

-develop spatial mapping plan

-construct intranet for syncing laptops

-create an "evernote.com for data" system whereby achham can create data offline but then sync to online database at low BW for US

Immediate Tasks

Solidify roles and start initial projects, with the understanding that these roles will evolve.  Key is to have some initial circumscribed, tangible 

Data Management: Jennifer

Demography: Dennis

Spatial Mapping and Device Management: Pon-Pon

Communications: Piali

Jennifer: learn the budget data system, and upload next month's data when it comes

Jennifer: learn about HMIS, and upload next month's data when it comes

Jennifer: learn about the ultrasound registry, and upload next month's data when it comes

Aditya: transition from msupply to Access for pharmacy database

Aditya: when new router comes, implement the new sync system across all the laptops

Duncan: develop the new pharmaceutical analysis protocol

Dennis: perform audit of Nyaya's data strategy for maternal health and identify key aspects for improvement, particularly with respect to public health outcomes measures.  Develop, with input from the rest of the team, a data vision statement of where Nyaya can go in the next few years.  

Dennis: review and perform basic calculations for our public health processes and outcomes measures (focus first maternal health)

Pon-pon:  investigate geospatial mapping strategies, outcomes, and next steps

Chhitij: CHW program data entry and transmission of updated database

Aram: patient confidentiality issues/vision