Measuring later health status of high risk infants: randomised comparison of two simple methods of data collection

Interventions

The essence of both interventions was that a small amount of clerical time was funded to enable follow up of eligible babies by using a core protocol and standardised data collection materials but with considerable flexibility to tailor activity to local circumstances. The study was coordinated on a regional basis, with one coordinator in Trent and one in Wessex. To prevent the coordinators becoming part of the intervention, their role with clerks was restricted to training, developing the local system, and, thereafter, triggering requests for information.

Parental assessment intervention—Before the infant was discharged from the neonatal unit, the clerk based in the unit asked parents if they would take part in the study. When agreement was obtained, the clerk noted basic demographic and clinical details about the baby6 and provided parents with supplementary pages to insert into their child health record. These pages contained forms for parents to complete with their assessment of the child's health at 2 years (corrected for gestation). The parents kept the child health record, which was marked with a study sticker, and the baby's general practitioner and health visitor were notified of the family's participation in the study. After discharge the unit clerk kept occasional informal contact with parents through birthday and Christmas cards to the child. On each contact, a reply paid card was enclosed for the parents to inform the clerk of receipt and any changes of address. At 2 years corrected age, the clerk wrote to the parents, inviting them to complete and return the supplementary pages in the child health record. Parents were free to ask for help in doing this, including help from their general practitioner or health visitor.

Community review intervention—Before the infant was discharged from the neonatal unit the clerk obtained consent and basic descriptive information, as described above. This information was then passed to a designated clerk in the community child health department. Over the next two years, the community based clerk developed and maintained a file of information on each child, using whatever sources of clinical information were routinely available. These might include hospital discharge summaries, information from routine child health surveillance, and outpatient letters. If the family moved out of the area, the clerk tried to obtain appropriate information from services in the new area. When the child reached 2 years corrected age, the clerk was asked to collate and review the information and complete a customised form to record equivalent items of information to those obtained by parental assessment.

Procedure

Although our primary aim was to test two different methods of data collection, we thought it was necessary to show that the items of information we collected were valid. To do this, we selected a 10% sample of children in each method by taking every child recorded as having a major health problem at 2 years and every fifth child who was reported as not having a major health problem, starting with births from 15 April 1997, until a quota of 47 children had been achieved. These children were visited at home within six weeks of their 2nd birthday (corrected age) by one of the study coordinators (either a health visitor or a neonatal nurse). The coordinator, who had not seen the information already recorded for the child at 2 years, reviewed the same areas of development by interview of the parents and assessment of the child. She then checked for any discrepancies between her information and that obtained in either intervention arm and attempted to obtain further information from the parents to resolve discrepancies.

We chose a sample size of 233 children in each arm of the trial to give 90% power, at the 5% significance level, to detect a 15% difference between the two groups in the proportion of children having outcome data at 2 years corrected age. Previous work with routine systems has shown that outcome data are available for about 80% of 2 year old children.8 We hypothesised that in this investigation one method might be similar to this level and the other might show an increase to 95%. We considered 95% as the target in terms of achieving a representative sample of children at 2 years.9 The calculation of sample size was based on the conservative assumption of four clusters within each arm and a maximum intraclass correlation of 0.025.10

To ensure the accuracy of the database we used a double data entry method. Data were stored on an Access database and analysed with SAS, version 8.0. We carried out a pragmatic analysis for the trial with a primary outcome of “an ability to ascertain the overall outcome of a child (at 2 years corrected age) into one of two categories: normal or severe functional loss.” To allow for the cluster randomised design we calculated the proportion of positive responses for each outcome of interest at each hospital and used the results as summary statistics for each hospital. We then compared the two groups using these summary statistics by weighted two sample t tests.11

We also carried out a health economic appraisal. Resources used within the data collection process consisted primarily of staff costs but additional variable resource items, such as services of the Office for National Statistics and consumables, were also included. To assess the additional staff input required in the data collection process we carried out an exercise on workload for data collection in each of the three years that the study was running. Clerks were asked to keep a diary of the time they spent working on the study. Study coordinators completed a questionnaire on their time input to the study, separating out tasks specific to the study (research) and ongoing tasks that formed part of the data collection process.

We assessed the value of staff time using standard pay scales applied to ward clerks, community clerks, and health visitors. 12 13 An additional overhead of 24% was applied to the ward/community clerk pay scale to cover employment costs (employers' national insurance contributions and superannuation) and standard overheads (for example, office space and equipment).

We obtained ethical approval from the eight local research ethics committees relating to the participating neonatal and community child health services.

Accuracy

More of the parents' questionnaires had all fields completed compared with the community questionnaires (difference 18%, P=0.048). Of the 214 questionnaires returned by parents, only 51.4% of were returned, as requested, within six weeks of their child reaching a corrected age of 2 years. This compared with 71.2% in the community arm. Data from the community arm were often not current, with clerks having access only to information about the child from the last time he or she was seen. Comments about the child's abilities were based on information recorded before 18 months of corrected age in 28% of cases. Because of the timing of routine screening this was a particular problem in relation to vision. However, even after we excluded this item the proportion of variables reported on the basis of data before 18 months of corrected age was still 22%. Information was retrieved from various sources available to the clerks. However, about two thirds was obtained either by contacting the health visitor or by reviewing the records of earlier assessments held on the community database.

From data returned by parents we identified 26 children (11.0%) with severe functional loss in one or more fields, while for 105 (44.5%) there were no concerns at all. Information obtained by community surveillance produced figures of 20 children (8.6%) with severe functional loss in one or more fields and 98 (42.2%) with apparently normal health.

Validation

Study coordinators visited and assessed 47 children to determine whether health status had been correctly assigned. All 23 for whom information had been provided by parents were found to have been correctly allocated by parental report (eight had severe functional loss, 15 were normal). Of the 24 children assessed from their community records (10 with reported severe functional loss, 14 normal), five were found to have been incorrectly allocated by the community clerks. All five were children recorded as having severe functional loss when in fact they were normal. In three cases this occurred because the clerk completed the form with data about development that had been obtained when the child was much younger and the record had not been updated in relation to later progress. In two cases the records were ambiguous in terms of the child's health.

Financial analysis

The costs of implementing the two approaches were an average of £2271 per centre (range £1520-£3170) with the parental questionnaire and £3709 (£2430-£4047) with the community review. The variation by centre largely reflected the working practices of the clerks involved, with marked differences in the time spent on each case. The equivalent figures for average cost per case recruited were £37 (£25-£52) for the parental questionnaire and £61 (£40-£67) for the community review.

Past problems

The failure to make progress in this area seems to result from several factors. The initial focus of neonatal intensive care was on improving survival. Therefore, although outcome was seen as important, procedures to gain these data were not established in a systematic fashion. Where cohorts of infants have been reviewed in later childhood the approach has generally been to use a detailed series of tests to look for minor as well as major variations from normal. These types of assessment require the use of trained professionals and hence are costly.15 The community services in the United Kingdom perform health surveillance as part of their core activity and collect information about the health status of all children.16 Currently there is no routine system for identifying subgroups, such as infants who have required intensive care, to allow their data to be abstracted. More importantly there is no national protocol with regard to how and when children are reviewed by community services. Similarly the approach to documentation and recording varies widely.8

New opportunities

We wanted to identify a simple method of obtaining data about the later health status of this group of children, who are at high risk of long term neurodevelopmental problems (although the methods should apply to any other group—for instance, those below a certain birth weight). We chose a corrected age of 2 years as we thought this would provide feedback to clinicians in a time frame that was still relevant to their practice. The level of detail was selected to be informative for all interested groups while being cheap to collect.

Although we aimed to compare the two methods, we were aware that even if modest numbers of children were lost to follow up serious under-reporting of impairments could occur.9 Because of this, our target for either method to succeed was to achieve outcome data for 95% of eligible children. This did not occur, and hence overall outcomes (rates of normality and serious impairment) must be interpreted with caution, particularly in the community arm where ascertainment was based on only 67% of the infants recruited. While this is disappointing, there are positive elements to emerge from both methods.

Most parents clearly found the questionnaire easy to use and were able to provide accurate information, although we thought some misunderstood the concept of corrected age. Further refinement, particularly for when English is not the family's first language, may improve the response rate. While we suggested to families at recruitment that they could get advice from a health visitor or general practitioner when they completed the form, we did not try actively to involve these professional groups. If the 35 families who could not complete all questions or did not return the form at all had asked for help, the level of ascertainment could have risen to above our threshold for success of 95% but would have added considerably to the costs.

In relation to the community review some information was available for 94.5% of children. This high return was achieved because existing systems allowed the clerks to track children even when they moved districts. However, the systems for collecting and recording information were variable, and hence there were areas of development for which either there was no information or the information recorded could not be interpreted. Electronic patient records will provide an opportunity to deal with the issues of definition and recording of outcome. The timing of reviews and decisions about who should be reviewed need to be considered by the relevant professional groups as a matter of urgency. It is hard to identify value for money within the current system. Similarly it is hard to justify the extent of local variation given previous recommendations. 17 18

Financial context

Given the financial constraints of the NHS it is often difficult to convince commissioners of health care that expenditure on obtaining follow up data is justified. The cost of neonatal intensive care is around £1000 a day and about 1.5% of births in the United Kingdom are at ≤32 weeks' gestation. The additional costs associated with the methods used in this study (highest estimate £67 per case) are trivial by comparison and might lead to information on the success or otherwise of neonatal care, which at present can be measured only by early mortality.

With further refinement, both of the approaches tested in this study seem to have the potential to provide this information, but at present neither is adequate.