The Future of Children, Princeton - Brookings: Providing research and analysis to promote effective policies and programs for children.

Assessing the Effects of Managed Care: Methodologic Considerations

Although there are a large number of studies which report the impact of managed care on the cost and quality of health care, it has been particularly difficult to measure accurately the effects of managed care. Difficulties have been encountered because of inadequate research design, limited data availability, and problems in measurement. In addition, differences in the perspective from which an analysis is developed frequently result in studies that address easily answered questions rather than questions which are important. In this Appendix, we review some methodologic issues that limit the utility of much of the research on managed care and children. We specifically focus on issues of research design, perspective, and measurement.

Research Design: In evaluating managed care plans, the fundamental challenge is to answer the question of whether outcomes are better or costs are lower than what they would have been had the people enrolled in managed care remained in a traditional fee-for-service plan. Accordingly, a study which aims to determine the impact of managed care must control for all of the factors which also might have caused the observed effect in order to isolate the true impact of managed care. There are, in general, two different techniques for making this kind of "all other things equal" comparison, experimental and quasi-experimental designs. Experimental designs, regarded as the “gold standard,” involve randomly assigning some individuals in a group to managed care and the others to traditional fee-for-service. Because people are randomly assigned to managed care or fee-for-service, it is assumed that the two groups do not differ in characteristics likely to affect the outcomes of interest (use, access, cost, health status). Accordingly, observed differences between the groups can be ascribed to the impact of managed care.

Randomized designs often are not practical because they need to be planned and implemented prior to enrolling people in managed care. Frequently, however, there is interest in discerning whether managed care has affected outcomes, cost, or quality after managed care has been introduced. In this instance, it is too late to randomize, and quasi-experimental approaches must be utilized. Using quasi-experimental designs, some studies of managed care look at groups of individuals who are new to managed care and have left fee-for-service. Studies of this kind determine the impact of managed care on outcomes such as use of services, costs, and health status by looking at the experience of the newly enrolled populations both before managed care and during it. If nothing else has changed, then any differences may be interpreted as effects of managed care.

Another technique may be employed where individuals have already been in managed care for a period of time and there is interest in determining how the program is working. Then the strategy for finding a control group changes; these individuals can no longer be compared to themselves prior to managed care. The tact often adopted is to choose a comparison group from another place (such as the same employer but a different locality, or in the case of Medicaid, a different geographic area) that is extremely similar in its health status and sociodemographic status but is not currently enrolled in managed care. If it can be shown that the two groups are very similar except for their enrollment in managed care, then a comparison of the two groups may indicate the effects of managed care.

The primary difficulty with quasi-experimental studies is that often it is difficult or even impossible to find a group of individuals for whom nothing else has changed except their entry into managed care. For example, it is possible that, had the newly enrolled group remained in fee-for-service, they would have been subject to a new type of utilization review introduced by the fee-for-service plan. This new utilization review program might also have reduced costs. If this is true, then a before-and-after study design may overstate the difference between managed care and fee-for-service because fee-for-service itself has changed. In fact, the fee-for-service world is a moving target, and it is often extremely difficult to select the appropriate comparison group.

Similarly, in the second scenario, it may be difficult to identify a group whose sociodemographic profile is truly identical to the managed care group. Statistical techniques may be used to make the populations look the same before drawing inferences about the impact of managed care, but these techniques are not always successful in accounting for all intergroup differences. Overall, the literature on the effects of managed care is weak, casting doubts on the findings because there are few randomized studies and because, in quasi-experimental studies, investigators rarely are able to select or do not select the appropriate control groups.

Perspective: Evaluation of managed care to date frequently has been further weakened by a lack of clarity in the perspective of the analysis. The analysis of the costs of care provides an illustration. To determine whether managed care controls costs and saves money, one needs to ask for whom—the employer, state or federal governments, the managed care entity, the enrollee, or society as a whole? Payers such as employers and state and federal governments typically view costs as what they expend on health insurance premiums for the populations they insure. Enrollees see costs as the dollars subtracted from their paychecks for insurance premiums, paid in taxes to support public programs, or paid out of pocket for their own care. To managed care entities, costs represent the value of the resources consumed in caring for the enrolled population. However, it is much easier to compute expenditures from the payer's viewpoint than to figure out what it really costs managed care companies to deliver health care. As a result, investigators often present data on expenditures when it is the resource costs of health care that may be of primary interest to policymakers. Expenditures may be appropriate if all the analyst cares about is whether a payer is saving money by paying out less than what was paid out last year or less than what would have been paid out under an alternative health plan. If, however, one wants to know whether managed care is more efficient than fee-for-service or if managed care companies make excess profits, then it is necessary to get true cost data. These are often impossible to obtain and difficult to interpret once they are acquired. Managed care plans are hesitant to release their cost data for fear of jeopardizing their competitive advantage. Also, cost data often are in the form of audited financial reports. Because accounting methods have not yet been standardized, there is much variability in these reports which makes comparing different managed care entities extremely precarious.

Measurement: As with cost, available data often are inadequate to answer important questions about managed care. This is particularly true with respect to questions regarding the impact of managed care on health status. For example, there are many global measures of health status available now, but such information is usually not available from managed care plans because they do not collect it for their own purposes. In addition, it is very difficult to demonstrate changes in health status for children because most children are healthy and because factors other than health care may affect children's health status more than choice of health plan. Because data on health status are not available and changes in health status are difficult to detect, researchers frequently use information available from encounter forms on what was done or not done for patients with a specific complaint. Guidelines are available for many aspects of health care in pediatrics and obstetrics, and these guidelines may be used to measure how levels of care achieved under managed care systems compare with care in other systems. The assumption is made that, if patients receive appropriate care, good outcomes will result. A limitation of this approach is that the evidence underlying guidelines may be weak so it may be hard to know whether improved adherence really improves outcomes.

An additional problem relates to the unit of observation. Most analyses of the impact of managed care require access to data on individuals, both enrollees and patients, and their resultant use of services. However, data from the traditional fee-for-service market frequently are available only in the form of claims paid. Transformation of information on claims into data on encounters is tricky, yet this is required to make comparisons with managed care. On the managed care side, individual data often are not collected or made available to researchers; only aggregate information on the plan as a whole may be available. Similarly, claims that are denied, for whatever reason, and claims for out-of-plan use in managed care are usually not recorded. Thus, information is lost on out-of-plan use of health care. As a result, potentially important information on health care needs that may not be met by either managed care or traditional fee-for-service is unavailable.