Flexible modelling of survival curves for censored data

For survival data with observed failure times t ₁, t ₂, t ₃, … t _m , let f(t) be the probability density function and F(t) be the cumulative probability density for survival time, then the survival curve is given by 1 − F(t). The problem is now to find an estimated F(t) for a given data under censoring. The problem of choosing f(t) and F(t) is solved by using generalized lambda distributions and the estimation of the parameters of f(t) and F(t) is achieved using one of the three methods described in Estimation Algorithms for Survival Data section. A brief introduction to generalized lambda distributions is given below.

The probability density function of FMKL GLD when either λ ₃ or λ ₄ or both are equal to zero can be easily derived and is not provided here. For FMKL and RS GLD, the numerical solution of u in F ^− 1(u) = t gives the cumulative density function F(t), since u = F(t). This is usually obtained using the Newton–Raphson method (see GLDEX package (Su 2007) in R). Once the corresponding u is found for a given t, the probability density function f(t) for RS and FMKL GLDs can be derived using (2) and (7) respectively.

Once a parametric model for survival curves is found using any of the above method, the confidence intervals for survival curves can be evaluated directly without using simulation owing to the work by Cramer (1963) and Su (2009). Detailed descriptions and performance of this method can be found in Su (2009).

Note that \( {\displaystyle {\int}_0^{x_0}g(x)dx={\left.\beta \left(w+1,n-w\right)\right|}_0^{F\left({x}_0\right)}} \) where β is the Euler’s incomplete beta function normalized by the complete Beta function. This procedure is known as the analytical-maximum likelihood GLD approach in Su (2009).

For illustration purposes, the cancer dataset from the survival library in R is used. The survival curve is estimated by RS GLD (λ ₁ = 4.746579e + 01, λ ₂ = 7.504002e-04, λ ₃ = 7.166621e-03, λ ₄ = 3.432945e-01) using SR method. To visually check the validity of the evaluated 95 % confidence intervals (CIs), 1000 simulated survival curves (grey area in Fig. 3) were generated along with the parametric CIs in Fig. 3. This is further validated by the almost indistinguishable result between the estimated CIs from simulated data and the parametrically evaluated CIs (Fig. 3). Extensive simulation studies on the performance of CIs in terms of coverage probability for different sample sizes and quantiles are covered in Su (2009).

In the absence of full information, the quality of parametric modelling can be visually examined by comparing the fitted parametric model with a non parametric survival curve such as the Kaplan Meier (KM) survival curve. In the case where the SR method is used, it is possible to compare the final model against the simulated data using QQ plots and more formally through Kolmogorov-Smirnoff test or Kolmogorov-Smirnoff resample test (Su 2007b, 2007a, 2010a, 2010b).

The use of direct MLE and PWM matching for censored data has been applied using well known distributions such as log Normal and Weibull distributions (Wang et al. 2010a). The theoretical justification for using MLE and PWM to fit any continuous distribution to data can be found elsewhere (Hosking 1990, Fisher 1922, Aldrich 1997). For the SR method, as long as the simulated data is sufficiently close to the true underlying distribution and the GLD can model the simulated uncensored data accurately, it will yield an accurate model.

Unlike many standard statistical distributions, GLDs are characterised by inverse quantile functions. This means to use ML under SR or MLE directly, it is necessary to get the probability density functions using numerical methods such as the Newton–Raphson procedure. Other methods such as PWM do not require this numerical step in the optimisation algorithm. However, all fitting methods are affected by the sudden shape change problem in GLD parameter estimation. For relatively small change in one of the four parameters, GLD can exhibit a dramatic change in shape. For example, RS GLD (0,1,0.5,1) is an increasing function from -1 to 1 but RS GLD (0,1,0.5,0.75) is a parabola shaped function from -1 to 1, even though the change in the fourth parameter is only 0.25. In other situations, RS GLD (0,1,1.5,2) and RS GLD (0,1,1.5,2.5) (with a 0.5 change in the fourth parameter) do exhibit similar shapes and there is a smoother transition in shapes as the fourth parameter changes. This property means the standard theory examining the lower bound for the variability of parameters is not particularly useful for GLDs. The theoretical examinations of fitting methods for GLD are drawn out by numerical computations, potential abrupt changes in shape of distributions from small changes in parameters and perhaps the most elegant strategy at the present time is to use simulations to compare between the methods. Instead of comparing whether the parameters of fitted GLD are close to the true GLD, the emphasis is on whether the fitted GLD quantiles are sufficiently close to the true quantiles from some other known statistical distribution. This is illustrated below.

To assess the performance of various estimation methods, survival curves were generated from 2000 observations from symmetric and skewed Normal distribution with parameters: location = 20, scale = 2, shape = -5 or 5 or 0 for left skewed, right skewed and symmetric shape respectively. The motivation for using skewed Normal rather than Weibull distribution is to facilitate a better comparison across different scenarios using the same distribution with different parameters. Also, it would be rather unfair comparison if one were to use extended Weibull distributions (exponentiated Weibull and four parameter Weibull) to fit Weibull distributed data. Instead, the primary focus here is to examine how well the GLDs and extended Weibull distributions fit data from other distributions, since the true distribution is never known in practice. Additionally, Gumbel distribution (an extreme value distribution) with location parameter 15 and scale parameter 5 is also used in this comparison. These distributions are primarily chosen to examine the behaviour of these fitting algorithms over a range of different shapes.

This entire process is repeated for 200 observations to allow assessment of effect of sample size on the performance of proposed fitting methods. To create right censored data, observations greater than quantile ranging from 0.5 (median) to 0.9 are censored. Similarly, to create left censored data, observations less than quantile ranging from 0.1 to 0.5 (median) are censored. Five estimation methods: ML (maximum likelihood)/LM (L moments)/QS (quantile matching) under SR and MLE and PPWM matching were applied over 100 simulation runs. When fitting RS GLD using MLE with half of the data being censored (i.e., at 0.5), sometimes it is desirable to use the SR-ML method to generate the initial values to start the optimisation process, rather than using randomised search as it would lead to a better performance. This strategy is used in this article.

To give reader an idea as to the degree of accuracy attained by GLDs in comparison to other distributions, this article also assesses the performance of the exponentiated Weibull distribution (Mudholkar and Srivastava 1993, Mudholkar et al. 1995, Singh et al. 2005) and the four parameter Weibull distribution (Wahed et al. 2009, Jeong 2006) under the same simulation scenarios. Maximum likelihood estimation via Newton–Raphson algorithm is used to fit both distributions to survival data. The set of initial values used to begin the optimisation process is obtained as follows: 1000 initial values from 0 to 100 are randomly using Sobol sequence generator for parameters of both distributions. From these 1000 set of values, the set of initial values that maximises the likelihood is used in the optimisation process.

The sample size 200 and 2000 were chosen to reflect that the number of patients in many Phase III and IV trials are in the vicinity of 200 and some large meta-analysis may combine several studies and reach around 2000 patients. The primary intention of 200 and 2000 sample size is to allow comparison as to the accuracy of the estimates as sample size increases. The general pattern of improved accuracy and numerical precision for larger sample sizes is seen from Figs. 4, 5, 6 and 7 and this is an expected result.

To compare the performance between methods, the relative error was computed. The relative error is defined as the absolute difference between fitted and true quantile divided by the true quantile. This is computed using 100 equally spaced quantiles from 1 % quantile up to the censored quantile for right censored data. For left censored data, this is computed using 100 equally spaced quantiles from the censored quantile up to the 99 % quantile. The log mean and log variance of the relative error among five estimation methods for different types of censoring and different statistical distributions are shown in Figs. 4, 5, 6 and 7. The log transformation is designed to solve the problem of extreme results, to ensure a fairer and clearer comparison across different methods.

Within Figs. 4, 5, 6 and 7, the emphasis is on the performance of different methods. In Fig. 4, it is clear that the exponentiated Weibull is among the worst performing distribution except with respect to fitting of Gumbel distribution data and the four parameter Weibull has fairly comparable performance with GLD but performs less well for Gumbel data. The precision comparison in Fig. 5 indicates similar conclusion as Fig. 4 and the general pattern in these two figures is reflected in Figs. 6 and 7 but with added variability.

Within GLD methods, Figs. 4 and 5 show that the most accurate methods appear to be MLE and PWM matching while quantile matching, ML and LM under SR method appear to be more variable in a number of cases. This is expected as ML/LM/QS under SR introduce extra variability through simulation, due to the nature of the SR algorithm in this article. However, this is not always true. The advantage of using SR is seen in Fig. 6 for Gumbel Distribution with 10 % left censored data (GB-LC0.1) for FMKL GLD PWM and FMKL GLD under SR-LM. It is clear that in this example, the direct use of PWM does not result in a fitting result as good as using SR-LM. This is likely due to the difficulty in ascertaining a suitable set of initial values to ensure proper convergence to find the best possible GLD fit, an area where SR can provide valuable guidance and input.

The log variability of relative error plot shows that the methods provided give quite precise results with direct MLE tends to outperforms PWM. ML/LM/QS under SR all have similar performance and often perform slightly worse than MLE or PWM (Figs. 5 and 7).

The perceived, generic pattern of superior performances of MLE or PWM over ML/LM SR should be interpreted with caution. There are also cases, as shown in the example above, where LM SR can in fact outperform from direct use of PWM. Also, note that the optimality of the simulation results comes from the fact that the true distributions are known and a single GLD is an adequate approximation to the true underlying distribution. In practical situations, the true underlying distribution is unknown and may require a mixture of GLDs. When dealing with mixture of GLDs, it is harder to fit a distribution to censored data using direct MLE or PWMs, since this requires maximising or minimising a much more complex objective function which can be difficult in practice. The ML/LM/QS under SR, on the other hand, can be adapted more easily fit mixture of GLDs and the success of these methods in fitting mixtures has already been documented elsewhere (Su 2007b, 2010a, 2010b). SR also tends to give better model for empirical data, as illustrated in Application in empirical data modelling. The main message is that the theoretical loss of efficiency and accuracy using SR is likely to be minimal as evident in these simulation studies but as illustrated below, SR can provide additional information to aid the fitting of a suitable distribution which is not attainable by using an estimation method such as PWM or MLE directly.

While the methods described in this article works well so far in controlled, simulation experiments, it is the successful modelling of survival curves for real life data set that will be most useful for practitioners. The European Blood Marrow Transplant (EBMT) registry (2205 patients) and Amsterdam Cohort Studies on AIDS infection data (329 participants) from Putter et al. (2007) are used for this illustration.

For EBMT data, the aim is to model the survival curve for those who experienced platelet recovery and subsequently died or had a relapse. For this dataset, both MLE and SR are used to estimate the survival curve. From Fig. 8 panel A and B, the SR estimation is better than using MLE. While the SR estimation is based on mixture of two GLDs and MLE is based on a single GLD, this comparison is not unfair. Since when we fit distribution to data using MLE with censored data, we have no information as to the shape of our target distribution. While it may be possible, for example, to use mixture of two GLDs with MLE to model censored data, this method will often fail because the likelihood often becomes too complex to maximize. In contrast, by using the SR method, users can clearly see that mixture of GLDs is needed in this case (Fig. 8-panel C). Also, it is much easier to fit a mixture of GLDs under the SR scheme since it is only necessary to maximize the usual uncensored likelihood. In practice, the additional information provided by SR method on the shape of the survival distribution to assist the identification of an appropriate probability density is what gives SR method an edge over direct use of PWM or MLE on censored data.

For Amsterdam Cohort Studies on AIDS infection, because there is competing risk (see Putter et al. (2007) for more details), the appropriate measure of probability is to use the cumulative incidence function rather than the naive KM curve. The SR-ML method is again used in this example and the resulting fit is very close to the non-parametric cumulative incidence function as shown in Fig. 8-panel D.

A further interest with the EBMT data in this article is to illustrate how the GLD fits could be improved and compared against well-known parametric models using mixture of Normal distributions. We consider three method in this example: fitting mixture of 3 RS GLDs (Su 2007a, 2010b) under SR scheme, fitting mixtures of Normal distributions (Fraley and Raftery 2002, 2007) under SR scheme, and fitting mixture of Normal distributions to censored data directly using Bayesian method (Komárek 2009). A fourth method, kernel density estimation (using Normal distribution as kernel), is provided as a benchmark as to how well these methods fit the target distribution in Fig. 9.

The survival curve fit to EBMT data is first refitted using mixture of 3 RS GLDs using SR method. Details on fitting mixture of GLDs can be obtained from Su (2007a, 2010b). This GLD mixture model (fitted using quantile matching method (Su 2010b)) now corresponds almost exactly to the survival curve (Fig. 9, (B)). Previously, the use of mixture of 2 RS GLDs in Fig. 8 (B) shows a slight departure (but still within the 95 % CI) at the start of survival curve. Note the number of GLDs to be chosen can be selected using AIC. For example, in this case, the AIC under mixtures of 3 RS GLDs give AIC is 179658.8 compared to 175950.5 under mixtures of 2 RS GLDs. The preference therefore, based on AIC, is to use mixtures of 2 RS GLDs.

In Fig. 9, the EBMT data was also fitted using Normal mixtures. Firstly, MCMC estimation of Normal mixtures for survival data with censoring is applied directly using 5 Normal distributions via mixAK package in R (Komárek 2009). Secondly, Normal mixtures is fitted directly using the SR method via mclust package (Fraley and Raftery 2002, 2007) in R. This involves finding the optimal Normal mixture using BIC under the EM algorithm by hierarchical clustering and a mixture of 8 Normal distributions was fitted onto the simulated data using the SR method. Figure 9, panel A shows that the mixAK Bayesian Normal mixture model failed to estimate the survival distribution accurately for survival times greater than 2500 in comparison to kernel density estimation. As a result, this model is not considered further in Fig. 9 panel (b) and (c). Fig. 9 shows that the GLD mixture model not only provides a compelling fit to the KM survival curve (Fig. 9, panel b) but also has the advantage of possessing a simpler shape and less parameters than the Normal mixture model (Fig. 9, panel a and c).

A common goal in multi-stage disease modelling is to find an overall survival curve, accounting for all different paths to a final outcome. Lo et al. (2009) discussed how this can be done using saddlepoint approximation in a semi-markov process setting, however, the problem of choosing a suitable statistical distribution is still unsolved. Their method also hinged upon the successful maximisation of the likelihood for the whole system; which can be difficult to achieve for a complex multi-stage disease scenario.

The method proposed below can be used quite effectively for complex multi-stage disease scenarios. Consider a simple multi-stage disease scenario in Fig. 10.

Figure 10 shows two possible paths leading to death for each patient in a clinical trial. Patients could develop a disease leading to death or go straight to death. There are two pathways leading to death and censoring could occur any time.

The above modelling strategy is simple yet effective. It avoids the problem of having a complicated likelihood in the event of a complex multi-stage system as was used in Lo et al. (2009) and it provides a visual check as to the overall goodness of fit between the estimated parametric survival curve and the non-parametric survival curve for the whole system. To illustrate, consider a simulated multi-stage disease modelling in Fig. 10. Let path 1 be the path with an intermediate event before death and path 2 be the path straight to death. One thousand survival times for path 1 and 2 are generated from Weibull (6,12) and Weibull (3,13) distributions respectively. Survival times greater than 20 are censored at 20 for the whole system. Additionally, for path 1, survival time greater than 14 are censored at 14. Under this simulation scenario, the overall survival curves estimated by RS GLD using MLE and SR methods are shown in Fig. 11. While both methods appeared to be effective, a slightly more accurate result is obtained using the direct ML estimation technique in this case.

This article illustrates the use of generalised lambda distributions in conjunction with MLE, PWMs matching and SR method to find an approximate probability density function and confidence interval for the survival curve. In the event where the use of MLE and PWMs matching failed to give convincing result, the SR method is often a useful alternative. The SR method converts the censored survival data into uncensored data, allowing users to improve the distributional fit using a wider range of fitting methods. The SR method can also provide initial values for a secondary optimisation for MLE and PWM matching, which sometimes provide a better fit to survival curves that is not attainable using only one method.

The development of these techniques is promising as it means statisticians are no longer limited to non- parametric techniques when analysing such data. Also, it is now possible to extract all the statistical information such as mean, quantile, variability in relation to survival times consistently under one parametric model and this opens up the prospect of developing more powerful statistical models and tests for censored survival data frequently used in engineering and medicine. Recent advances in GLD regression (Su 2015) also opens the possibility of extending this work into accelerated failure models, which will further enhance statistician’s toolbox in practice.