## Estimating survival rates

Here’s the set-up: looking at a data set of individuals’ lifespans you would like to infer the distributions—Analysing when people die, or things break etc. The statistical problem of estimating how long people’s lives are is complicated somewhat by the particular structure of the data – loosely, “every person dies at most one time”, and there are certain characteristic difficulties that arise, such as right-censorship. (If you are looking at data from an experiment and not all your subjects have died yet, they presumably die later, but you don’t know when.)

Handily, the tools one invents to solve this kind of problem end up being useful to solve other problems, such as point process inference.

So let’s say you have a a random variable \(X\) of positive support according to which the lifetime of your people (components, machines, whatever) are distributed, which possesses a pdf \(f_X(t)\) and cdf \(F_X(T)\).

We define several useful functions:

- The survival function
\[S(t):=1-F(t)\]

- the hazard function
\[\lambda(t):=f(t)/S(t)\]

- the cumulative hazard function
\[\Lambda(t) :=\int_0^t\lambda(s) \textrm{d} s\].

Why? Because it happens to come out nicely if we do that, and these functions acquire intuitive interpretations once we squint at them a bit. The hazard function will turn out to be the probability density for a death at time \(t\) given that one has not yet occurred. The survival function is the probability of an individual surviving to time \(t\) etc.

Using the chain rule we can find the following useful relation:

\[S(t)=\exp[-\Lambda (t)]={\frac {f(t)}{\lambda (t)}}\]

### Cox proportional hazards

A classic, in which we don’t care about the baseline rate, just treatment effects. We assume the following model for our data, with some measured predictors \(X_j\). The magical trick is that we cancel out the nuisance baseline hazard rate \(\lambda_0\) which is nice in medial application as its the thing we definitionally can’t change.

\[ λ(t) = λ_0(t)\exp(β_1X_1 +β_2X_2 +\dots+β_pX_p) = λ_0(t) \exp(β′X), \] The resulting partial likelihood is

\[ L(\beta)=\prod_{r\in D}\frac{\exp\beta'x_r}{\sum_{j\in R_r}\exp\beta'x_j} \]

It *seems* one could get a more general effects model than a basic linear link and have everything still work, but I won’t look into that here; my purposes for now involve also identifying the baseline rate and not necessarily treatment effects.

### Nelson-Aalen estimates

a.k.a. Empirical Cumulative Hazard Function estimator.

The original Aalen paper on this is notoriously beautiful because of clever construction of a life point process and associated martingale. Clear and worth reading. Spoiler, despite the elegant derivation, the actual estimator is something a high-school student could probably discover by guessing.

TBC.

## Other reliability stuff

Reliawiki has good stuff, e.g.comprehensive docs on the Weibull law. It’s in support of some software package their are trying to sell, I think?

## Refs

- LaOl81: Nan Laird, Donald Olivier (1981) Covariance Analysis of Censored Survival Data Using Log-Linear Analysis Techniques.
*Journal of the American Statistical Association*, 76(374), 231–240. DOI - SyTa00: Judy P. Sy, Jeremy M. G. Taylor (2000) Estimation in a Cox Proportional Hazards Cure Model.
*Biometrics*, 56(1), 227–236. DOI - Pete77: Arthur V. Peterson (1977) Expressing the Kaplan-Meier Estimator as a Function of Empirical Subsurvival Functions.
*Journal of the American Statistical Association*, 72(360), 854–858. DOI - Nels69: Wayne Nelson (1969) Hazard Plotting for Incomplete Failure Data.
*Journal of Quality Technology*, 1(1), 27–52. DOI - Scho03: Frederic Paik Schoenberg (2003) Multidimensional Residual Analysis of Point Process Models for Earthquake Occurrences.
*Journal of the American Statistical Association*, 98(464), 789–795. DOI - Nels00: (n.d.) Nelson - 2000 - Theory and Applications of Hazard Plotting for Cen.pdf.
- Aale78: Odd Aalen (1978) Nonparametric Inference for a Family of Counting Processes.
*The Annals of Statistics*, 6(4), 701–726. DOI - Hjor92: Nils Lid Hjort (1992) On Inference in Parametric Survival Data Models.
*International Statistical Review / Revue Internationale de Statistique*, 60(3), 355–387. DOI - LuGF12: W. LU, Y. GOLDBERG, J. P. FINE (2012) On the robustness of the adaptive lasso to model misspecification.
*Biometrika*, 99(3), 717–731. DOI - Cox72: D. R. Cox (1972) Regression Models and Life-Tables.
*Journal of the Royal Statistical Society: Series B (Methodological)*, 34(2), 187–202. DOI - SFHT11: Noah Simon, Jerome Friedman, Trevor Hastie, Rob Tibshirani (2011) Regularization Paths for Cox’s Proportional Hazards Model via Coordinate Descent.
*Journal of Statistical Software*, 39(5). - HjWL92: Nils Lid Hjort, Mike West, Sue Leurgans (1992) Semiparametric Estimation Of Parametric Hazard Rates. In Survival Analysis: State of the Art (pp. 211–236). Springer Netherlands
- Tibs97: Robert Tibshirani (1997) The Lasso Method for Variable Selection in the Cox Model.
*Statistics in Medicine*, 16(4), 385–395. DOI - Nels00: Wayne Nelson (2000) Theory and Applications of Hazard Plotting for Censored Failure Data.
*Technometrics*, 42(1), 12–25. DOI