Hadley Wickham: Managing many models with R

What people said

Strong rec Shorpy · 2017-08 · Stack Overflow · on “run multiple model and save model comparison results in dataframe in r”

This is a really good application of the tidyr::nest() function in conjunction with purrr and broom . What you do is: - Group the data frame - Apply a model with mutate(mod = map(data, model) - summarize the model using broom::tidy() - extract the relevant statistics. For more on this here's a great talk by Hadley on the subject: [this video] In your case I think you can do something like this: library(tidyverse) library(broom) diamonds %>% group_by(cut) %>% nest() %>% mutate( model1 = map(data, ~lm(price~carat, data=.)), model2 = map(data, ~lm(price~carat+depth, data=.)) ) %>% mutate(anova = map2(model1, model2, ~anova(.x,.y))) %>% mutate(tidy_anova = map(anova, broom::tidy)) %>% mutate(p_val =

Recommends aosmith · 2016-07 · Stack Overflow · on “advice on Usage of dplyr:: do vs purrr: map, tidy::nest, for predictions”

Package broom's `augment` is useful for adding the residuals to the dataset used to fit the model. For making predictions with the test dataset you can use `predict` in `map`, using the `newdata` argument. If you haven't already seen it, you might also be interested in this talk by Hadley Wickham [this video] that goes through an example of fitting many models with dplyr/tidyr/purrr/broom.

Recommends roman · 2016-10 · Stack Overflow · on “How to calculate the slopes of different linear regression lines on multiple plots in R”

rawr [LINK] has a perfect answer for you in the comments but in case you're interested... If you are doing reasonably complex stuff to subsets of your data then its well worth checking out the group_by() %>% nest() %>% map() workflow in R. I will demonstrate with a simple example: library(dplyr) library(tidyr) library(purrr) data(iris) these are some functions that you want to apply to your subsets. doModel <- function(dat) lm(Sepal.Length ~ Sepal.Width, dat) getSlope <- function(mod) coef(mod)[2] You can apply them as follows. models <- iris %>% group_by(Species) %>% nest %>% mutate(model = map(data, doModel)) %>% mutate(slope = map(model, getCoef)) models$slope What happened is that you cr

Recommends simitpatel · 2017-07 · Stack Overflow · on “A function to create multiple models based on column value in R”

i would recommend watching this video from hadley wickham [this video] when you have the time. it relates very much to your challenge. this also seems like a classic split-apply-combine problem, so my first thought is to consider the tidyverse . here is some code that might help you: library(tidyverse) library(randomForest) df2 <- df %>% group_by(cl) %>% mutate(rfcol=list(randomForest(x=., formula=.$cl~.$Work+.$Age))) basically a new column has been created that contains the randomforest algorithm appropriate for that row based on its value in cl . you can explore the details of each model by looking at df2$rfcol[[2]] to summarize what's going on, the group_by function gets you started with creat

Recommends ohnoplus · 2017-12 · Stack Overflow · on “purrr with linear model in R: invalid 'envir' argument of type 'character'”

It helped me to look at this post [LINK] , and this video [this video] to understand how to best use purrr and broom together. As G. Grothendeik points out, I can add a column with models to a data frame (where each cell is a full model). The way to do this with the map function is duneJ %>% group_by(Species) %>% nest %>% mutate(Mod = map(data, my_lm0)) -> test Here, nest is a key function that makes a column that is a list of data frames, each of which contain the data about each species and saves it to a default column named "data". I run map inside of the mutate funciton to save the models to yet another column, where each cell is a new model. If I want to look at model results, I can combine

Recommends Dom · 2018-03 · Stack Overflow · on “Forcing the use of a for loop with group_by and mutate()”

The thing that you're missing in your dplyr pipe is `purrr::map`. Can I suggest that you check out this video of Hadley explaining how to solve exactly this problem while you wait for potential answers to be posted below: [this video] Slide deck here: [LINK]

Cites danilinares · 2016-05 · Stack Overflow · on “dplyr, do(), extracting parameters from model without losing grouping variable”

Using the approach of Hadley Wickham in this video [this video] : library(dplyr) library(purrr) library(broom) fitmodel <- function(d) lm(mpg ~ disp, data = d) by_cyl <- mtcars %>% group_by(cyl) %>% nest() %>% mutate(mod = map(data, fitmodel), pars = map(mod, tidy), pred = map(mod, augment)) pars <- by_cyl %>% unnest(pars) prediction <- by_cyl %>% unnest(pred)

Cites r2evans · 2016-06 · Stack Overflow · on “Import a .dat file from CDC mortality 2003 into Rstudio”

Regardless of the file extension, it's merely a text file (88MB, with 2.4M rows). On inspection, it looks fixed-width (at 430 chars wide, a bit wide to show here, though your picture does it justice). The only "trick" to reading in FWF files is to find the column widths. In some "dat" files, this is well-defined/documented, but it can often be inferred by looking at the first few lines. In this case, I just counted characters. (I only grab the first 100 lines for brevity.) Some of the fields appear to be space-delimited. That is, some (even most) of the rows had a space between two groupings but some rows had none. Since I don't know how to decode the logic, I keep those together (with the s

Cites Rentrop · 2016-12 · Stack Overflow · on “How to create for loop for multiple dataset”

There are multiple error here: 1. Lets start with the general for -loop structure: companies<-c("a", "b", "c") for(i in c){ What should c be here? c is not defined, so R interprets c as the function c (see ?c ) which i am sure you dont mean. So a for loop always needs something to loop through: a list, a vector... I guess you mean companies<-c("a", "b", "c") for(i in companies){ 2. How to address the data/companies : You try to get the data by [i] . This is not meaningful to R . Normally [ is used for subsetting as you correctly do using [4] to address the fourth column. I guess what you want is to get the global variable named "a", "b" or "c" as you defined in companies . To do this use get

Cites user8229029 · 2020-10 · Stack Overflow · on “How to extract the slope coefficient from tidy data frame made with Broom Package R”

I'm trying out the tidy methods that Hadley Wickham describes in this video: [this video] . It's straightforward to get certain statistics this way, as long as there is only 1 row in the tidied data frame, but the slope of each linear model is buried in the 2nd row of a tidied data frame made with Broom. My code is very similar to Hadley's, and looks like this. library(tidyverse) corn_by_county <- corn_final_long %>% group_by(County) %>% nest() # define & run linear models for each county corn_county <- function(df){ lm(Yield ~ Year, data = df)} corn_models <- corn_by_county %>% mutate(model = map(data, corn_county)) corn_output <- corn_models %>% mutate(tidy = map(model, broom::tidy), glance = m