Document training_options for the udpipe engine in step_tokenize()

NEWS.md

@@ -12,6 +12,8 @@
 
 * Added a vignette, "Using textrecipes as a sparse matrix engine", documenting which steps produce sparse data and how to use them. (#302)
 
+* The documentation for `step_tokenize()` now states that `training_options` also applies to `engine = "udpipe"`, where it must contain the loaded `model`, and includes an example. (#237)
+
 ## Bug Fixes
 
 * All steps now work with 0 and 1 row data in `bake()`. This fixes errors in `step_texthash()`, `step_dummy_hash()`, `step_lda()`, and `step_pos_filter()` with 0 rows, and zero-length tokenlists no longer lose their lemma and pos attributes, which broke `step_lemma()`. (#290)

R/tokenize.R

@@ -9,8 +9,10 @@
 #' @template args-trained
 #' @template args-columns
 #' @param training_options A list of options passed to the tokenizer when it is
-#'   being trained. Only applicable when `engine = "tokenizers.bpe"`. Note that
-#'   the argument `x` should not be passed here. Defaults to `list()`.
+#'   being trained. Only applicable when `engine = "tokenizers.bpe"` or
+#'   `engine = "udpipe"`. For `engine = "udpipe"` the list must contain `model`,
+#'   a loaded udpipe model as returned by [udpipe::udpipe_load_model()]. Note
+#'   that the argument `x` should not be passed here. Defaults to `list()`.
 #' @param options A list of options passed to the tokenizer. Note that the
 #'   argument `x` should not be passed here. For
 #'   `engine = "tokenizers.bpe"` these options only apply when the tokenizer
@@ -166,6 +168,22 @@
 #'
 #' * "words"
 #'
+#' The udpipe engine requires a pre-trained udpipe model. Load one with
+#' [udpipe::udpipe_load_model()] (use [udpipe::udpipe_download_model()] to
+#' download one first) and pass it as the `model` element of the
+#' `training_options` argument.
+#'
+#' ```{r, eval=FALSE}
+#' model <- udpipe::udpipe_load_model("english-ewt-ud-2.5-191206.udpipe")
+#'
+#' recipe(~ text, data = text_tibble) |>
+#'   step_tokenize(
+#'     text,
+#'     engine = "udpipe",
+#'     training_options = list(model = model)
+#'   )
+#' ```
+#'
 #' ## custom_token
 #'
 #' Sometimes you need to perform tokenization that is not covered by the