tokenizers

R 패키지 메타데이터와 수집 신호를 모아 봅니다.

Packages / CRAN / tokenizers

tokenizers

v0.3.0

tokenizers

Repository CRANLicense MIT + file LICENSELifecycle activeNeeds compilation yes

Package page Reference Source DOI README NEWS

DOI

10.32614/CRAN.package.tokenizers

Task views

Natural Language Processing

Reverse imports

497

Core Signals

첫 화면에서 판단해야 할 수집 신호를 먼저 배치합니다.

Task views

Natural Language Processing

Reverse imports

497

Supported Backends

DESCRIPTION에서 감지한 backend 관련 package입니다.

backend package 신호가 없습니다.

Quick Facts

기본 메타데이터를 작은 카드와 토큰으로 압축합니다.

profile

Repository

CRAN

Version

0.3.0

License

MIT + file LICENSE

Lifecycle

active

Needs compilation

yes

Reverse imports

497

Last observed

2026-05-30

CRAN: cran.r-project.org/package=tokenizers

Build fields

LinkingTo

Rcpp

수집 소스별 패키지 정보

2개 소스

CRAN

0.3.0

2026-05-30

License

MIT + file LICENSE

Depends

R (>= 3.1.3)

Imports

stringi (>= 1.0.1), Rcpp (>= 0.12.3), SnowballC (>= 0.5.1)

Suggests

covr, knitr, rmarkdown, stopwords (>= 0.9.0), testthat

LinkingTo

Rcpp

Needs compilation

yes

Reverse imports

497

Lifecycle

active

Last observed

2026-05-30 10:45:11

R-universe

0.3.1

2026-05-30

Depends

Imports

stringi, Rcpp, SnowballC

Suggests

covr, knitr, rmarkdown, stopwords, testthat

Last observed

2026-05-30 11:26:48

이 패키지가 의존하는 패키지

5개 표시전체 9개

Package	Type	Spec
Rcpp CRAN · 0.3.0 · 2026-05-30	Imports	Rcpp (>= 0.12.3)
SnowballC CRAN · 0.3.0 · 2026-05-30	Imports	SnowballC (>= 0.5.1)
stringi CRAN · 0.3.0 · 2026-05-30	Imports	stringi (>= 1.0.1)
Rcpp CRAN · 0.3.0 · 2026-05-30	LinkingTo	Rcpp
covr CRAN · 0.3.0 · 2026-05-30	Suggests	covr
knitr CRAN · 0.3.0 · 2026-05-30	Suggests	knitr
rmarkdown CRAN · 0.3.0 · 2026-05-30	Suggests	rmarkdown
stopwords CRAN · 0.3.0 · 2026-05-30	Suggests	stopwords (>= 0.9.0)
testthat CRAN · 0.3.0 · 2026-05-30	Suggests	testthat
검색 결과가 없습니다.

1 / 2

이 패키지를 쓰는 패키지

5개 표시전체 18개

Package	Type	Spec
blocking 1.0.2 CRAN · 2026-05-30	Imports	tokenizers
covfefe 1.0.0 CRAN · 2026-05-30	Imports	tokenizers
deeplr 2.1.0 CRAN · 2026-05-30	Imports	tokenizers
DramaAnalysis 3.0.2 CRAN · 2026-05-18	Imports	tokenizers (>= 0.2.1)
pdfsearch 0.4.3 CRAN · 2026-05-30	Imports	tokenizers
proustr 0.4.0 CRAN · 2026-05-30	Imports	tokenizers
rslp 0.2.0 CRAN · 2026-05-30	Imports	tokenizers
rwig 0.1.0 CRAN · 2026-05-30	Imports	tokenizers
textrecipes 1.1.0 CRAN · 2026-05-30	Imports	tokenizers
tidypmc 2.0 CRAN · 2026-05-30	Imports	tokenizers
tidytext 0.4.3 CRAN · 2026-05-30	Imports	tokenizers
wactor 0.0.1 CRAN · 2026-05-30	Imports	tokenizers
WhatsR 1.0.6 CRAN · 2026-05-30	Imports	tokenizers
edgarWebR 1.1.0 CRAN · 2026-05-30	Suggests	tokenizers
llmshieldr 0.1.0 CRAN · 2026-05-30	Suggests	tokenizers
sumup 1.0.2 CRAN · 2026-05-30	Suggests	tokenizers
torchdatasets 0.3.2 CRAN · 2026-05-30	Suggests	tokenizers
quanteda 4.4 CRAN · 2026-05-30	Enhances	tokenizers
검색 결과가 없습니다.

1 / 4

Reverse dependency summary

3 types

Type	Packages
Imports	13
Suggests	4
Enhances	1

패키지 페이지

Reverse imports

Reverse suggests

Reverse enhances

All links

Repository: CRAN
Version: 0.3.0
Collected: 2026-05-19 20:35:12
Package page: https://cran.r-project.org/web/packages/tokenizers/index.html
DOI: 10.32614/CRAN.package.tokenizers
Citation: https://cran.r-project.org/web/packages/tokenizers/citation.html
CRAN checks: https://cran.r-project.org/web/checks/check_results_tokenizers.html
README: https://cran.r-project.org/web/packages/tokenizers/readme/README.html
NEWS: https://cran.r-project.org/web/packages/tokenizers/news/news.html
Reference HTML: https://cran.r-project.org/web/packages/tokenizers/refman/tokenizers.html
Reference PDF: https://cran.r-project.org/web/packages/tokenizers/tokenizers.pdf
Source package: https://cran.r-project.org/src/contrib/tokenizers_0.3.0.tar.gz
Archive: https://CRAN.R-project.org/src/contrib/Archive/tokenizers

In views

NaturalLanguageProcessing

Page fields

Author: Lincoln Mullen [aut, cre], Os Keyes [ctb], Dmitriy Selivanov [ctb], Jeffrey Arnold [ctb], Kenneth Benoit [ctb]
BugReports: https://github.com/ropensci/tokenizers/issues
CRAN Checks: tokenizers results
Citation: tokenizers citation info
DOI: 10.32614/CRAN.package.tokenizers
In Views: NaturalLanguageProcessing
License: MIT + file LICENSE
LinkingTo: Rcpp
Maintainer: Lincoln Mullen <lincoln at lincolnmullen.com>
Materials: README , NEWS
NeedsCompilation: yes
Old Sources: tokenizers archive
Package Source: tokenizers_0.3.0.tar.gz
Published: 2022-12-22
Reference Manual: tokenizers.html , tokenizers.pdf
Reverse Enhances: quanteda
Reverse Imports: blocking , covfefe , deeplr , DeepPINCS , pdfsearch , proustr , rslp , rwig , textrecipes , tidypmc , tidytext , ttgsea , wactor , WhatsR
Reverse Suggests: edgarWebR , sumup , torchdatasets
URL: https://docs.ropensci.org/tokenizers/ , https://github.com/ropensci/tokenizers
Version: 0.3.0
Vignettes: Introduction to the tokenizers Package ( source , R code ) The Text Interchange Formats and the tokenizers Package ( source , R code )
Windows Binaries: r-devel: tokenizers_0.3.0.zip , r-release: tokenizers_0.3.0.zip , r-oldrel: tokenizers_0.3.0.zip
MacOS Binaries: r-release (arm64): tokenizers_0.3.0.tgz , r-oldrel (arm64): tokenizers_0.3.0.tgz , r-release (x86_64): tokenizers_0.3.0.tgz , r-oldrel (x86_64): tokenizers_0.3.0.tgz

Version

0.3.0

LinkingTo

Rcpp

Published

2022-12-22

DOI

10.32614/CRAN.package.tokenizers

Author

Lincoln Mullen [aut, cre], Os Keyes [ctb], Dmitriy Selivanov [ctb], Jeffrey Arnold [ctb], Kenneth Benoit [ctb]

https://orcid.org/0000-0001-5103-6917 https://orcid.org/0000-0001-5196-609X https://orcid.org/0000-0001-9953-3904 https://orcid.org/0000-0002-0797-564X

Maintainer

Lincoln Mullen <lincoln at lincolnmullen.com>

BugReports

https://github.com/ropensci/tokenizers/issues

License

MIT + file LICENSE

MIT LICENSE

URL

https://docs.ropensci.org/tokenizers/ , https://github.com/ropensci/tokenizers

https://docs.ropensci.org/tokenizers/https://github.com/ropensci/tokenizers

NeedsCompilation

yes

Citation

tokenizers citation info

Materials

README , NEWS

README NEWS

In Views

NaturalLanguageProcessing

CRAN Checks

tokenizers results

Reference Manual

tokenizers.html , tokenizers.pdf

tokenizers.html tokenizers.pdf

Vignettes

Introduction to the tokenizers Package ( source , R code ) The Text Interchange Formats and the tokenizers Package ( source , R code )

Introduction to the tokenizers Package source R code The Text Interchange Formats and the tokenizers Package source R code

Package Source

tokenizers_0.3.0.tar.gz

Windows Binaries

r-devel: tokenizers_0.3.0.zip , r-release: tokenizers_0.3.0.zip , r-oldrel: tokenizers_0.3.0.zip

tokenizers_0.3.0.zip tokenizers_0.3.0.zip tokenizers_0.3.0.zip

MacOS Binaries

r-release (arm64): tokenizers_0.3.0.tgz , r-oldrel (arm64): tokenizers_0.3.0.tgz , r-release (x86_64): tokenizers_0.3.0.tgz , r-oldrel (x86_64): tokenizers_0.3.0.tgz

tokenizers_0.3.0.tgz tokenizers_0.3.0.tgz tokenizers_0.3.0.tgz tokenizers_0.3.0.tgz

Old Sources

tokenizers archive

Reverse Imports

blocking , covfefe , deeplr , DeepPINCS , pdfsearch , proustr , rslp , rwig , textrecipes , tidypmc , tidytext , ttgsea , wactor , WhatsR

blocking covfefe deeplr DeepPINCS pdfsearch proustr rslp rwig textrecipes tidypmc tidytext ttgsea wactor WhatsR

Reverse Suggests

edgarWebR , sumup , torchdatasets

edgarWebR sumup torchdatasets

Reverse Enhances

quanteda

Page sections 4

Documentation

Heading

Documentation

Links

[{"label":"tokenizers.html","section":"","type":"","url":"https://cran.r-project.org/web/packages/tokenizers/refman/tokenizers.html"},{"label":"tokenizers.pdf","section":"","type":"","url":"https://cran.r-project.org/web/packages/tokenizers/tokenizers.pdf"},{"label":"Introduction to the tokenizers Package","section":"","type":"","url":"https://cran.r-project.org/web/packages/tokenizers/vignettes/introduction-to-tokenizers.html"},{"label":"source","section":"","type":"","url":"https://cran.r-project.org/web/packages/tokenizers/vignettes/introduction-to-tokenizers.Rmd"},{"label":"R code","section":"","type":"","url":"https://cran.r-project.org/web/packages/tokenizers/vignettes/introduction-to-tokenizers.R"},{"label":"The Text Interchange Formats and the tokenizers Package","section":"","type":"","url":"https://cran.r-project.org/web/packages/tokenizers/vignettes/tif-and-tokenizers.html"},{"label":"source","section":"","type":"","url":"https://cran.r-project.org/web/packages/tokenizers/vignettes/tif-and-tokenizers.Rmd"},{"label":"R code","section":"","type":"","url":"https://cran.r-project.org/web/packages/tokenizers/vignettes/tif-and-tokenizers.R"}]

Text

Reference manual: tokenizers.html , tokenizers.pdf Vignettes: Introduction to the tokenizers Package ( source , R code ) The Text Interchange Formats and the tokenizers Package ( source , R code )

Downloads

Heading

Downloads

Links

[{"label":"tokenizers_0.3.0.tar.gz","section":"","type":"","url":"https://cran.r-project.org/src/contrib/tokenizers_0.3.0.tar.gz"},{"label":"tokenizers_0.3.0.zip","section":"","type":"","url":"https://cran.r-project.org/bin/windows/contrib/4.7/tokenizers_0.3.0.zip"},{"label":"tokenizers_0.3.0.zip","section":"","type":"","url":"https://cran.r-project.org/bin/windows/contrib/4.6/tokenizers_0.3.0.zip"},{"label":"tokenizers_0.3.0.zip","section":"","type":"","url":"https://cran.r-project.org/bin/windows/contrib/4.5/tokenizers_0.3.0.zip"},{"label":"tokenizers_0.3.0.tgz","section":"","type":"","url":"https://cran.r-project.org/bin/macosx/sonoma-arm64/contrib/4.6/tokenizers_0.3.0.tgz"},{"label":"tokenizers_0.3.0.tgz","section":"","type":"","url":"https://cran.r-project.org/bin/macosx/big-sur-arm64/contrib/4.5/tokenizers_0.3.0.tgz"},{"label":"tokenizers_0.3.0.tgz","section":"","type":"","url":"https://cran.r-project.org/bin/macosx/big-sur-x86_64/contrib/4.6/tokenizers_0.3.0.tgz"},{"label":"tokenizers_0.3.0.tgz","section":"","type":"","url":"https://cran.r-project.org/bin/macosx/big-sur-x86_64/contrib/4.5/tokenizers_0.3.0.tgz"},{"label":"tokenizers archive","section":"","type":"","url":"https://CRAN.R-project.org/src/contrib/Archive/tokenizers"}]

Text

Package source: tokenizers_0.3.0.tar.gz Windows binaries: r-devel: tokenizers_0.3.0.zip , r-release: tokenizers_0.3.0.zip , r-oldrel: tokenizers_0.3.0.zip macOS binaries: r-release (arm64): tokenizers_0.3.0.tgz , r-oldrel (arm64): tokenizers_0.3.0.tgz , r-release (x86_64): tokenizers_0.3.0.tgz , r-oldrel (x86_64): tokenizers_0.3.0.tgz Old sources: tokenizers archive

Reverse dependencies

Heading

Reverse dependencies

Links

[{"label":"blocking","section":"","type":"","url":"https://cran.r-project.org/web/packages/blocking/index.html"},{"label":"covfefe","section":"","type":"","url":"https://cran.r-project.org/web/packages/covfefe/index.html"},{"label":"deeplr","section":"","type":"","url":"https://cran.r-project.org/web/packages/deeplr/index.html"},{"label":"DeepPINCS","section":"","type":"","url":"https://www.bioconductor.org/packages/release/bioc/html/DeepPINCS.html"},{"label":"pdfsearch","section":"","type":"","url":"https://cran.r-project.org/web/packages/pdfsearch/index.html"},{"label":"proustr","section":"","type":"","url":"https://cran.r-project.org/web/packages/proustr/index.html"},{"label":"rslp","section":"","type":"","url":"https://cran.r-project.org/web/packages/rslp/index.html"},{"label":"rwig","section":"","type":"","url":"https://cran.r-project.org/web/packages/rwig/index.html"},{"label":"textrecipes","section":"","type":"","url":"https://cran.r-project.org/web/packages/textrecipes/index.html"},{"label":"tidypmc","section":"","type":"","url":"https://cran.r-project.org/web/packages/tidypmc/index.html"},{"label":"tidytext","section":"","type":"","url":"https://cran.r-project.org/web/packages/tidytext/index.html"},{"label":"ttgsea","section":"","type":"","url":"https://www.bioconductor.org/packages/release/bioc/html/ttgsea.html"},{"label":"wactor","section":"","type":"","url":"https://cran.r-project.org/web/packages/wactor/index.html"},{"label":"WhatsR","section":"","type":"","url":"https://cran.r-project.org/web/packages/WhatsR/index.html"},{"label":"edgarWebR","section":"","type":"","url":"https://cran.r-project.org/web/packages/edgarWebR/index.html"},{"label":"sumup","section":"","type":"","url":"https://cran.r-project.org/web/packages/sumup/index.html"},{"label":"torchdatasets","section":"","type":"","url":"https://cran.r-project.org/web/packages/torchdatasets/index.html"},{"label":"quanteda","section":"","type":"","url":"https://cran.r-project.org/web/packages/quanteda/index.html"}]

Text

Reverse imports: blocking , covfefe , deeplr , DeepPINCS , pdfsearch , proustr , rslp , rwig , textrecipes , tidypmc , tidytext , ttgsea , wactor , WhatsR Reverse suggests: edgarWebR , sumup , torchdatasets Reverse enhances: quanteda

Linking

Heading

Linking

Links

[{"label":"https://CRAN.R-project.org/package=tokenizers","section":"","type":"","url":"https://CRAN.R-project.org/package=tokenizers"}]

Text

Please use the canonical form https://CRAN.R-project.org/package=tokenizers to link to this page.

Materials 2

README NEWS

Documentation 8

tokenizers.html tokenizers.pdf Introduction to the tokenizers Package source R code The Text Interchange Formats and the tokenizers Package source R code

Vignettes 6

Introduction to the tokenizers Package source R code The Text Interchange Formats and the tokenizers Package source R code

Downloads 9

tokenizers_0.3.0.tar.gz tokenizers_0.3.0.zip tokenizers_0.3.0.zip tokenizers_0.3.0.zip tokenizers_0.3.0.tgz tokenizers_0.3.0.tgz tokenizers_0.3.0.tgz tokenizers_0.3.0.tgz tokenizers archive

All page links 59

stringi Rcpp SnowballC covr knitr rmarkdown stopwords testthat 10.32614/CRAN.package.tokenizers https://orcid.org/0000-0001-5103-6917 https://orcid.org/0000-0001-5196-609X https://orcid.org/0000-0001-9953-3904 https://orcid.org/0000-0002-0797-564X https://github.com/ropensci/tokenizers/issues MIT LICENSE https://docs.ropensci.org/tokenizers/https://github.com/ropensci/tokenizers tokenizers citation info README NEWS NaturalLanguageProcessing tokenizers results tokenizers.html tokenizers.pdf Introduction to the tokenizers Package source R code The Text Interchange Formats and the tokenizers Package source R code tokenizers_0.3.0.tar.gz tokenizers_0.3.0.zip tokenizers_0.3.0.zip tokenizers_0.3.0.zip tokenizers_0.3.0.tgz tokenizers_0.3.0.tgz tokenizers_0.3.0.tgz tokenizers_0.3.0.tgz tokenizers archive blocking covfefe deeplr DeepPINCS pdfsearch proustr rslp rwig textrecipes tidypmc tidytext ttgsea wactor WhatsR edgarWebR sumup torchdatasets quanteda https://CRAN.R-project.org/package=tokenizers

버전 이력

Repository	Version	Published	First seen	Last seen	Docs
CRAN	0.3.0		2026-05-29	2026-05-30
R-universe	0.3.1		2026-05-29	2026-05-30

보안

표시할 OSV 데이터가 없습니다.

문헌 신호

표시할 OpenAlex 데이터가 없습니다.

CRAN Task Views

Natural Language Processing · 2026-05-30