Word and sentence tokenization can be done easily using the spacy library in python. In this NLP tutorial, we will cover tokenization and a few related topics.
NLP platform: https://www.firstlanguage.in/
⭐️ Timestamps ⭐️
00:00 What is tokenization
02:35 Install spacy
02:49 Coding starts
03:23 Basic English word tokenization
14:15 Span object
15:00 Token attributes
18:40 Grab emails from the student information doc
23:58 Tokenization in Hindi
26:13 Customize tokenization rule
29:52 Sentence tokenization (or segmentation)
33:15 Exercise
Code: https://github.com/codebasics/nlp-tutorials/blob/main/4_tokenization/spacy_tokenizer_tutorial.ipynb
Exercise: In the above code, go to the end and you will find exercises
Complete NLP Playlist: https://www.youtube.com/playlist?list=PLeo1K3hjS3uuvuAXhYjV2lMEShq2UYSwX
🔖Hashtags🔖
#nlp #nlptutorial #nlppython #spacytutorial #spacytutorialnlp #spacytutorialnlp #wordtokenization #tokenizerspacy #tokenizationnlp #wordtokenizerspacy #tokenizationandspacy #spacynlp
Do you want to learn technology from me? Check https://codebasics.io/?utm_source=description&utm_medium=yt&utm_campaign=description&utm_id=description for my affordable video courses.
Need help building software or data analytics and AI solutions? My company https://www.atliq.com/ can help. Click on the Contact button on that website.
🎥 Codebasics Hindi channel: https://www.youtube.com/channel/UCTmFBhuhMibVoSfYom1uXEg
#️⃣ Social Media #️⃣
🔗 Discord: https://discord.gg/r42Kbuk
📸 Instagram: https://www.instagram.com/codebasicshub/
🔊 Facebook: https://www.facebook.com/codebasicshub
📱 Twitter: https://twitter.com/codebasicshub
📝 Linkedin (Personal): https://www.linkedin.com/in/dhavalsays/
📝 Linkedin (Codebasics): https://www.linkedin.com/company/codebasics/
🔗 Patreon: https://www.patreon.com/codebasics?fan_landing=true
❗❗ DISCLAIMER: All opinions expressed in this video are of my own and not that of my employers'.