NLP & Language AI Engineer

Module 1 · NLP Fundamentals · Lesson 1 of 1

Module overview — NLP Fundamentals

Module overview — NLP Fundamentals Goal : Tokenize, lemmatize and clean a text corpus without destroying meaning. Study time : Durée d'étude : 3 h Prerequisites : Bases de Python : variables, listes et boucles (formation AI Engineer, module 1) Running thread : Awa, in Dakar. Awa runs the Facebook page of a small business in Dakar. She exports five hundred customer comments: emojis everywhere, typos, missing accents, a few phone numbers. A program cannot analyze this as it is. In this module, she learns to split text without destroying it, to clean her corpus with rules she can explain, and to choose between lemmatization and stemming. Lesson plan : 1. NLP beyond LLMs and chatbots 2. Tokenizing without destroying meaning 3. Cleaning a real corpus 4. Lemmatization vs stemming on commercial ads What you will be able to do : - Compare word, subword and character tokenization on the same sentence - Clean a batch of raw comments without destroying emojis or useful information - Tell lemmatization from stemming on conjugated verbs taken from ads End-of-module questions (Answer in writing, with figures.) : 1. How many tokens does the sentence of your choice give when split by word, then by character? Give both numbers. 2. Which cleaning rules did you apply to your sample, and which one did you refuse to apply? 3. Among your ten comments, how many contained an emoji, a phone number or a word missing an accent? Give the three numbers. 4. What difference do you see between lemmatization and stemming? Answer in two sentences. Module deliverables : - Your NGO corpus sheet with its five cleaning rules - The lab report: tokens, two cleaned comments, tallies and the justification for keeping emojis

Free preview, no account needed — the rest of this module and the following modules unlock after enrolling.