String Tokenization

String Tokenization • What is String Tokenization? • The StringTokenizer class • Examples

What is String Tokenization? • So far we have been reading our input one value at a time. • Sometimes it is more natural to read a group of input at a time. • For example, when reading records of students from a text file, it is natural to read a whole record at a time. "995432 Al-Suhaim Adil 3.5" • The readLine() method of the Buffered Reader class can read a group of input as a single string object. • The problem is, how do we break this string object into individual words known as tokens? "995432""Al-Suhaim Adil”"3.5“ • This process is what String tokenization is about.

The StringTokenizer Class • The StringTokenizer class, of the java.util package, is used to break a string object into individual tokens. • It has the following constructors:

StringTokenizer Methods • The following are the main methods of the StringTokenizer class:

How to apply the methods • To break a string into tokens, first, a StringTokenizer object is created. String myString = "I like Java very much"; StringTokenizer tokenizer = new StringTokenizer(myString); • Then any of the following loops can be used to process the tokens: while(tokenizer.hasMoreTokens()){ String token = tokenizer.nextToken(); // process token } or int tokenCount = tokenizer.countTokens(); for(int k = 1; k <= tokenCount; k++){ String token = tokenizer.nextToken(); // process token }

Example 1 • The following program reads grades from the keyboard and finds the average. The grades are read in one line. import java.io.*; import java.util.StringTokenizer; public class TokenizerExamplel{ public static void main(String[] args)throws IOException{ BufferedReader stdin = new BufferedReader(new InputStreamReader(System.in)); System.out.print("Enter grades in one line:"); String inputLine = stdin.readLine(); StringTokenizer tokenizer = new StringTokenizer(inputLine); int count = tokenizer.countTokens(); double sum = 0; while(tokenizer.hasMoreTokens()) sum += Double.parseDouble(tokenizer.nextToken()); System.out.println("\nThe average = "+ sum / count); } }

Example 2 • This example shows how to use the second constructor of StringTokenizer class. • It tokenizes the words in a string, such that the punctuation characters following the words are not appended to the resulting tokens. • import java.util.StringTokenizer; • public class TokenizerExample2{ • public static void main(String[] args){ • String inputLine = • "Hi there, do you like Java? I do;very much."; • StringTokenizer tokenizer = • new StringTokenizer (inputLine, ",.?;:! \t\r\n"); • while(tokenizer.hasMoreTokens()) • System.out.println(tokenizer.nextToken()); • } • } Output: Hi there do you like Java I do very much

Example 3 • This example shows how to use the third constructor of StringTokenizer class. • It tokenizes an arithmetic expression based on the operators and returns both the operands and the operators as tokens. import java.util.StringTokenizer; public class TokenizerExample3{ public static void main(String[] args){ String inputLine = "(2+5)/(10-1)"; StringTokenizer tokenizer = new StringTokenizer(inputLine,“+—*/()",true); while(tokenizer.hasMoreTokens()) System.out.println(tokenizer.nextToken()); } } Output: ( 2 + 5 ) / 10 - 1 )

String Tokenization