word_tokens - BigQuery

Function

word_tokens(string, symbol)

Description

Word Tokens - Splits a <string> into words using the specified <symbol>.

Example Query 


SELECT `justfunctions.eu.word_tokens`("this is a sentence", "\\s+")
                                            
/*--Output--
['this', 'is', 'a', 'sentence']
*/

Statement 

CREATE OR REPLACE FUNCTION `your_project_id.your_dataset_id.word_tokens`(`string` string, `symbol` string) 
  RETURNS ARRAY<STRING> AS (SPLIT(REGEXP_REPLACE(LOWER(string),symbol," "), " "))
  OPTIONS ( description = '''Splits a <string> into words using the specified <symbol>.''')

Regions

justfunctions.eu.word_tokens(string, symbol),
justfunctions.us.word_tokens(string, symbol)

Type

SQL User Defined Function (SQL UDF)

Related Functions

How to Use

Frequently Asked Questions

User-Defined Functions (UDFs) in Google BigQuery are custom functions that you can create to perform operations that aren't available through the standard SQL functions. These UDFs allow you to extend BigQuery's SQL capabilities to suit your specific data processing needs. JustFunctions is a collection of open-source user-defined functions (UDFs).

JustFunctions is a collection of open-source User-Defined Functions (UDFs) designed to extend the capabilities of Google BigQuery. These functions cover a wide range of applications, including text manipulation, URL processing, date processing, email handling, similarity measures, and more. Moreover, JustFunctions is frequently updated to include more use cases.

We welcome any feedback or questions you may have. You can Contact us or report an issue on Github.

Functions and procedures from JustFunctions can be used directly in any of your projects.
To start, simply click 👆 on any function,  Copy  the 'Example Query' and run it in your BigQuery console.
You can also  Copy  the 'Statement' to create your own private user-defined function.

Yes, JustFunctions is completely free to use.

Yes, currently JustFunctions is only available for Google BigQuery. In the future, we will also support PostgreSQL.

See something wrong? Contact us or report an issue on Github.